Tom brings up good points here.
One thing I will say is that the next year or two are key in his career. If he is truly an average QB, once the league makes their adjustments to stop you, then he will begin to suffer. The good to great QB's stay ahead of the curve and make adjustments to the league's adjustments. Part of this also falls upon the offensive coordinator as well, but look at Eli Manning and how much better he is now than he was a few years ago.
Another thing is that public perception of him is a bit out of control - the dropped snap in the playoffs really turned people against him - should he have caught the snap, of course, but things happen sometimes, and in all reality, he should not even have been out there anyway. People forget that he almost scored on that play, and then what kind of a hero he would have been if he did. Dating Jessica Simpson and Carrie Underwood and going to Mexico before the playoffs have all been big headlines - what other QB faces that kind of scrutiny? Brady possibly, but that's it. Big Ben was accused of rape and that isn't nearly the story in the public eye.
My point is that the rest of the team, coaching staff, etc. hasn't really put him in a spot yet that is fair to judge his performance. He has played 2 playoff games in 3 full years of starting at QB - one game he should have won, and the other wasn't his fault that they lost. They had 11 penalties that game, and Patrick Crayton dropped a sure TD that would have been the decider. In those 2 playoff games, 1 interception, and that was on the last play against the Giants on 4th down where he was trying to win them the game.
Come back to me in 2-3 years and ask the same question. Bigger problems right now:
Defense
Coach
WR play