People always downgrade Brady because of the much better defenses, but they do that because they love to hate the Patriots. While Brady and the offense put up 500 yards and 350+ passing yards against the NFL's top defense in the 2004 SB, the Patriots defense was crapping the bed by giving up 29 points in 30 minutes. Vinatieri's 2 missed chip shot field goals didn't help either. And yet somehow Brady was a game manager! Absurd.
Meanwhile, Manning's 2006 Colts had the #1 defense in the NFL. In the playoffs, Manning threw 3 interceptions against the Chiefs, but the Colts defense got 3 turnovers, 4 sacks, and they didn't even give up a single first down for 3 quarters in their 23-8 victory. The next week against the Ravens, Peyton Manning threw another 2 interceptions and no TDs. Vinatieri kicked 5 field goals. The Colts defense got 4 turnovers (2 fumbles, 2 INTs) from the Ravens as the Colts won 15-6.
In the Super Bowl, Peyton Manning threw 1 INT and 1 TD. The Colts defense, however, got 3 fumble recoveries and 2 interceptions off Rex Grossman. The Colts D gave up 10 points the entire game.
So, in those 3 games, the Colts defense gave up a total of 24 points, got 12 turnovers. Meanwhile, Manning threw 6 interceptions and 2 TDs, and he had a sub 50 passer rating.
Yet somehow Brady is the one who needed the defense to win, even though he put up top 5 Super Bowl performance.
I'd say you're a victim of media hype.