I never referred to the the Nielsen data as "TV sets". I was on auto-pilot earlier and referred to them as ratings, rather than total viewers. Ratings are not TV sets, but rather, the percentage of households w/ a TV that are watch a specific program/game.
The extrapolation between TV sets to viewers changes from situation to situation. So, two programs with the same exact rating can actually have different "total viewers". That's b/c when Nielsen logs TV usage, it also records total viewers, based either on demographic information required when you sign up to allow Nielsen to watch your viewing habits, or by keying in the number of viewers when you turn on the TV (Nielsen has several different systems). Generally, for college sporting events, there are 1.35 - 1.55 viewers per TV set.
Also, despite the popularity of college football, it is a fraction of a fraction of a fraction of many other categories. The NFL absolutely obliterates college football in viewership. So, drawing in 4M+ viewers on average for a football game is pretty darn good, especially when you're talking about ALL games across ALL channels. The conference measures aren't solely Tier 1 games. They include Tier 2, which would include all of the ESPN brand, Fox Sports, etc. And, it also includes Tier 3, which are your PPV, Raycom, Sun Sports, Longhorn Network, etc.