It's easy for me to understand why people might not care about UConn football, but it's a lot harder for me to grasp why the people who do are taking this comment so personally.
You should go to games because you want to go to games. That's it. This is entertainment, not the armed forces. Nobody's getting a ribbon at the end because they walked the darkest road. If you're not going to have a good time regardless of the result, then don't go. The tooth fairy isn't going to show up at 3 AM and peel the Husky logo off your jacket in revenge.
All else being equal, he's saying yeah, people who come to the games do matter. You're helping the school by showing up, and in the event that we do become Auburn or Michigan, people like you will be the reason for it.
I thought he did a bad job wording it, but he's obviously not actively trying to get people to dislike him. It was probably a throwaway cliche that he'd like to have back - saying "hey I know the program has sucked forever now and will probably suck forever, but our athletic department is bleeding and we need your money. please!" would probably present problems of its own.