The endowment at the Ivies is huge and they don't have big time fan support.
Your premise, and others in this thread, was that a better quality student is a lower quality fan, and that jobs/careers aren't readily available in the area to cultivate a bigger/better fanbase.
I don't necessarily disagree with either of those statements, but I tend to think they are more excuses and I agree with
@August_West more that it's just a front runner syndrome.
I think a lot things conspired to enter into it. When we were playing at the 3500 seat Field House and had a brand new and innovative 15,000 (then, IIRC) arena it made sense to go there. But seeing a game in a pro arena in Hartford has a different feel than seeing a game on campus. We also played games at the New Haven Civic Center. When Gampel was built, they initially held the capacity to 8,500 to make sure that the new on campus venue would not compete with the Civic Center. Calhoun almost immediately bumped it to 10,000. At that point most, if not all home games should have been played at Storrs, but somehow playing in Hartford and supporting the XL became a claim of right. Add to that some truly dumb pricing policy decisions from Jeff Hathaway and the traditional fan became alienated. If we had guys sitting in the same seats and not being moved around and (some) price out of tickets, there would be more mutual loyalty.
Also Storrs and our big population center in the SW are inconveniently not sitting right on top of each other. So getting there takes time and people show up late and like to get an early head start out. Likewise Hartford is a half hour away from Storrs. College kids would be far more groomed to see games if it was just a walk across campus. And make no mistake about it today's college kids are the future super fans of the program. The games we play in Hartford sacrifice on future a bit.
If that weren't enough, Kraft's dalliance Hartford made the notion of playing football off campus seem palatable and normal. College football gets a lot of ambiance and pageantry from being on campus. Once again. A more convenient location comes at the cost of having kids get up and wander over to the stadium.
CT's economy is struggling and HDTV and every game being broadcast makes it easier to pass on the hassles of going to a game. Don't get me wrong, nothing, nothing beats being at a game, but in today's harried world with a limited discretionary budget access to games on TV makes it easier to rationalize not going.
Finally, our teams are in a down cycle. The cost and hassle of going to game is harder to justify when your are frustrated with the play. All these things combine into a stew of malaise and disinterest in our program. We need to win and rethink the way we treat the fans to build back the loyalty and energy we once had.