With all due respect, Yale and Harvard (and Stanford) are somewhat unique. I went to one of their peer institutions, and had a lot of contact with the folks at the AD's fundraising arm, and with some folks at our fellow Ivies.
What you have to understand is that those endowment funds aren't unencumbered. Most of them are actually legally directed to particular bugaboos for which they were given.
Here's a silly example from Harvard. Harvard's band turns out to have a very large endowment of its own (as these things go--when I worked with them a couple decades ago it was close to $10MM). As a result, they didn't get much funding from the student activities budget or the athletic department budget, which is fair. (They're not going to get any from the music department.) You'd think everything's paid for, right? Wrong. They never had money to pay for things like (legally) copying music in their library. Music to which they owned the copyright. They had lots and lots of money to buy instruments and stuff, but they had lots of instruments—more than they could ever use. So what they didn't spend stayed "in the bank" as it were. But they never had enough money to do things like purchase music, repair uniforms, or even copy music.
I can tell you that the biggest battle that our university's fundraisers (living saints, IMO) had was raising money for athletics that didn't have lots of strings attached. Instead of getting money to refurbish the football weight room, an alumnus will think it's a good idea to have a big, foreboding bell for the game day staff to ring down at the stadium (which they use only a few times per year). It drove the foundation people not to mention the athletic director and coaching staff absolutely nuts.
Now do I think that the Ivies would ever get a lot of TV money for their athletic programs? Heck no. But any new revenues they could get, that aren't encumbered would be seen by all as something of a godsend.