I think you are partially correct. You are correct that media rights have not been negotiated down after defections (not impositions like the BE), but conference networks have muddied the waters. For example, the B1G gets a much higher rate for the BTN in markets that it has schools than in markets it doesn't. So if a big market B1G school left and was replaced by a small market school, there could be a negative impact on BTN revenues. (Just an example, I'm not saying any B1G school would leave the conference.)
Today, there are 3 conference networks: SEC, PAC12, BTN. I don't think anyone is leaving those three conferences so defections are a very low probability. Big 12 and ACC do not have networks yet, but the ACC is trying to get one off the ground. For the ACC, the loss of big market schools like FSU, UNC, UVA, or VT would have a very negative impact on the value of an ACC Network. In my opinion, the next possible catalyst for conference movement is the possibility of an ACC Network.