The Grant of Rights isn't to ESPN...it is to the conference. The conference is the receiver of that grant and would decide whether to enforce it, or not. I suspect nobody in any GOR conference wants a challenge that may affect their GOR,
The Business of College Sports says...
In these agreements, all of the conference members grant their television broadcast rights to their athletic contests to the conference for a certain period of time. If a member leaves the conference during that time, the conference retains the member’s television rights. Because the value of a school to a conference is the television revenue it can help generate, a grant of rights agreement makes the members essentially worthless to another conference that is looking for new members.