Although our current conference position is not ideal, there is another major issue that people are not talking about. Private universities playing major college football. If you look at the top 5 conferences + ND, there are currently 11 private universities playing major college football, and 5 of them are in the ACC. Over time, each of these schools will have to self evaluate their commitment to playing high level football.
First, look at the attendance (from 2011):
Notre Dame: 80,795
Stanford: 49,997
Miami: 48,654
Baylor: 41,368
Syracuse: 40,504
BC: 35,709
TCU: 33,686
Northwestern: 33,442
Vanderbilt: 32,873
Wake Forest: 31,967
Duke: 24,393
Next, think about football stadiums. How many of these schools can afford or can commit to do a major upgrade or replace their football stadiums? ND, Stanford, Baylor, TCU, probably Vandy
In addition, how many of these schools are big football draws on TV? Notre Dame and Miami. In a pay cable channel or pay per view world, how many of these schools can lock up their local geography? Maybe a couple.
As college presidents and BOTs look to the future, they not only have to think about their school's position in the future, but those of their current conference mates. If I was UNC, I would look at the private schools and think they are not the right conference mates in 10 to 20 years. Duke is attractive to conferences for a number of reasons, but they have to decide if they want to and can they compete in football LT. For Duke, competing against BC, WF, Syracuse, Virginia, UNC,... means that they can still try to compete in football, but they are not going to be competitive and they probably wouldn't want to try in the SEC or Big 1G.