It's always been my understanding that the conference of the host institution assigns the referees.
Not necessarily. It depends on what is agreed when the teams sign the contact to play. If both teams are strong and have relatively equal clout, it is not unusual for the visiting team to have the right to assign officials. If it is a major program playing a weaker one, then the major one usually assigns them -
in what is likely their home game.