It's not only the total number of fouls, it's also how and when they were called and on whom.
One of my big pet peeves is when refs call nothing in the first half and then everything in the second half. It messes up the flow of the game. It matters less to me how the game is called than that it's called consistently.
Similarly, when they were in the bonus and only had three fouls called against them, that made it really hard for us in the second half. A lot of of our key players were on the bench and the refs took away our ability to aggressively guard people. That changed the flow of the game dramatically and greatly contributed to their second half run. As often is the case, it seems like at one point they looked up at the scoreboard and said "oh crap this is really lopsided" and started calling more fouls on BYU. But, by that time, the damage was done and we had what four guys with four fouls?
No that I'm not going all conspiracy theory on this and saying that the refs secretly hate us or in BYU's pocket. I am saying that the game was inconsistently called half to half and in the second half the fouls were very lopsided against Connecticut until about eight minutes to go or so. It is what it is, and we ended up winning, but it is a pet peeve.