I think the "you can't call everything" thought is basically ridiculous. The issue isn't with touch fouls, it's with letting teams that live on inside physical stuff get away with it. If all of what goes on inside were called, just maybe the coaches would decide having the big guy in foul trouble 5 minutes into every game isn't worth it.
For certain, I've been at games where the refs blow the whistle so often, that being at the game is like having a bad toothache. Those tend to be games where even the smallest contact is called. But then there are way too many games where a guy gets clobbered underneath and no whistle and in the next 20 seconds there is a whistle for a hand check. Body fouls seems to be rarely called on "blocks" these days. There was a play in the SU game where, I think it was Olander, was supposedly blocked by two guys, wound up on the floor, no whistle. No way that play happened without one of the 2 defenders bumping him. And in fact, that was the play where UConn was whistled for a touch foul on the next trip down court. The lack of 3 second calls doesn't help either. You can go to many games in a row with no 3 second calls.