So take that one step further. If the Yankees sweep a 3 game set against the Nationals at home, winning each game by 1 run, and the Red Sox sweep the Nationals at home winning each game by 10, would the margin be relevant in trying to predict who would win a Yankees--Red Sox series the following week? Of course it would be relevant. Not determinative, but certainly relevant.
On the other hand, no one would seriously propose that, in the standings, the Sox should get more wins for their sweep than the Yankees get from theirs. So when folks object to putting much weight on the KenPom/Net type numbers that's all they're saying. Because schedules in college sports are far less balanced than in a professional league, it's fine to reward and punish teams for their strengths of schedule when you compare their records, but just because margins are useful in making predictions doesn't mean they should be used in ranking teams for anything meaningful.