Some stats are misleading and in need of context. Other stats - like the ones cited in the OP - are literally a reflection of what happened, and arguing them is about as worthwhile as arguing time ("I'm 34, but that doesn't tell the whole story"...actually, it does, it means you're 34).
Using the schedule as a crutch to explain away these numbers is mostly bogus. No, they're not in the ACC or Big 12, but they've still played five top 50 offenses and held three of them to 40% or less from the field.
Michigan, the #11 offense in the country, shot 32% against us. Some of it was luck, sure, but there's also an infrastructure in place defensively that prohibits us from ever allowing opponents too far north of 40% (I wrote about this in detail before the season).
37% is a startlingly low number (last years Kentucky team, which was considered among the best of all-time defensively, was around that number). Obviously, it's probably not that low if we're in a power conference - but we got a look at what that was like in the OOC schedule, and it isn't as if we got lit up (if you take the aggregate of the Michigan, Syracuse, Gonzaga, Maryland, Ohio State, Texas, and Georgetown games, I bet we're under 4o%).
There's still some ambiguity regarding how good the defense is, but there's no doubt that it has been among the best in America thus far.