You're talking about a kid in Oriakhi who has a fragile psyche. Yes, Drummond gave us 10, 8, and 3, which is pretty good for a freshman and definitely had a positive effect on the team. What didn't have a positive effect, was his cluelessness on defense, his inability to set good screens, his inconsistency, and his inability to score unless he had the ball above the rim. He had a lot of limitations as a player, and although Roscoe Smith isn't exactly Kevin McChale, you could probably make the case that him on the floor gave us more flexiblity because of his ability to step away from the basket.
I'm not saying that the seemingly negative effect he had on Oriakhi is his fault, but it certainly seemed like Oriakhi grew tired of playing with Drummond real quick. Last year he was the one throwing down the alley oops, blocking shots, scoring on putbacks, etc. He was the force inside during last years title run. Why Drummond and Oriakhi could not co-exist, I'll never know, however, would you agree Drummond's presense seemd to have an inverse effect on Oriakhi's game rather than a positive effect? Likewise, the addition of Drummond deprived Roscoe minutes at the four. And then you have to consider the psychological element of the perimeter players knowing they have a freak of nature behind them to block shots.
The defense as a whole regressed mightily this season. I would wager that this was more because of Kemba's departure and less because of Drummond's arrival. But still, it's another factor you have to consider.