You talk like someone with no familiarity with the modern university environment.
The reality is that many minorities who fail to graduate, or take many years to graduate, from elite institutions would graduate just fine in four years from a less competitive institution. They are accepted with more lenient admission standards but then they take the same classes with the same grading standards as other students. Or, they self-segregate into majors with minimal academic standards, like African-American Studies, when they would have been better off majoring in engineering or some such useful major, which they might well have done if they had been accepted into a better fit.
There's no reason to think that people of color would have fewer opportunities to earn college degrees if admissions were designed to ensure that they were competitive in the classroom. Just as likely it would be the reverse - more would obtain degrees.