By having Affirmative Action only apply to less fortunate individuals rather than a specific race we allow the people who don't have the same opportunities the wealthy have to further educate themselves and put them into a position where they can eventually have a decent career at a job which doesn't have a dead-end.

Let's say there are more African-Americans which are poor (keeping in mind it's percentage of their own population group) then the majority. By allowing Affirmative Action to work for the economically weak, rich African-Americans don't get some super advantage because of their race and poor white student's won't lack the support they deserve because they are white.