Ah, here is your simple, rational response: It isn't an issue, and you're making a problem out of nothing.

How is being attracted to specifically a certain race fine, but specifically not being attracted to a certain race racist? Attraction is not taught, and gay conversion therapy victims can attest to that. Racism, however, is a viewpoint entirely caused by life experience or is taught by people around the individual. If your argument is that disliking a certain race is based in prejudice (which is not an identical term to racism), you wouldn't be wrong, since we are prejudiced toward features we find attractive in a mate, and skin tone is a physical feature, however I find it a stretch to claim that that equates to racism.

Then again, maybe I'm wrong about racism being learned and both of us are actually right, meaning that racism is as natural as (and directly affects) sexual preference.