I am not of the opinion that a fetish for a specific race is racist if you are using the textbook definition of it, which is inherently negative. Sexual fantasies can be creepy, and you're entirely justified in being put off by a fetish that you disagree with, but intense sexual attraction toward a race is, in my opinion, entirely understandable, as is any other fetish.
I do think it should come down to intent, though.
Attraction to a demographic can be problematic and damaging if not kept in check, the same as any attraction could be. Fetishes toward certain races will probably damage your chances at a real relationship with them, because if it's something you genuinely struggle to suppress, I'm sure many people would feel that you can only view them in a sexual sense. However, I think that if your ambitions don't lie in simply pursuing your object of desire, most people can live totally normal lives without any risk at offending anyone, because what you nut to should never be a point of offense on anyone's track record unless you are literally considering having sex with them.