It's a tough subject, but those cases are definitely in the minority. A homosexual could also have "inappropriate behavior" with a member of the same gender, and that to me is no different than a trans person having intimate relations with a member of their new gender. By separating males and females you can't prevent these sexual acts taking place if you allow homosexuals in, and that's sort of the same logic when you let a transgender person in to the military.

If you allow one, then you should allow the other. People who argue that being transgender is a mental illness are thinking that it is unnatural and out of the ordinary, but so is homosexuality.
Also, it might be expensive to accommodate for transsexuals as opposed to homosexuals, as trans require extra measures in place, but I don't think that money is something to be niggardly over when people's rights are concerned.