You mean you game doesn't work with more than 4 ppl connected?
Well, that is strange... I have played with 8 ppl in a room without any problems.
But, also, when I have hosted my game with 6 ppl, they had a big lag. Not unplayable, but certainly annoying. I guess it really depends where you are and how good is your internet connection.
Sometimes you need to find a guy from the group that can host games for large amount of ppl.
Me, personalty, rarely play with more than 5 ppl in a room, so haven't had any big problems.

That's why when groups are looking for players they tend to stick to people with same time zones, and that are close together.
I mean, I have played a game with some dudes from AU, bit every time, it was like 400 ping for them or for me. I don't even want to tell you how large ping is when you play with someone from China, lol.