It can freeze just because Toribash is a terrible game as it is.

The most likely cause is that someone has shitty connection. No need to point fingers at anyone when you don't know what's going on.

Nothing you can do about it.