My suggestion is to add an option to make your server hidden from the Multiplayer menu.

So instead of having people joining and getting the pass, only people with the /join pass will know the server.