Adding login for users does not work if server accessed with ip-address

I am running jitsi on ubuntu server 18.04 installed from jitsi stable repository.

I successfully added authentication for users when accessing jitsi meet by using the servers domain name.
But the problem is that if I access the server by its ip-address prosody authentication does not work and I can access the meeting without authentication.

I guess that there is some additional configuration that needs to be added, but i cant find any documentation for this anywhere.

Help would be appreciated.