Jitsi Audio/Video call is not working after hosting it in NAT

Hi All,

I m new in Jitsi ecosystem and for my use case I followed this official doc Self-Hosting Guide - Debian/Ubuntu server | Jitsi Meet to deploy jitsi on my EC2 instance with NAT configuration.

I was able to successfully deploy jitsi on my ec2 instance all 3 components i.e. JVB, jicofo and prosody are up and running without any error.
However when I tried to place a conference call between 2 or 3 users, in this case - users are not able to hear or see in the conference call.

Below is the jvb.log details of this above call :-

JVB 2023-04-24 14:13:26.660 INFO: [1384] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=860db73d stats_id=Marlen-bhN local_ufrag=7ae241gupppeo3] IceTransport.iceStateChanged#342: ICE state changed old=Waiting new=Running

JVB 2023-04-24 14:13:26.660 INFO: [1384] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=860db73d stats_id=Marlen-bhN local_ufrag=7ae241gupppeo3 ufrag=7ae241gupppeo3] ConnectivityCheckClient.startChecks#147: Start connectivity checks.

JVB 2023-04-24 14:13:29.316 INFO: [1388] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=beb63537 stats_id=Mike-9nC] AbstractEndpoint.expire#289: Expiring.

JVB 2023-04-24 14:13:29.318 INFO: [1388] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=beb63537 stats_id=Mike-9nC] Endpoint.expire#1114: Spent 0 seconds oversending

JVB 2023-04-24 14:13:29.318 INFO: [1388] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=beb63537 stats_id=Mike-9nC] Transceiver.teardown#357: Tearing down

JVB 2023-04-24 14:13:29.318 INFO: [1388] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=beb63537 stats_id=Mike-9nC] RtpReceiverImpl.tearDown#347: Tearing down

JVB 2023-04-24 14:13:29.318 INFO: [1388] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=beb63537 stats_id=Mike-9nC] RtpSenderImpl.tearDown#318: Tearing down

JVB 2023-04-24 14:13:29.319 INFO: [1388] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=beb63537 stats_id=Mike-9nC] DtlsTransport.stop#178: Stopping

JVB 2023-04-24 14:13:29.319 INFO: [1388] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=beb63537 stats_id=Mike-9nC local_ufrag=6rufl1guppp0eh] IceTransport.stop#252: Stopping

JVB 2023-04-24 14:13:29.320 INFO: [1388] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=beb63537 stats_id=Mike-9nC local_ufrag=6rufl1guppp0eh ufrag=6rufl1guppp0eh] Agent.setState#946: ICE state changed from Running to Terminated.

JVB 2023-04-24 14:13:29.321 INFO: [1388] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=beb63537 stats_id=Mike-9nC local_ufrag=6rufl1guppp0eh ufrag=6rufl1guppp0eh name=stream-beb63537 componentId=1] MergingDatagramSocket.close#142: Closing.

JVB 2023-04-24 14:13:29.321 INFO: [1388] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=beb63537 stats_id=Mike-9nC] Endpoint.expire#1132: Expired.

JVB 2023-04-24 14:13:29.323 INFO: [1388] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=beb63537 local_ufrag=6qedo1gupppi3a ufrag=6qedo1gupppi3a] Agent.gatherCandidates#647: Gathering candidates for component stream-beb63537.RTP.

JVB 2023-04-24 14:13:29.325 INFO: [1388] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=beb63537] Endpoint.<init>#328: Created new endpoint isUsingSourceNames=true, iceControlling=true

JVB 2023-04-24 14:13:30.050 INFO: [1388] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=beb63537 stats_id=Mike-9nC] DtlsTransport.setSetupAttribute#120: The remote side is acting as DTLS client, we'll act as server

JVB 2023-04-24 14:13:30.050 INFO: [1388] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=beb63537 stats_id=Mike-9nC local_ufrag=6qedo1gupppi3a] IceTransport.startConnectivityEstablishment#199: Starting the Agent without remote candidates.

JVB 2023-04-24 14:13:30.050 INFO: [1388] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=beb63537 stats_id=Mike-9nC local_ufrag=6qedo1gupppi3a ufrag=6qedo1gupppi3a] Agent.startConnectivityEstablishment#736: Start ICE connectivity establishment.

JVB 2023-04-24 14:13:30.050 INFO: [1388] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=beb63537 stats_id=Mike-9nC local_ufrag=6qedo1gupppi3a ufrag=6qedo1gupppi3a] Agent.initCheckLists#972: Init checklist for stream stream-beb63537

JVB 2023-04-24 14:13:30.050 INFO: [1388] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=beb63537 stats_id=Mike-9nC local_ufrag=6qedo1gupppi3a ufrag=6qedo1gupppi3a] Agent.setState#946: ICE state changed from Waiting to Running.

JVB 2023-04-24 14:13:30.050 INFO: [1388] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=beb63537 stats_id=Mike-9nC local_ufrag=6qedo1gupppi3a] IceTransport.iceStateChanged#342: ICE state changed old=Waiting new=Running

JVB 2023-04-24 14:13:30.053 INFO: [1388] [confId=999f4285986e5b44 conf_name=test123456@conference.jitsi.xyz.com meeting_id=ade2f123 epId=beb63537 stats_id=Mike-9nC local_

Please suggest or help me on this ? Let me know if you need any further info.

Have you open udp 10000 port on your EC2 instance in the security group?

Yes, I enabled UDP port 10000 in my security group as well as opened in my EC2 instance.

root@jitsi:/usr/bin# sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
80/tcp                     ALLOW IN    Anywhere
443/tcp                    ALLOW IN    Anywhere
10000/udp                  ALLOW IN    Anywhere
22/tcp                     ALLOW IN    Anywhere
3478/udp                   ALLOW IN    Anywhere
5349/tcp                   ALLOW IN    Anywhere
80/tcp (v6)                ALLOW IN    Anywhere (v6)
443/tcp (v6)               ALLOW IN    Anywhere (v6)
10000/udp (v6)             ALLOW IN    Anywhere (v6)
22/tcp (v6)                ALLOW IN    Anywhere (v6)
3478/udp (v6)              ALLOW IN    Anywhere (v6)
5349/tcp (v6)              ALLOW IN    Anywhere (v6)

One thing here is I have a Network load balancer where I created a listener for port TCP- 443 and under that listener I registered my EC2 instance where jitsi is running.

So the flow is like below :-
Client → NLB [TCP(443)] → EC2 (jitsi) [All desired ports are open as mentioned in doc]

Below is the detail of service running on my ec2 ;-

root@jitsi:/usr/bin# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      33445/nginx: master
tcp        0      0 0.0.0.0:5280            0.0.0.0:*               LISTEN      33321/lua5.2
tcp        0      0 0.0.0.0:5222            0.0.0.0:*               LISTEN      33321/lua5.2
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      33445/nginx: master
tcp        0      0 0.0.0.0:5269            0.0.0.0:*               LISTEN      33321/lua5.2
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      670/sshd: /usr/sbin
tcp6       0      0 :::443                  :::*                    LISTEN      33445/nginx: master
tcp6       0      0 :::5280                 :::*                    LISTEN      33321/lua5.2
tcp6       0      0 :::9090                 :::*                    LISTEN      33323/java
tcp6       0      0 :::5222                 :::*                    LISTEN      33321/lua5.2
tcp6       0      0 :::80                   :::*                    LISTEN      33445/nginx: master
tcp6       0      0 127.0.0.1:8080          :::*                    LISTEN      33323/java
tcp6       0      0 :::5269                 :::*                    LISTEN      33321/lua5.2
tcp6       0      0 :::22                   :::*                    LISTEN      670/sshd: /usr/sbin
tcp6       0      0 :::8888                 :::*                    LISTEN      33334/java
udp        0      0 XX.XX.XX.XX:3478       0.0.0.0:*                           18653/turnserver
udp        0      0 XX.XX.XX.XX:3478       0.0.0.0:*                           18653/turnserver
udp        0      0 127.0.0.1:3478          0.0.0.0:*                           18653/turnserver
udp        0      0 127.0.0.1:3478          0.0.0.0:*                           18653/turnserver
udp        0      0 XX.XX.XX.XX:3479       0.0.0.0:*                           18653/turnserver
udp        0      0 XX.XX.XX.XX:3479       0.0.0.0:*                           18653/turnserver
udp        0      0 127.0.0.1:3479          0.0.0.0:*                           18653/turnserver
udp        0      0 127.0.0.1:3479          0.0.0.0:*                           18653/turnserver
udp        0      0 0.0.0.0:68              0.0.0.0:*                           458/dhclient
udp        0      0 127.0.0.1:323           0.0.0.0:*                           672/chronyd
udp        0      0 0.0.0.0:5000            0.0.0.0:*                           33323/java
udp6       0      0 ::1:3478                :::*                                18653/turnserver
udp6       0      0 ::1:3478                :::*                                18653/turnserver
udp6       0      0 ::1:3479                :::*                                18653/turnserver
udp6       0      0 ::1:3479                :::*                                18653/turnserver
udp6       0      0 ::1:323                 :::*                                672/chronyd
udp6       0      0 fe80::8a7:b9ff:fed4:546 :::*                                551/dhclient
udp6       0      0 :::5000                 :::*                                33323/java
udp6       0      0 XX.XX.XX.XX:10000      :::*                                33323/java
root@jitsi:/usr/bin#

Note - I replaced EC2 private IP with XX.XX.XX.XX

Does NLB support UDP?

Yes It support UDP protocol as well but to use it I have to create a listener.
Do you think I should use UDP instead of TCP in my NLB listener ?
If yes then what port should I use for UDP in NLB ?

Or is there something else I need to do.

You need to make sure UDP packages on port 10000 reach your jvb. I have no experience with NLB to give you advice on that.
Try without NLB and assign a public address to your instance and allow port 10000 UDP in the security group.

Hi @damencho ,

Greeting,

I m trying to debug above issue and now trying to deploy it in instance without NLB.

And there I m getting error in nginx :-

Error -

you should increase server_names_hash_bucket_size: 64
root@ip-XX.XX.XX.XX:~# nginx -t
nginx: [emerg] could not build server_names_hash, you should increase server_names_hash_bucket_size: 64
nginx: configuration file /etc/nginx/nginx.conf test failed
root@ip-XX.XX.XX.XX:~#
root@ip-XX.XX.XX.XX:~# systemctl status nginx
â—Ź nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2023-04-25 15:23:08 UTC; 14min ago
       Docs: man:nginx(8)
    Process: 18397 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)
        CPU: 17ms

Apr 25 15:23:08 ip-XX.XX.XX.XX systemd[1]: Starting A high performance web server and a reverse proxy server...
Apr 25 15:23:08 ip-XX.XX.XX.XX nginx[18397]: nginx: [emerg] could not build server_names_hash, you should increase server_names_hash_bucket_size: 64
Apr 25 15:23:08 ip-XX.XX.XX.XX nginx[18397]: nginx: configuration file /etc/nginx/nginx.conf test failed
Apr 25 15:23:08 ip-XX.XX.XX.XX systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Apr 25 15:23:08 ip-XX.XX.XX.XX systemd[1]: nginx.service: Failed with result 'exit-code'.
Apr 25 15:23:08 ip-XX.XX.XX.XX systemd[1]: Failed to start A high performance web server and a reverse proxy server.

Any idea how to fix this error ?

Is it possible if we update value from 64 to 128 under this file : :

Like - server_names_hash_bucket_size: 128 in below config file -
/etc/nginx/sites-enabled/jitsi.xyz.com.conf ??

Yep.

Hi @damencho,

I installed jitsi on my public EC2 instance and able to access the application through URL and 2 or 3 user able to connect the conference call and access the audio and video.

However I would like to understand the role of UDP PORT 10000.
because when I m trying to do telnet to my jitsi app url - I m able to connect with 443.
But telnet is not connecting to 10000 for jitsi app ?

UDP 10000 is the port where clients send media to the bridge and from there they are getting the media for the rest of the participants.
Port 443 TCP is used to download client and to connect to prosody for the signalling.

Regarding NLB, you cannot route media to JVB through NLB, because NLB always preserves the client IP when forwarding UDP, so JVB will send media back directly to the client (not through the NLB), so the source IP will not match what the client expects.

2 Likes

@jbg ,

Thanks for replying on this, I was really looking for someone to discussion on this and guide.

I did a quick setup in my AWS environment, where I have an EC2 instance (Jitsi installed) running in Public subnet and I created a Network load balancer with two listener ( TCP 443 and UDP 10000) and registered my EC2 instance on it.

I did a testing of above setup and I confirm you that It is working for 3+ user.
Users are able to join the call and access the audio/Video without any issue so far.

Any idea How it is working then ?

Most likely, the JVB is advertising its public address and media is flowing directly to it. (This is how JVB works by default — if you wanted media to flow to the NLB you would have needed to manually configure a mapping in JVB to tell it to advertise one of the NLB’s addresses — and then you would have hit the problem with packets sourcing from the “wrong” address — the JVB’s public IP — in the other direction.)

You can likely just delete the udp/10000 listener on your NLB.

1 Like

So you are saying that we can’t directly use NLB in front of EC2 instance ?

the flow is like below -
Incoming Traffic :
Client (anyone fron Internet) —>Jitsi URL(DNS Record) → AWS NLB (TCP/UDP Listener) → EC2 instance(Jitsi running).

Can you explain the outbound in this case ?
BTW - NLB use private IP of ec2 instance for routing.

It doesn’t matter how the NLB is configured, you can’t use it for JVB.

You can use it for nginx and/or prosody if you want to (web traffic for serving the frontend and XMPP websocket/BOSH for the signalling), although an ALB really makes more sense in that context.

In brief: after client connects to Prosody over XMPP websocket/BOSH, Jicofo allocates channels for them on the JVB, and tells the client the IP address of the JVB. If you don’t override it manually, that will be the public IP address of the JVB (detected automatically by JVB on startup using AWS IMDS or STUN), so your NLB will be bypassed. If you do override it manually by configuring a NAT mapping on JVB to tell it that its public IP is the NLB, then Jicofo will signal JVB’s address as the NLB, and client will connect to the NLB, but JVB will send packets in the other direction from its own IP, and connectivity will be broken.

I’m curious, why do you want to use an NLB in front of JVB?

2 Likes

We have Private instance and we can’t directly access it through URL over the internet.(if I’m not wrong)
And we have no idea about the consequences of using NLB.

Thank you so much for detail explanation and saving my time. :pray:
We are now redesigning the architecture.

Btw, Is there any Jitsi official document to install it on Kubernetes cluster ?

Not official, but there’s this in the contrib repo. I haven’t looked at it yet, so can’t vouch for it one way or another.

@jbg ,

Greeting,

So we redesign our architecture and now trying to put HAProxy instead of NLB.

Our problem is we are deploying Jitsi in Private subnet with NAT, hence we need something in public subnet front for Jitsi instance.

Ques - Is it possible to successfully implement HAProxy ?
Do you have any other suggestion or solution ?

Thanks in advance.