New Jitsi Meet Install - No audio/video - Websockets failing

Install & Config
,
1

Hi, having 2 issues on a new install on Ubuntu 20.04.2.

  • P2P 2 participant meeting works great.
  • 3+ participant meeting not working with audio or video. Not working with all 3 participants on same local network as JMS. Not working with all 3 participants on an external network.
  • Always have colibri-ws websocket 403 failures.

Any help is appreciated in figuring out why there is no audio/video for 3+ participant sessions.

Expectation:

  • 3+ participant audio and video seamless transition of audio/video when the third participant joins the p2p session.
  • /colibri-ws webscket calls with status 101 switching protocols, then changing to 200 okay status codes.

Re: no audio/video with 3+ people.

  • I have verified port 10001/udp is externally accessible trough meet.collectivetech.ca to the local machine through the reverse proxy. Externally and not from within the local network. Form this article. (Tip: how to check UDP/10000 connectivity)
  • I have been researching and scouring this site for over a week with no resolutions to this audio/video problem and I’m failing to see any issues within the logs or .conf.

Re: colibri-ws websocket failing.

  • On the same server, my mattemost instance’s websockets are connecting and are also running through the same reverse proxy from the public IP to this local machine. So i’m ruling the reverse proxy out.
  • 403 errors from Private-machine’s nginx/access.log
  • Request headers have “Connection: Upgrade” and “Upgrade: websocket”.

Server Setup:
Only one jitsi-meet, jvb and prosody instance running behind an nginx stream reverse proxy on a different machine. I believe this is the simplest setup but with port 10001 instead of 10000 and using manual SSL certificates are the only changes.

  • Openjdk 11.0.17 2022-10-18

External network:

  • 80, 443, 10001/udp open.

Reverse Proxy:

  • Simple nginx stream using SNI to pass port 443 requests to local-machine’s port 8445
  • Simple nginx http proxy passing 80 to local machine’s port 8083
  • Firewall is open for recommended ports
Status: active
To                         Action      From
--                         ------      ----
10001                      ALLOW       Anywhere                  
10001/udp                  ALLOW       Anywhere                  
9090                       ALLOW       Anywhere                  
5280                       ALLOW       Anywhere                  
5281                       ALLOW       Anywhere                  
10000                      ALLOW       Anywhere                  
80                         ALLOW       Anywhere                  
443                        ALLOW       Anywhere                  
8445                       ALLOW       Anywhere                  
10001 (v6)                 ALLOW       Anywhere (v6)             
10001/udp (v6)             ALLOW       Anywhere (v6)             
9090 (v6)                  ALLOW       Anywhere (v6)             
5280 (v6)                  ALLOW       Anywhere (v6)             
5281 (v6)                  ALLOW       Anywhere (v6)             
10000 (v6)                 ALLOW       Anywhere (v6)             
80 (v6)                    ALLOW       Anywhere (v6)             
443 (v6)                   ALLOW       Anywhere (v6)             
8445 (v6)                  ALLOW       Anywhere (v6)

Private local server running JMS, JVB and Prosody:

  • Nginx server listening on port http on 8083 and https on 8445.
  • Internal Hashed authenticated server running at https :// meet.collectivetech.ca
  • JVB listening on port 10001.
  • Have verified 10001/udp messages are received on this server with netcat tests.
  • Firewall is open for recommended ports
Status: active

To                         Action      From
--                         ------      ----
5900                       ALLOW       Anywhere                  
80/tcp                     ALLOW       Anywhere                  
443/tcp                    ALLOW       Anywhere                  
10000/udp                  ALLOW       Anywhere                  
3478/udp                   ALLOW       Anywhere                  
5349/tcp                   ALLOW       Anywhere                  
10001/udp                  ALLOW       Anywhere                  
8445                       ALLOW       Anywhere                  
9090                       ALLOW       Anywhere                  
5280                       ALLOW       Anywhere                  
5281                       ALLOW       Anywhere                  
5900 (v6)                  ALLOW       Anywhere (v6)             
80/tcp (v6)                ALLOW       Anywhere (v6)             
443/tcp (v6)               ALLOW       Anywhere (v6)             
10000/udp (v6)             ALLOW       Anywhere (v6)             
3478/udp (v6)              ALLOW       Anywhere (v6)             
5349/tcp (v6)              ALLOW       Anywhere (v6)             
10001/udp (v6)             ALLOW       Anywhere (v6)             
8445 (v6)                  ALLOW       Anywhere (v6)             
9090 (v6)                  ALLOW       Anywhere (v6)             
5280 (v6)                  ALLOW       Anywhere (v6)             
5281 (v6)                  ALLOW       Anywhere (v6)

Config Files

  • jicofo.conf
  • meet.collectivetech.ca.cfg.lua
  • meet.collectivetech.ca-config.js
  • jvb.conf w/ port 10001 change
  • jvb - sip-communicator.properties
  • nginx/sites-available/meet.collectivetech.ca.conf
  • prosody.cfg.lua

Logs from a fresh start
Context: Stopped the tree servers. Removed old logs. Restarted. Connected one participant. Captured logs. Added a second participant over P2P. Captured logs. Connected third participant over P2P. No video/audio. Captured logs. Chrome Version 107.0.5304.121 (Official Build) (x86_64)

    • Chrome console Log after one participant (private chrome window)
    • Chrome console Log after second participant (private chrome window)
    • Chrome console Log after third participant (private chrome window)
    • jvb.log
    • jicofo.log
    • prosody.log
    • prosody.err
    • Local machine’s nginx access.log showing 403s for /colibri-ws websocket calls.

All redacted logs and config files

JITSI versions

dpkg -l | grep jitsi
ii  jitsi-meet                                 2.0.8044-1                                 all          WebRTC JavaScript video conferences
ii  jitsi-meet-prosody                         1.0.6776-1                                 all          Prosody configuration for Jitsi Meet
ii  jitsi-meet-turnserver                      1.0.6776-1                                 all          Configures coturn to be used with Jitsi Meet
ii  jitsi-meet-web                             1.0.6776-1                                 all          WebRTC JavaScript video conferences
ii  jitsi-meet-web-config                      1.0.6776-1                                 all          Configuration for web serving of Jitsi Meet
ii  jitsi-videobridge2                         2.2-61-g98c9f868-1                         all          WebRTC compatible Selective Forwarding Unit (SFU)
ii  lua-basexx                                 0.4.1-jitsi1                               all          baseXX encoding/decoding library for Lua
ii  lua-cjson:amd64                            2.1.0.10-jitsi1                            amd64        JSON parser/encoder for Lua
2

I’m not sure if SINGLE_PORT_HARVESTER_PORT is still valid. You may try the new format by setting port using hocon

hocon -f /etc/jitsi/videobridge/jvb.conf set videobridge.ice.udp.port 10001
3

Tried it and didn’t make a difference. In my old jvb.log there are all these logs stating jvb is using 10001.

Videobridge.createConference#282: create_conf, id=1a58f14858d48ff3 meetingId=9a19aaee-c09b-4302-b9a2-607c78a60340
JVB 2022-11-27 12:20:31.418 INFO: [69] org.ice4j.ice.harvest.AbstractUdpListener.<init>: Initialized AbstractUdpListener with address 172.17.0.1:10001/udp. Receive buffer size 10485760 (asked for 10485760)

my jvb.conf is now with no luck:

videobridge {
    http-servers {
        public {
            port = 9090
        }
    }
    websockets {
        enabled = true
        domain = "meet.collectivetech.ca"
        tls = true
    }
    ice: {
      udp: {
        port: 10001
      }
    }
}
4

Did you check “ICE Candidate pair” in chrome://webrtc-internals/ (in chrome browser)?

5

Hmmm, seems to not be transmitting any data. All the STUN requests are failing.

ICE candidate pair: (not connected)

After 3rd participant joins:

2840×1886 659 KB

6

Did you check UDP/10001 connectivity for these pairs?

7

Not sure how to test that on one local machine. It seems to be always using local IP address.

I noticed my instance isn’t using meet-jit-si-turnrelay.jitsi.net:443 even though it is configured in sip-cmmunicator.properties. Could this be the issue? Is there another setting required to enable org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES?

When using meet.jit.si I have:
https://meet.jit.si/torrin4, { iceServers: [turns:meet-jit-si-turnrelay.jitsi.net:443?transport=tcp], iceTransportPolicy: all, bundlePolicy: max-bundle, rtcpMuxPolicy: require, iceCandidatePoolSize: 0}

When using my server with sip-communicator.properties below
iceServers: [turns:meet.collectivetech.ca:5349?transport=tcp], iceTransportPolicy: all, bundlePolicy: max-bundle, rtcpMuxPolicy: require, iceCandidatePoolSize: 0 }

org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=true
org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=meet-jit-si-turnrelay.jitsi.net:443
org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_TRANSPORT=muc
org.jitsi.videobridge.xmpp.user.shard.HOSTNAME=localhost
org.jitsi.videobridge.xmpp.user.shard.DOMAIN=auth.meet.collectivetech.ca
org.jitsi.videobridge.xmpp.user.shard.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard.PASSWORD=Qp6LYeid
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS=JvbBrewery@internal.auth.meet.collectivetech.ca
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME=87c33380-470e-40ef-8a07-ed98bf9d00d2

org.jitsi.videobridge.SINGLE_PORT_HARVESTER_PORT=10001
org.jitsi.videobridge.xmpp.user.shard.DISABLE_CERTIFICATE_VERIFICATION=true
org.jitsi.videobridge.DISABLE_TCP_HARVESTER=true
8

If remote-candidate IPs are correct then you don’t need to worry about STUN. This means that JVB interfaces are published correctly.

On JVB server run ngrep

ngrep -q 'is accessable' udp port 10001

and try to send a message from the client machine. Repeat this step for each published remote-candidate IPs.

echo 'yes, it is accessable' | nc -u REMOTE_CANDIDATE_IP 10001

If you cannot see the same message on the server side, this means that you have a network/NAT/firewall problem.

9

I believe my remote candidate ips are wrong when using meet.collectivetech.ca. They differ between 3-person conferences.

With meet.jit.si showing my client machine’s public IPs and 1 pair on average.

Whereas meet.collectivetech.ca is using local private IPs and attempting 3-4 pairs that never connect.

2640×1090 325 KB

2604×1458 374 KB

Tried this process with ngrep -q 'is accessable' udp port 10001 on the server and on my client machine echo 'yes, it is accessable' | nc -u 172.19.0.1 10001 with no success.

10

Did you test for each possible IP? You need only one of them.

11

Don’t you have an Internet IP in the list when there are 3+ participants in the same room?

12

I do when I add back these :

org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=X.X.X.X-Local-Private-IP
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=X.X.X.X-Public-IP

2592×1380 372 KB

When trying:
Tried this process with ngrep -q 'is accessable' udp port 10001 on the server and on my external client machine echo 'yes, it is accessable' | nc -u <JVB_PUBLIC_IP> 10001 with no success.

  • This was working when I originally as I tested this many many times last week. But right now, I can’t even get this to work locally on the same network as the JVB’s local private IP. Need to figure this out and get this 10001 port listening again locally, and remotely.
13
  • This is fixed. Noticed that ngrep -q 'is accessable' udp port 10001 was listening on the wrong interface. Had to use ngreg -d <ethernet_device> -q 'is accessable' udp port 10001. Now I can echo 'yes, it is accessable' | nc -u <JVB_PUBLIC_IP> 10001 and I receive, yes, it is accessable on the JVB server.
14

If there is no problem with UDP/10001 then you should have audio/video even websocket fails.

15

Exact. This is where am im stumped.

Is there specific UDP packets I can sniff on <JVB_PUBLIC_IP> that can give me a hint if audio/video is being at least transmitted? I am using Wireshark and filtering IP.addr by my <JVB_PUBLIC_IP> and i’m not seeing any UDP traffic heading out of my external client machine.

16

On JVB server - I get LOTS of traffic with, sudo ngrep -d enp9s0 udp port 10001.
and also verified with Wireshark on JVB server that im getting STUN traffic on port 10001 coming from my remote client’s public IP.

On external client machine i’m actually seeing port 10001 udp traffic. (I thought it had to say UDP instead of STUN, my bad).

2206×1178 614 KB