Audio/Video/NAT problems after fresh install

I’d like to have a server on the LAN which is available to local clients as well as those from the Internet. The server will have a different local IP address than the public IP address.

I followed the quickstart guide: https://jitsi.github.io/handbook/docs/devops-guide/devops-guide-quickstart

I can see all other participants of the chat, use the text chat, see the icon change when they mute/unmute their audio and video, but I do not get any audio or video.

The error from the Firefox Dev Console:
2020-09-27T21:57:58.712Z [modules/RTC/BridgeChannel.js] <_send>: Bridge Channel send: no opened channel.

The router is forwarding 10000/UDP, 443/TCP, and 4443/TCP and there is no firewall on the server. I have a deploy script (Vagrant) that makes it reasonably easy to redeploy when trying different options. I tried a deploy which modified my hosts file, as well as one without. I’ve tried this on Ubuntu 18.04 and Debian 10.

I’ve tried it without modifying /etc/jitsi/videobridge/sip-communicator.properties (as I’ve seen reports that NOT changing that file would help).

I also tried commenting out STUN_MAPPING_HARVESTER_ADDRESSES and specifying NAT_HARVESTER_LOCAL_ADDRESS/NAT_HARVESTER_PUBLIC_ADDRESS (as that has also been reported to help), and once also specifying org.ice4j.ice.harvest.TCP_HARVESTER=4443. If I am not doing a full re-deploy, I restart the videobridge service after each change (systemctl restart jitsi-videobridge2) and reconnect all the clients.

My DNS server will return the LAN IP to clients on the LAN and queries coming from the Internet will get the public IP address. The 443/TCP traffic from the internet does go through a HAProxy box using a TCP proxy (does not terminate TLS connections, only one back end). Local 443/TCP traffic goes directly to the jitsi server. The UDP/10000 traffic goes directly to the jitsi server whether coming from the LAN or the Internet (no HAProxy).

In terms of clients, I’ve tried using the mobile app, and the mobile version of Chrome to see if any of those clients work. They all act the same. I have been testing with 3 clients at a time to make sure I am not going into p2p mode (once I get the server working, I’ll re-test p2p mode).

Right now I’m using Ubuntu 18.04 using the standard configuration (no Advanced Configuration). The audio and video does not work when all clients are local, nor when all clients are remote, nor when there is a mix of local and remote clients. I’m testing with Firefox 68 and Chrome 85. I am not testing using the mobile app because I’m using a self-signed cert until I can get this debugged.

On a related note, I’d like to help contribute to a troubleshooting guide for people facing NAT problems (or just audio/video problems in general). I’m not sure the order of operations is ideal, and these didn’t solve my problem yet, but here’s what I have so far:

General Troubleshooting: The mobile client will silently refuse to connect to self-signed certificates, so it’s best to use web-based clients for initial testing.

Capturing packets using tcpdump or wireshark on a client will be a common debugging technique. You should see UDP packets on port 10000 (wireshark identifies this as “STUN” protocol) going to the server. The traffic on 443 is used for text chat, metadata (like when someone is muted), seeing the list of participants and setting up media streams. All audio and video will be on 10000/UDP in the standard setup (Note: there are server-side options to support clients on networks that block UDP).

Local Troubleshooting: Try connecting to the server with 3 clients from the local network. This eliminates any potential problems with port forwarding. Packet dump on a client to see if the media traffic is going to the internal IP address of the server. (TODO: explain why media wouldn’t be working in this case). If the server is working for local clients, it’s time to move on to testing with clients connecting from the Internet.

Internet Troubleshooting: Try connecting from the internet with 3 clients. Capture traffic to see if the packets are being sent to the internal IP address of the server. If the are, this is a problem (TODO: explain why this happened… e.g., did the videobridge hand this out, meet, another peer, etc.? How did the component that handed it out determine whether the client should get the internal or external).
Solution: see Advanced Configuration. TODO: this didn’t solve my issue, so there should be more troubleshooting steps here.

As a final note, here’s the installation script I am using in case there is an error:

#!/bin/sh -e
# This will install a Jitsi server
# Usage: $0 INTERNAL_IP [EXTERNAL_IP]
# Instructions from multiple sources:
# https://jitsi.github.io/handbook/docs/devops-guide/devops-guide-quickstart
# https://github.com/jitsi/jitsi-meet/issues/6130
export DEBIAN_FRONTEND=noninteractive
FQDN=`hostname -f`
INTERNAL_IP="$1"
EXTERNAL_IP="$2"
if [ -z $EXTERNAL_IP ]; then
        EXTERNAL_IP=`curl icanhazip.com`
fi

# Dependencies
apt-get update
apt-get install -y apt-transport-https software-properties-common
apt-add-repository universe || true  # Ubuntu only
apt-get update

# Import their GPG key and add the repo
curl https://download.jitsi.org/jitsi-key.gpg.key | sudo sh -c 'gpg --dearmor > /usr/share/keyrings/jitsi-keyring.gpg'
echo 'deb [signed-by=/usr/share/keyrings/jitsi-keyring.gpg] https://download.jitsi.org stable/' | sudo tee /etc/apt/sources.list.d/jitsi-stable.list > /dev/null
apt-get update

# Install the package
echo "jitsi-videobridge jitsi-videobridge/jvb-hostname string $FQDN" | debconf-set-selections
echo "jitsi-meet jitsi-meet/cert-choice select Generate a new self-signed certificate (You will later get a chance to obtain a Let's encrypt certificate)" | debconf-set-selections
apt-get install -qy jitsi-meet

# Do advanced configuration in case the server is NATed
conf="/etc/jitsi/videobridge/sip-communicator.properties"
#sed -i 's/\(.*STUN_MAPPING_HARVESTER_ADDRESSES\)/#\1/g' $conf
#echo "org.ice4j.ice.harvest.TCP_HARVESTER=4443" >> $conf
#echo "org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=$INTERNAL_IP" >> $conf
#echo "org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=$EXTERNAL_IP" >> $conf
#systemctl restart jitsi-videobridge2

TL;DR Configuration with UDP working, TCP-only configuration still not working, looking for help debugging this.

I switched from Chrome in private mode to Firefox in private mode and it worked fine from the LAN and the Internet. I was going to document how one could test that their browser works with Jitsi, but now Chrome work fine in private mode too and I’m not sure why I am unable to reproduce my previous results. I’ll still write it up as something like “make sure your browser works with meet.jit.si to rule out any client-side incompatibilities”. (I will edit that section of my original post)

This confirms that what is in the quickstart guide is correct, but it does not contain any information about how to get jitsi to work where UDP isn’t available for all clients. The manual install page does mention configuring TCP_HARVESTER_PORT (https://jitsi.github.io/handbook/docs/devops-guide/devops-guide-manual#install-jitsi-videobridge) but it does not mention that TCP is disabled by default (https://jitsi.github.io/handbook/docs/devops-guide/devops-guide-manual#install-jitsi-videobridge), which means setting the port has no effect. After setting DISABLE_TCP_HARVESTER=false, setting TCP_HARVESTER_PORT=4443, and rebooting, the service does appear to be listening on port 4443.

When I try to connect from a client that can not establish any UDP connections, tcpdump on the client side does show it completes a TCP handshake to port 4333, however the client closes the connection before sending any data. On the server side, I found the following log entry that looks relevant:

2020-10-04 15:22:01.145 INFO: [21] Harvesters.initializeStaticConfiguration#99: Initialized TCP harvester on port 4443, ssltcp=true

I also see that when there are “new Pair Added” entries, none of them use the public IP specified in NAT_HARVESTER_PUBLIC_ADDRESS. I believe that if jitsi-videobridge would use the public IP address for a pair (in addition to the NAT_HARVESTER_LOCAL_ADDRESS for another pair), it should work from both the LAN/INTERNAL_ADDRESS and the Internet/PUBLIC_ADDRESS without relying on UDP.

Does anyone know of any guides on how to configure jitsi-videobridge to work over pure TCP?