Jitsi-videobridge: Voice/video not working for client on public LTE network

mr_johnson22 · February 16, 2020, 11:37pm

My jitsi-videobridge is hosted on a Linux machine in my local network, and is connected to a publically-accessible endpoint via an OpenVPN tunnel. The companion Jitsi components (prosody, jicofo, jitsi-meet) all work, as clients from any network can join a conference room, see each other’s microphone/camera status, and send text messages.

The only problem is that when I join a conference from Jitsi Android via a mobile LTE connection, no audio or video is sent or received. The other member of the conference is a desktop web browser on my local network connecting to my self-hosted jitsi-meet. Only when my phone is on the local network (via Wi-Fi) does voice/video work properly.

I have already set $HOME/.sip-communicator/sip-communicator.properties to include the videobridge host’s private & public IP addresses:

org.jitsi.impl.neomedia.transform.srtp.SRTPCryptoContext.checkReplay=false
org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=<local IP address>
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=<public IP address>

(It doesn’t make a difference whether a space is included after the equals sign, as suggested by [Solved]NAT STUN ICE4J traffic goes to private IP address intead of public one)

Despite having done this, the videobridge’s logs show that the host’s local IPv6 address is used to connect to the Android client:

ConnectivityCheckClient$PaceMaker.run#922: Pair failed: [<jvb host's private IPv6 address>]:10000/udp/host -> [<phone's public IPv6 address>]:54789/udp/host (stream-6e810773.RTP)

I did not configure the videobridge to use turn/stun, but that shouldn’t matter because I disabled peer-to-peer mode in both my jitsi-meet’s config.js, and in my Jitsi Android’s advanced settings.

I have already opened all necessary ports on my firewall. The same firewall rules for Jitsi are used for both the local network interface and the VPN tunnel interface, and since local connections work, the firewall cannot be the issue. I also used iptables on the VPN server to properly forward traffic to my host. I have several other public web services running under this setup, so it’s not as if my host isn’t reachable via public networks at all.

The only explanation I can think of is that sip-communicator.properties isn’t being used. How can I ensure that it is being seen by the videobridge process?

EDIT: I should also mention that my host is Debian 10, and since it has an ARM CPU, I had to follow the manual install guide to install everything.

damencho · February 17, 2020, 8:24am

When you start jvb it prints the used config. Port 10000 udp is forwarded from public address to the private?

mr_johnson22 · February 17, 2020, 4:00pm

This is what the log prints right after jvb starts up:

INFO: [1] LegacyConfigFileLoader$Companion.load#40: Attempting to load legacy config file at path null, null, sip-communicator.properties
INFO: [1] LegacyConfigFileLoader$Companion.load#61: No legacy config file found: java.lang.NullPointerException
INFO: [1] LegacyConfigFileLoader$Companion.load#40: Attempting to load legacy config file at path null, null, sip-communicator.properties
INFO: [1] LegacyConfigFileLoader$Companion.load#61: No legacy config file found: java.lang.NullPointerException
INFO: [1] LegacyConfigFileLoader$Companion.load#40: Attempting to load legacy config file at path null, null, sip-communicator.properties
INFO: [1] LegacyConfigFileLoader$Companion.load#61: No legacy config file found: java.lang.NullPointerException
INFO: [1] LegacyConfigFileLoader$Companion.load#40: Attempting to load legacy config file at path null, null, sip-communicator.properties
INFO: [1] LegacyConfigFileLoader$Companion.load#61: No legacy config file found: java.lang.NullPointerException
INFO: [12] ConfigurationActivator.start#45: Registered the LegacyConfigurationServiceShim in OSGi.
INFO: [12] AbstractVersionActivator.start#91: VersionService registered: JVB 2.1.SNAPSHOT
INFO: [12] AbstractJettyBundleActivator.start#613: Not starting the Jetty service for org.jitsi.videobridge.rest.RESTBundleActivator(port=8080)
INFO: [12] AbstractJettyBundleActivator.start#613: Not starting the Jetty service for org.jitsi.videobridge.rest.PublicRESTBundleActivator(port=-1)
INFO: [12] AbstractJettyBundleActivator.start#613: Not starting the Jetty service for org.jitsi.videobridge.rest.PublicClearPortRedirectBundleActivator(port=8080)
INFO: [12] UlimitCheck.printUlimits#115: Running with open files limit 1048576 (hard 1048576), thread limit 13714 (hard 13714).
INFO: [12] VideobridgeExpireThread.start#92: Starting with 60 second interval.
WARNING: [12] Videobridge.start#918: No authorized source regexp configured. Will accept requests from any source.
SEVERE: [16] Health.doRun#300: Health check failed in 37ms:
java.lang.Exception: No XMPP components
	at org.jitsi.videobridge.health.Health.checkXmppConnection(Health.java:186)
	at org.jitsi.videobridge.health.Health.doCheck(Health.java:146)
	at org.jitsi.videobridge.health.Health.doRun(Health.java:266)
	at org.jitsi.utils.concurrent.PeriodicRunnableWithObject.run(PeriodicRunnableWithObject.java:87)
	at org.jitsi.utils.concurrent.RecurringRunnableExecutor.run(RecurringRunnableExecutor.java:216)
	at org.jitsi.utils.concurrent.RecurringRunnableExecutor.runInThread(RecurringRunnableExecutor.java:292)
	at org.jitsi.utils.concurrent.RecurringRunnableExecutor.access$000(RecurringRunnableExecutor.java:36)
	at org.jitsi.utils.concurrent.RecurringRunnableExecutor$1.run(RecurringRunnableExecutor.java:328)
INFO: [12] LegacyConfigFileLoader$Companion.load#40: Attempting to load legacy config file at path null, null, sip-communicator.properties
INFO: [12] LegacyConfigFileLoader$Companion.load#61: No legacy config file found: java.lang.NullPointerException
INFO: [12] LegacyConfigFileLoader$Companion.load#40: Attempting to load legacy config file at path null, null, sip-communicator.properties
INFO: [12] LegacyConfigFileLoader$Companion.load#61: No legacy config file found: java.lang.NullPointerException
INFO: [12] OctoRelayService.start#62: Octo relay is disabled.
INFO: [21] ComponentBase.loadConfig#202: Component org.jitsi.videobridge. config: 
INFO: [21] ComponentBase.loadConfig#203:   ping interval: 10000 ms
INFO: [21] ComponentBase.loadConfig#204:   ping timeout: 5000 ms
INFO: [21] ComponentBase.loadConfig#205:   ping threshold: 3
INFO: [17] org.ice4j.ice.harvest.AwsCandidateHarvester.obtainEC2Addresses: We failed to obtain EC2 instance addresses for the following reason: 
java.lang.IllegalArgumentException: hostname can't be null
	at java.base/java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
	at java.base/java.net.InetSocketAddress.<init>(InetSocketAddress.java:216)
	at org.ice4j.TransportAddress.<init>(TransportAddress.java:61)
	at org.ice4j.ice.harvest.AwsCandidateHarvester.obtainEC2Addresses(AwsCandidateHarvester.java:112)
	at org.ice4j.ice.harvest.AwsCandidateHarvester.getFace(AwsCandidateHarvester.java:153)
	at org.ice4j.ice.harvest.MappingCandidateHarvesters.maybeAdd(MappingCandidateHarvesters.java:222)
	at org.ice4j.ice.harvest.MappingCandidateHarvesters.prune(MappingCandidateHarvesters.java:206)
	at org.ice4j.ice.harvest.MappingCandidateHarvesters.initialize(MappingCandidateHarvesters.java:180)
	at java.base/java.lang.Thread.run(Thread.java:834)
INFO: [17] org.ice4j.ice.harvest.AwsCandidateHarvester.obtainEC2Addresses: String for local IP: <*MY PUBLIC IPv4 ADDRESS*>
INFO: [17] org.ice4j.ice.harvest.AwsCandidateHarvester.obtainEC2Addresses: String for public IP: null
INFO: [17] org.ice4j.ice.harvest.MappingCandidateHarvesters.maybeAdd: Discarding a mapping harvester: org.ice4j.ice.harvest.AwsCandidateHarvester, face=/<*MY PUBLIC IPv4 ADDRESS*>, mask=null
INFO: [17] org.ice4j.ice.harvest.MappingCandidateHarvesters.initialize: Initialized mapping harvesters (delay=1858ms).  stunDiscoveryFailed=false
INFO: [16] Videobridge.createConference#326: create_conf, id=177ce0e49a9094e1 gid=null logging=false
INFO: [16] TaskPools.<clinit>#81: TaskPools detected 4 processors, creating the CPU pool with that many threads
INFO: [16] org.ice4j.ice.harvest.AbstractUdpListener.<init>: Initialized AbstractUdpListener with address <*MY PRIVATE IPv4 ADDRESS ON tun0*>:10000/udp. Receive buffer size 163840 (asked for 10485760)
INFO: [16] org.ice4j.ice.harvest.SinglePortUdpHarvester.<init>: Initialized SinglePortUdpHarvester with address <*MY PRIVATE IPv4 ADDRESS ON tun0*>:10000/udp
INFO: [16] org.ice4j.ice.harvest.AbstractUdpListener.<init>: Initialized AbstractUdpListener with address [<*MY PRIVATE IPv6 ADDRESS ON eth0*>]:10000/udp. Receive buffer size 163840 (asked for 10485760)
INFO: [16] org.ice4j.ice.harvest.SinglePortUdpHarvester.<init>: Initialized SinglePortUdpHarvester with address [<*MY PRIVATE IPv6 ADDRESS ON eth0*>]:10000/udp
INFO: [16] org.ice4j.ice.harvest.AbstractUdpListener.<init>: Initialized AbstractUdpListener with address <*MY PRIVATE IPv4 ADDRESS ON eth0*>:10000/udp. Receive buffer size 163840 (asked for 10485760)
INFO: [16] org.ice4j.ice.harvest.SinglePortUdpHarvester.<init>: Initialized SinglePortUdpHarvester with address <*MY PRIVATE IPv4 ADDRESS ON eth0*>:10000/udp

tun0 is the OpenVPN interface.

Is the problem is that tun0's private IPv6 address isn’t used? But then again, isn’t NAT only an IPv4 concept?

If it makes a difference, this same server has coturn configured to allow p2p WebRTC calls (not handled by jitsi), and calls made from my phone on LTE work just fine.

damencho · February 17, 2020, 8:19pm

Missing public ip, will not work. So definitely your config is not used.
How do you start jvb?

mr_johnson22 · February 17, 2020, 8:36pm

In the jitsi-videobridge folder cloned from git, I ran:

mvn package -DskipTests -Dassembly.skipAssembly=false

and unzipped the SNAPSHOT-archive.zip that was created. Then I cd into the unzipped directory and run jvb with

sudo -u my_jitsi_user ./jvb.sh --host=localhost --domain=jitsi.mydomain.tld --port=5347 --secret=YOURSECRET1_from_manual_install_guide

The .sip-communicator folder is in the home directory of my_jitsi_user.

I originally tried the mvn compile exec:exec command suggested by JVB’s README, but doing that gives me wrong ELF class errors (likely because of using ARM). I also had to rebuild & install jitsi-sctp in order for even the packaged jvb to work.

damencho · February 17, 2020, 9:19pm

You can use some params settings to point the folder where the properties file is: https://github.com/jitsi/jitsi-videobridge/blob/66995a1a492da34fd8d10bedbf740e0050cf5aff/debian/postinst#L89

mr_johnson22 · February 17, 2020, 9:46pm

I tried changing some params but the problem is still there.

In jvb.sh, I added the following line:

JAVA_SYS_PROPS="-Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/var/lib/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=.sip-communicator -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi"

I did this because jvb.sh uses $JAVA_SYS_PROPS as an argument to the java process it launches. The config file is indeed located at /var/lib/jitsi/.sip-communicator/sip-communicator.properties.

Those params do get picked up by the JVB process, but a public IP is still missing:

INFO: [12] LegacyConfigFileLoader$Companion.load#40: Attempting to load legacy config file at path /var/lib/jitsi, .sip-communicator, sip-communicator.properties
INFO: [12] LegacyConfigFileLoader$Companion.load#40: Attempting to load legacy config file at path /var/lib/jitsi, .sip-communicator, sip-communicator.properties
INFO: [12] OctoRelayService.start#62: Octo relay is disabled.
INFO: [17] org.ice4j.ice.harvest.AwsCandidateHarvester.obtainEC2Addresses: We failed to obtain EC2 instance addresses for the following reason: 
java.lang.IllegalArgumentException: hostname can't be null
	at java.base/java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
	at java.base/java.net.InetSocketAddress.<init>(InetSocketAddress.java:216)
	at org.ice4j.TransportAddress.<init>(TransportAddress.java:61)
	at org.ice4j.ice.harvest.AwsCandidateHarvester.obtainEC2Addresses(AwsCandidateHarvester.java:112)
	at org.ice4j.ice.harvest.AwsCandidateHarvester.getFace(AwsCandidateHarvester.java:153)
	at org.ice4j.ice.harvest.MappingCandidateHarvesters.maybeAdd(MappingCandidateHarvesters.java:222)
	at org.ice4j.ice.harvest.MappingCandidateHarvesters.prune(MappingCandidateHarvesters.java:206)
	at org.ice4j.ice.harvest.MappingCandidateHarvesters.initialize(MappingCandidateHarvesters.java:180)
	at java.base/java.lang.Thread.run(Thread.java:834)
INFO: [17] org.ice4j.ice.harvest.AwsCandidateHarvester.obtainEC2Addresses: String for local IP: <MY PUBLIC IPv4 ADDRESS>
INFO: [17] org.ice4j.ice.harvest.AwsCandidateHarvester.obtainEC2Addresses: String for public IP: null
INFO: [17] org.ice4j.ice.harvest.MappingCandidateHarvesters.maybeAdd: Discarding a mapping harvester: org.ice4j.ice.harvest.AwsCandidateHarvester, face=/<MY PUBLIC IPv4 ADDRESS>, mask=null
INFO: [17] org.ice4j.ice.harvest.MappingCandidateHarvesters.initialize: Using org.ice4j.ice.harvest.MappingCandidateHarvester, face=/<MY PRIVATE IPv4 ADDRESS ON tun0>, mask=/<MY PUBLIC IPv4 ADDRESS>
INFO: [17] org.ice4j.ice.harvest.MappingCandidateHarvesters.initialize: Initialized mapping harvesters (delay=2469ms).  stunDiscoveryFailed=false

The public IP is null as it was before. I confirmed that voice/video still doesn’t work from my phone on LTE, and that the same Pair failed message got printed in the logs.

mr_johnson22 · February 18, 2020, 2:11pm

Actually, since my setup doesn’t use AWS at all, is the AwsCandidateHarvester really needed?

With that said, do the messages for the MappingCandidateHarvesters look correct?

damencho · February 18, 2020, 9:08pm

Yep, that is not needed. You can even enable just stun harvester to auto discover addresses.

github.com

jitsi/docker-jitsi-meet/blob/8bfe7fbb922edbbfa38e2058c05334d3665577b7/jvb/rootfs/defaults/sip-communicator.properties#L5


org.jitsi.videobridge.SINGLE_PORT_HARVESTER_PORT={{ .Env.JVB_PORT }}
org.jitsi.videobridge.DISABLE_TCP_HARVESTER={{ .Env.JVB_TCP_HARVESTER_DISABLED }}
org.jitsi.videobridge.TCP_HARVESTER_PORT={{ .Env.JVB_TCP_PORT }}
{{ if .Env.JVB_STUN_SERVERS }}
org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES={{ .Env.JVB_STUN_SERVERS }}
{{ end }}


org.jitsi.videobridge.xmpp.user.shard.HOSTNAME={{ .Env.XMPP_SERVER }}
org.jitsi.videobridge.xmpp.user.shard.DOMAIN={{ .Env.XMPP_AUTH_DOMAIN }}
org.jitsi.videobridge.xmpp.user.shard.USERNAME={{ .Env.JVB_AUTH_USER }}
org.jitsi.videobridge.xmpp.user.shard.PASSWORD={{ .Env.JVB_AUTH_PASSWORD }}
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS={{ .Env.JVB_BREWERY_MUC }}@{{ .Env.XMPP_INTERNAL_MUC_DOMAIN }}
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME={{ .Env.HOSTNAME }}
org.jitsi.videobridge.xmpp.user.shard.DISABLE_CERTIFICATE_VERIFICATION=true

github.com

jitsi/docker-jitsi-meet/blob/8bfe7fbb922edbbfa38e2058c05334d3665577b7/env.example#L182


# MUC for the JVB pool.
JVB_BREWERY_MUC=jvbbrewery


# XMPP user for JVB client connections.
JVB_AUTH_USER=jvb


# XMPP password for JVB client connections.
JVB_AUTH_PASSWORD=passw0rd


# STUN servers used to discover the server's public IP.
JVB_STUN_SERVERS=stun.l.google.com:19302,stun1.l.google.com:19302,stun2.l.google.com:19302


# Media port for the Jitsi Videobridge
JVB_PORT=10000


# TCP Fallback for Jitsi Videobridge for when UDP isn't available
JVB_TCP_HARVESTER_DISABLED=true
JVB_TCP_PORT=4443


# A comma separated list of APIs to enable when the JVB is started. The default is none.
# See https://github.com/jitsi/jitsi-videobridge/blob/master/doc/rest.md for more information

mr_johnson22 · February 19, 2020, 12:03am

Isn’t STUN/TURN only needed for p2p connections, though?

mr_johnson22 · February 20, 2020, 3:19am

Aha, disabling the IPv6 listener fixes the problem (org.ice4j.ipv6.DISABLED=true in sip-communicator.properties). There must be something wrong with my server’s IPv6 configuration. I can deal with that later though.

Thanks for the help!

Frank21777 · February 26, 2020, 10:26pm

I have the same issue on my default installed jitsi Server. My Server is hosted on a IONOS dedicated Server based in Germany. (ipv4 and ipv6 enabled)
I have installed the jitsi components according https://www.howtoforge.com/tutorial/how-to-create-your-own-video-conference-using-jitsi-meet-on-ubuntu-1804/

Issue1: as soon as one of to participants are connected via mobile network (no Wifi) voice and video are not working

issue2 as soon a third (ore more) participant joins the conference voice and cideo are not working.

damencho · February 27, 2020, 7:58am

This sounds like not fully configured jvb for poblic and private address, not forwarded port or firewall issue. Checkout the official quick install guide, advanced section for more details. https://jitsi.org/qi

mr_johnson22 · February 28, 2020, 5:55pm

I’m back again; the problem isn’t fully solved. Voice/video only works sometimes.

I have ruled out the following as being problems:

IPv6: I disabled the listener, and there are even times when my phone’s IP address on LTE is IPv4.
Peer-to-peer: Disabling or enabling p2p settings in the Jitsi Meet Android app or in the Meet server’s config.js doesn’t singlehandedly fix/break anything.
TURN: I copied some settings from a recent pull request to connect my videobridge to a coTURN server (which I already had running properly for another VoIP service). jvb logs suggest that the TURN setup isn’t broken, but voice/video still doesn’t work. (I’m using default ports for TURN, not 443.)
Firewall settings: All required ports are open, and I even disabled listeners on local interfaces to ensure that peers may only connect via public interfaces.

Maybe this jvb log message is a clue of what’s going wrong:

WARNING: [40] ComponentBase.verifyProcessingTime#540: PROCESSING TIME LIMIT EXCEEDED - it took 101ms to process: <blablabla...>

How could 101ms possibly exceed any sort of time limit…?

mr_johnson22 · February 29, 2020, 8:57am

Good news: by using nginx’s ssl prereading to let the TURN TLS listener use port 443, calls work consistently! I even get to keep hosting all of my other web services, and the new TURN setup still works for the other VoIP service I’m hosting.

Logs still print the PROCESSING TIME LIMIT messages, so that was a red herring in the end.

@damencho Your pull request is what guided me through setting all of this up; thank you very much for writing it! (If I was able to use the Debian package, I probably never would have had all of this trouble, but alas, I’m on ARM. On the bright side, I got to learn a lot from this!)

mr_johnson22 · March 1, 2020, 4:50am

Argh…! I spoke too soon. It’s still not working consistently…

I’ll keep my mouth shut until things stay fixed for more than just a few quick tests.

Maybe a relevant comment: I had to use SNI instead of ALPN when using nginx’s ssl prereading, since 1) there were no ALPN sections in coturn’s ClientHello messages (which I confirmed with Wireshark), 2) sending all non-HTTP traffic to coturn also routed what appeared to be incoming DNS queries to it (which I didn’t expect to be getting in the first place…), and 3) I use a dedicated subdomain for turn traffic and coturn does include a SNI section in ClientHello (confirmed with Wireshark).

And I know for sure that coturn is correctly listening on 443, as it passes tests on the WebRTC troubleshooter and Trickle ICE.

mr_johnson22 · March 4, 2020, 6:23am

The only reasonable explanation is that something is wrong with my jvb setup, or with Jitsi Meet on Android.

It can’t be that the network is blocking packets, as I was able to have a sound-enabled conference between my phone on LTE and a laptop on a corporate firewall that blocks non-web traffic.
I know for sure that my nginx & coturn are set up correctly, as I’m able to host a Matrix server that can always facilitate WebRTC calls using the same TURN settings that jvb is using.
What seems to always fix the problem is to connect my phone to my local WiFi, then connecting back to LTE and joining a conference. Calls continue to work until I force-quit Jitsi Android and re-open it.

This might be a key clue: for any failed jvb connection, this error appears in my coturn logs:

closed (2nd stage), user <> realm <turn.domain.tld> origin <>, local 10.8.0.4:5349, remote 10.8.0.4:39940, reason: TLS/TCP socket buffer operation error (callback)

Note that nginx is forwarding encrypted turn traffic on public port 443 to private port 5349.

I also tried using the non-TLS listener, but that sometimes hits different coturn errors (I don’t have the log for that at the moment). It seems that Jitsi Meet on Android is sometimes creating malformed TURN connections.

Is there some problem with the Android version of Jitsi Meet where it doesn’t make TURN connections properly? I tried both the F-Droid version and the Google Play version.

mr_johnson22 · March 26, 2020, 10:51pm

The problem might be due to coturn. I’ll open a new thread.