Videobridge under NAT doesn't works since JVB2

Jerome_LEMAN_GARIN · July 3, 2020, 3:22pm

Hello !

Since we updated our production servers to JVB2, without changing any configuration, we are now unable to run videobridge behind NAT.

Here is our ~/.sip-communicator/sip-communicator.properties file :

org.ice4j.ipv6.DISABLED=true
org.jitsi.videobridge.TCP_HARVESTER_PORT=80
org.jitsi.videobridge.SINGLE_PORT_HARVESTER_PORT=10000
org.jitsi.impl.neomedia.transform.srtp.SRTPCryptoContext.checkReplay=false
org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=InternalIP
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=ExternalIP

I also tried a lot of configuration (using videobridge directives instead of ice4j, removing the checkReplay directive…)

For the network part : 10000/TCP and 80/TCP are opened (+ 443/TCP for the web part) and checked from end to end using the SocketTest tool.

We also tried to open the 4443/TCP port as mentioned here even if it should be overwriten by the TCP_HARVESTER_PORT directive.

In the logs there is no errors except videobridge trying without success to pair :

INFOS: Add remote candidate for stream-46c8cfa3.RTP: 192.168.8.37:51701/udp/host
INFOS: Starting the agent with remote candidates.
INFOS: Start ICE connectivity establishment.
INFOS: Add remote candidate for stream-46c8cfa3.RTP: 192.168.8.37:51701/udp/host
INFOS: Init checklist for stream stream-46c8cfa3
INFOS: Starting the agent with remote candidates.
INFOS: ICE state changed from Waiting to Running.
INFOS: ICE state changed old=Waiting new=Running
INFOS: Start connectivity checks.

INFOS: Transport description:
 <transport xmlns='urn:xmpp:jingle:transports:ice-udp:1' pwd='721d8u70s9rhe4utb06csslttf' ufrag='679pl1ecaips2h'><rtcp-mux/><fingerprint xmlns='urn:xmpp:jingle:apps:dtls:0' setup='actpass' hash='sha-256'>61:5C:82:C6:CE:C8:00:4E:DA:72:CB:CD:B6:53:D9:81:34:CC:FD:56:92:F4:24:E8:51:AE:B0:15:B4:37:BD:C1</fingerprint><candidate component='1' foundation='1' generation='0' id='288968ff21f11ee70437d8d3e' network='0' priority='2130706431' protocol='udp' type='host' ip='InternalIP' port='10000'/></transport>

INFOS: Pair failed: InternalIP:10000/udp/host -> 192.168.8.37:51701/udp/host (stream-46c8cfa3.RTP)
INFOS: Pair failed: InternalIP:10000/udp/host -> 192.168.8.37:51701/udp/host (stream-46c8cfa3.RTP)
INFOS: create_conf, id=8dd8307281cfa01f gid=null logging=false
INFOS: Performed a successful health check in PT0.006S. Sticky failure: false
INFOS: Endpoint's ICE connection has neither failed nor connected after PT2M34.951S, expiring
INFOS: Expiring endpoint 0f6b78db

After reading this issue I tried to look at the SetRemoteDescription in chrome and it seems that the public IP configured in the server is not published to the client because I only have candidate with the internal IP address (verified by wireshark, requests are sent using the internal IP) :

a=candidate:1 1 udp 2130706431 InternalIP 10000 typ host generation 0

Thanks in advance for your help ! Don’t hesitate to ask if you need more informations.

Also Jitsi meet is really important to us and our customers, especially during the COVID crisis. Thank you for your work !

Best

damencho · July 3, 2020, 3:36pm

Can you show the jvb log on startup there it prints some stuff around the harvesters … maybe it does not read your config or something and uses just the private IP, which is the default behaviour if you do not set public/private one.

emrah · July 3, 2020, 5:24pm

I didn’t test using a subfolder in jvb's home before. I’m using the /etc/jitsi/videobridge/sip-communicator.properties file.

damencho · July 3, 2020, 6:42pm

Yeah maybe its bot picking it from there … Default is in etc, I think does should be visible in the log file after restart

Jerome_LEMAN_GARIN · July 6, 2020, 8:14am

You are right, Jitsi videobridge was not searching for my sip-communicator.properties file anywhere :

juil. 06, 2020 9:33:54 AM org.jitsi.utils.logging2.LoggerImpl log
INFOS: Attempting to load legacy config file at path null, null, sip-communicator.properties
juil. 06, 2020 9:33:54 AM org.jitsi.utils.logging2.LoggerImpl log
INFOS: No legacy config file found: java.lang.NullPointerException

Maybe the “No legacy config file found” or the “config file at path null” should be considered as warnings instead of infos.

After adding “-Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/root -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=.sip-communicator” to jvb.sh" (I set it to /root because I’m starting jvb as root but I also tried in /etc/jitsi/videobridge before) it worked :

INFOS: Attempting to load legacy config file at path /root, .sip-communicator, sip-communicator.properties
INFOS: Registered the LegacyConfigurationServiceShim in OSGi.
INFOS: Using org.ice4j.ice.harvest.MappingCandidateHarvester, face=/InternalIP, mask=/ExternalIP

Did I missed a documentation about it ? I also think the debug part (check the logs for MappingCandidateHarvester) should be in the documentation.

Thank you for your help !

ricsc2 · July 15, 2020, 1:41pm

@Jerome_LEMAN_GARIN can do you share what settings and files to run videobridge under nat. I didi the same configuration, on the internal network its worked perfectly, but on the internet it only works on the main jvb, on the room in the jvb2 open, more restarts after a few seconds.

Thanks you for any help!

Jerome_LEMAN_GARIN · July 15, 2020, 2:29pm

Hey !

Please share your logs when jvb starts to check if this is the same issue

Thx !

ricsc2 · July 15, 2020, 8:07pm

At the start of jvb there is no error, the error in the jvb log presents when a client (coming from the internet) logs into a room and in this case generates the logs in the jvb below:

2020-07-15 12:55:47.460 INFO: [53] [confId=3f5e03e4c2c06f03 epId=08fb2f4c gid=ff9982 stats_id=Skye-nrl conf_name=teste] DtlsTransport.setSetupAttribute#120: The remote side is acting as DTLS server, we’ll act as client
2020-07-15 12:55:47.462 INFO: [53] [confId=3f5e03e4c2c06f03 gid=ff9982 stats_id=Skye-nrl componentId=1 conf_name=teste ufrag=8aig51ed9j0imp name=stream-08fb2f4c epId=08fb2f4c local_ufrag=8aig51ed9j0imp] Component.addRemoteCandidate#330: Add remote candidate for stream-08fb2f4c.RTP: 192.168.13.4:49999/udp/host
2020-07-15 12:55:47.462 INFO: [53] [confId=3f5e03e4c2c06f03 epId=08fb2f4c local_ufrag=8aig51ed9j0imp gid=ff9982 stats_id=Skye-nrl conf_name=teste] IceTransport.startConnectivityEstablishment#176: Starting the agent with remote candidates.
2020-07-15 12:55:47.462 INFO: [53] [confId=3f5e03e4c2c06f03 gid=ff9982 stats_id=Skye-nrl conf_name=teste ufrag=8aig51ed9j0imp epId=08fb2f4c local_ufrag=8aig51ed9j0imp] Agent.startConnectivityEstablishment#753: Start ICE connectivity establishment.
2020-07-15 12:55:47.463 INFO: [53] [confId=3f5e03e4c2c06f03 gid=ff9982 stats_id=Skye-nrl conf_name=teste ufrag=8aig51ed9j0imp epId=08fb2f4c local_ufrag=8aig51ed9j0imp] Agent.initCheckLists#996: Init checklist for stream stream-08fb2f4c
2020-07-15 12:55:47.463 INFO: [53] [confId=3f5e03e4c2c06f03 gid=ff9982 stats_id=Skye-nrl conf_name=teste ufrag=8aig51ed9j0imp epId=08fb2f4c local_ufrag=8aig51ed9j0imp] Agent.setState#963: ICE state changed from Waiting to Running.
2020-07-15 12:55:47.463 INFO: [53] [confId=3f5e03e4c2c06f03 epId=08fb2f4c local_ufrag=8aig51ed9j0imp gid=ff9982 stats_id=Skye-nrl conf_name=teste] IceTransport.iceStateChanged#321: ICE state changed old=Waiting new=Running
2020-07-15 12:55:47.464 INFO: [53] [confId=3f5e03e4c2c06f03 gid=ff9982 stats_id=Skye-nrl conf_name=teste ufrag=8aig51ed9j0imp epId=08fb2f4c local_ufrag=8aig51ed9j0imp] ConnectivityCheckClient.startChecks#142: Start connectivity checks.
2020-07-15 12:55:47.464 INFO: [53] [confId=3f5e03e4c2c06f03 epId=08fb2f4c gid=ff9982 stats_id=Skye-nrl conf_name=teste] Endpoint.describe#1293: Transport description:
EC:88:B6:71:82:90:2E:24:E7:F8:75:D3:6C:F1:EA:D3:8D:13:59:4C:6B:5A:9F:C3:6C:8E:48:0A:F9:3D:D7:54
2020-07-15 12:55:47.485 INFO: [56] [confId=3f5e03e4c2c06f03 gid=ff9982 stats_id=Skye-nrl conf_name=teste ufrag=8aig51ed9j0imp epId=08fb2f4c local_ufrag=8aig51ed9j0imp] ConnectivityCheckClient$PaceMaker.run#919: Pair failed: 192.168.11.7:10000/udp/host -> 192.168.13.4:49999/udp/host (stream-08fb2f4c.RTP)
2020-07-15 12:55:48.363 INFO: [53] [confId=3f5e03e4c2c06f03 gid=ff9982 stats_id=Skye-nrl componentId=1 conf_name=teste ufrag=8aig51ed9j0imp name=stream-08fb2f4c epId=08fb2f4c local_ufrag=8aig51ed9j0imp] Component.addUpdateRemoteCandidates#347: Update remote candidate for stream-08fb2f4c.RTP: 192.168.13.4:49999/udp
2020-07-15 12:55:48.364 INFO: [53] [confId=3f5e03e4c2c06f03 gid=ff9982 stats_id=Skye-nrl componentId=1 conf_name=teste ufrag=8aig51ed9j0imp name=stream-08fb2f4c epId=08fb2f4c local_ufrag=8aig51ed9j0imp] Component.addUpdateRemoteCandidates#369: Not adding duplicate remote candidate: 192.168.13.4:49999/udp
2020-07-15 12:55:48.364 INFO: [53] [confId=3f5e03e4c2c06f03 gid=ff9982 stats_id=Skye-nrl componentId=1 conf_name=teste ufrag=8aig51ed9j0imp name=stream-08fb2f4c epId=08fb2f4c local_ufrag=8aig51ed9j0imp] Component.addUpdateRemoteCandidates#347: Update remote candidate for stream-08fb2f4c.RTP: 192.168.13.4:49999/udp
2020-07-15 12:55:48.364 INFO: [53] [confId=3f5e03e4c2c06f03 gid=ff9982 stats_id=Skye-nrl componentId=1 conf_name=teste ufrag=8aig51ed9j0imp name=stream-08fb2f4c epId=08fb2f4c local_ufrag=8aig51ed9j0imp] Component.addUpdateRemoteCandidates#369: Not adding duplicate remote candidate: 192.168.13.4:49999/udp
2020-07-15 12:55:48.365 INFO: [53] [confId=3f5e03e4c2c06f03 epId=08fb2f4c gid=ff9982 stats_id=Skye-nrl conf_name=teste] Endpoint.describe#1293: Transport description:
EC:88:B6:71:82:90:2E:24:E7:F8:75:D3:6C:F1:EA:D3:8D:13:59:4C:6B:5A:9F:C3:6C:8E:48:0A:F9:3D:D7:54
2020-07-15 12:55:49.371 INFO: [53] [confId=3f5e03e4c2c06f03 gid=ff9982 stats_id=Skye-nrl componentId=1 conf_name=teste ufrag=8aig51ed9j0imp name=stream-08fb2f4c epId=08fb2f4c local_ufrag=8aig51ed9j0imp] Component.addUpdateRemoteCandidates#347: Update remote candidate for stream-08fb2f4c.RTP: 192.168.13.4:49999/udp
2020-07-15 12:55:49.371 INFO: [53] [confId=3f5e03e4c2c06f03 gid=ff9982 stats_id=Skye-nrl componentId=1 conf_name=teste ufrag=8aig51ed9j0imp name=stream-08fb2f4c epId=08fb2f4c local_ufrag=8aig51ed9j0imp] Component.addUpdateRemoteCandidates#369: Not adding duplicate remote candidate: 192.168.13.4:49999/udp
2020-07-15 12:55:49.372 INFO: [53] [confId=3f5e03e4c2c06f03 gid=ff9982 stats_id=Skye-nrl componentId=1 conf_name=teste ufrag=8aig51ed9j0imp name=stream-08fb2f4c epId=08fb2f4c local_ufrag=8aig51ed9j0imp] Component.addUpdateRemoteCandidates#347: Update remote candidate for stream-08fb2f4c.RTP: 192.168.13.4:49999/udp
2020-07-15 12:55:49.372 INFO: [53] [confId=3f5e03e4c2c06f03 gid=ff9982 stats_id=Skye-nrl componentId=1 conf_name=teste ufrag=8aig51ed9j0imp name=stream-08fb2f4c epId=08fb2f4c local_ufrag=8aig51ed9j0imp] Component.addUpdateRemoteCandidates#369: Not adding duplicate remote candidate: 192.168.13.4:49999/udp
2020-07-15 12:55:49.373 INFO: [53] [confId=3f5e03e4c2c06f03 gid=ff9982 stats_id=Skye-nrl componentId=1 conf_name=teste ufrag=8aig51ed9j0imp name=stream-08fb2f4c epId=08fb2f4c local_ufrag=8aig51ed9j0imp] Component.addUpdateRemoteCandidates#347: Update remote candidate for stream-08fb2f4c.RTP: 127.0.0.1:65404/udp
2020-07-15 12:55:49.373 INFO: [53] [confId=3f5e03e4c2c06f03 gid=ff9982 stats_id=Skye-nrl componentId=1 conf_name=teste ufrag=8aig51ed9j0imp name=stream-08fb2f4c epId=08fb2f4c local_ufrag=8aig51ed9j0imp] Component.updateRemoteCandidates#481: new Pair added: 192.168.11.7:10000/udp/host -> 127.0.0.1:65404/udp/relay (stream-08fb2f4c.RTP).
2020-07-15 12:55:49.373 INFO: [53] [confId=3f5e03e4c2c06f03 epId=08fb2f4c gid=ff9982 stats_id=Skye-nrl conf_name=teste] Endpoint.describe#1293: Transport description:
EC:88:B6:71:82:90:2E:24:E7:F8:75:D3:6C:F1:EA:D3:8D:13:59:4C:6B:5A:9F:C3:6C:8E:48:0A:F9:3D:D7:54
2020-07-15 12:55:49.384 INFO: [56] [confId=3f5e03e4c2c06f03 gid=ff9982 stats_id=Skye-nrl conf_name=teste ufrag=8aig51ed9j0imp epId=08fb2f4c local_ufrag=8aig51ed9j0imp] ConnectivityCheckClient$PaceMaker.run#919: Pair failed: 192.168.11.7:10000/udp/host -> 127.0.0.1:65404/udp/relay (stream-08fb2f4c.RTP)

…

2020-07-15 12:56:03.451 INFO: [53] [confId=3f5e03e4c2c06f03 epId=08fb2f4c gid=ff9982 stats_id=Skye-nrl conf_name=teste] AbstractEndpoint.expire#303: Expiring.
2020-07-15 12:56:03.453 INFO: [53] [confId=3f5e03e4c2c06f03 epId=08fb2f4c gid=ff9982 stats_id=Skye-nrl conf_name=teste] Transceiver.teardown#315: Tearing down
2020-07-15 12:56:03.453 INFO: [53] [confId=3f5e03e4c2c06f03 epId=08fb2f4c gid=ff9982 stats_id=Skye-nrl conf_name=teste] RtpReceiverImpl.tearDown#287: Tearing down
2020-07-15 12:56:03.453 INFO: [53] [confId=3f5e03e4c2c06f03 epId=08fb2f4c gid=ff9982 stats_id=Skye-nrl conf_name=teste] RtpSenderImpl.tearDown#263: Tearing down
2020-07-15 12:56:03.453 INFO: [53] [confId=3f5e03e4c2c06f03 epId=08fb2f4c gid=ff9982 stats_id=Skye-nrl conf_name=teste] DtlsTransport.stop#180: Stopping
2020-07-15 12:56:03.454 INFO: [53] [confId=3f5e03e4c2c06f03 epId=08fb2f4c local_ufrag=8aig51ed9j0imp gid=ff9982 stats_id=Skye-nrl conf_name=teste] IceTransport.stop#235: Stopping
2020-07-15 12:56:03.454 INFO: [53] [confId=3f5e03e4c2c06f03 gid=ff9982 stats_id=Skye-nrl conf_name=teste ufrag=8aig51ed9j0imp epId=08fb2f4c local_ufrag=8aig51ed9j0imp] Agent.setState#963: ICE state changed from Running to Terminated.
2020-07-15 12:56:03.454 INFO: [53] [confId=3f5e03e4c2c06f03 epId=08fb2f4c local_ufrag=8aig51ed9j0imp gid=ff9982 stats_id=Skye-nrl conf_name=teste] IceTransport.iceStateChanged#321: ICE state changed old=Running new=Terminated
2020-07-15 12:56:03.455 INFO: [53] [confId=3f5e03e4c2c06f03 gid=ff9982 stats_id=Skye-nrl componentId=1 conf_name=teste ufrag=8aig51ed9j0imp name=stream-08fb2f4c epId=08fb2f4c local_ufrag=8aig51ed9j0imp] MergingDatagramSocket.close#142: Closing.
2020-07-15 12:56:03.455 INFO: [53] [confId=3f5e03e4c2c06f03 epId=08fb2f4c gid=ff9982 stats_id=Skye-nrl conf_name=teste] Endpoint.expire#809: Expired.

Você quis dizer: [Clientes da rede interno faz o login normalmente , sem erros no jvb.](javascript:void(0))

67/5000

Internal network clients log in nomarlly, with no jvb errors.

Thanks for the support!

Jerome_LEMAN_GARIN · July 16, 2020, 7:25am

Could you please search for the string “Attempting to load legacy config file at path” in your logs ?

Thanks !

ricsc2 · July 16, 2020, 1:00pm

The string “Attempting to load legacy config file at path” was correcct.

But my problem solved!!!,

Thanks for your attention and support, being able to configure two videobridge with two public IPs, using oct (NAT) and I only had a problem with the symmetry of return to the internet(TCP/10000). In this case it was published on one link and returning via another link. With the correction of symmetry the videoconference in the jitsi meet with 2 JVB working perfectly.

Thanks a lot for the help.

uvwild · February 26, 2021, 1:03pm

I am struggling with this problem also.
But I am running JVB in a k8s pod and the public IP is not directly on the network interface but routed from the cloud provider.
This I have PUBLICIP → DockerHostIP== NodeIP → JVB-container-IP/podIP

I tried various combinations but the pairing always fails.
So I am wondering if the Harvester can handle this scenario?!?

damencho · February 26, 2021, 1:47pm

Yep, it can. If packets reach your jvb it maybe outgoing packets are stopped somewhere