Jitsi with ext. coturn: massive struggeling

Hi there,

me and my collegue are struggeling for 2 days getting our internal Jitsi server to work with an external hosted coturn server.

Problem
Sessions with 2 participants are working, when a third person joines everything is broken.

Some info on the setup
We are using Jitsi for several years. Now we rented a external linux machine to get turn to work via port 443.

  • Internal Jitsi server: Debian 10

jitsi-meet 2.0.6433-1
ii jitsi-meet-prosody 1.0.5415-1
ii jitsi-meet-turnserver 1.0.5415-1
ii jitsi-meet-web 1.0.5415-1
ii jitsi-meet-web-config 1.0.5415-1
ii jitsi-videobridge2 2.1-570-gb802be83-1

The server runs behind our corporate firewall (sophos) in a DMZ, website is secured via WAF. Server has full internet access.

  • External Coturn server: Debian 11

Version Coturn-4.5.2 ‘dan Eider’

The server runs external with no restrictions.

Configfiles
*In the configfiles I use placeholders for the DNS names of the servers.

/etc/prosody/conf.d/JITSI-SERVER.cfg.lua

unlimited_jids = { “focus@auth.JITSI-SERVER”, “jvb@auth.JITSI-SERVER” }
plugin_paths = { “/usr/share/jitsi-meet/prosody-plugins/” }

– domain mapper options, must at least have domain base set to use the mapper
muc_mapper_domain_base = “JITSI-SERVER”;

turncredentials_secret = “5eb13ecdc2c8d9f7318f464e21d08b8c”;
turncredentials_port = 5349;
turncredentials_ttl = 86400;

turncredentials = {
{ type = “stun”, host = “TURNSERVER”, port = “3478” },
{ type = “turn”, host = “TURNSERVER”, port = “5349”, transport = “udp” },
{ type = “turns”, host = “TURNSERVER”, port = “5349”, transport = “tcp” }
};

cross_domain_bosh = false;
consider_bosh_secure = true;

VirtualHost “JITSI-SERVER”
– enabled = false – Remove this line to enable this host
authentication = “anonymous”
– Properties below are modified by jitsi-meet-tokens package config
– and authentication above is switched to “token”
–app_id=“example_app_id”
–app_secret=“example_app_secret”
– Assign this host a certificate for TLS, otherwise it would use the one
– set in the global section (if any).
– Note that old-style SSL on port 5223 only supports one certificate, and will always
– use the global one.
ssl = {
key = “/etc/prosody/certs/JITSI-SERVER.key”;
certificate = “/etc/prosody/certs/JITSI-SERVER.crt”;
}
speakerstats_component = “speakerstats.kirnach.stwa.local”
conference_duration_component = “conferenceduration.kirnach.stwa.local”
– we need bosh
modules_enabled = {
“bosh”;
“pubsub”;
“ping”; – Enable mod_ping
“speakerstats”;
“turncredentials”;
“conference_duration”;
“muc_lobby_rooms”;
}
c2s_require_encryption = false
lobby_muc = “lobby.JITSI-SERVER”
main_muc = “conference.JITSI-SERVER”

/etc/jitsi/meet/JITSI-SERVER-config.js

useStunTurn: true
p2p: {
enabled: true,
useStunTurn: true,
stunServers: [
{ urls: ‘TURNSERVER:5349’ }
],
},

/etc/jitsi/videobridge/sip-communicator.properties

org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=TURNSERVER
org.jitsi.videobridge.DISABLE_TCP_HARVESTER=false

/etc/turnserver.conf

#listening-port=3478
#tls-listening-port=5349
listening-ip=TURNSERVER-IP
relay-ip=TURNSERVER-IP
use-auth-secret
static-auth-secret=5eb13ecdc2c8d9f7318f464e21d08b8c
server-name=TURNSERVER
realm=TURNSERVER
no-udp
cert=/etc/turnserver/fullchain.pem
pkey=/etc/turnserver/privkey.pem
no-multicast-peers
keep-address-family
no-cli
no-tlsv1
no-tlsv1_1
no-tlsv1_2

turnserver.log

0: : log file opened: /etc/turnserver/turn.log
0: : Config file found: /root/…/etc/turnserver.conf
0: : Listener address to use: TURNSERVER-IP
0: : Relay address to use: TURNSERVER-IP
0: : Config file found: /root/…/etc/turnserver.conf
0: :
RFC 3489/5389/5766/5780/6062/6156 STUN/TURN Server
Version Coturn-4.5.2 ‘dan Eider’
0: :
Max number of open files/sockets allowed for this process: 1048576
0: :
Due to the open files/sockets limitation,
max supported number of TURN Sessions possible is: 524000 (approximately)
0: :

==== Show him the instruments, Practical Frost: ====

0: : TLS supported
0: : DTLS supported
0: : DTLS 1.2 supported
0: : TURN/STUN ALPN supported
0: : Third-party authorization (oAuth) supported
0: : GCM (AEAD) supported
0: : OpenSSL compile-time version: OpenSSL 1.1.1k 25 Mar 2021 (0x101010bf)
0: :
0: : SQLite supported, default database location is /var/lib/turn/turndb
0: : Redis supported
0: : PostgreSQL supported
0: : MySQL supported
0: : MongoDB is not supported
0: :
0: : Default Net Engine version: 3 (UDP thread per CPU core)

=====================================================

0: : Domain name:
0: : Default realm: TURNSERVER
0: : SSL23: Certificate file found: /etc/turnserver/fullchain.pem
0: : SSL23: Private key file found: /etc/turnserver/privkey.pem
0: : TLS cipher suite: DEFAULT
0: : DTLS: Certificate file found: /etc/turnserver/fullchain.pem
0: : DTLS: Private key file found: /etc/turnserver/privkey.pem
0: : DTLS1.2: Certificate file found: /etc/turnserver/fullchain.pem
0: : DTLS1.2: Private key file found: /etc/turnserver/privkey.pem
0: : DTLS cipher suite: DEFAULT
0: : pid file created: /var/run/turnserver.pid
0: : IO method (main listener thread): epoll (with changelist)
0: : WARNING: I cannot support STUN CHANGE_REQUEST functionality because only one IP address is provided
0: : Wait for relay ports initialization…
0: : relay TURNSERVER-IP initialization…
0: : relay TURNSERVER-IP initialization done
0: : Relay ports initialization done
0: : IO method (general relay thread): epoll (with changelist)
0: : turn server id=1 created
0: : Cannot create TLS listener
0: : Cannot create TLS listener
0: : IO method (general relay thread): epoll (with changelist)
0: : turn server id=0 created
0: : IO method (general relay thread): epoll (with changelist)
0: : turn server id=2 created
0: : Cannot create TLS listener
0: : Cannot create TLS listener
0: : Cannot create TLS listener
0: : Cannot create TLS listener
0: : IO method (general relay thread): epoll (with changelist)
0: : turn server id=3 created
0: : IPv4. DTLS listener opened on: TURNSERVER-IP:5349
0: : Total General servers: 4
0: : Cannot create TLS listener
0: : Cannot create TLS listener
0: : IO method (auth thread): epoll (with changelist)
0: : IO method (auth thread): epoll (with changelist)
0: : SQLite DB connection success: /var/lib/turn/turndb
0: : IO method (admin thread): epoll (with changelist)
2115: : IPv4. tcp or tls connected to: FIREWALL-IP:61343
2115: : IPv4. tcp or tls connected to: FIREWALL-IP:61344
2115: : IPv4. tcp or tls connected to: FIREWALL-IP:52642
2115: : IPv4. tcp or tls connected to: FIREWALL-IP:61345
2133: : IPv4. tcp or tls connected to: FIREWALL-IP:61360
2133: : IPv4. tcp or tls connected to: FIREWALL-IP:61361
2133: : IPv4. tcp or tls connected to: FIREWALL-IP:61362
2150: : IPv4. tcp or tls connected to: FIREWALL-IP:61372
2150: : IPv4. tcp or tls connected to: FIREWALL-IP:61373
2150: : IPv4. tcp or tls connected to: FIREWALL-IP:61374
2168: : IPv4. tcp or tls connected to: FIREWALL-IP:61384
2171: : IPv4. tcp or tls connected to: FIREWALL-IP:61387
2171: : IPv4. tcp or tls connected to: FIREWALL-IP:61388
2175: : session 002000000000000006: usage: realm=, username=<>, rp=1, rb=497, sp=0, sb=0
2175: : session 002000000000000006: peer usage: realm=, username=<>, rp=0, rb=0, sp=0, sb=0
2175: : session 002000000000000006: closed (2nd stage), user <> realm origin <>, local TURNSERVER-IP:5349, remote FIREWALL-IP:61343, reason: allocation watchdog determined stale session state
2175: : session 003000000000000004: usage: realm=, username=<>, rp=1, rb=497, sp=0, sb=0
2175: : session 003000000000000004: peer usage: realm=, username=<>, rp=0, rb=0, sp=0, sb=0
2175: : session 003000000000000004: closed (2nd stage), user <> realm origin <>, local TURNSERVER-IP:5349, remote FIREWALL-IP:61344, reason: allocation watchdog determined stale session state
2175: : session 001000000000000005: usage: realm=, username=<>, rp=1, rb=497, sp=0, sb=0
2175: : session 001000000000000005: peer usage: realm=, username=<>, rp=0, rb=0, sp=0, sb=0
2175: : session 001000000000000005: closed (2nd stage), user <> realm origin <>, local TURNSERVER-IP:5349, remote FIREWALL-IP:52642, reason: allocation watchdog determined stale session state
2175: : session 003000000000000005: usage: realm=, username=<>, rp=1, rb=497, sp=0, sb=0
2175: : session 003000000000000005: peer usage: realm=, username=<>, rp=0, rb=0, sp=0, sb=0
2175: : session 003000000000000005: closed (2nd stage), user <> realm origin <>, local TURNSERVER-IP:5349, remote FIREWALL-IP:61345, reason: allocation watchdog determined stale session state
2193: : session 001000000000000006: usage: realm=, username=<>, rp=1, rb=497, sp=0, sb=0
2193: : session 001000000000000006: peer usage: realm=, username=<>, rp=0, rb=0, sp=0, sb=0
2193: : session 001000000000000006: closed (2nd stage), user <> realm origin <>, local TURNSERVER-IP:5349, remote FIREWALL-IP:61360, reason: allocation watchdog determined stale session state
2193: : session 002000000000000007: usage: realm=, username=<>, rp=1, rb=497, sp=0, sb=0
2193: : session 002000000000000007: peer usage: realm=, username=<>, rp=0, rb=0, sp=0, sb=0
2193: : session 002000000000000007: closed (2nd stage), user <> realm origin <>, local TURNSERVER-IP:5349, remote FIREWALL-IP:61361, reason: allocation watchdog determined stale session state
2193: : session 001000000000000007: usage: realm=, username=<>, rp=1, rb=497, sp=0, sb=0
2193: : session 001000000000000007: peer usage: realm=, username=<>, rp=0, rb=0, sp=0, sb=0
2193: : session 001000000000000007: closed (2nd stage), user <> realm origin <>, local TURNSERVER-IP:5349, remote FIREWALL-IP:61362, reason: allocation watchdog determined stale session state
2210: : session 003000000000000006: usage: realm=, username=<>, rp=1, rb=497, sp=0, sb=0
2210: : session 003000000000000006: peer usage: realm=, username=<>, rp=0, rb=0, sp=0, sb=0
2210: : session 003000000000000006: closed (2nd stage), user <> realm origin <>, local TURNSERVER-IP:5349, remote FIREWALL-IP:61372, reason: allocation watchdog determined stale session state
2210: : session 000000000000000003: usage: realm=, username=<>, rp=1, rb=497, sp=0, sb=0
2210: : session 000000000000000003: peer usage: realm=, username=<>, rp=0, rb=0, sp=0, sb=0
2210: : session 000000000000000003: closed (2nd stage), user <> realm origin <>, local TURNSERVER-IP:5349, remote FIREWALL-IP:61373, reason: allocation watchdog determined stale session state
2210: : session 000000000000000004: usage: realm=, username=<>, rp=1, rb=497, sp=0, sb=0
2210: : session 000000000000000004: peer usage: realm=, username=<>, rp=0, rb=0, sp=0, sb=0
2210: : session 000000000000000004: closed (2nd stage), user <> realm origin <>, local TURNSERVER-IP:5349, remote FIREWALL-IP:61374, reason: allocation watchdog determined stale session state
2228: : session 003000000000000007: usage: realm=, username=<>, rp=1, rb=497, sp=0, sb=0
2228: : session 003000000000000007: peer usage: realm=, username=<>, rp=0, rb=0, sp=0, sb=0
2228: : session 003000000000000007: closed (2nd stage), user <> realm origin <>, local TURNSERVER-IP:5349, remote FIREWALL-IP:61384, reason: allocation watchdog determined stale session state
2231: : session 000000000000000005: usage: realm=, username=<>, rp=1, rb=497, sp=0, sb=0
2231: : session 000000000000000005: peer usage: realm=, username=<>, rp=0, rb=0, sp=0, sb=0
2231: : session 000000000000000005: closed (2nd stage), user <> realm origin <>, local TURNSERVER-IP:5349, remote FIREWALL-IP:61387, reason: allocation watchdog determined stale session state
2231: : session 000000000000000006: usage: realm=, username=<>, rp=1, rb=497, sp=0, sb=0
2231: : session 000000000000000006: peer usage: realm=, username=<>, rp=0, rb=0, sp=0, sb=0
2231: : session 000000000000000006: closed (2nd stage), user <> realm origin <>, local TURNSERVER-IP:5349, remote FIREWALL-IP:61388, reason: allocation watchdog determined stale session state

WebRTC-internals

https://JITSISERVER-URL/test, { iceServers: [turn:TURNSERVER:5349, turns:TURNSERVER:5349?transport=tcp, stun:TURNSERVER:3478], iceTransportPolicy: all, bundlePolicy: max-bundle, rtcpMuxPolicy: require, iceCandidatePoolSize: 0, sdpSemantics: “unified-plan”, extmapAllowMixed: true }

Perhaps anyone can point us to the solution. We are going slightly mad.

Thank you for any help!!!

If 2 participants works but 3 doesn’t then you need to look at the JVB. What error do you get in the browser console? Is the JVB up and working?

Hi saghul!

Thanks for the fast reply. I was already out of the office yesterday.

After adding the stun-port in /etc/jitsi/videobridge/sip-communicator.properties conferencing with 2 or more participants worked.

/etc/jitsi/videobridge/sip-communicator.properties

org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=true
org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=TURNSERVER:3478
org.jitsi.videobridge.DISABLE_TCP_HARVESTER=false
org.jitsi.videobridge.ENABLE_STATISTICS=false
org.jitsi.videobridge.STATISTICS_TRANSPORT=muc
org.jitsi.videobridge.xmpp.user.shard.HOSTNAME=localhost
org.jitsi.videobridge.xmpp.user.shard.DOMAIN=auth.JITSISERVER
org.jitsi.videobridge.xmpp.user.shard.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard.PASSWORD=kXHNh84P
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS=JvbBrewery@internal.auth.JITSISERVER
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME=04b455b5-c1c4-491a-b6d4-a191f0978862
org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=JITSISERVER-INTIP
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=JITSISERVER-EXTIP

For testing the turnserver I blocked port 10000/udp on our firewall for my external test client (IP: 178.24.249.75). On that client I have no video and sound.

browser console on external test client

2022-07-21T12:02:34.423Z [modules/statistics/AvgRTPStatsReporter.js] <m.addNext>:
bandwidth_upload - invalid value for idx: 1 undefined
2022-07-21T12:02:24.900Z [modules/xmpp/JingleSessionPC.js]
<I.sendIceCandidates>:  
JingleSessionPC[session=JVB,initiator=false,sid=45jim6tsa2qt2]
sendIceCandidates [{"candidate":"candidate:3415476587 1 udp 2122260223
100.80.109.26 50570 typ host generation 0 ufrag YPry network-id 1 network-cost 10","sdpMid":"0","sdpMLineIndex":0},
{"candidate":"candidate:2232939931 1 tcp 1518280447 100.80.109.26 9 typ host tcptype active generation 0 ufrag YPry network-id 1 network-cost 10","sdpMid":"0","sdpMLineIndex":0}] 

turnserver.log

1144: : IPv4. tcp or tls connected to: 178.24.249.75:37700
1164: : IPv4. tcp or tls connected to: 178.24.249.75:37714
1182: : IPv4. tcp or tls connected to: 178.24.249.75:37694
1204: : session 001000000000000005: usage: realm=<TURNSERVER>, username=<>, rp=1, rb=517, sp=0, sb=0
1204: : session 001000000000000005: peer usage: realm=<TURNSERVER>, username=<>, rp=0, rb=0, sp=0, sb=0
1204: : session 001000000000000005: closed (2nd stage), user <> realm <TURNSERVER> origin <>, local TURNSERVER-IP:5349, remote 178.24.249.75:37700, reason: allocation watchdog determined stale session state
1224: : session 003000000000000006: usage: realm=<TURNSERVER>, username=<>, rp=1, rb=517, sp=0, sb=0
1224: : session 003000000000000006: peer usage: realm=<TURNSERVER>, username=<>, rp=0, rb=0, sp=0, sb=0
1224: : session 003000000000000006: closed (2nd stage), user <> realm <TURNSERVER> origin <>, local TURNSERVER-IP:5349, remote 178.24.249.75:37714, reason: allocation watchdog determined stale session state
1242: : session 000000000000000010: usage: realm=<TURNSERVER>, username=<>, rp=1, rb=517, sp=0, sb=0
1242: : session 000000000000000010: peer usage: realm=<TURNSERVER>, username=<>, rp=0, rb=0, sp=0, sb=0
1242: : session 000000000000000010: closed (2nd stage), user <> realm <TURNSERVER> origin <>, local TURNSERVER-IP:5349, remote 178.24.249.75:37694, reason: allocation watchdog determined stale session s

jvb.log

JVB 2022-07-21 14:01:35.893 INFORMATION: [104] [confId=4a27f28b761ad913 gid=52920 stats_id=Glennie-CnN conf_name=test@conference.JITSISERVER-LOCAL ufrag=ad8a41g8g9jl3t epId=5a126568 local_ufrag=ad8a41g8g9jl3t] ConnectivityCheckClient.processTimeout#874: timeout for pair: FIREWALL-EXTIP:10000/udp/srflx -> 178.24.249.75:37706/udp/prflx (stream-5a126568.RTP), failing.

WebRTC-internals

https://JITSISERVER-URL/test, { iceServers: [turns:TURNSERVER:5349?transport=tcp], iceTransportPolicy: all, bundlePolicy: max-bundle, rtcpMuxPolicy: require, iceCandidatePoolSize: 0, sdpSemantics: "unified-plan", extmapAllowMixed: true }
ICE connection state: new => disconnected
Connection state: new => disconnected => failed => closed
Signaling state: new => have-remote-offer => stable => have-remote-offer => stable
ICE Candidate pair:
setRemoteDescription (type: "offer", 6 sections)
a=candidate:1 1 udp 2130706431 JITSISERVER-LOCALIP 10000 typ host generation 0
a=candidate:2 1 udp 1694498815 FIREWALL-EXTIP 10000 typ srflx raddr JITSISERVER-LOCALIP rport 10000 generation 0

I guess somehow the turnserver isn’t used at all?

Does nobody have an idea?

Sorry for my impatience…

Wait, you want to use the TURN server also when connecting with the JVB?

Although I’m afraid I don’t quite understand your question, here’s my answer:
The aim is that participants with restricted internet access (udp port 10000 blocked) can take part in a meeting. That’s why we set up the extra turnserver.

Your TURN server would still be able to connect to port 10000 on the JVB, right?

Yes, we have a NAT/firewall rule to allow connections via port 10000/udp to the JVB.
Source-IPs are not restricted to allow connections from any unrestricted participant.

Hum here I see the candidate uses TURNS but you have disabled TLS in coturn.

Hi saghul,

big thanks for your hint, I think I got it running now.

In turnserver.conf it says:

For secure TCP connections, Coturn currently supports
TLS version 1.0, 1.1 and 1.2

But I disabled those tls versions by

no-tlsv1
no-tlsv1_1
no-tlsv1_2

Where there’s a will there’s a way. - Thanks a million, saghul!

2 Likes

Happy to have helped!