Can't get jitsi-meet to work with TURN (solved)

Hi folks,

I’m trying to set up an entirely self-hosted jitsi-meet, and I think I’m almost there.

The last thing that I would like to test is whether using my own TURN server works. I searched up and down the interwebs and tried most things I found, including on this forum, but somehow, I’m still stuck…

General system information

root@bender:~# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"

root@bender:~# cat /etc/apt/sources.list.d/jitsi-stable.list
deb https://download.jitsi.org stable/

root@bender:~# dpkg -l jitsi* prosody coturn
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                 Version                 Architecture            Description
+++-====================================-=======================-=======================-==============================================================================
ii  coturn                               4.5.0.7-1ubuntu2.18.04. amd64                   TURN and STUN server for VoIP
ii  jitsi-meet                           2.0.4384-1              all                     WebRTC JavaScript video conferences
ii  jitsi-meet-prosody                   1.0.3969-1              all                     Prosody configuration for Jitsi Meet
un  jitsi-meet-tokens                    <none>                  <none>                  (no description available)
un  jitsi-meet-turnserver                <none>                  <none>                  (no description available)
ii  jitsi-meet-web                       1.0.3969-1              all                     WebRTC JavaScript video conferences
ii  jitsi-meet-web-config                1.0.3969-1              all                     Configuration for web serving of Jitsi Meet
un  jitsi-videobridge                    <none>                  <none>                  (no description available)
ii  jitsi-videobridge2                   2.1-164-gfdce823f-1     all                     WebRTC compatible Selective Forwarding Unit (SFU)
ii  prosody                              0.10.0-1build1          amd64                   Lightweight Jabber/XMPP server

Configuration

I have (more or less) followed this excellent guide to get started.

NOTE: All of the configuration files below are copied verbatim, including passwords. The only thing I replaced is the actual domain (replaced by example.org), and the public IP (replaced by 111.222.333.444). I have also removed comments from some of the configuration files in order to keep them short.

Firewall configuration

root@bender:~# ufw status verbose
Status: active
Logging: off
Default: reject (incoming), allow (outgoing), allow (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW IN    Anywhere
80/tcp                     ALLOW IN    Anywhere
443/tcp                    ALLOW IN    Anywhere
25/tcp                     ALLOW IN    Anywhere
465/tcp                    ALLOW IN    Anywhere
587/tcp                    ALLOW IN    Anywhere
53/tcp                     ALLOW IN    Anywhere
53/udp                     ALLOW IN    Anywhere
143/tcp                    ALLOW IN    Anywhere
993/tcp                    ALLOW IN    Anywhere
5349                       ALLOW IN    Anywhere
10000/udp                  ALLOW IN    Anywhere

Note: This server provides a couple of other services like mail etc., hence the other rules. The prosody ports (5280 etc.) are not permitted to be accessed from outside, but the turn (5349) and jvb2 (10000) ports are.

/etc/turnserver.conf

listening-port=3478
tls-listening-port=5349
listening-ip=111.222.333.444
verbose
fingerprint
use-auth-secret
static-auth-secret=0123456789abcdef
realm=example.org
total-quota=256
bps-capacity=0
stale-nonce=600
cert=/var/lib/letsencrypt/certs/example.org/fullchain.pem
pkey=/var/lib/letsencrypt/certs/example.org/privkey.pem
cipher-list="ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384"
no-stdout-log
log-file=/var/log/coturn.log
simple-log
no-loopback-peers
no-multicast-peers
proc-user=turnserver
proc-group=turnserver
no-tlsv1
no-tlsv1_1

As far as I can tell, this configuration works in principle:

  1. I used https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/ to test stun (using stun:example.org:5349), and I get back rtp host and rtp srflx replies.
  2. I don’t really know how to test turn there, because I don’t know how to generate a valid username/password pair for the auth-secret. Any hints for that are welcome!
  3. However, I also tested the turn functionality with a (also self-hosted) Nextcloud Talk installation, and Nextcloud Talk is working properly and using my TURN server.

/etc/jitsi/videobridge/sip-communicator.properties

org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=true
org.jitsi.videobridge.DISABLE_TCP_HARVESTER=true
org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=example.org:5349
org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_TRANSPORT=muc
org.jitsi.videobridge.xmpp.user.shard.HOSTNAME=localhost
org.jitsi.videobridge.xmpp.user.shard.DOMAIN=auth.example.org
org.jitsi.videobridge.xmpp.user.shard.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard.PASSWORD=diB1Giwx
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS=JvbBrewery@internal.auth.example.org
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME=755cac30-4f24-45da-9d29-f9169de2331c

Note: the org.jitsi.videobridge.DISABLE_TCP_HARVESTER=true line was inserted according to this guide, but I don’t know if it has any effect.

/etc/jitsi/jicofo/sip-communicator.properties

org.jitsi.jicofo.BRIDGE_MUC=JvbBrewery@internal.auth.example.org
org.jitsi.jicofo.auth.URL=XMPP:example.org
org.jitsi.jicofo.auth.DISABLE_AUTOLOGIN=true

Note: the org.jitsi.jicofo.auth.DISABLE_AUTOLOGIN=true line was inserted to prevent client-side credentials caching, and it works (and I don’t think it would be related to the turn functionality anyway).

/etc/jitsi/jicofo/config

JICOFO_HOST=localhost
JICOFO_HOSTNAME=example.org
JICOFO_SECRET=@ybAGmtw
JICOFO_PORT=5347
JICOFO_AUTH_DOMAIN=auth.example.org
JICOFO_AUTH_USER=focus
JICOFO_AUTH_PASSWORD=EOxbxqRh
JICOFO_OPTS=""
JAVA_SYS_PROPS="-Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=jicofo -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/jicofo/logging.properties"

/etc/prosody/conf.avail/example.org.cfg.lua

plugin_paths = { "/usr/share/jitsi-meet/prosody-plugins/" }

-- domain mapper options, must at least have domain base set to use the mapper
muc_mapper_domain_base = "example.org";

turncredentials_secret = "0123456789abcdef";

turncredentials = {
  { type = "stun", host = "example.org", port = "5349" },
  { type = "turn", host = "example.org", port = "5349", transport = "udp" },
  { type = "turns", host = "example.org", port = "5349", transport = "tcp" }
};

cross_domain_bosh = false;
consider_bosh_secure = true;

VirtualHost "example.org"
		-- enabled = false -- Remove this line to enable this host
		authentication = "internal_plain"
		-- Properties below are modified by jitsi-meet-tokens package config
		-- and authentication above is switched to "token"
		--app_id="example_app_id"
		--app_secret="example_app_secret"
		-- Assign this host a certificate for TLS, otherwise it would use the one
		-- set in the global section (if any).
		-- Note that old-style SSL on port 5223 only supports one certificate, and will always
		-- use the global one.
		ssl = {
				key = "/etc/prosody/certs/example.org.key";
				certificate = "/etc/prosody/certs/example.org.crt";
		}
		speakerstats_component = "speakerstats.example.org"
		conference_duration_component = "conferenceduration.example.org"
		-- we need bosh
		modules_enabled = {
			"bosh";
			"pubsub";
			"ping"; -- Enable mod_ping
			"speakerstats";
			"turncredentials";
			"conference_duration";
		}
		c2s_require_encryption = false

Component "conference.example.org" "muc"
	storage = "none"
	modules_enabled = {
		"muc_meeting_id";
		"muc_domain_mapper";
		-- "token_verification";
	}
	admins = { "focus@auth.example.org" }

-- internal muc component
Component "internal.auth.example.org" "muc"
	storage = "none"
	modules_enabled = {
	  "ping";
	}
	admins = { "focus@auth.example.org", "jvb@auth.example.org" }

VirtualHost "auth.example.org"
	ssl = {
		key = "/etc/prosody/certs/auth.example.org.key";
		certificate = "/etc/prosody/certs/auth.example.org.crt";
	}
	authentication = "internal_plain"

Component "focus.example.org"
	component_secret = "@ybAGmtw"

Component "speakerstats.example.org" "speakerstats_component"
	muc_component = "conference.example.org"

Component "conferenceduration.example.org" "conference_duration_component"
	muc_component = "conference.example.org"

VirtualHost "guest.example.org"
	authentication = "anonymous"
	c2s_require_encryption = false

Note: I have verified that the file /usr/share/jitsi-meet/prosody-plugins/mod_turncredentials.lua exists, however I don’t know how to check whether the plugin/module is really enabled.

/etc/jitsi/meet/example.org-config.js

var config = {
	hosts: {
		domain: 'example.org',
		anonymousdomain: 'guest.example.org',
		authdomain: 'example.org',
		muc: 'conference.example.org'
	},
	bosh: '//example.org/http-bind',
	clientNode: 'http://jitsi.org/jitsimeet',
	focusUserJid: 'focus@auth.example.org',
	testing: {
		enableFirefoxSimulcast: false,
		p2pTestMode: false
	},
	enableNoAudioDetection: true,
	enableNoisyMicDetection: true,
	startAudioMuted: 3,
	desktopSharingChromeExtId: null,
	desktopSharingChromeSources: [ 'screen', 'window', 'tab' ],
	desktopSharingChromeMinExtVersion: '0.1',
	channelLastN: -1,
	useStunTurn: true,
	requireDisplayName: true,
	enableWelcomePage: true,
	defaultLanguage: 'de',
	enableUserRolesBasedOnToken: false,
	disableThirdPartyRequests: true,
	p2p: {
		enabled: true,
		useStunTurn: true,
		stunServers: [
			{ urls: 'stun:example.org:5349' },
		],
		preferH264: true
	},
	analytics: {
	},
	deploymentInfo: {
	},
	makeJsonParserHappy: 'even if last key had a trailing comma'
};

Note: comments removed for brevity

How to test if TURN is working?

So, first off, thanks for reading up to here :slightly_smiling_face: - and sorry for having started with a barrage of configuration files.

But I guess that this is the information required to troubleshoot, and I must have made a mistake somewhere in there – and someone more knowledgeable than me might be able to easily spot that mistake.

Finally, just to recapitulate:

  • jitsi-meet is actually working fine (or at least, it seems to). But all communication, as far as I can tell, happens over UDP port 10000 using the videobridge2.
  • I want to make sure that it would also work, by falling back to the TURN server, if the videobridge wasn’t working (e.g. for people behind firewalls).
  • I have tried to simulate that case by blocking udp port 10000 (incoming, on the server), and would have expected a fallback to TURN, but this is precisely what is never happening, and what I expect to happen. If I block port 10000, then no audio or video works at all.
  • (side note: yes, I know that port 5349 isn’t ideal for that situation either, because it might also be blocked, and that port 443 would be preferrable. But I can’t use that for now, because there’s a real HTTP server there, and I only have one IP).

The entire setup is currently running in “test mode”, so I can change the configuration and restart things at any time. Also, in case I have missed something that I should have provided (like other configuration), feel free to ask.

That said, I’m really amazed by jitsi and its community so far, and I’d be really grateful if someone could help me solve that last mystery.

Ok, this post was long enough already, so: does anybody know what I did wrong?

Thanks a ton!

So the easiest way to test it is by opening 3 tabs from your browser to the deployment and on the deployment block access to port udp 10000 only to your ip-address. Blocking complete access to 10000 you will block turn sending traffic there. This is the jvb use of turn.
For p2p … you need to block traffic between the two peers in the network, which is hard, sometimes …

1 Like

Wow, thank you for the incredibly fast reply!

So, I have now tried the following:

Blocked UDP port 10000 (incoming) from my home IP, but allowed for everything else:

root@bender:~# ufw status
Status: active

To                         Action      From
--                         ------      ----
5349                       ALLOW       Anywhere
10000                      REJECT      55.66.77.88
10000                      ALLOW       Anywhere

(Note: shortened to relevant ports, anonymized IP)

And then set up a conference using

  1. A local browser (using the blocked public IP)
  2. Another browser running in virtual machine, connected to a VPN
  3. My cellphone using the jitsi-meet app

From a server point of view, these clients all connect from different IPs.

The result is that:

  1. The browser on the blocked IP doesn’t get or send any audio or video
  2. The browser on the VPN, and the phone work fine (with each other), but they both only use port 10000.
  3. There is no traffic whatsoever on port 5349 (turn)

This was checked using tcpdump on the server: tcpdump -i any -n port 5349.

Any more ideas?

Thanks!

Next step is to verify you see the turns server used in webrtc-internals.

So open the two non-blocked clients, and first open chrome://webrtc-internals on the PC with blocked access to 10000, then open the participant link.
Now in webrtc-internals there are tabs per ever peer connection, and on the top you should see turns in ice servers.

FWIW, I also tried using 3 tabs in the same browser, connected to the same conference (from the “publicly blocked” IP) as you suggested. The result is that the chat is working, but A/V isn’t, each tab only sees their own picture.

Looking at chrome://webrtc-internals/, I find four different tabs, but only one of them has:

iceServers: [stun:example.org:5349]

(note: only stun, no turn)

and the three others have:
iceServers: []

And in coturn.log on the server, I only see stun BINDING stuff, but no TURN stuff…

That is exactly why I’ve been banging my head on the wall for a few days… :slightly_frowning_face:

OK, I did as you suggested.

Made a conference between phone and VPN, works.

Joined using the blocked client:

  • it “knows” the other participants, but no audio or video from them
  • no audio or video from that client on the other clients (but again, they “know” the blocked client exists)
  • chrome://webrtc-internals on the blocked client shows one tab only, with iceServers: []

:frowning:

So it seems turn is not used for the jvb connections. Check whether you see any errors in the prosody log when restarting it, something about turncredentials module.
Check do you see the file /usr/share/jitsi-meet/prosody-plugins/mod_turncredentials.lua.

Yes, that was my suspicion as well, that maybe the turncredentials plugin wasn’t working…

But everything looks fine on the surface:

root@bender:~# ls -la /usr/share/jitsi-meet/prosody-plugins/mod_turncredentials.lua
-rw-r--r-- 1 root root 2965 Jan  9 18:08 /usr/share/jitsi-meet/prosody-plugins/mod_turncredentials.lua
root@bender:~# /etc/init.d/prosody restart
[ ok ] Restarting prosody (via systemctl): prosody.service.
root@bender:~# grep -i turncred /var/log/prosody/prosody.*

Is there any way to “manually” check that the plugin is working? Like, sending a raw XMPP request on the command line or via curl, or telnet, or so?

Otherwise, from looking at the source, maybe I should explicitly remove the turncredentials_secret from the configuration, in the hope that the module should then fail?

What do you think?

Yeah, try so you can see messages, to see it is trying to load.

Hmm…

root@bender:~# /etc/init.d/prosody restart
[ ok ] Restarting prosody (via systemctl): prosody.service.
root@bender:~# grep -i turncred /var/log/prosody/prosody.
prosody.err  prosody.log
root@bender:~# grep -i turncred /var/log/prosody/prosody.*
/var/log/prosody/prosody.err:Apr 09 23:48:27 example.org:turncredentials        error   turncredentials not configured
/var/log/prosody/prosody.log:Apr 09 23:48:27 example.org:turncredentials        error   turncredentials not configured

So it does seem to get loaded.

Any way to manually check if it’s actually working?

Not that I’m aware of … There are xmpp messages sent from lib-jitsi-meet … Maybe check the JavaScript console for errors or something … Search for turn or iceserver

Oh, you’re on to something here. This is what the chrome console shows:

Logger.js:154 2020-04-09T22:15:55.896Z [modules/xmpp/strophe.jingle.js] getting turn credentials failed <iq xmlns=​"jabber:​client" type=​"error" to=​"be8f6129-c4da-40b7-9391-39c3735ee64e@guest.example.org/​eaf0aa63-3a30-4b78-a998-44f8bdbab736" from=​"guest.example.org" id=​"36adfc7e-440d-4437-8300-9fbed14f576d:​sendIQ">​<error type=​"cancel">​<service-unavailable xmlns=​"urn:​ietf:​params:​xml:​ns:​xmpp-stanzas">​</service-unavailable>​</error>​</iq>​
Logger.js:154 2020-04-09T22:15:55.897Z [modules/xmpp/strophe.jingle.js] is mod_turncredentials or similar installed?

The question is, why am I seeing this and how can I fix it?

There is no indication in the server-side logs of anything going wrong, at least as far as I can tell…

BTW, in case I didn’t say so yet – your help and patience is really appreciated, so thank you again!

Thank you.
Which prosody is this?

But wait, you don’t have it enabled under that host. I just saw you have authentication.
Enabled it and for this virtualhost.

2 Likes

Straight from the repositories…

root@bender:~# dpkg -S /etc/prosody/README
prosody: /etc/prosody/README
root@bender:~# apt-cache policy prosody
prosody:
  Installed: 0.10.0-1build1
  Candidate: 0.10.0-1build1
  Version table:
 *** 0.10.0-1build1 500
		500 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
		100 /var/lib/dpkg/status

Yep, I spotted the problem enable the module and you should be fine.

1 Like

Oh, that definitely makes sense.

So, in the /etc/prosody/conf.avail/example.org.cfg.lua

I currently have all of this enabled for VirtualHost "example.org":

		-- we need bosh
		modules_enabled = {
			"bosh";
			"pubsub";
			"ping"; -- Enable mod_ping
			"speakerstats";
			"turncredentials";
			"conference_duration";
		}

… but pretty much nothing for VirtualHost "guest.example.org".

Should I just enable all the modules, just to be on the safe side, or is that not recommended?

Go one by one :slight_smile: If something is not working, now you know :slight_smile:

@damencho,

you are officially the hero of the day. (well, at least for me)

IT WORKS!

Thank you very very much.

(yes, now in hindsight it’s actually a rather trivial configuration thing, but that’s how most problems appear after they’ve been solved :stuck_out_tongue: )

I just copy-pasted the section modules_enabled part from the example.org VirtualHost to the guest.example.org one, to enable all of the “original” modules, but for future readers of this thread, YMMV.

1 Like

Hi,

I have some issues with coturn as well. Since I enabled LDAP auth, i cant see any candidates going through port 443. I did not tested it with original config…

So, when auth is enabled, what is needed to get coturn working? prosody has a guest.host section with now same modules from the main domain - but nothing changed. No errors in prosody regarding coturn.

Current setup:

plugin_paths = { "/usr/share/jitsi-meet/prosody-plugins/" }

-- domain mapper options, must at least have domain base set to use the mapper
muc_mapper_domain_base = "meeting.host.tld";

turncredentials_secret = "secret";

turncredentials = {
  { type = "stun", host = "meeting.host.tld", port = "4446" },
  { type = "turn", host = "meeting.host.tld", port = "4446", transport = "udp" },
  { type = "turns", host = "meeting.host.tld", port = "443", transport = "tcp" }
};

cross_domain_bosh = false;
consider_bosh_secure = true;

VirtualHost "meeting.host.tld"
        -- enabled = false -- Remove this line to enable this host
        authentication = "cyrus"
        -- Properties below are modified by jitsi-meet-tokens package config
        -- and authentication above is switched to "token"
        --app_id="example_app_id"
        --app_secret="example_app_secret"
        -- Assign this host a certificate for TLS, otherwise it would use the one
        -- set in the global section (if any).
        -- Note that old-style SSL on port 5223 only supports one certificate, and will always
        -- use the global one.
        ssl = {
                key = "/etc/prosody/certs/meeting.host.tld.key";
                certificate = "/etc/prosody/certs/meeting.host.tld.crt";
        }
        speakerstats_component = "speakerstats.meeting.host.tld"
        conference_duration_component = "conferenceduration.meeting.host.tld"
        -- we need bosh
        modules_enabled = {
            "bosh";
            "pubsub";
            "ping"; -- Enable mod_ping
            "speakerstats";
            "turncredentials";
            "conference_duration";
			"auth_cyrus";
        }
        c2s_require_encryption = false
		cyrus_application_name = "xmpp"
		allow_unencrypted_plain_auth = true

VirtualHost "guest.meeting.host.tld"
    authentication = "anonymous"
		modules_enabled = {
		"bosh";
		"pubsub";
		"ping"; -- Enable mod_ping
		"speakerstats";
		"turncredentials";
		"conference_duration";
	}
    c2s_require_encryption = false

Component "conference.meeting.host.tld" "muc"
    storage = "none"
    modules_enabled = {
        "muc_meeting_id";
        "muc_domain_mapper";
        -- "token_verification";
    }
    admins = { "focus@auth.meeting.host.tld" }
    muc_room_locking = false
    muc_room_default_public_jids = true

-- internal muc component
Component "internal.auth.meeting.host.tld" "muc"
    storage = "none"
    modules_enabled = {
      "ping";
    }
    admins = { "focus@auth.meeting.host.tld", "jvb@auth.meeting.host.tld" }
    muc_room_locking = false
    muc_room_default_public_jids = true

VirtualHost "auth.meeting.host.tld"
    ssl = {
        key = "/etc/prosody/certs/auth.meeting.host.tld.key";
        certificate = "/etc/prosody/certs/auth.meeting.host.tld.crt";
    }
    authentication = "internal_plain"

Component "focus.meeting.host.tld"
    component_secret = "secret"

Component "speakerstats.meeting.host.tld" "speakerstats_component"
    muc_component = "conference.meeting.host.tld"

Component "conferenceduration.meeting.host.tld" "conference_duration_component"
    muc_component = "conference.meeting.host.tld"

The guest-section had no modules at first!