Problem on no fallback to TURN when port 10000 of the meeting user is blocked

Self Hosted Environment:

Server 1 (192.168.83.228):
Oracle Linux 8.4 with SELinux disabled.
Jitsi Meet, JVB, Jicofo version 6689.
prosody version 0.11.11-1

Server 2 (192.168.83.250):
coturn version 4.5.2-1 (hosted on port 443)

Problem Description:
I have done the setup and everything works fine.
In order for me to setup and test the TURN server is working as expected, I now tested with 4 meeting users.

  • Windows Firefox (192.168.83.36)
  • Windows Chrome (192.168.83.36)
  • Android Phone Jitsi App from Play Store (192.168.83.27)
  • Android Phone Jitsi App from Play Store (192.168.83.28)

I block Windows Firefox and Windows Chrome outgoing UDP Port 10000, and immediately lost the Audio and Video, it didn’t fallback using TURN server that I have setup.
As soon as I unblock the outgoing UDP Port 10000, the Audio and Video come back immediately.

I have checked the followings and I cant figure out anythings wrong.

A. coturn
coturn is up and running without any error.
Port 443 is open.

0: : IPv4. TLS/SCTP listener opened on : 0.0.0.0:443
0: : IPv4. TLS/TCP listener opened on : 0.0.0.0:443
0: : IPv4. TLS/SCTP listener opened on : 0.0.0.0:443
0: : IPv4. TLS/TCP listener opened on : 0.0.0.0:443

turnserver.conf

listening-port=443
tls-listening-port=443
listening-ip=0.0.0.0
min-port=40001
max-port=59999
verbose
fingerprint
use-auth-secret
static-auth-secret=Admin123
realm=coturn.jitsi.example.com
cert=/appl/ssl/crt/star.jitsi.example.com.crt
pkey=/appl/ssl/key/star.jitsi.example.com.nopwd.private.pem
log-file=/var/log/coturn/turnserver.log
simple-log
no-multicast-peers
no-cli
no-tlsv1
no-tlsv1_1

B. Jitsi VideoBridge
Jitsi VideoBridge also able to discover STUN server without any problem.

Jan 10 18:08:11 jitsi java[2106]: JVB 2022-01-10 18:08:11.003 INFO: [12] org.ice4j.ice.harvest.StunMappingCandidateHarvester.discover: Discovered public address 192.168.83.228:49174/udp from STUN server 192.168.83.250:443/udp using local address org.ice4j.socket.IceUdpSocketWrapper@5f2f4c9f

Jitsi VideoBridge config.

org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=true
org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=coturn.jitsi.example.com:443

C. turnutils_stunclient
Using turnutils to verify TURN server and all responded ok.

# turnutils_stunclient -p 443 coturn.jitsi.example.com
0: : IPv4. UDP reflexive addr: 192.168.83.228:54737
0: : IPv4. UDP reflexive addr: 192.168.83.228:54737

D. prosody
Tested prosody did load the mod_turncredentials.lua by removing the turncredentials_secret and got the error message.

Jan 10 17:57:15 xmpp.jitsi.example.com:turncredentials    error   turncredentials not configured

Full prosody configuration:

use_libevent = true;

muc_mapper_domain_base = "xmpp.jitsi.example.com";

cross_domain_bosh = false;
consider_bosh_secure = true;
cross_domain_websocket = true;
consider_websocket_secure = true;
https_ports = { };

ssl = {
  protocol = "tlsv1_2+";
  ciphers = "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384"
}

unlimited_jids = {
  "focus@auth.jitsi.example.com",
  "jvb@auth.jitsi.example.com"
}

turncredentials_secret = "Admin123";
turncredentials_port = 443;
turncredentials_ttl = 86400;
turncredentials = {
  { type = "stun", host = "coturn.jitsi.example.com" },
  { type = "turns", host = "coturn.jitsi.example.com", port = 443, transport = "udp" },
  { type = "turns", host = "coturn.jitsi.example.com", port = 443, transport = "tcp" }
}

VirtualHost "xmpp.jitsi.example.com"
  authentication = "internal_hashed"
  ssl = {
    key = "/appl/ssl/key/star.jitsi.example.com.nopwd.private.pem";
    certificate = "/appl/ssl/crt/star.jitsi.example.com.withca.crt";
  }
  av_moderation_component = "avmoderation.xmpp.jitsi.example.com"
  speakerstats_component = "speakerstats.xmpp.jitsi.example.com"
  conference_duration_component = "conferenceduration.xmpp.jitsi.example.com"
  modules_enabled = {
    "bosh";
    "websocket";
    "smacks";
    "pubsub";
    "ping";
    "speakerstats";
    "external_services";
    "conference_duration";
    "muc_lobby_rooms";
    "muc_breakout_rooms";
    "av_moderation";
        "turncredentials";
  }
  c2s_require_encryption = false
  lobby_muc = "lobby.xmpp.jitsi.example.com"
  breakout_rooms_muc = "breakout.xmpp.jitsi.example.com"
  main_muc = "conference.xmpp.jitsi.example.com"
  muc_lobby_whitelist = { "recorder.xmpp.jitsi.example.com" }
  smacks_max_unacked_stanzas = 5;
  smacks_hibernation_time = 60;
  smacks_max_hibernated_sessions = 1;
  smacks_max_old_sessions = 1;

Component "conference.xmpp.jitsi.example.com" "muc"
  restrict_room_creation = true
  storage = "memory"
  modules_enabled = {
    "muc_meeting_id";
    "muc_domain_mapper";
    "polls";
    "muc_rate_limit";
  }
  admins = { "focus@auth.jitsi.example.com" }
  muc_room_locking = false
  muc_room_default_public_jids = true

Component "breakout.xmpp.jitsi.example.com" "muc"
  restrict_room_creation = true
  storage = "memory"
  modules_enabled = {
    "muc_meeting_id";
    "muc_domain_mapper";
    "muc_rate_limit";
  }
  admins = { "focus@auth.jitsi.example.com" }
  muc_room_locking = false
  muc_room_default_public_jids = true

Component "internal-auth.xmpp.jitsi.example.com" "muc"
  storage = "memory"
  modules_enabled = {
    "ping";
  }
  admins = { "focus@auth.jitsi.example.com", "jvb@auth.jitsi.example.com" }
  muc_room_locking = false
  muc_room_default_public_jids = true

VirtualHost "auth.jitsi.example.com"
  authentication = "internal_hashed"
  ssl = {
    key = "/appl/ssl/key/star.jitsi.example.com.nopwd.private.pem";
    certificate = "/appl/ssl/crt/star.jitsi.example.com.withca.crt";
  }
  modules_enabled = {
    "limits_exception";
  }

Component "focus.xmpp.jitsi.example.com" "client_proxy"
  target_address = "focus@auth.jitsi.example.com"

Component "speakerstats.xmpp.jitsi.example.com" "speakerstats_component"
  muc_component = "conference.xmpp.jitsi.example.com"

Component "conferenceduration.xmpp.jitsi.example.com" "conference_duration_component"
  muc_component = "conference.xmpp.jitsi.example.com"

Component "avmoderation.xmpp.jitsi.example.com" "av_moderation_component"
  muc_component = "conference.xmpp.jitsi.example.com"

Component "avmoderation.xmpp.jitsi.example.com" "av_moderation_component"
  muc_component = "conference.xmpp.jitsi.example.com"

Component "lobby.xmpp.jitsi.example.com" "muc"
  storage = "memory"
  restrict_room_creation = true
  muc_room_locking = false
  muc_room_default_public_jids = true
  modules_enabled = {
    "muc_rate_limit";
    "polls";
  }

VirtualHost "guest.xmpp.jitsi.example.com"
  authentication = "anonymous"
  c2s_require_encryption = false
  modules_enabled = {
    "bosh";
    "websocket";
    "smacks";
    "pubsub";
    "ping";
    "speakerstats";
    "external_services";
    "conference_duration";
    "muc_breakout_rooms";
    "av_moderation";
        "turncredentials";
  }
  c2s_require_encryption = false
  breakout_rooms_muc = "breakout.xmpp.jitsi.example.com"
  main_muc = "conference.xmpp.jitsi.example.com"
  smacks_max_unacked_stanzas = 5;
  smacks_hibernation_time = 60;
  smacks_max_hibernated_sessions = 1;
  smacks_max_old_sessions = 1;

E. Chrome browser
The only things from here in chrome browser chrome://webrtc-internals , the iceServers:[] is always empty regardless whether UDP port 10000 is blocked or unblocked.
Tried even clear cache and refresh the page still same, always empty on iceServers: []

https://www.jitsi.example.com/test, { iceServers: [], iceTransportPolicy: all, bundlePolicy: max-bundle, rtcpMuxPolicy: require, iceCandidatePoolSize: 0, sdpSemantics: "unified-plan", extmapAllowMixed: true }, {advanced: [{googScreencastMinBitrate: {exact: 100}}, {googCpuOveruseDetection: {exact: true}}]}

Any clue please enlighten me. Many thanks.

Take a look inside your turnserver.conf file, you’ll see the following

# jitsi-meet coturn relay disable config. Do not modify this line
denied-peer-ip=0.0.0.0-0.255.255.255
#denied-peer-ip=10.0.0.0-10.255.255.255
denied-peer-ip=100.64.0.0-100.127.255.255
denied-peer-ip=127.0.0.0-127.255.255.255
denied-peer-ip=169.254.0.0-169.254.255.255
denied-peer-ip=127.0.0.0-127.255.255.255
denied-peer-ip=172.16.0.0-172.31.255.255
denied-peer-ip=192.0.0.0-192.0.0.255
denied-peer-ip=192.0.2.0-192.0.2.255
denied-peer-ip=192.88.99.0-192.88.99.255
denied-peer-ip=192.168.0.0-192.168.255.255
denied-peer-ip=198.18.0.0-198.19.255.255
denied-peer-ip=198.51.100.0-198.51.100.255
denied-peer-ip=203.0.113.0-203.0.113.255
denied-peer-ip=240.0.0.0-255.255.255.255

notice this one ? denied-peer-ip=192.168.0.0-192.168.255.255

Do you see a log in the webclient why it is not loading the iceServers?
You see them if you test on meet.jit.si, right?

I have checked, I dont have any denied-peer-ip.

I tried with meet.jit.si and able to see the iceServers.
iceServers: [turns:meet-jit-si-turnrelay.jitsi.net:443?transport=tcp]

I will further check my webclient for any errors, will update here again.

@damencho , I tried using F12 Developer Tools to look for “why it is not loading the iceServers” but I didn’t see any clue.
I also compared the traffic in meet.jit.si, also didn’t see any clue and diff of the iceServers.
May you please tell me what pattern I should be looking into ?

@damencho , so far these are the difference I saw from the meet.jit.si vs what I have setup.
Also, want to ask, which version is currently hosted in meet.jit.si ?
because I saw the difference of the xmpp websocket message passing …
in current meet.jit.si, clearly see that it responded with the service of the TURN, however from my setup using version 6689, it responded with features after the identity.

from meet.jit.si:

<message from='meet.jit.si' xmlns='jabber:client' to='c04a6998-ebbe-4cbb-858c-d88e186f5403@meet.jit.si/XN8mPiX5'>
	<query xmlns='http://jabber.org/protocol/disco#info'>
		<identity name='Prosody' category='server' type='im'/>
		<identity name='conferenceduration.meet.jit.si' category='component' type='conference_duration'/>
		<identity name='lobby.meet.jit.si' category='component' type='lobbyrooms'/>
		<identity name='breakout.meet.jit.si' category='component' type='breakout_rooms'/>
		<identity name='meet-jit-si-ap-south-1a-s8' category='server' type='shard'/>
		<identity name='ap-south-1' category='server' type='region'/>
		<identity name='speakerstats.meet.jit.si' category='component' type='speakerstats'/>
		<identity name='avmoderation.meet.jit.si' category='component' type='av_moderation'/>
	</query>
	<services>
		<service host='meet-jit-si-turnrelay.jitsi.net' port='443' type='stun'/>
		<service host='meet-jit-si-turnrelay.jitsi.net' restricted='1' port='443' type='turn' transport='udp' password='b0JP27OurEMb3Vz1mLueGsUC10s=' username='1641952882' expires='2022-01-12T02:01:22Z'/>
		<service host='meet-jit-si-turnrelay.jitsi.net' restricted='1' port='443' type='turns' transport='tcp' password='b0JP27OurEMb3Vz1mLueGsUC10s=' username='1641952882' expires='2022-01-12T02:01:22Z'/>
	</services>
</message>

from my problematic environment:

<iq from='xmpp.jitsi.example.com' id='288a3c2a-df87-406f-ac48-ebae61dea18b:sendIQ' to='dj_5xmeudkufq7gg@guest.xmpp.jitsi.example.com/0iWgl2RW' xmlns='jabber:client' type='result'>
	<query xmlns='http://jabber.org/protocol/disco#info'>
		<identity category='pubsub' type='service' name='Prosody PubSub Service'/>
		<identity category='server' type='im' name='Prosody'/>
		<identity category='component' type='conference_duration' name='conferenceduration.xmpp.jitsi.example.com'/>
		<identity category='component' type='breakout_rooms' name='breakout.xmpp.jitsi.example.com'/>
		<identity category='component' type='av_moderation' name='avmoderation.xmpp.jitsi.example.com'/>
		<identity category='component' type='lobbyrooms' name='lobby.xmpp.jitsi.example.com'/>
		<identity category='pubsub' type='pep' name='Prosody'/>
		<identity category='component' type='speakerstats' name='speakerstats.xmpp.jitsi.example.com'/>
		
		<feature var='urn:xmpp:extdisco:2'/>
		<feature var='urn:xmpp:extdisco:1'/>
		<feature var='http://jabber.org/protocol/commands'/>
		<feature var='http://jabber.org/protocol/pubsub'/>
		<feature var='http://jabber.org/protocol/pubsub#item-ids'/>
		<feature var='http://jabber.org/protocol/pubsub#delete-items'/>
		<feature var='http://jabber.org/protocol/pubsub#publish'/>
		<feature var='http://jabber.org/protocol/pubsub#persistent-items'/>
		<feature var='http://jabber.org/protocol/pubsub#instant-nodes'/>
		<feature var='http://jabber.org/protocol/pubsub#retrieve-subscriptions'/>
		<feature var='http://jabber.org/protocol/pubsub#subscribe'/>
		<feature var='http://jabber.org/protocol/pubsub#retrieve-items'/>
		<feature var='http://jabber.org/protocol/pubsub#meta-data'/>
		<feature var='http://jabber.org/protocol/pubsub#config-node'/>
		<feature var='http://jabber.org/protocol/pubsub#create-nodes'/>
		<feature var='http://jabber.org/protocol/pubsub#subscription-options'/>
		<feature var='http://jabber.org/protocol/pubsub#access-open'/>
		<feature var='http://jabber.org/protocol/pubsub#publisher-affiliation'/>
		<feature var='http://jabber.org/protocol/pubsub#create-and-configure'/>
		<feature var='http://jabber.org/protocol/pubsub#multi-items'/>
		<feature var='http://jabber.org/protocol/pubsub#retrieve-default'/>
		<feature var='http://jabber.org/protocol/pubsub#member-affiliation'/>
		<feature var='http://jabber.org/protocol/pubsub#outcast-affiliation'/>
		<feature var='http://jabber.org/protocol/pubsub#publish-options'/>
		<feature var='http://jabber.org/protocol/pubsub#purge-nodes'/>
		<feature var='http://jabber.org/protocol/pubsub#modify-affiliations'/>
		<feature var='http://jabber.org/protocol/pubsub#delete-nodes'/>
		<feature var='http://jabber.org/protocol/pubsub#retract-items'/>
		<feature var='jabber:iq:register'/>
		<feature var='http://jabber.org/protocol/disco#info'/>
		<feature var='http://jabber.org/protocol/disco#items'/>
		<feature var='vcard-temp'/><feature var='urn:xmpp:blocking'/>
		<feature var='jabber:iq:version'/>
		<feature var='jabber:iq:last'/>
		<feature var='msgoffline'/>
		<feature var='jabber:iq:private'/>
		<feature var='jabber:iq:roster'/>
		<feature var='urn:xmpp:carbons:2'/>
		<feature var='urn:xmpp:ping'/>
		<feature var='urn:xmpp:time'/>
		<feature var='jabber:iq:time'/>
	</query>
</iq>

That doesn’t matter. That is a optimization we had done for large calls?

You have disco info enabled and there should be queries discovering the turnserver around joining the room.

That’s exactly I am wondering too.
Because videobridge able to discover the STUN server but not the client.
Hence, it makes me think that the problem is not on the coturn configuration since many evidence showed that coturn is up and running properly.

Would it be possible somewhere in my prosody config is wrong ?
I have updated in my original post for full prosody config.

Do you see the extdiso query and the response with turnservers in your deployment?

Finally I able to figure it out, but I dont really understand …
May be @damencho if you dont mind to explain it to us ?

At first I am using turncredentials and my prosody config as this,

turncredentials_secret = "Admin123";
turncredentials_port = 443;
turncredentials_ttl = 86400;
turncredentials = {
  { type = "stun", host = "coturn.jitsi.example.com" },
  { type = "turns", host = "coturn.jitsi.example.com", port = 443, transport = "udp" },
  { type = "turns", host = "coturn.jitsi.example.com", port = 443, transport = "tcp" }
}

After referring to this github prosody sample,

I changed my prosody config to use external_services and the prosody config now

external_service_secret = "Admin123";
external_services = {
  { type = "stun", host = "coturn.jitsi.example.com", port = 443 },
  { type = "turn", host = "coturn.jitsi.example.com", port = 443, transport = "udp", secret = true, ttl = 86400, algorithm = "turn" },
  { type = "turns", host = "coturn.jitsi.example.com", port = 443, transport = "tcp", secret = true, ttl = 86400, algorithm = "turn" }
};

And all works like a charm …

Not sure, both are picky about some of the params, but not sure which … so I would guess it is the config and some of the params were strings instead of number or the other way around.