Jitsi-meet configured with turn gives wrong IPs for webrtc candidates

Hello,

We have been able to setup a self-hosted jitsi-meet infrastructure which is working great!
Now we need to add a TURN server because some firewalls are not acting friendly and block UPD traffic.
I have searched this forum and found many tutorials/questions/… about this topic and I have tried many configurations. Now I have an issue which I can’t seem to resolve.

When I join a meet it crashes after a few seconds (doesn’t matter if it is with 2 or more persons). I checked the webrtc-internals and noticed that the candidates being passed are not correct. The only candidate I see is the following:
a=candidate:1 1 udp 2130706431 10.x.x.x 10000 typ host generation 0

So it seems the jvb is not giving the proper candidate IPs to the clients which causes a crash as they cant connect to the private IP. The IP (10.x.x.x) shown above is the private IP of the jvb, but I am not sure which IPs should be shown there. Should it be the public IP of the TURN server, or the public IP of the JVB ?

System info
All servers are using the following OS

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.2 LTS
Release:        18.04
Codename:       bionic

Packages

coturn                          4.5.0.7-1ubuntu2.18.
jitsi-videobridge2              2.1-202-g5f9377b9-1
jitsi-meet-prosody              1.0.4127-1
jitsi-meet-web                  1.0.4127-1
jitsi-meet-web-config           1.0.4127-1
prosody                         0.11.5-1~bionic6

Setup
We are running everything on AWS EC2 instances.
The videobridge is on a dedicated EC2 instance
The jitsi-meet/prosody is running on a dedicated EC2 instance
The coturn is running on a dedicated EC2 instance

Configuration

All configuration is more or less copied directly. I only changed the actual domain, password and IPs

/etc/turnserver.conf

listening-port=443
tls-listening-port=443
listening-ip=0.0.0.0
external-ip=PUBLIC_IP_TURN_SERVER/PRIVATE_IP_TURN_SERVER
min-port=10000
max-port=20000
Verbose
fingerprint
use-auth-secret
static-auth-secret=THE_STATIC_AUTH_SECRET
server-name=jitsi_jitsi1_turn_server_node1.mydomain.com
oauth
realm=jitsi_jitsi1_turn_server_node1.mydomain.com
cert=/etc/ssl/mydomain/mydomain_com.crt
pkey=/etc/ssl/mydomain/mydomain_com.crt
no-stdout-log
log-file=/var/log/turnserver.log
no-tlsv1
no-tlsv1_1

/etc/jitsi/videobridge/sip-communicator.properties

org.jitsi.jicofo.ALWAYS_TRUST_MODE_ENABLED=true
org.jitsi.videobridge.ENABLE_REST_SHUTDOWN=true
org.jitsi.videobridge.DISABLE_TCP_HARVESTER=true
org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=true
org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=jitsi_jitsi1_turn_server_node1.mydomain.com:443
org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_TRANSPORT=muc
org.jitsi.videobridge.xmpp.user.shard.HOSTNAME=PRIVATE_IP_XMPP_PROSODY
org.jitsi.videobridge.xmpp.user.shard.DOMAIN=auth.jitsi1.mydomain.com
org.jitsi.videobridge.xmpp.user.shard.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard.PASSWORD=jvb_password
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS=JvbBrewery@internal.auth.jitsi1.mydomain.com
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME=jitsi_jitsi1_shard1_node0
org.jitsi.videobridge.xmpp.user.shard.DISABLE_CERTIFICATE_VERIFICATION=true

/etc/jitsi/videobridge/config

JVB_HOSTNAME=jitsi1.mydomain.com
JVB_HOST=
JVB_PORT=5347
JVB_SECRET=jvb_password
JVB_OPJAVA_SYS_PROPS="-Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/videobridge/logging.properties"TS="--apis=rest,"

/etc/jitsi/jicofo/sip-communicator.properties

org.jitsi.jicofo.BRIDGE_MUC=JvbBrewery@internal.auth.jitsi1.mydomain.com
org.jitsi.jicofo.ALWAYS_TRUST_MODE_ENABLED=true

/etc/jitsi/jicofo/config

JICOFO_HOST=localhost
JICOFO_HOSTNAME=jitsi1.mydomain.com
JICOFO_SECRET=jicofo_secret
JICOFO_PORT=5347
JICOFO_AUTH_DOMAIN=auth.jitsi1.mydomain.com
JICOFO_AUTH_USER=focus
JICOFO_AUTH_PASSWORD=jicofo_password
JICOFO_OPTS=""
JAVA_SYS_PROPS="-Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=jicofo -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/jicofo/logging.properties"

/etc/prosody/conf.avail/jitsi1.mydomain.com.lua

plugin_paths = { "/usr/share/jitsi-meet/prosody-plugins/" }

-- domain mapper options, must at least have domain base set to use the mapper
muc_mapper_domain_base = "jitsi1.mydomain.com";


turncredentials_secret = "THE_STATIC_AUTH_SECRET";
turncredentials_port = 443;
turncredentials_ttl = 86400;
turncredentials = {
   { type = "stun", host = "jitsi_jitsi1_turn_server_node1.mydomain.com", port = "443" },
   { type = "turn", host = "jitsi_jitsi1_turn_server_node1.mydomain.com", port = "443", transport = "udp" },
   { type = "turns", host = "jitsi_jitsi1_turn_server_node1.mydomain.com", port = "443", transport = "tcp" }
};

cross_domain_bosh = false;
consider_bosh_secure = true;

VirtualHost "jitsi1.mydomain.com"
        -- enabled = false -- Remove this line to enable this host
        authentication = "anonymous"
        -- Properties below are modified by jitsi-meet-tokens package config
        -- and authentication above is switched to "token"
        app_id="jitsi_app_id"
        app_secret="jitsi_app_secret"
        -- Assign this host a certificate for TLS, otherwise it would use the one
        -- set in the global section (if any).
        -- Note that old-style SSL on port 5223 only supports one certificate, and will always
        -- use the global one.
        ssl = {
                key = "/etc/prosody/certs/jitsi1.mydomain.com.key";
                certificate = "/etc/prosody/certs/jitsi1.mydomain.com.crt";
        }
        speakerstats_component = "speakerstats.jitsi1.mydomain.com"
        conference_duration_component = "conferenceduration.jitsi1.mydomain.com"
        -- we need bosh
        modules_enabled = {
            "bosh";
            "pubsub";
            "ping"; -- Enable mod_ping
            "speakerstats";
            "turncredentials";
            "conference_duration";
            "muc_status";
        }
        c2s_require_encryption = false

Component "conference.jitsi1.mydomain.com" "muc"
    storage = "memory"
    modules_enabled = {
        "muc_meeting_id";
        "muc_domain_mapper";
        "muc_max_occupants";
        -- "token_verification";
    }
    admins = { "focus@auth.jitsi1.mydomain.com" }
    muc_room_locking = false
    muc_room_default_public_jids = true
    muc_max_occupants = 9
    muc_access_whitelist = { }
    muc_room_default_persistent = false

-- internal muc component
Component "internal.auth.jitsi1.mydomain.com" "muc"
    storage = "memory"
    modules_enabled = {
      "ping";
    }
    admins = { "focus@auth.jitsi1.mydomain.com", "jvb@auth.jitsi1.mydomain.com" }

VirtualHost "auth.jitsi1.mydomain.com"
    ssl = {
        key = "/etc/prosody/certs/auth.jitsi1.mydomain.com.key";
        certificate = "/etc/prosody/certs/auth.jitsi1.mydomain.com.crt";
    }
    authentication = "internal_plain"

Component "focus.jitsi1.mydomain.com"
    component_secret = "jicofo_secret"

Component "speakerstats.jitsi1.mydomain.com" "speakerstats_component"
    muc_component = "conference.jitsi1.mydomain.com"

Component "conferenceduration.jitsi1.mydomain.com" "conference_duration_component"
    muc_component = "conference.jitsi1.mydomain.com"

/etc/jitsi/meet/jitsi1.mydomain.com-config.js

/* eslint-disable no-unused-vars, no-var */

var config = {
    hosts: {
        domain: 'jitsi1.mydomain.com',
        muc: 'conference.jitsi1.mydomain.com'
    },
    bosh: '//jitsi1.mydomain.com/http-bind',
    clientNode: 'http://jitsi.org/jitsimeet',
    testing: {
        p2pTestMode: false
    },
    disableAudioLevels: true,
    enableNoAudioDetection: true,
    enableNoisyMicDetection: true,
    startWithAudioMuted: true,
    resolution: 480,
    constraints: {
        video: {
            aspectRatio: 16 / 9,
            frameRate: {
                max: 15
            },
            height: {
                ideal: 480,
                max: 480,
                min: 240
            }
        }
    },
    disableSimulcast: false,
    disableH264: true,
    desktopSharingChromeExtId: null,
    desktopSharingChromeDisabled: true,
    desktopSharingFirefoxDisabled: true,
    startScreenSharing: false,
    fileRecordingsEnabled: false,
    liveStreamingEnabled: false,
    transcribingEnabled: false,
    autoCaptionOnRecord: false,
    channelLastN: -1,
    useStunTurn: true,
    useNicks: false,
    requireDisplayName: false,
    enableWelcomePage: true,
    enableUserRolesBasedOnToken: false,
    enableCalendarIntegration: false,
    gatherStats: false,
    enableDisplayNameInStats: false,
    enableEmailInStats: false,
    p2p: {
        enabled: true,
        useStunTurn: true,
        stunServers: [
            { urls: 'stun:jitsi_jitsi1_turn_server_node1.mydomain.com:443' },
                    ],
        preferH264: true,
        disableH264: true,
    },
    analytics: {
    },
    deploymentInfo: {
    },
    disableInviteFunctions: true,
    remoteVideoMenu: {
        disableKick: true
    },
    disableRemoteMute: true,
    makeJsonParserHappy: 'even if last key had a trailing comma'
};

/* eslint-enable no-unused-vars, no-var */

How I tried to test the TURN server is working locally
I have disabled outgoing UDP ports 10000-20000 and all incoming UDP ports using Windows Defender Firewall. Then when I joined a meet with another person I cant see/hear the other person and vice versa.
On my windows I used Firefox v77.0.1
On my mac I used Chrome 83.0.4103.97

I have also places some log files in the turncredentials plugin to make sure it returns the stun/turn/turns configuration and that seems to be working. The correct credentials are return backed there.
I also checked the log files (jvb, prosody, jicofo, turnserver) and I didn’t noticed any errors or warnings which could indicate a configuration fault.
The only thing I noticed was in the webrtc-internals where it show the candidates that is only returns the following
a=candidate:1 1 udp 2130706431 10.x.x.x 10000 typ host generation 0
which is the private IP of the jvb.

So to summarize my questions:

  • why does the candidate only show the private IP of the JVB
  • which IP should be returned for the candidates (the IP of the turn server or the JVB)

Thanks in advance to anyone willing to help with this issue. And if I need to provide more information just ask and I will provide it.

1 Like

I have partially fixed my problem. The reason why the candidate only showed the private IP of the JVB was because it resolved the STUN_MAPPING_HARVESTER_ADDRESSES (jitsi_jitsi1_turn_server_node1.mydomain.com) to its private IP and thus it never gave the public IP of the JVB.
Once I changed DNS records so the JVB resolved the jitsi_jitsi1_turn_server_node1.mydomain.com to its public IP I got back both the private IP and the public IP of the JVB in the candidates.

Now the only issue I am still facing is why it doesn’t give back the public IP of the TURNS server for clients which have the UDP ports blocked.

I have a working TURN server on port 80/443 which relays the traffic to the JVB.
The outoing port on the TURN server was not open so the TURN could not connect to the JVB on port 10000. I opened the outoing port on the TURN server so it could make the connection to the JVB.
We tested it with an actual user which could not directly connect to the JVB and was able to join the conference and I saw the connection on the TURN server and it was relaying the traffic to the JVB.