Incoming web socket request with an invalid password

I’m trying to set up a Meet + multi-JVB setup that will work when some clients are behind a corporate VPN that blocks non-HTTPS traffic. My understanding is that websockets should allow this to work, but I’ve hit a roadblock when trying to get multiple JVBs involved.

My setup consists of a “main” server running Meet, Jicofo, and one JVB; and two other servers (all in a common OpenVPN subnet with the main server), each running a JVB. The JVBs & Jicofo use a MUC so that each JVB can reach Jicofo. Most importantly, I followed the instructions to set up websockets, including giving each JVB a unique server_id value in jvb.conf, and updating the main server’s nginx.conf accordingly. I also configured the firewalls on each server to open all required ports.

However, only the JVB on the server running Meet & Jicofo is able to work properly. When one of the other JVBs is active, starting a voice/video chat (with P2P disabled) between my phone and my work laptop while connected to a corp VPN displays this error in their logs:

Endpoint.acceptWebSocket#661: Incoming web socket request with an invalid password. Expected: <something> received <something_else>

Stranger still, later logs show that the correct password sometimes does eventually get sent, but by then, the JVB expects a different password!

And in the console of the web browser running Meet (either Chrome or Firefox), this error appears:

<WebSocket.e.onclose>: Channel closed: 1006

Disconnecting my work laptop from the corp VPN allows everything to work, though I suspect that’s only it’s because the connection doesn’t have to rely on websockets anymore.

Any hints as to what is going wrong?

Thanks!

Does it work when JVB is stopped on JMS?

Show your JVB & nginx configs; you may be routing all Colibri websocket requests to the first JVB rather than directing them to the correct JVB based on the server-id.

(And note that using WebSocket for Colibri is not really related to getting around strict firewalls that block non-HTTPS traffic. Your main problem will be the actual media (audio & video), which normally uses DTLS-SRTP on a UDP port. You’ll need a working TURN server if the firewall blocks that.)

No, things break when I turn off the JVB on the JMS server.

Using any of the other JVMs allows a meeting to run, but unless a P2P connection is in use, voice/video is not sent/received, and the Meet web client shows “Connection lost”.

Trying to start a meeting with no JVMs running causes meetings to crash, which makes sense.

I have a coturn server to handle P2P connections.

@mr_johnson22

Can you reply this?

Here are my configs. Apologies for the delay.

Server descriptions

Server 1

Debian 11 armhf
Present in my local home network as 192.168.0.x
Runs jitsi-meet-*, jicofo, jitsi-videobridge, all installed from the jitsi-stable repository
Runs coturn, installed & configured independently of Jitsi
Connected to an OpenVPN server on Server 2 as 10.8.0.x / fd00::x
Exposed to the Internet via a transparent proxy on Server 2

Server 2

Debian 11 amd64
VPS with static & publicly routable addresses
Runs jitsi-videobridge2, installed from jitsi-stable
Runs OpenVPN server with address 10.8.0.1, which Servers 1 and 3 are connected to as clients
Acts as a transparent proxy for public services on Servers 1 and 3

Server 3

Fedora 35 x86_64
Present in my local home network as 192.168.0.y
Runs jitsi-videobridge2 built from source
Connected to the OpenVPN server on Server 2 as 10.8.0.y / fd00::y

Server 1 config

nginx

/etc/nginx/sites-enabled/jitsi.domain.tld.conf

(Some settings, like SSL certificates, are managed in /etc/nginx/nginx.conf to be shared by all virtual hosts on the system)

types {
    application/wasm     wasm;
}
server {
    listen 80;
    listen [::]:80;
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name jitsi.domain.tld jitsi.domain.local;

    set $prefix "";

    root /usr/share/jitsi-meet;

    # ssi on with javascript for multidomain variables in config.js
    ssi on;
    ssi_types application/x-javascript application/javascript;

    index index.html index.htm;
    error_page 404 /static/404.html;

    gzip on;
    gzip_types text/plain text/css application/javascript application/json image/x-icon application/octet-stream application/wasm;
    gzip_vary on;
    gzip_proxied no-cache no-store private expired auth;
    gzip_min_length 512;

    location = /config.js {
        alias /etc/jitsi/meet/jitsi.domain.tld-config.js;
    }

    location = /external_api.js {
        alias /usr/share/jitsi-meet/libs/external_api.min.js;
    }

    # ensure all static content can always be found first
    location ~ ^/(libs|css|static|images|fonts|lang|sounds|connection_optimization|.well-known)/(.*)$
    {
        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
        add_header 'Access-Control-Allow-Origin' '*';
        alias /usr/share/jitsi-meet/$1/$2;

        # cache all versioned files
        if ($arg_v) {
            expires 1y;
        }
    }

    # BOSH
    location = /http-bind {
        proxy_pass http://127.0.0.1:5280/http-bind?prefix=$prefix&$args;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $http_host;
    }

    # xmpp websockets
    location = /xmpp-websocket {
        proxy_pass http://127.0.0.1:5280/xmpp-websocket?prefix=$prefix&$args;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $http_host;
        tcp_nodelay on;
    }

    # colibri (JVB) websockets for each videobridge
    # https://github.com/jitsi/jitsi-videobridge/blob/master/doc/web-sockets.md#proxy-configuration
    location ~ ^/colibri-ws/jvb1/(.*) {
        proxy_pass http://127.0.0.1:9090/colibri-ws/jvb1/$1$is_args$args;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        tcp_nodelay on;
    }
    location ~ ^/colibri-ws/jvb2/(.*) {
        proxy_pass http://10.8.0.1:9090/colibri-ws/jvb2/$1$is_args$args;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        tcp_nodelay on;
    }
    location ~ ^/colibri-ws/jvb3/(.*) {
        proxy_pass http://192.168.0.y:9090/colibri-ws/jvb3/$1$is_args$args;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        tcp_nodelay on;
    }

    # load test minimal client, uncomment when used
    #location ~ ^/_load-test/([^/?&:'"]+)$ {
    #    rewrite ^/_load-test/(.*)$ /load-test/index.html break;
    #}
    #location ~ ^/_load-test/libs/(.*)$ {
    #    add_header 'Access-Control-Allow-Origin' '*';
    #    alias /usr/share/jitsi-meet/load-test/libs/$1;
    #}

    location ~ ^/([^/?&:'"]+)$ {
        try_files $uri @root_path;
    }

    location @root_path {
        rewrite ^/(.*)$ / break;
    }

    location ~ ^/([^/?&:'"]+)/config.js$
    {
        set $subdomain "$1.";
        set $subdir "$1/";

        alias /etc/jitsi/meet/jitsi.domain.tld-config.js;
    }

    # BOSH for subdomains
    location ~ ^/([^/?&:'"]+)/http-bind {
        set $subdomain "$1.";
        set $subdir "$1/";
        set $prefix "$1";

        rewrite ^/(.*)$ /http-bind;
    }

    # websockets for subdomains
    location ~ ^/([^/?&:'"]+)/xmpp-websocket {
        set $subdomain "$1.";
        set $subdir "$1/";
        set $prefix "$1";

        rewrite ^/(.*)$ /xmpp-websocket;
    }

    # Anything that didn't match above, and isn't a real file, assume it's a room name and redirect to /
    location ~ ^/([^/?&:'"]+)/(.*)$ {
        set $subdomain "$1.";
        set $subdir "$1/";
        rewrite ^/([^/?&:'"]+)/(.*)$ /$2;
    }
}

prosody

/etc/prosody/conf.d/jitsi.domain.tld.cfg.lua

plugin_paths = { "/usr/share/jitsi-meet/prosody-plugins/" }
-- domain mapper options, must at least have domain base set to use the mapper
muc_mapper_domain_base = "jitsi.domain.tld";

external_service_secret = "REDACTED_TURN_SECRET";
external_services = {
     { type = "stun", host = "turn.domain.tld", port = 3478 },
     { type = "turn", host = "turn.domain.tld", port = 3478, transport = "udp", secret = true, ttl = 86400, algorithm = "turn" },
     { type = "turn", host = "turn.domain.tld", port = 3478, transport = "tcp", secret = true, ttl = 86400, algorithm = "turn" }
     -- { type = "turns", host = "turn.domain.tld", port = 5349, transport = "tcp", secret = true, ttl = 86400, algorithm = "turn" }
};

cross_domain_bosh = false;
consider_bosh_secure = true;
https_ports = { }; -- Remove this line to prevent listening on port 5284

The rest of the file is unchanged from what was installed by the jitsi-meet-prosody package.

I disabled DTLS because of issues with using Let’s Encrypt certificates (see Coturn fails to connect: "TLS/TCP socket buffer operation error (callback)").

coturn

/etc/turnserver.conf (comments removed)

listening-ip=10.8.0.x
listening-ip=fd00::x
external-ip=REDACTED_PUBLIC_IPv4_ADDRESS/10.8.0.x
external-ip=REDACTED_PUBLIC_IPv6_ADDRESS/fd00::x
verbose
use-auth-secret
static-auth-secret=REDACTED_TURN_SECRET
realm=turn.domain.tld
no-tls
no-dtls
no-tcp-relay
cert=/etc/letsencrypt/live/domain.tld/fullchain.pem
pkey=/etc/letsencrypt/live/domain.tld/privkey.pem
dh-file=/etc/letsencrypt/ssl-dhparams.pem
syslog
no-multicast-peers
denied-peer-ip=10.0.0.0-10.255.255.255
denied-peer-ip=172.16.0.0-172.31.255.255
denied-peer-ip=192.168.0.0-192.168.255.255
no-cli
no-tlsv1
no-tlsv1_1

jicofo

/etc/jitsi/jicofo/jicofo.conf

# Jicofo HOCON configuration. See /usr/share/jicofo/jicofo.jar/reference.conf for
#available options, syntax, and default values.
jicofo {
  xmpp: {
    client: {
      client-proxy: focus.jitsi.domain.tld
    }
    trusted-domains: [ "recorder.jitsi.domain.tld" ]
  }
  bridge: {
    brewery-jid: "JvbBrewery@internal.auth.jitsi.domain.tld"
    selection-strategy = RegionBasedBridgeSelectionStrategy
  }

  # https://github.com/jitsi/jitsi-videobridge/blob/master/doc/octo.md#jicofo-configuration
  octo {
    // Whether or not to use Octo. Note that when enabled, its use will be determined by
    // $jicofo.bridge.selection-strategy. There's a corresponding flag in the JVB and these
    // two MUST be in sync (otherwise bridges will crash because they won't know how to
    // deal with octo channels).
    enabled = true

    // An identifier of the Jicofo instance, used for the purpose of generating conference IDs unique across a set of
    // Jicofo instances. Valid values are [1, 65535]. The value 0 is used when none is explicitly configured.
    id = "1"
  }
  # https://community.jitsi.org/t/p2p-ok-third-party-joins-conference-videobridgenotavailable/104633/5
  sctp {
    enabled = false
  }
}

I disabled SCTP because the native dependencies for using SCTP aren’t available on armhf.

/etc/jitsi/jicofo/sip-communicator.properties

# https://github.com/jitsi/jitsi-videobridge/blob/master/doc/muc.md#jicofo-configuration
org.jitsi.jicofo.BRIDGE_MUC=JvbBrewery@internal.auth.jitsi.domain.tld

/etc/jitsi/jicofo/config (for the systemd service)

# Jitsi Conference Focus settings
# sets the host name of the XMPP server
JICOFO_HOST=localhost

# sets the XMPP domain (default: none)
JICOFO_HOSTNAME=jitsi.domain.tld

# sets the XMPP domain name to use for XMPP user logins
JICOFO_AUTH_DOMAIN=auth.jitsi.domain.tld

# sets the username to use for XMPP user logins
JICOFO_AUTH_USER=focus

# sets the password to use for XMPP user logins
JICOFO_AUTH_PASSWORD=REDACTED_JICOFO_SECRET

# extra options to pass to the jicofo daemon
JICOFO_OPTS=""

# adds java system props that are passed to jicofo (default are for home and logging config file)
JAVA_SYS_PROPS="-Dconfig.file=/etc/jitsi/jicofo/jicofo.conf -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=jicofo -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/jicofo/logging.properties"

# NEW
JICOFO_MAX_MEMORY=256m

jitsi-meet

/etc/jitsi/meet/jitsi.domain.tld-config.js (comments removed)

var config = {
    hosts: {
        domain: 'jitsi.domain.tld',
        muc: 'conference.<!--# echo var="subdomain" default="" -->jitsi.domain.tld'
    },
    bosh: '//jitsi.domain.tld/http-bind',
    testing: {
        octo: {
            probability: 1
        },
    },
    enableNoAudioDetection: true,
    enableNoisyMicDetection: true,
    startAudioOnly: true,
    fileRecordingsEnabled: false,
    fileRecordingsServiceEnabled: false,
    fileRecordingsServiceSharingEnabled: false,
    liveStreamingEnabled: false,
    transcribingEnabled: false,
    autoCaptionOnRecord: false,
    channelLastN: -1,
    enableWelcomePage: true,
    p2p: {
        enabled: true,
        stunServers: [
            { urls: 'stun:turn.domain.tld:3478' }
        ]
    },
    analytics: {
    },
    deploymentInfo: {
        region: "region1",
    },
    mouseMoveCallbackInterval: 1000,
    makeJsonParserHappy: 'even if last key had a trailing comma'
};

jitsi-videobridge2

/etc/jitsi/videobridge/jvb.conf

# https://jitsi.github.io/handbook/docs/devops-guide/faq
videobridge {
  http-servers {
    public {
      port = 9090
    }
  }
  websockets {
    enabled = true
    domain = "jitsi.domain.tld:443"
    tls = true
    # https://github.com/jitsi/jitsi-videobridge/blob/master/doc/web-sockets.md
    server-id = "jvb1"
  }
  # https://community.jitsi.org/t/posibility-of-proper-arm-packaging-for-jvb2/102703/8
  sctp {
    enabled = false
  }
}
ice4j {
  harvest {
    mapping {
      aws {
        enabled = false
      }
      static-mappings = [
        {
          local-address = "10.8.0.x"
          public-address = "REDACTED_PUBLIC_IPv4_ADDRESS"
          local-port = 10000
          public-port = 10002
        },
        {
          local-address = "fd00::x"
          public-address = "REDACTED_PUBLIC_IPv6_ADDRESS"
          local-port = 10000
          public-port = 10002
        }
      ]
    }
  }
}

/etc/jitsi/videobridge/sip-communicator.properties

org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_TRANSPORT=muc
org.jitsi.videobridge.xmpp.user.shard.HOSTNAME=localhost
org.jitsi.videobridge.xmpp.user.shard.DOMAIN=auth.jitsi.domain.tld
org.jitsi.videobridge.xmpp.user.shard.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard.PASSWORD=REDACTED_XMPP_SECRET
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS=JvbBrewery@internal.auth.jitsi.domain.tld
org.jitsi.videobridge.xmpp.user.shard.MUC=JvbBrewery@internal.auth.jitsi.domain.tld
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME=jvb1
org.jitsi.videobridge.xmpp.user.shard.DISABLE_CERTIFICATE_VERIFICATION=true
org.jitsi.videobridge.octo.BIND_ADDRESS=10.8.0.x
org.jitsi.videobridge.octo.BIND_PORT=4096
org.jitsi.videobridge.REGION=region1

/etc/jitsi/videobridge/config (for the systemd service)

# Jitsi Videobridge settings

# sets the XMPP domain (default: none)
JVB_HOSTNAME=jitsi.domain.tld

# sets the hostname of the XMPP server (default: domain if set, localhost otherwise)
JVB_HOST=

# sets the port of the XMPP server (default: 5275)
JVB_PORT=5347

# sets the shared secret used to authenticate to the XMPP server
JVB_SECRET=REDACTED_XMPP_SECRET

# extra options to pass to the JVB daemon
JVB_OPTS="--apis=,"


# adds java system props that are passed to jvb (default are for home and logging config file)
JAVA_SYS_PROPS="-Dconfig.file=/etc/jitsi/videobridge/jvb.conf -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/videobridge/logging.properties"

# NEW
VIDEOBRIDGE_MAX_MEMORY=512m

Firewall settings

Open ports:

  • 80,443/TCP
  • 3478-3479/TCP+UDP, 5349-5350/TCP+UDP, 49152-65535/UDP (for coturn)
  • 10000/UDP (for jvb)
  • 5222/TCP (for prosody)
  • 5347/TCP (for jicofo)
  • 4096/TCP (for octo)
  • 9090/TCP (for websockets)

Server 2 config

jitsi-videobridge2

jvb.conf

# https://jitsi.github.io/handbook/docs/devops-guide/faq
videobridge {
  http-servers {
    public {
      port = 9090
    }
  }
  websockets {
    enabled = true
    domain = "jitsi.domain.tld:443"
    tls = true
    # https://github.com/jitsi/jitsi-videobridge/blob/master/doc/web-sockets.md
    server-id = "jvb2"
  }
  # https://community.jitsi.org/t/posibility-of-proper-arm-packaging-for-jvb2/102703/8
  sctp {
    enabled = false
  }
}
ice4j {
  harvest {
    mapping {
      aws {
        enabled = false
      }
    }
  }
}

/etc/jitsi/videobridge/sip-communicator.properties

org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_TRANSPORT=muc
org.jitsi.videobridge.xmpp.user.shard.HOSTNAME=10.8.0.x
org.jitsi.videobridge.xmpp.user.shard.DOMAIN=auth.jitsi.domain.tld
org.jitsi.videobridge.xmpp.user.shard.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard.PASSWORD=REDACTED_XMPP_SECRET
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS=JvbBrewery@internal.auth.jitsi.domain.tld
org.jitsi.videobridge.xmpp.user.shard.MUC=JvbBrewery@internal.auth.jitsi.domain.tld
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME=jvb2
org.jitsi.videobridge.xmpp.user.shard.DISABLE_CERTIFICATE_VERIFICATION=true
org.jitsi.videobridge.octo.BIND_ADDRESS=10.8.0.1
org.jitsi.videobridge.octo.BIND_PORT=4096
org.jitsi.videobridge.REGION=region2

/etc/jitsi/videobridge/config (for the systemd service)

# Jitsi Videobridge settings

# sets the XMPP domain (default: none)
JVB_HOSTNAME=jitsi.domain.tld

# sets the hostname of the XMPP server (default: domain if set, localhost otherwise)
JVB_HOST=10.8.0.x

# sets the port of the XMPP server (default: 5275)
JVB_PORT=5347

# sets the shared secret used to authenticate to the XMPP server
JVB_SECRET=REDACTED_XMPP_SECRET

# extra options to pass to the JVB daemon
JVB_OPTS="--apis=,"


# adds java system props that are passed to jvb (default are for home and logging config file)
JAVA_SYS_PROPS="-Dconfig.file=/etc/jitsi/videobridge/jvb.conf -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.   communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/  etc/jitsi/videobridge/logging.properties"

# NEW
VIDEOBRIDGE_MAX_MEMORY=1536m

Firewall settings

Open ports: all (default allow)

Forwarded ports:

  • 10001/UDP → to Server 3
  • 10002/UDP → to Server 2

nginx

Configured with a transparent proxy that sends HTTPS traffic to either Server 2 or 3 based on hostname.

This might not be relevant, but I’ll include it in case it’s breaking anything.

stream {
    map $ssl_preread_server_name $name {
        a.domain.tld server3;
        b.domain.tld server3;
        default server2;
    }
    map $ssl_preread_server_name $name6 {
        a.domain.tld server3v6;
        b.domain.tld server3v6;
        default server2v6;
    }

    upstream server3 {
        server 10.8.0.y:444;
    }
    upstream server3v6 {
        server [fd00::y]:444;
    }

    upstream server2 {
        server 10.8.0.x:443;
    }
    upstream server2v6 {
        server [fd00::x]:443;
    }

    server {
        listen 443;
        proxy_bind $remote_addr transparent;
        proxy_pass $name;
        ssl_preread on;
    }
    server {
        listen [::]:443;
        proxy_bind $remote_addr transparent;
        proxy_pass $name6;
        ssl_preread on;
    }
}

Server 3 config

jitsi-videobridge2

jvb.conf

# https://jitsi.github.io/handbook/docs/devops-guide/faq
videobridge {
  http-servers {
    public {
      port = 9090
    }
  }
  websockets {
    enabled = true
    domain = "jitsi.domain.tld:443"
    tls = true
    # https://github.com/jitsi/jitsi-videobridge/blob/master/doc/web-sockets.md
    server-id = "jvb3"
  }
  # https://community.jitsi.org/t/posibility-of-proper-arm-packaging-for-jvb2/102703/8
  sctp {
    enabled = false
  }
}
ice4j {
  harvest {
    mapping {
      aws {
        enabled = false
      }
      static-mappings = [
        {
          local-address = "10.8.0.y"
          public-address = "REDACTED_PUBLIC_IPv4_ADDRESS"
          local-port = 10000
          public-port = 10001
        },
        {
          local-address = "fd00::y"
          public-address = "REDACTED_PUBLIC_IPv6_ADDRESS"
          local-port = 10000
          public-port = 10001
        }
      ]
    }
  }
}

/etc/jitsi/videobridge/sip-communicator.properties

org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_TRANSPORT=muc
org.jitsi.videobridge.xmpp.user.shard.HOSTNAME=192.168.0.x
org.jitsi.videobridge.xmpp.user.shard.DOMAIN=auth.jitsi.domain.tld
org.jitsi.videobridge.xmpp.user.shard.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard.PASSWORD=REDACTED_XMPP_SECRET
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS=JvbBrewery@internal.auth.jitsi.domain.tld
org.jitsi.videobridge.xmpp.user.shard.MUC=JvbBrewery@internal.auth.jitsi.domain.tld
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME=jvb3
org.jitsi.videobridge.xmpp.user.shard.DISABLE_CERTIFICATE_VERIFICATION=true
org.jitsi.videobridge.octo.BIND_ADDRESS=10.8.0.y
org.jitsi.videobridge.octo.BIND_PORT=4096
org.jitsi.videobridge.REGION=region3

Firewall settings

Open ports: Same as Server 1, but without the coturn ports, and with 444/TCP for HTTPS traffic (unrelated to Jitsi)

Launching command

Instead of being managed by a systemd service, the JVB is launched as per the recommendation in the source’s README:

mvn compile exec:exec -Dexec.args=-Djava.library.path=./lib/native/linux-64 -Djava.util.logging.config.file=./lib/logging.properties -Dconfig.file=/home/USERNAME/.config/jitsi/videobridge/jvb.conf -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/home/USERNAME/.config/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=videobridge -cp %classpath org.jitsi.videobridge.MainKt

  • TURNS is disable. So some participants behind a corporate firewall cannot connect to the meeting.

  • JICOFO_MAX_MEMORY seems very low

  • VIDEOBRIDGE_MAX_MEMORY seems low

  • I don’t understand why jvb.conf is so complicated in JMS

  • org.jitsi.videobridge.xmpp.user.shard.HOSTNAME in my setup is an FQDN, not IP… I never tried with IP

  • why is the region different in JVB2?

  • JVB2 and JVB3 listen the default port UDP/10000. There is no SINGLE_PORT_HARVESTER_PORT line in their config. I don’t know if static-mappings can handle this situation.

I won’t comment on all your setup (well I will, it’s terribly complicated; I hope that it is not your first try at Jitsi-meet with several Jvbs and you have first setup a simple configuration to test how it all works and grow from a firm base - usually trying to solve all problems at the same time often finish in a ditch) but are you really using ARM processors as seem to be implied in your jvb.conf ? If not, try to enable sctp back on. When websockets are not working, sctp can save the day (until it’s finally disabled by Jitsi devs that is)

Haven’t had time to look over the config in detail, but if non-P2P calls are not working at all (“Connection lost” as stated in post #3) then it’s a bigger issue than just SCTP/colibri websocket. With no Colibri websocket you’ll usually still get audio & video, you’ll just have problems with constraints etc not being signalled.

@mr_johnson22 you can look at the XMPP websocket in developer tools to see the Jingle signalling, look for Jingle session-initiate, source-add and/or transport-info messages. In those you will find the JVB IP(s) (“candidates”) that Jicofo is telling the browser to use for audio & video. Check that they are the correct public IP of one of your JVBs. And check that you have nothing like a firewall blocking udp/10000 on the JVB servers.

If you’re trying to do a test with UDP blocked to verify that your TURN setup is working, you’ll have to fix TURNS. Browsers won’t do WebRTC over TURN without TLS & a valid certificate on the TURN server.

That’s what I was thinking until I disabled both sctp and websockets. I got no more video and sound. You are right that it did not lead to disconnects (at least immediately); however I did try it with current unstable, I don’t know what could happen with current stable.

Unfortunately, the Jitsi Meet app on Android (among other apps) uses a library that rejects Let’s Encrypt certificates: Android App Rejects Let's Encrypt Chain on TURNS Connection · Issue #5589 · jitsi/jitsi-meet · GitHub. With that said, TURN seems to be working okay, as P2P calls work correctly. I’ll consider switching to ZeroSSL if this proves to be an issue after all.

Yes, my ARM server is memory-constrained, so I have to lower the max memory, lest jicofo/jvb don’t launch at all! Calls are still able to work, thankfully–at least if only one is running at a time, which is all I need.

IMO it looks more complicated than it really is. The only changes I made from the default installation are:

  • Set videobridge.websockets.server-id (here, and in the every other jvb.conf too) that the nginx config can distinguish between each JVB
  • Set videobridge.sctp.enabled=false to prevent using SCTP, which doesn’t work on armhf due to missing native dependencies
  • Added the ice4j section to configure both IPv4 and IPv6 static mappings (as an alternative to org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL/PUBLIC_ADDRESS, which only allows a single mapping)

It does work, thankfully! I’m glad it does, as it makes it possible to connect to another machine in the local network without having to fiddle with DNS names.

No real reason; I just set each JVB’s region to something unique for debugging purposes.

You might be right about this. I thought ice4j settings would configure this (ice4j/reference.conf at master · jitsi/ice4j · GitHub), but it seems to be unrelated to the JVB’s SINGLE_PORT_HARVESTER_PORT setting. I’ll look into it!

Rest assured, I’ve tested this with one JVB at a time first :slight_smile:

Only one of my servers is ARM, so I tried turning off its JVB and re-enabling sctp in the other JVBs, but it still doesn’t work. So, either I made a mistake in my configs, or websockets aren’t the problem after all.

TURN is only needed for P2P with certain kinds of restrictive firewall/NAT. In many cases the audio and video can flow directly, even with some corporate networks. So working P2P calls are not necessarily an indication that TURN is working. You can use your browser’s WebRTC debug view to see which candidate was selected, or look at the TURN server logs/stats.

1 Like

ZeroSSL is only needed for Windows servers IMO. With other kinds of server you can control the certification chain completely and Let’sEncrypt certificates should work. All that is needed is to add the right certiicates in the chain.
Second point; TURN use for clients who have their port 10000 blocked has nothing to do with P2P. It’s a very different use case. P2P can work correctly with coturn stopped.

Edit: did not notice that this was already answered.

1 Like

Quite possible. Did you ensure that your clients can connect directly to each JVB ? there are untold amount of posts describing how it’s done on this forum, hint: nc. This is absolutely needed if Coturn don’t work.

Aha, it was a TURN problem after all! Thanks for turning my attention to it (pun indended :wink: )

I tried moving coturn from Server 1, which is behind a NAT, to Server 2, which isn’t. This fixed the JVB running on Server 2, but broke the JVB running on Server 1.

This revealed that JVBs would only work when running on the same server that runs coturn.

The fix for this was to simply configure coturn to allow servers in the OpenVPN subnet to be peers:

denied-peer-ip=10.0.0.0-10.255.255.255
denied-peer-ip=172.16.0.0-172.31.255.255
denied-peer-ip=192.168.0.0-192.168.255.255

# NEW
allowed-peer-ip=10.8.0.1-10.8.0.x

Does this mean that JVBs act as peers of the TURN server?

In any case, I’m happy! Thanks again for all of the advice.

This means someone can relay traffic using your turnserver to that network.
Your turnserver config should look like jitsi-meet/turnserver.conf at master · jitsi/jitsi-meet · GitHub

If you have specified extern-ip setting, that forces the turnserver to contact the jvb on the internal address, if you drop that the turnserver will contact jvb on its public address on port 10000 as the normal clients and you don’t need the allowed-peer-ip=10.8.0.1-10.8.0.x.

Even when I put coturn on my non-NATted server and unset external-ip, the problem remains, unless I include the hosts in my OpenVPN subnet as allowed peers. Since the other servers are each behind a NAT (with the non-NATted server acting as their public access point), is this an acceptable usage of allowed-peer-ip?

A TURN server is basically an open proxy (to any allowed-peer-ip) for anyone who has the credentials. It’s OK as long as you’re aware that this internal network 10.8.0.0/24 is now open to anyone who can authenticate to the TURN server.

1 Like

Ok, that should be fine then, as the TURN server is configured with a shared secret. I’ve also limited the allowed IPs to be the individual hosts running the other JVBs, instead of the entire 10.8.0.0/24 subnet.