JVBs on Docker Swarm

Install & Config
,
#1

JVB on Docker Swarm

Hi!

I am setting up a jitsi meet server with docker. The main server (jm01) runs jicofo, jvb, prosody and meet-web with an nginx proxy. The configuration is working fine. I’ve deployed another vm with same specs to add another JVB to the server, jm02.

jm01 is Docker Swarm Master node 189.XXX.XXX.26

  • jitsi/jicofo:latest
    → none
  • jitsi/jvb:latest
    → 127.0.0.1:8080->8080/tcp, 0.0.0.0:10000->10000/udp, :::10000->10000/udp
  • jitsi-meet-prosody-1
    → 5222/tcp, 5280/tcp, 5347/tcp
  • jitsi-meet-web-1
    → 127.0.0.1:180->80/tcp, 127.0.0.1:1443

jm02 is Docker Swarm Worker node 189.XXX.XXX.27

  • alpine dummy
    → for the overlay network
  • jitsi-meet-jvb2-1
    → 127.0.0.1:8080->8080/tcp, 0.0.0.0:10001->10001/udp, :::10001->10001/udp

However jvb2 on jm02 seem not be able to connect and i don’t why. The error from the container log:
JVB 2022-03-21 16:20:48.280 WARNING: [22] [hostname=xmpp.meet.jitsi id=shard] MucClient.lambda$getConnectAndLoginCallable$7#622: [MucClient id=shard hostname=xmpp.meet.jitsi] error connecting

org.jivesoftware.smack.SmackException$EndpointConnectionException: The following addresses failed: 'RFC 6120 A/AAAA Endpoint + [xmpp.meet.jitsi:5222] (xmpp.meet.jitsi/10.0.1.26:5222)' failed because: java.net.SocketTimeoutException: connect timed out

I’ve tried a few things and got it working for a short amount of time, but after a reboot i am back at the same problem again, and need some help understanding.

The dummy-alpine on jm02 seem to reach the containers on jm01 just fine with the overlay network.

Here are the docker-compose files from each host:

jm01:

version: '3.5'

services:
    # Frontend
    web:
        image: jitsi/web:latest
        restart: ${RESTART_POLICY}
        ports:
            - '${HTTP_PORT}:80'
            - '${HTTPS_PORT}:443'
        volumes:
            - ${CONFIG}/web:/config:Z
            - ${CONFIG}/web/crontabs:/var/spool/cron/crontabs:Z
            - ${CONFIG}/transcripts:/usr/share/jitsi-meet/transcripts:Z
        environment:
            - AMPLITUDE_ID
            - ANALYTICS_SCRIPT_URLS
            - ANALYTICS_WHITELISTED_EVENTS
            - CALLSTATS_CUSTOM_SCRIPT_URL
            - CALLSTATS_ID
            - CALLSTATS_SECRET
            - CHROME_EXTENSION_BANNER_JSON
            - CONFCODE_URL
            - CONFIG_EXTERNAL_CONNECT
            - DEFAULT_LANGUAGE
            - DEPLOYMENTINFO_ENVIRONMENT
            - DEPLOYMENTINFO_ENVIRONMENT_TYPE
            - DEPLOYMENTINFO_REGION
            - DEPLOYMENTINFO_SHARD
            - DEPLOYMENTINFO_USERREGION
            - DESKTOP_SHARING_FRAMERATE_MIN
            - DESKTOP_SHARING_FRAMERATE_MAX
            - DIALIN_NUMBERS_URL
            - DIALOUT_AUTH_URL
            - DIALOUT_CODES_URL
            - DISABLE_AUDIO_LEVELS
            - DISABLE_DEEP_LINKING
            - DISABLE_GRANT_MODERATOR
            - DISABLE_HTTPS
            - DISABLE_KICKOUT
            - DISABLE_POLLS
            - DISABLE_REACTIONS
            - DROPBOX_APPKEY
            - DROPBOX_REDIRECT_URI
            - DYNAMIC_BRANDING_URL
            - ENABLE_AUDIO_PROCESSING
            - ENABLE_AUTH
            - ENABLE_BREAKOUT_ROOMS
            - ENABLE_CALENDAR
            - ENABLE_COLIBRI_WEBSOCKET
            - ENABLE_FILE_RECORDING_SERVICE
            - ENABLE_FILE_RECORDING_SERVICE_SHARING
            - ENABLE_FLOC
            - ENABLE_GUESTS
            - ENABLE_HSTS
            - ENABLE_HTTP_REDIRECT
            - ENABLE_IPV6
            - ENABLE_LETSENCRYPT
            - ENABLE_LIPSYNC
            - ENABLE_NO_AUDIO_DETECTION
            - ENABLE_NOISY_MIC_DETECTION
            - ENABLE_PREJOIN_PAGE
            - ENABLE_P2P
            - ENABLE_WELCOME_PAGE
            - ENABLE_CLOSE_PAGE
            - ENABLE_RECORDING
            - ENABLE_REMB
            - ENABLE_REQUIRE_DISPLAY_NAME
            - ENABLE_SIMULCAST
            - ENABLE_STATS_ID
            - ENABLE_STEREO
            - ENABLE_SUBDOMAINS
            - ENABLE_TALK_WHILE_MUTED
            - ENABLE_TCC
            - ENABLE_TRANSCRIPTIONS
            - ENABLE_XMPP_WEBSOCKET
            - ETHERPAD_PUBLIC_URL
            - ETHERPAD_URL_BASE
            - GOOGLE_ANALYTICS_ID
            - GOOGLE_API_APP_CLIENT_ID
            - HIDE_PREMEETING_BUTTONS
            - INVITE_SERVICE_URL
            - JICOFO_AUTH_USER
            - LETSENCRYPT_DOMAIN
            - LETSENCRYPT_EMAIL
            - LETSENCRYPT_USE_STAGING
            - MATOMO_ENDPOINT
            - MATOMO_SITE_ID
            - MICROSOFT_API_APP_CLIENT_ID
            - NGINX_RESOLVER
            - NGINX_WORKER_PROCESSES
            - NGINX_WORKER_CONNECTIONS
            - PEOPLE_SEARCH_URL
            - PUBLIC_URL
            - P2P_PREFERRED_CODEC
            - RESOLUTION
            - RESOLUTION_MIN
            - RESOLUTION_WIDTH
            - RESOLUTION_WIDTH_MIN
            - START_AUDIO_MUTED
            - START_AUDIO_ONLY
            - START_BITRATE
            - START_SILENT
            - START_WITH_AUDIO_MUTED
            - START_VIDEO_MUTED
            - START_WITH_VIDEO_MUTED
            - TESTING_CAP_SCREENSHARE_BITRATE
            - TESTING_OCTO_PROBABILITY
            - TOKEN_AUTH_URL
            - TOOLBAR_BUTTONS
            - TZ
            - VIDEOQUALITY_BITRATE_H264_LOW
            - VIDEOQUALITY_BITRATE_H264_STANDARD
            - VIDEOQUALITY_BITRATE_H264_HIGH
            - VIDEOQUALITY_BITRATE_VP8_LOW
            - VIDEOQUALITY_BITRATE_VP8_STANDARD
            - VIDEOQUALITY_BITRATE_VP8_HIGH
            - VIDEOQUALITY_BITRATE_VP9_LOW
            - VIDEOQUALITY_BITRATE_VP9_STANDARD
            - VIDEOQUALITY_BITRATE_VP9_HIGH
            - VIDEOQUALITY_ENFORCE_PREFERRED_CODEC
            - VIDEOQUALITY_PREFERRED_CODEC
            - XMPP_AUTH_DOMAIN
            - XMPP_BOSH_URL_BASE
            - XMPP_DOMAIN
            - XMPP_GUEST_DOMAIN
            - XMPP_MUC_DOMAIN
            - XMPP_RECORDER_DOMAIN
        networks:
            jitsi-meet:

    # XMPP server
    prosody:
        image: jitsi/prosody:latest
        restart: ${RESTART_POLICY}
        expose:
            - '5222'
            - '5347'
            - '5280'
        volumes:
            - ${CONFIG}/prosody/config:/config:Z
            - ${CONFIG}/prosody/prosody-plugins-custom:/prosody-plugins-custom:Z
        environment:
            - AUTH_TYPE
            - DISABLE_POLLS
            - ENABLE_AUTH
            - ENABLE_AV_MODERATION
            - ENABLE_BREAKOUT_ROOMS
            - ENABLE_GUESTS
            - ENABLE_LOBBY
            - ENABLE_XMPP_WEBSOCKET
            - GLOBAL_CONFIG
            - GLOBAL_MODULES
            - JIBRI_RECORDER_USER
            - JIBRI_RECORDER_PASSWORD
            - JIBRI_XMPP_USER
            - JIBRI_XMPP_PASSWORD
            - JICOFO_AUTH_USER
            - JICOFO_AUTH_PASSWORD
            - JICOFO_COMPONENT_SECRET
            - JIGASI_XMPP_USER
            - JIGASI_XMPP_PASSWORD
            - JVB_AUTH_USER
            - JVB_AUTH_PASSWORD
            - JWT_APP_ID
            - JWT_APP_SECRET
            - JWT_ACCEPTED_ISSUERS
            - JWT_ACCEPTED_AUDIENCES
            - JWT_ASAP_KEYSERVER
            - JWT_ALLOW_EMPTY
            - JWT_AUTH_TYPE
            - JWT_TOKEN_AUTH_MODULE
            - MATRIX_UVS_URL
            - MATRIX_UVS_ISSUER
            - MATRIX_UVS_AUTH_TOKEN
            - MATRIX_UVS_SYNC_POWER_LEVELS
            - LOG_LEVEL
            - LDAP_AUTH_METHOD
            - LDAP_BASE
            - LDAP_BINDDN
            - LDAP_BINDPW
            - LDAP_FILTER
            - LDAP_VERSION
            - LDAP_TLS_CIPHERS
            - LDAP_TLS_CHECK_PEER
            - LDAP_TLS_CACERT_FILE
            - LDAP_TLS_CACERT_DIR
            - LDAP_START_TLS
            - LDAP_URL
            - LDAP_USE_TLS
            - PUBLIC_URL
            - TURN_CREDENTIALS
            - TURN_HOST
            - TURNS_HOST
            - TURN_PORT
            - TURNS_PORT
            - TZ
            - XMPP_DOMAIN
            - XMPP_AUTH_DOMAIN
            - XMPP_GUEST_DOMAIN
            - XMPP_MUC_DOMAIN
            - XMPP_INTERNAL_MUC_DOMAIN
            - XMPP_MODULES
            - XMPP_MUC_MODULES
            - XMPP_INTERNAL_MUC_MODULES
            - XMPP_RECORDER_DOMAIN
            - XMPP_CROSS_DOMAIN
        networks:
            jitsi-meet:
                aliases:
                    - ${XMPP_SERVER}

    # Focus component
    jicofo:
        image: jitsi/jicofo:latest
        restart: ${RESTART_POLICY}
        volumes:
            - ${CONFIG}/jicofo:/config:Z
        environment:
            - AUTH_TYPE
            - BRIDGE_AVG_PARTICIPANT_STRESS
            - BRIDGE_STRESS_THRESHOLD
            - ENABLE_AUTH
            - ENABLE_AUTO_OWNER
            - ENABLE_CODEC_VP8
            - ENABLE_CODEC_VP9
            - ENABLE_CODEC_H264
            - ENABLE_OCTO
            - ENABLE_RECORDING
            - ENABLE_SCTP
            - ENABLE_AUTO_LOGIN
            - JICOFO_AUTH_USER
            - JICOFO_AUTH_PASSWORD
            - JICOFO_ENABLE_BRIDGE_HEALTH_CHECKS
            - JICOFO_CONF_INITIAL_PARTICIPANT_WAIT_TIMEOUT
            - JICOFO_CONF_SINGLE_PARTICIPANT_TIMEOUT
            - JICOFO_ENABLE_HEALTH_CHECKS
            - JICOFO_SHORT_ID
            - JICOFO_RESERVATION_ENABLED
            - JICOFO_RESERVATION_REST_BASE_URL
            - JIBRI_BREWERY_MUC
            - JIBRI_REQUEST_RETRIES
            - JIBRI_PENDING_TIMEOUT
            - JIGASI_BREWERY_MUC
            - JIGASI_SIP_URI
            - JVB_BREWERY_MUC
            - MAX_BRIDGE_PARTICIPANTS
            - OCTO_BRIDGE_SELECTION_STRATEGY
            - SENTRY_DSN="${JICOFO_SENTRY_DSN:-0}"
            - SENTRY_ENVIRONMENT
            - SENTRY_RELEASE
            - TZ
            - XMPP_DOMAIN
            - XMPP_AUTH_DOMAIN
            - XMPP_INTERNAL_MUC_DOMAIN
            - XMPP_MUC_DOMAIN
            - XMPP_RECORDER_DOMAIN
            - XMPP_SERVER
        depends_on:
            - prosody
        networks:
            jitsi-meet:

    # Video bridge
    jvb:
        image: jitsi/jvb:latest
        hostname: jvb
        restart: ${RESTART_POLICY}
        ports:
            - '${JVB_PORT}:${JVB_PORT}/udp'
            - '127.0.0.1:8080:8080'
        volumes:
            - ${CONFIG}/jvb:/config:Z
        environment:
            - DOCKER_HOST_ADDRESS
            - ENABLE_COLIBRI_WEBSOCKET
            - ENABLE_OCTO
            - JVB_AUTH_USER
            - JVB_AUTH_PASSWORD
            - JVB_BREWERY_MUC
            - JVB_PORT
            - JVB_MUC_NICKNAME
            - JVB_STUN_SERVERS
            - JVB_OCTO_BIND_ADDRESS
            - JVB_OCTO_PUBLIC_ADDRESS
            - JVB_OCTO_BIND_PORT
            - JVB_OCTO_REGION
            - JVB_WS_DOMAIN
            - JVB_WS_SERVER_ID
            - PUBLIC_URL
            - SENTRY_DSN="${JVB_SENTRY_DSN:-0}"
            - SENTRY_ENVIRONMENT
            - SENTRY_RELEASE
            - COLIBRI_REST_ENABLED
            - SHUTDOWN_REST_ENABLED
            - TZ
            - XMPP_AUTH_DOMAIN
            - XMPP_INTERNAL_MUC_DOMAIN
            - XMPP_SERVER
        depends_on:
            - prosody
        networks:
            jitsi-meet:

# Custom network so all services can communicate using a FQDN
networks:
    jitsi-meet:
      name: jitsi-meet
      external: true

jm02:

version: '3.5'

services:
    # Video bridge
    jvb:
        image: jitsi/jvb:latest
        hostname: jvb
        restart: ${RESTART_POLICY}
        ports:
            - '${JVB_PORT}:${JVB_PORT}/udp'
            - '127.0.0.1:8080:8080'
        volumes:
            - ${CONFIG}/jvb:/config:Z
        environment:
            - DOCKER_HOST_ADDRESS
            - ENABLE_COLIBRI_WEBSOCKET
            - ENABLE_OCTO
            - JVB_AUTH_USER
            - JVB_AUTH_PASSWORD
            - JVB_BREWERY_MUC
            - JVB_PORT
            - JVB_MUC_NICKNAME
            - JVB_STUN_SERVERS
            - JVB_OCTO_BIND_ADDRESS
            - JVB_OCTO_PUBLIC_ADDRESS
            - JVB_OCTO_BIND_PORT
            - JVB_OCTO_REGION
            - JVB_WS_DOMAIN
            - JVB_WS_SERVER_ID
            - PUBLIC_URL
            - SENTRY_DSN="${JVB_SENTRY_DSN:-0}"
            - SENTRY_ENVIRONMENT
            - SENTRY_RELEASE
            - COLIBRI_REST_ENABLED
            - SHUTDOWN_REST_ENABLED
            - TZ
            - XMPP_AUTH_DOMAIN
            - XMPP_INTERNAL_MUC_DOMAIN
            - XMPP_SERVER
        networks:
            jitsi-meet:

# Custom network so all services can communicate using a FQDN
networks:
    jitsi-meet:
      external: true
      name: jitsi-meet

Docker Config

root@jm01:/opt/docker/jitsi-meet# docker node ls
ID                            HOSTNAME       STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
66cxmnw8ne84j7t36fo4820ap *   jm01           Ready     Active         Leader           20.10.13
nyqabwdc4cs28kw6ztyezfuvq     jm02           Ready     Active                          20.10.13

root@jm01:/opt/docker/jitsi-meet# docker network list
NETWORK ID     NAME              DRIVER    SCOPE
f720d189b3e0   bridge            bridge    local
d8c0154aab07   docker_gwbridge   bridge    local
c19c89191b45   host              host      local
saiv9ha7qaj6   ingress           overlay   swarm
v3i344414g4m   jitsi-meet        overlay   swarm
a6d282477deb   none              null      local

root@jm01:/opt/docker/jitsi-meet# docker container ls
CONTAINER ID   IMAGE                                COMMAND                  CREATED        STATUS                  PORTS                                                                     NAMES
28339da2a2ff   grafana/grafana                      "/run.sh"                18 hours ago   Up 18 hours             0.0.0.0:3000->3000/tcp, :::3000->3000/tcp                                 grafana-grafana-1
f9fe5cb0be46   prom/prometheus                      "/bin/prometheus --c…"   18 hours ago   Up 18 hours             9090/tcp                                                                  grafana-prometheus-1
c56f4d4c91b0   prom/node-exporter:latest            "/bin/node_exporter …"   18 hours ago   Up 18 hours             9100/tcp                                                                  grafana-node-exporter-1
b127c3706271   gcr.io/cadvisor/cadvisor:latest      "/usr/bin/cadvisor -…"   18 hours ago   Up 18 hours (healthy)   8080/tcp                                                                  grafana-cadvisor-1
8a76bddaf606   goberle/jitsi-prom-exporter:latest   "exporter"               18 hours ago   Up 18 hours             8080/tcp                                                                  grafana-jitsi-exporter-1
50568c4666bf   jitsi/jicofo:latest                  "/init"                  18 hours ago   Up 18 hours                                                                                       jitsi-meet-jicofo-1
9255a64e1930   jitsi/jvb:latest                     "/init"                  18 hours ago   Up 18 hours             127.0.0.1:8080->8080/tcp, 0.0.0.0:10000->10000/udp, :::10000->10000/udp   jitsi-meet-jvb-1
23334913b617   jitsi/web:latest                     "/init"                  18 hours ago   Up 18 hours             127.0.0.1:180->80/tcp, 127.0.0.1:1443->443/tcp                            jitsi-meet-web-1
63b045157b86   jitsi/prosody:latest                 "/init"                  18 hours ago   Up 18 hours             5222/tcp, 5280/tcp, 5347/tcp                                              jitsi-meet-prosody-1

Can anyone help me figure out how to properly configure the setup?

I’m stuck and don’t know what to do at this point.

#2

You have changed the name so the default hostnames won’t work. Just leave it as meet.jitsi so container-name.meet.jitsi works.

1 Like
#3

Thank you for the quick answer.

I can’t leave it default. Docker Swarm dosen’t support dots(".") in the network name. In which configs do i have to change the default hostnames, or is that even possible?

#4

Yes, you can change all of the default values, look in env.sample. Now, how does a FQDN look like in Swarm?

#5

Looks like you can just go ahead an specify the hostname like so: docker-swarm files by prayagsingh · Pull Request #1134 · jitsi/docker-jitsi-meet · GitHub

1 Like
#6

Thank you for the link. I will test the Stack deployment as described and report back.

#7

Sadly adding the "hostname: xmpp.meet.jitsi" dosn’t seem to have any effect.
I’ve also tried to add the aliases section:

            aliases:
               - meet.jitsi
               - xmpp.meet.jitsi
               - auth.meet.jitsi
               - guest.meet.jitsi
               - muc.meet.jitsi
               - internal-muc.meet.jitsi
               - focus.meet.jitsi

but again no luck. I am a bit lost. I need to get this running with scalable JVBs. Any other idea to solve the problem?

#8

I’d suggest you try the PR and post your comments there.

#9

This seem like a totaly normal setup what i am trying to archive. And i know jitsi is capable of it. But i am messing something up here.

#10

Our setup does not support Swarm out of the box, so while totally normal, it’s not something we test so it might be missing some stuff.

#11

Well ok i didn’t know that. Is there an easier soultion to archive this setup. Both Servers are in the same network which is reachable by the outside world(only needed ports are exposed to the public).

#12

With more than one machine Swarm is the simplest way to go that I’m aware of.

I’d start with the PR I mentioned and check how it differs from your own files.

#13

Well i’ve came a bit closer to finding the error within the setup.

root@jm01:/opt/docker/jitsi-meet# docker network inspect jitsi-meet
[
...
            "9a649bfbea200602da9b74d8ad82c45923765dbd28e57c135a2acf97c6060138": {
                "Name": "jitsi-meet_prosody_1",
                "EndpointID": "d8f01d4825007aeee6332b8db36a83eb2a05b1fe382d7f411a0046c740fb0b2d",
                "MacAddress": "02:42:0a:00:01:0e",
                "IPv4Address": "10.0.1.14/24",
                "IPv6Address": ""
            },
...

As we can see the prosody container got the IP 10.0.1.14. When i check the logs on jvb2 on the jm02 host i can see the following message:

org.jivesoftware.smack.SmackException$EndpointConnectionException: The following addresses failed: 'RFC 6120 A/AAAA Endpoint + [xmpp.meet.jitsi:5222] (xmpp.meet.jitsi/10.0.1.14:5222)' failed because: java.net.SocketTimeoutException: connect timed out

So jvb2 is able to get the correct IP Adresse 10.0.1.14 but the connection timed out. A ping from my dummy container on jm02 is also working. Any Ideas?

Edit: Is there a way to tell prosody on which IPs it should listin on?

#14

The issue is fixed.

I’ve check back the whole configuration and performed a clean install. The problem still existet.
To get a better understanding of what is happening we’ve installed portainer and check there. The containers of the worker nodes were not visible.

We’ve found a workaround online to init the swarm with a different dataport.

So we removed our current swarm and recreate the swarm with

docker swarm init --data-path-port=9789

Joined our worker node as usual and volia containers are visible in portainer. And the jvb2 on jm02 connects just fine.

INFO: [29] BridgeSelector.addJvbAddress#95: Added new videobridge: Bridge[jid=jvbbrewery@internal-muc.meet.jitsi/jvb2, relayId=null, region=null, stress=0.00]

@saghul anyway, thank you for the help and clearning things up :slight_smile: Cheers!

#15

Happy to hear that!

1 Like