Docker/Traefik network issues

I’m trying to setup my jitsi stack behind traefik on a Ubuntu 20.04 host. I feel like I’m so close to getting this working but I have two problems:

  1. If I have more than 2 participants in a conference then everyone loses video
  2. My 3rd party app can’t connect (see below)

I think this is due to a misconfiguration in my networking and I’m hoping for some pointers!

I have a bit of a weird network topology, but I think this should be solvable:
Internet -> ISP Gateway -> Host -> Docker | Traefik -> Jitsi stack

Ports are forwarded on the gateway to the host. UFW has rules allowing the relevant ports (80, 443, 10000/udp).

In Traefik I have setup entrypoints for 80, 443 and 10000/udp

I have stood up the jitsi stack ok (can reach web), but connecting from my 3rd party app (Foundry VTT) on the same server I see this in the browser log:

Failed to load resource: the server responded with a status of 504 ()
15:23:45.009 Logger.js:154 2020-05-07T22:23:45.009Z [modules/xmpp/strophe.util.js] <Object.r.Strophe.log>:  Strophe: request id 2.1 error 504 happened

and in the web container logs:

2020/05/07 15:23:43 [error] 246#246: *3 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.18.0.2, server: _, request: "POST /http-bind HTTP/1.1", upstream: "http://172.19.0.2:5280/http-bind", host: "voice.my.domain", referrer: "https://vtt.my.domain/game"


172.18.0.2 - - [07/May/2020:15:23:43 -0700] "POST /http-bind HTTP/1.1" 504 590 "https://vtt.my.domain/game" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36 Edg/81.0.416.68"

The upstream piece tells me that the containers aren’t talking to each other/configured correctly?

Jitsi specific config:
env:

#
# Basic configuration options
#

# Directory where all configuration will be stored
CONFIG=~/.jitsi-meet-cfg

# Exposed HTTP port
HTTP_PORT=80

# Exposed HTTPS port
HTTPS_PORT=443

# System time zone
TZ=America/Vancouver

# Public URL for the web service
PUBLIC_URL=https://voice.my.domain

# IP address of the Docker host
# See the "Running behind NAT or on a LAN environment" section in the README
DOCKER_HOST_ADDRESS=192.168.1.69

my jitsi stack compose.yml:

version: '3'

services:
    # Frontend
    web:
        image: jitsi/web:latest
        restart: ${RESTART_POLICY}
        volumes:
            - ${CONFIG}/web:/config:Z
            - ${CONFIG}/transcripts:/usr/share/jitsi-meet/transcripts:Z
        environment:
            - ENABLE_AUTH
            - ENABLE_GUESTS
            - ENABLE_LETSENCRYPT
            - ENABLE_HTTP_REDIRECT
            - ENABLE_TRANSCRIPTIONS
            - DISABLE_HTTPS
            - JICOFO_AUTH_USER
            - LETSENCRYPT_DOMAIN
            - LETSENCRYPT_EMAIL
            - PUBLIC_URL
            - XMPP_DOMAIN
            - XMPP_AUTH_DOMAIN
            - XMPP_BOSH_URL_BASE
            - XMPP_GUEST_DOMAIN
            - XMPP_MUC_DOMAIN
            - XMPP_RECORDER_DOMAIN
            - ETHERPAD_URL_BASE
            - TZ
            - JIBRI_BREWERY_MUC
            - JIBRI_PENDING_TIMEOUT
            - JIBRI_XMPP_USER
            - JIBRI_XMPP_PASSWORD
            - JIBRI_RECORDER_USER
            - JIBRI_RECORDER_PASSWORD
            - ENABLE_RECORDING
        networks:
            proxy:
            meet.jitsi:
                aliases:
                    - ${XMPP_DOMAIN}
        labels:
            traefik.enable: 'true'
            traefik.docker.network: proxy
            traefik.http.routers.jitsi.entrypoints: http
            traefik.http.routers.jitsi.rule: 'Host(`voice.my.domain`)'
            traefik.http.routers.jitsi.middlewares: jitsi-https-redirect
            traefik.http.routers.jitsi-secure.entrypoints: https
            traefik.http.routers.jitsi-secure.rule: Host(`voice.my.domain`)
            traefik.http.routers.jitsi-secure.tls: 'true'
            traefik.http.routers.jitsi-secure.tls.certresolver: http
            traefik.http.routers.jitsi-secure.service: jitsi
            traefik.http.routers.jitsi-secure.middlewares: cors@docker
            traefik.http.services.jitsi.loadbalancer.server.port: 80
            traefik.http.middlewares.jitsi-https-redirect.redirectscheme.scheme: https

    # XMPP server
    prosody:
        image: jitsi/prosody:latest
        restart: ${RESTART_POLICY}
        expose:
            - '5222'
            - '5347'
            - '5280'
        volumes:
            - ${CONFIG}/prosody/config:/config:Z
            - ${CONFIG}/prosody/prosody-plugins-custom:/prosody-plugins-custom:Z
        environment:
            - AUTH_TYPE
            - ENABLE_AUTH
            - ENABLE_GUESTS
            - GLOBAL_MODULES
            - GLOBAL_CONFIG
            - LDAP_URL
            - LDAP_BASE
            - LDAP_BINDDN
            - LDAP_BINDPW
            - LDAP_FILTER
            - LDAP_AUTH_METHOD
            - LDAP_VERSION
            - LDAP_USE_TLS
            - LDAP_TLS_CIPHERS
            - LDAP_TLS_CHECK_PEER
            - LDAP_TLS_CACERT_FILE
            - LDAP_TLS_CACERT_DIR
            - LDAP_START_TLS
            - XMPP_DOMAIN
            - XMPP_AUTH_DOMAIN
            - XMPP_GUEST_DOMAIN
            - XMPP_MUC_DOMAIN
            - XMPP_INTERNAL_MUC_DOMAIN
            - XMPP_MODULES
            - XMPP_MUC_MODULES
            - XMPP_INTERNAL_MUC_MODULES
            - XMPP_RECORDER_DOMAIN
            - JICOFO_COMPONENT_SECRET
            - JICOFO_AUTH_USER
            - JICOFO_AUTH_PASSWORD
            - JVB_AUTH_USER
            - JVB_AUTH_PASSWORD
            - JIGASI_XMPP_USER
            - JIGASI_XMPP_PASSWORD
            - JIBRI_XMPP_USER
            - JIBRI_XMPP_PASSWORD
            - JIBRI_RECORDER_USER
            - JIBRI_RECORDER_PASSWORD
            - JWT_APP_ID
            - JWT_APP_SECRET
            - JWT_ACCEPTED_ISSUERS
            - JWT_ACCEPTED_AUDIENCES
            - JWT_ASAP_KEYSERVER
            - JWT_ALLOW_EMPTY
            - JWT_AUTH_TYPE
            - JWT_TOKEN_AUTH_MODULE
            - LOG_LEVEL
            - TZ
        networks:
            #proxy:
            meet.jitsi:
                aliases:
                    - ${XMPP_SERVER}

    # Focus component
    jicofo:
        image: jitsi/jicofo:latest
        restart: ${RESTART_POLICY}
        volumes:
            - ${CONFIG}/jicofo:/config:Z
        environment:
            - ENABLE_AUTH
            - XMPP_DOMAIN
            - XMPP_AUTH_DOMAIN
            - XMPP_INTERNAL_MUC_DOMAIN
            - XMPP_SERVER
            - JICOFO_COMPONENT_SECRET
            - JICOFO_AUTH_USER
            - JICOFO_AUTH_PASSWORD
            - JICOFO_RESERVATION_REST_BASE_URL
            - JVB_BREWERY_MUC
            - JIGASI_BREWERY_MUC
            - JIGASI_SIP_URI
            - JIBRI_BREWERY_MUC
            - JIBRI_PENDING_TIMEOUT
            - TZ
        depends_on:
            - prosody
        networks:
            #proxy:
            meet.jitsi:

    # Video bridge
    jvb:
        image: jitsi/jvb:latest
        #network_mode: host
        #ports:
            #- '${JVB_PORT}:${JVB_PORT}/udp'
            #- '${JVB_TCP_MAPPED_PORT}:${JVB_TCP_PORT}'
        expose:
            - '10000/udp'
            - '4443'
        restart: ${RESTART_POLICY}
        volumes:
            - ${CONFIG}/jvb:/config:Z
        environment:
            - DOCKER_HOST_ADDRESS
            - XMPP_AUTH_DOMAIN
            - XMPP_INTERNAL_MUC_DOMAIN
            - XMPP_SERVER
            - JVB_AUTH_USER
            - JVB_AUTH_PASSWORD
            - JVB_BREWERY_MUC
            - JVB_PORT
            - JVB_TCP_HARVESTER_DISABLED
            - JVB_TCP_PORT
            - JVB_STUN_SERVERS
            - JVB_ENABLE_APIS
            - TZ
        depends_on:
            - prosody
        networks:
            proxy:
            meet.jitsi:
        labels:
            traefik.enable: true
            traefik.docker.network: proxy
            traefik.udp.routers.jitsi-vb.service: jitsi-vb
            traefik.udp.routers.jitsi-vb.entrypoints: jitsi-video-bridge
            traefik.udp.services.jitsi-vb.loadbalancer.server.port: 10000

# Custom network so all services can communicate using a FQDN
networks:
    meet.jitsi:
    proxy:
        external: true

The documentation is quite confusing about this, but did you try to set the DOCKER_HOST_ADDRESS pointing to your external public ip - instead of the internal ip address?

In various attempts to fix this I have done that. I can try it again.

Right now I’m leaning towards two possibilities:

  1. the user-defined? meet.jitsi network is not behaving correctly alongside my default proxy network (therefore the containers aren’t talking/jvb gets the wrong hostname internally?)
  2. my cors header middleware in traefik is breaking the headers
- "traefik.http.middlewares.cors.headers.accesscontrolallowmethods=*"
      - "traefik.http.middlewares.cors.headers.accesscontrolalloworigin=*"
      - "traefik.http.middlewares.cors.headers.accesscontrolmaxage=100"
      - "traefik.http.middlewares.cors.headers.addvaryheader=true"
      - "traefik.http.middlewares.cors.headers.accesscontrolallowcredentials=true"
      - "traefik.http.middlewares.cors.headers.accesscontrolallowheaders=Access-Control-Allow-Headers, Origin,Accept, X-Requested-With, Content-Type, Access-Control-Request-Method, Access-Control-Request-Headers"

@Franky1 no dice I’m afraid. I’m basically at the point where I’m going to pull jitsi from behind of traefik and just let it interface with the host directly. Kind of defeats the purpose but I’m at a loss.

Hi, did you ever resolve this? I am facing the same issue