2 user P2P works, but with three users all AV is disabled

Hi,
I tried to set up jitsi meet using the docker compose template from https://github.com/jitsi/docker-jitsi-meet . Since I want to run it with docker swarm on a public server, I moved all env values into the docker-compose.yml
So far, the stack deploys fine. Webinterface works, chat works, even to clients in the same LAN can (sometimes?!) use video.
However, when a third client connects from another network (cellular network) all AV gets disabled.

What I don’t understand is: What tells the browser where the JVB is and when/where to send AV to it?

Using wireshark, I can confirm that my browser never tries to send a single byte to UDP port 10000 or TCP port 4443, so I’m not surprised that nothing works.

Within the stack, all components are assigned to the same overlay network and all containers run on the same node.

I don’t think it’s a firewall issue between client and server, since packets that aren’t sent cannot be blocked. Ports 4443 TCP and 10000 UDP are open on the server.

I already found out that using XMPP_BOSH_URL_BASE: http://prosody:5280 works fine, while using XMPP_BOSH_URL_BASE: http://xmpp.meet.${DOMAIN}:5280 does not - it throws “cannot connect UPSTREAM” errors on “web” and meeting won’t start, despite “prosody” also has the network alias name xmpp.meet.${DOMAIN} attached. “web” tries to connect to the XMPP service via its external IP resolved from xmpp.meet.${DOMAIN} (*.${DOMAIN} resolves to the same IP), which fails, because port 5280 is not exposed to the public. If I expose 5280 on “prosody”, this error goes away.

For now, I use XMPP_BOSH_URL_BASE: http://prosody:5280.

For “jicofo” and “jvb” using XMPP_SERVER: xmpp.meet.${DOMAIN} or XMPP_SERVER: prosody doesn’t seem to make any difference…

I tried setting DOCKER_HOST_ADDRESS: jvb.meet.${DOMAIN} but that didn’t change anything.

How do I debug this?

The channels the jicofo opens in jvb has the data, and jicofo sends that to clients. So basically the ip address and port are coming from jvb, through jicofo to clients. If you open chrome://webrtc-internals and find the peer connection to the bridge looking at the setRemoteDescription you will see what is announced.

Well… there is no setRemoteDescription.

When I open chrome://webrtc-internals I have one Tab named GetUserMedia Requests. When a 2nd user joins, I get one additional tab named by the URL of my jitsi server.
On this tab there is a setLocalDescription but no setRemoteDescription… The event list on the left ends with icegatheringstatechange: complete 40s after the last icecandidate (srflx) (timeout?).
All stats there show that 0 bytes are sent.

On a working jitsi setup, I get two tabs with the URL (guess one incoming one outgoing) and a lot more events after icecandidate (host) including setRemoteDescription on each of them.

These are 3 tabs in chrome:


This is p2p peer connection:

This is jvb peer connection:

This is the setRemoteDescription and the jvb candidates:

Hmm, that’s all I get when a 2nd client is connected:

This is the jicofo log when 2nd client joined:

Jicofo 2020-03-22 17:57:01.692 INFO: [46] org.jitsi.jicofo.xmpp.FocusComponent.handleConferenceIq().401 Focus request for room: xx5@muc.meet.*domain*
Jicofo 2020-03-22 17:57:01.919 INFO: [36] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Chat room event ChatRoomMemberPresenceChangeEvent[type=MemberJoined sourceRoom=org.jitsi.impl.protocol.xmpp.ChatRoomImpl@59f051aa member=ChatMember[xx5@muc.meet.*domain*/98140321, jid: null]@2024065939]
Jicofo 2020-03-22 17:57:01.922 INFO: [36] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Member xx5@muc.meet.*domain*/98140321 joined.
Jicofo 2020-03-22 17:57:01.925 INFO: [36] org.jitsi.impl.protocol.xmpp.colibri.OperationSetColibriConferenceImpl.log() Conference created: org.jitsi.impl.protocol.xmpp.colibri.ColibriConferenceImpl@2302a82c
Jicofo 2020-03-22 17:57:01.931 INFO: [36] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Added participant jid= xx5@muc.meet.*domain*/3ccdb265, bridge=jvb@auth.meet.*domain*/2zQH7qQj
Jicofo 2020-03-22 17:57:01.931 INFO: [36] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=ffe6b1 octo_enabled= false: [[null, null]]
Jicofo 2020-03-22 17:57:01.933 INFO: [36] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Added participant jid= xx5@muc.meet.*domain*/98140321, bridge=jvb@auth.meet.*domain*/2zQH7qQj
Jicofo 2020-03-22 17:57:01.933 INFO: [36] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=ffe6b1 octo_enabled= false: [[null, null, null]]
Jicofo 2020-03-22 17:57:02.518 INFO: [94] org.jitsi.jicofo.AbstractChannelAllocator.log() Using jvb@auth.meet.*domain*/2zQH7qQj to allocate channels for: Participant[endpointId=3ccdb265]
Jicofo 2020-03-22 17:57:02.520 INFO: [43] org.jitsi.jicofo.Bridge.log() Adding 1 video streams on jvb@auth.meet.*domain*/2zQH7qQj video streams: 0 diff: 1 (estimated: 1)
Jicofo 2020-03-22 17:57:03.348 INFO: [54] org.jitsi.jicofo.AbstractChannelAllocator.log() Using jvb@auth.meet.*domain*/2zQH7qQj to allocate channels for: Participant[endpointId=98140321]
Jicofo 2020-03-22 17:57:03.350 INFO: [43] org.jitsi.jicofo.Bridge.log() Adding 3 video streams on jvb@auth.meet.*domain*/2zQH7qQj video streams: 0 diff: 4 (estimated: 4)
Jicofo 2020-03-22 17:57:04.052 WARNING: [36] org.jitsi.protocol.xmpp.AbstractOperationSetJingle.processJingleIQ().368 unsupported action session-terminate
Jicofo 2020-03-22 17:57:05.575 INFO: [36] org.jitsi.jicofo.Bridge.log() Reset video stream diff on jvb@auth.meet.*domain*/2zQH7qQj video streams: 0 (estimation error: 4)
Jicofo 2020-03-22 17:57:10.580 INFO: [36] org.jitsi.jicofo.Bridge.log() Video stream count for: jvb@auth.meet.*domain*/2zQH7qQj: 4
Jicofo 2020-03-22 17:57:17.556 SEVERE: [94] org.jitsi.protocol.xmpp.AbstractOperationSetJingle.wasInviteAccepted().243 Timeout waiting for RESULT response to 'session-initiate' request from xx5@muc.meet.*domain*/3ccdb265
Jicofo 2020-03-22 17:57:17.557 INFO: [94] org.jitsi.jicofo.ParticipantChannelAllocator.log() Expiring xx5@muc.meet.*domain*/3ccdb265 channels - no RESULT for session-initiate
Jicofo 2020-03-22 17:57:17.560 WARNING: [94] org.jitsi.jicofo.AbstractParticipant.log() Canceling ParticipantChannelAllocator[BridgeSession[id=ffe6b1_e637f3, bridge=Bridge[jid=jvb@auth.meet.*domain*/2zQH7qQj, relayId=null, region=null]]@50104514, Participant[endpointId=3ccdb265]]@846064878
Jicofo 2020-03-22 17:57:17.561 INFO: [94] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=ffe6b1 octo_enabled= false: [[null, null]]
Jicofo 2020-03-22 17:57:17.561 INFO: [94] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Expiring channels for: xx5@muc.meet.*domain*/3ccdb265 on: Bridge[jid=jvb@auth.meet.*domain*/2zQH7qQj, relayId=null, region=null]
Jicofo 2020-03-22 17:57:17.570 INFO: [43] org.jitsi.jicofo.Bridge.log() Removing 3 video streams on jvb@auth.meet.*domain*/2zQH7qQj video streams: 4 diff: -3 (estimated: 1)
Jicofo 2020-03-22 17:57:17.572 INFO: [94] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Removed participant: true, xx5@muc.meet.*domain*/3ccdb265
Jicofo 2020-03-22 17:57:18.365 SEVERE: [54] org.jitsi.protocol.xmpp.AbstractOperationSetJingle.wasInviteAccepted().243 Timeout waiting for RESULT response to 'session-initiate' request from xx5@muc.meet.*domain*/98140321
Jicofo 2020-03-22 17:57:18.366 INFO: [54] org.jitsi.jicofo.ParticipantChannelAllocator.log() Expiring xx5@muc.meet.*domain*/98140321 channels - no RESULT for session-initiate
Jicofo 2020-03-22 17:57:18.367 WARNING: [54] org.jitsi.jicofo.AbstractParticipant.log() Canceling ParticipantChannelAllocator[BridgeSession[id=ffe6b1_e637f3, bridge=Bridge[jid=jvb@auth.meet.*domain*/2zQH7qQj, relayId=null, region=null]]@50104514, Participant[endpointId=98140321]]@447399530
Jicofo 2020-03-22 17:57:18.367 INFO: [54] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=ffe6b1 octo_enabled= false: [[null]]
Jicofo 2020-03-22 17:57:18.368 INFO: [54] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Expiring channels for: xx5@muc.meet.*domain*/98140321 on: Bridge[jid=jvb@auth.meet.*domain*/2zQH7qQj, relayId=null, region=null]
Jicofo 2020-03-22 17:57:18.375 INFO: [54] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Removed participant: true, xx5@muc.meet.*domain*/98140321
Jicofo 2020-03-22 17:57:18.380 INFO: [43] org.jitsi.jicofo.Bridge.log() Removing 1 video streams on jvb@auth.meet.*domain*/2zQH7qQj video streams: 4 diff: -4 (estimated: 0)
Jicofo 2020-03-22 17:57:20.592 INFO: [36] org.jitsi.jicofo.Bridge.log() Reset video stream diff on jvb@auth.meet.*domain*/2zQH7qQj video streams: 4 (estimation error: -4)

This is Chromes console when a 2nd client joined (different try):
screenhots removed - can only post one

I’m still unable to figure out where to start debugging…

What is your config.js? You should have a config error or something…why would your client reject the focus requests…

I don’t have one. My understanding is that the config is created from ENV when the docker container starts.

Thats my complete jitsiweb configuration within the stack:

jitsiweb:
    image: jitsi/web
    deploy:
        replicas: 1
        placement:
            constraints:
                - node.role == manager
        labels:
            - traefik.enable=true
            - traefik.http.services.jitsiweb.loadbalancer.server.port=80
            - traefik.http.routers.jitsiweb_https.rule=Host(`meet.${DOMAIN}`)
            - traefik.http.routers.jitsiweb_https.service=jitsiweb
            - traefik.http.routers.jitsiweb_https.tls.certresolver=leResolver
            - traefik.http.routers.jitsiweb_https.entrypoints=https
    environment:
        ENABLE_AUTH: 0
        DISABLE_HTTPS: 1
        JICOFO_AUTH_USER: focus
        PUBLIC_URL: "https://meet.${DOMAIN}"
        XMPP_DOMAIN: meet.${DOMAIN}
        XMPP_AUTH_DOMAIN: auth.meet.${DOMAIN}
        XMPP_BOSH_URL_BASE: http://prosody:5280
        XMPP_GUEST_DOMAIN: guest.meet.${DOMAIN}
        XMPP_MUC_DOMAIN: muc.meet.${DOMAIN}
        XMPP_RECORDER_DOMAIN: recorder.meet.${DOMAIN}
        TZ: Europe/Amsterdam
        JIBRI_BREWERY_MUC: jibribrewery
        JIBRI_XMPP_USER: jibri
        JIBRI_XMPP_PASSWORD: $PASSWORD
        JIBRI_RECORDER_USER: recorder
        JIBRI_RECORDER_PASSWORD: $PASSWORD
        ENABLE_RECORDING: 0
    networks:
        traefik-public:
            aliases:
                - meet.${DOMAIN}

I removed the volumes: section, so all config should be generated from scratch every time the container starts.
I disabled https and did not expose port 80/443 because Traefik is handling https.

I’ve attached to the running container and extracted the config.js, it looks like it was generated properly (comments removed):

var config = {
    hosts: {
        domain: 'meet.*****',
        muc: 'muc.meet.*****',
    },
    bosh: '/http-bind',
    clientNode: 'http://jitsi.org/jitsimeet',
    focusUserJid: 'focus@auth.meet.*****',
    testing: {
        enableFirefoxSimulcast: false,
        p2pTestMode: false
    },
    disableSuspendVideo: true,
    desktopSharingChromeExtId: null,
    desktopSharingChromeSources: [ 'screen', 'window', 'tab' ],
    desktopSharingChromeMinExtVersion: '0.1',
    channelLastN: -1,
    enableWelcomePage: true,
    enableUserRolesBasedOnToken: false,
    p2p: {
        enabled: true,
        stunServers: [
            { urls: 'stun:stun.l.google.com:19302' },
            { urls: 'stun:stun1.l.google.com:19302' },
            { urls: 'stun:stun2.l.google.com:19302' }
        ],
        preferH264: true
    },
    analytics: {
    },
    deploymentInfo: {
    }
};

@saghul any idea?

如果用域名,请核对域名证书文件的正确性。

Google Translate says:

If using a domain name, check the correctness of the domain name certificate file.

https:// for the domain works fine. I use Lets Encrypt which is handled by traefik. All incoming https:// on the domain meet.$DOMAIN is proxied through traefik as http:// to the service “jitsiweb” on port 80. The webinterface works fine. I don’t think there is an issue… I don’t need certificates on the other docker containers, right?

Iss your jitsiweb container part of the meet.jitsi network?

No, all containers are part of the network traefik-public in my case. Using networks with “.” in the name throws an error on a Docker swarm.
I’m pretty sure at least jitsiweb, jicofo and prosody can communicate, since jicofos logs show when somebody connects.

This is the complete stack file I’m trying at the moment:

version: '3'

services:
    # Frontend
    jitsiweb:
        image: jitsi/web
        deploy:
            replicas: 1
            placement:
                constraints:
                    - node.role == manager
            labels:
                - traefik.enable=true
                - traefik.http.services.jitsiweb.loadbalancer.server.port=80
                - traefik.http.routers.jitsiweb_https.rule=Host(`meet.${DOMAIN}`)
                - traefik.http.routers.jitsiweb_https.service=jitsiweb
                - traefik.http.routers.jitsiweb_https.tls.certresolver=leResolver
                - traefik.http.routers.jitsiweb_https.entrypoints=https
        environment:
            ENABLE_AUTH: 0
            DISABLE_HTTPS: 1
            JICOFO_AUTH_USER: focus
            JICOFO_AUTH_PASSWORD: $PASSWORD
            PUBLIC_URL: "https://meet.${DOMAIN}"
            XMPP_DOMAIN: meet.${DOMAIN}
            XMPP_AUTH_DOMAIN: auth.meet.${DOMAIN}
            XMPP_BOSH_URL_BASE: http://prosody:5280
            XMPP_GUEST_DOMAIN: guest.meet.${DOMAIN}
            XMPP_MUC_DOMAIN: muc.meet.${DOMAIN}
            XMPP_RECORDER_DOMAIN: recorder.meet.${DOMAIN}
            TZ: Europe/Amsterdam
            JIBRI_BREWERY_MUC: jibribrewery
            JIBRI_XMPP_USER: jibri
            JIBRI_XMPP_PASSWORD: $PASSWORD
            JIBRI_RECORDER_USER: recorder
            JIBRI_RECORDER_PASSWORD: $PASSWORD
            ENABLE_RECORDING: 0
        networks:
            traefik-public:
                aliases:
                    - meet.${DOMAIN}

    # XMPP server
    prosody:
        image: jitsi/prosody
        ports: 
            - '5280:5280'
        deploy:
            replicas: 1
            placement:
                constraints:
                    - node.role == manager
        environment:
            ENABLE_AUTH: 0
            TZ: Europe/Amsterdam
            XMPP_DOMAIN: meet.${DOMAIN}
            XMPP_AUTH_DOMAIN: auth.meet.${DOMAIN}
            XMPP_GUEST_DOMAIN: guest.meet.${DOMAIN}
            XMPP_MUC_DOMAIN: muc.meet.${DOMAIN}
            XMPP_INTERNAL_MUC_DOMAIN: internal-muc.meet.${DOMAIN}
            XMPP_RECORDER_DOMAIN: recorder.meet.${DOMAIN}
            JICOFO_COMPONENT_SECRET: MyCicofoSecretValue
            JICOFO_AUTH_USER: focus
            JICOFO_AUTH_PASSWORD: $PASSWORD
            JVB_AUTH_USER: jvb
            JVB_AUTH_PASSWORD: $PASSWORD
            JIBRI_XMPP_USER: jibri
            JIBRI_XMPP_PASSWORD: $PASSWORD
            JIBRI_RECORDER_USER: recorder
            JIBRI_RECORDER_PASSWORD: $PASSWORD
        networks:
            traefik-public:
                aliases: 
                    - xmpp.meet.${DOMAIN}

    # Focus component
    jicofo:
        image: jitsi/jicofo
        deploy:
            replicas: 1
            placement:
                constraints:
                    - node.role == manager
        environment:
            ENABLE_AUTH: 0
            XMPP_DOMAIN: meet.${DOMAIN}
            XMPP_AUTH_DOMAIN: auth.meet.${DOMAIN}
            XMPP_INTERNAL_MUC_DOMAIN: internal-muc.meet.${DOMAIN}
            XMPP_SERVER: xmpp.meet.${DOMAIN}
            JICOFO_COMPONENT_SECRET: MyCicofoSecretValue
            JICOFO_AUTH_USER: focus
            JICOFO_AUTH_PASSWORD: $PASSWORD
            JVB_BREWERY_MUC: jvbbrewery
            JIGASI_BREWERY_MUC: jigasibrewery
            JIBRI_BREWERY_MUC: jibribrewery
            TZ: Europe/Amsterdam
        depends_on:
            - prosody
        networks:
            traefik-public:

    # Video bridge
    jvb:
        image: jitsi/jvb
        ports:
            - '10000:10000/udp'
            - '4443:4443'
        deploy:
            replicas: 1
            placement:
                constraints:
                    - node.role == manager
        environment:
            XMPP_AUTH_DOMAIN: auth.meet.${DOMAIN}
            XMPP_INTERNAL_MUC_DOMAIN: internal-muc.meet.${DOMAIN}
            XMPP_SERVER: xmpp.meet.${DOMAIN}
            JVB_AUTH_USER: jvb
            JVB_AUTH_PASSWORD: $PASSWORD
            JVB_BREWERY_MUC: jvbbrewery
            JVB_PORT: 10000
            JVB_TCP_PORT: 4443
            JVB_STUN_SERVERS: stun.l.google.com:19302,stun1.l.google.com:19302,stun2.l.google.com:19302
            JVB_ENABLE_APIS: rest,colibri
            TZ: Europe/Amsterdam
            JVB_TCP_HARVESTER_DISABLED: 'true'
        depends_on:
            - prosody
        networks:
            traefik-public:

# Custom network so all services can communicate using a FQDN
networks:
    traefik-public:
        external: true