[SOLVED, WORKING] Docker OCTO - 5870

Hi Guys, the vanilla OCTO installation was successful, up and running. After that I wanted to achieve docker OCTO and this is the scenario. Please note - All everything happening below is done on a single VM

Version - 5870 - Release stable-5870 · jitsi/docker-jitsi-meet · GitHub

My virtual machine is One-to-one NAT’ed with the Public IP and all ports are forwarded as expected.

Looking at the environment variables listed in JVB service in the docker-compose, I have added some missing environment variables in .env file which can be seen below

# OCTO SETTINGS

TESTING_OCTO_PROBABILITY=1
ENABLE_OCTO=true
OCTO_BRIDGE_SELECTION_STRATEGY=SplitBridgeSelectionStrategy
JVB_OCTO_BIND_ADDRESS=0.0.0.0
JVB_OCTO_PUBLIC_ADDRESS=<PUBLIC-IP>
JVB_OCTO_BIND_PORT=4096
JVB_OCTO_REGION=Region1

In the same folder, I have the main docker-compose.yaml file for the JMS Stack which looks similar to the github repo.

    # Video bridge
    jvb:
        image: jitsi/jvb:stable-5870
        restart: ${RESTART_POLICY}
        ports:
            - '${JVB_PORT}:${JVB_PORT}/udp'
            - '${JVB_TCP_PORT}:${JVB_TCP_PORT}'
        volumes:
            - ${CONFIG}/jvb:/config:Z
        environment:
            - ENABLE_COLIBRI_WEBSOCKET
            - ENABLE_OCTO
            - DOCKER_HOST_ADDRESS
            - XMPP_AUTH_DOMAIN
            - XMPP_INTERNAL_MUC_DOMAIN
            - XMPP_SERVER
            - JVB_AUTH_USER
            - JVB_AUTH_PASSWORD
            - JVB_BREWERY_MUC
            - JVB_PORT
            - JVB_TCP_HARVESTER_DISABLED
            - JVB_TCP_PORT
            - JVB_TCP_MAPPED_PORT
            - JVB_STUN_SERVERS
            - JVB_ENABLE_APIS
            - JVB_WS_DOMAIN
            - JVB_WS_SERVER_ID
            - PUBLIC_URL
            - JVB_OCTO_BIND_ADDRESS
            - JVB_OCTO_PUBLIC_ADDRESS
            - JVB_OCTO_BIND_PORT
            - JVB_OCTO_REGION
            - TZ
        depends_on:
            - prosody
        networks:
            meet.jitsi:
                aliases:
                    - jvb.meet.jitsi

For convenience, I have added a second jvb.yaml file besides .env which looks something like this

version: '3'

services:

    # Video bridge
    jvb2:
        container_name: jvb2
        image: jitsi/jvb:stable-5870
        restart: ${RESTART_POLICY}
        ports:
            - '10001:10001/udp'
            #- '${JVB_TCP_PORT}:${JVB_TCP_PORT}'
        volumes:
            - ${CONFIG}/jvb2:/config:Z
        environment:
            - ENABLE_COLIBRI_WEBSOCKET
            - ENABLE_OCTO
            - DOCKER_HOST_ADDRESS
            - XMPP_AUTH_DOMAIN
            - XMPP_INTERNAL_MUC_DOMAIN
            - XMPP_SERVER
            - JVB_AUTH_USER
            - JVB_AUTH_PASSWORD
            - JVB_BREWERY_MUC
            - JVB_PORT=10001
            - JVB_TCP_HARVESTER_DISABLED
            - JVB_TCP_PORT
            - JVB_TCP_MAPPED_PORT
            - JVB_STUN_SERVERS
            - JVB_ENABLE_APIS
            - JVB_WS_DOMAIN
            - JVB_WS_SERVER_ID
            - PUBLIC_URL
            - JVB_OCTO_BIND_ADDRESS
            - JVB_OCTO_PUBLIC_ADDRESS
            - JVB_OCTO_BIND_PORT
            - JVB_OCTO_REGION=Region2
            - TZ
        networks:
            meet.jitsi:
                aliases:
                    - jvb2.meet.jitsi

# Custom network so all services can communicate using a FQDN
networks:
    meet.jitsi:

Changes in jvb2.yaml

  1. Service, Container Name = jvb2
  2. Changed UDP port to 10001 / udp
  3. Commented - '${JVB_TCP_PORT}:${JVB_TCP_PORT}'
  4. Changed config directory
  5. In environment variables -
    a. Changed - JVB_PORT=10001
    b. Changed - JVB_OCTO_REGION=Region2
  6. Changed aliases to - jvb2.meet.jitsi

All this can be seen in the file **jvb2.yaml** above, and I up this docker-compose

At this stage, from prosody logs, I can see that the new JVB connects to prosody

info       Client connected
info       Authenticated as jvb@auth.meet.jitsi

The container (jvb2) is UP and running without the jvb-tcp port because commented in jvb2.yaml

97c697318dce   jitsi/jvb:stable-5870                             "/init"                  About a minute ago   Up About a minute   0.0.0.0:10001->10001/udp, :::10001->10001/udp                                              jvb2
00aa119d2079   jitsi/jvb:stable-5870                             "/init"                  7 minutes ago        Up 7 minutes        0.0.0.0:4443->4443/tcp, :::4443->4443/tcp, 0.0.0.0:10000->10000/udp, :::10000->10000/udp   jvb

Disabled p2p from config.js

Good thing -

All the participants that join the meeting are split on JVB1 and JVB2 round-robin way

Smalll Issue

Participants placed on JVB1 cannot see participants placed on JVB2

Web Screenshot

App Screenshot

Surprisingly, I don’t see port 4096 exposed internally when I do docker ps for both the jvb’s

Firewall doesn’t seem to be an issue

nc -z -u -v <public IP> 10000
Connection to <public IP> 10000 port [udp/*] succeeded!

nc -z -u -v <public IP> 10001
Connection to <public IP> 10001 port [udp/*] succeeded!

Inside both the JVB’s

I don’t see JVB_OCTO_BIND_PORT in the ports section of Docker compose so it looks like it’s being exported maybe?

Hi @saghul

Thanks for your reply :slight_smile:

I tried adding a line

expose:
    - 4096/udp

in both the jvb’s docker-compose.yaml

The port was successfully exposed for both the containers, inside the internal docker network. But still the same issue.

Added octo expose 4096

No idea then, sorry.

Issue sorted.

One Machine, multiple JVB’s, participants split in different JVB’s <3

What was the problem? Anything we can improve in the documentation / Docker configuration?

About docker, yes.

My env file for OCTO settings looks like this

TESTING_OCTO_PROBABILITY=1
ENABLE_OCTO=true
OCTO_BRIDGE_SELECTION_STRATEGY=SplitBridgeSelectionStrategy
JVB_OCTO_BIND_ADDRESS=jvb1
#JVB_OCTO_PUBLIC_ADDRESS=<public ip>
JVB_OCTO_PUBLIC_ADDRESS=jvb1
JVB_OCTO_BIND_PORT=4096
JVB_OCTO_REGION=Region1

Since the docker-compose comes with a jvb service, my default docker-compose looks like this

jvb1:
        container_name: jvb1
        image: jitsi/jvb:stable-5870
        restart: ${RESTART_POLICY}
        expose:
            - '4096/udp'
        ports:
            - '${JVB_PORT}:${JVB_PORT}/udp'
            - '${JVB_TCP_PORT}:${JVB_TCP_PORT}'
        volumes:
            - ${CONFIG}/jvb1:/config:Z
        environment:
            - ENABLE_COLIBRI_WEBSOCKET
            - ENABLE_OCTO
            - DOCKER_HOST_ADDRESS
            - XMPP_AUTH_DOMAIN
            - XMPP_INTERNAL_MUC_DOMAIN
            - XMPP_SERVER
            - JVB_AUTH_USER
            - JVB_AUTH_PASSWORD
            - JVB_BREWERY_MUC
            - JVB_PORT
            - JVB_TCP_HARVESTER_DISABLED
            - JVB_TCP_PORT
            - JVB_TCP_MAPPED_PORT
            - JVB_STUN_SERVERS
            - JVB_ENABLE_APIS
            - JVB_WS_DOMAIN
            - JVB_WS_SERVER_ID
            - PUBLIC_URL
            - JVB_OCTO_BIND_ADDRESS
            - JVB_OCTO_PUBLIC_ADDRESS
            - JVB_OCTO_BIND_PORT
            - JVB_OCTO_REGION
            - TZ
        depends_on:
            - prosody
        networks:
            meet.jitsi:
                aliases:
                    - jvb.meet.jitsi

My second jvb.yaml looks like this

jvb2:
        container_name: jvb2
        image: jitsi/jvb:stable-5870
        restart: ${RESTART_POLICY}
        expose:
            - '4097/udp'
        ports:
            - '10002:10002/udp'
            #- '${JVB_TCP_PORT}:${JVB_TCP_PORT}'
        volumes:
            - ${CONFIG}/jvb2:/config:Z
        environment:
            - ENABLE_COLIBRI_WEBSOCKET
            - ENABLE_OCTO
            - DOCKER_HOST_ADDRESS
            - XMPP_AUTH_DOMAIN
            - XMPP_INTERNAL_MUC_DOMAIN
            - XMPP_SERVER
            - JVB_AUTH_USER
            - JVB_AUTH_PASSWORD
            - JVB_BREWERY_MUC
            - JVB_PORT=10002
            - JVB_TCP_HARVESTER_DISABLED
            - JVB_TCP_PORT
            - JVB_TCP_MAPPED_PORT
            - JVB_STUN_SERVERS
            - JVB_ENABLE_APIS
            - JVB_WS_DOMAIN
            - JVB_WS_SERVER_ID
            - PUBLIC_URL
            - JVB_OCTO_BIND_ADDRESS=jvb2
            - JVB_OCTO_PUBLIC_ADDRESS=jvb2
            - JVB_OCTO_BIND_PORT=4097
            - JVB_OCTO_REGION=Region2
            - TZ
        networks:
            meet.jitsi:
                aliases:
                    - jvb2.meet.jitsi

and the values change as you go on adding jvb’s. Now this I did for a single machine, which was one-to-one NAT’ed

@saghul - Just wanted to know, now that I have 5 jvb’s running in the same machine and all the participants are getting splitted in Round robin between bridges, what is the max participant capacity that I can reach ?

My targets are 500+.

It’s hard to tell because it depends on the machine of course. Generally we a single smaller machine fully dedicated to be a JVB. TBH I’m not sure what would perform better: 32 cores + 32 GB of RAM vs 4 x (8 cores + 8GB of RAM) for example.