Octo with 3 regions

Install & Config
#4

Hi Jerry,

As Damencho mentioned we’ve been using Octo with more than 2 regions and haven’t seen any problems so far.

Unfortunately I don’t see anything obvious that would pinpoint the issue. One difference I notice: you seem to be using XMPP components for the bridges, while we use XMPP clients/MUX. I don’t think this should have any effect, but it hasn’t been tested.

Perhaps you’re allocating channels on the 3 bridges at nearly the same time? I think in our case we add them one by one as participants join.

You can debug further by looking at the logs on the bridge that failed to respond – did it receive the request? Did it throw any exceptions or print anything interesting?

Boris

#5

Hi Boris,

Is it correct that I should first set things up as load-balance (jicofo/doc) then turn on octo (jvb/doc)? That’s what I had done.

I’ve tried several times using stables (3508 and 3186), running 1 full meet, and 3 jvb in VMs (Debian Stretch) but still can’t get it to go more than 2 reliably.

I found that if I restart jvb#3, then 3-region octo would work, but only for one conference.

I then compared the jvb logs for the working and non-working cases.

Working jvb#3

44.403 INFO: [94] org.jitsi.videobridge.Videobridge.log() CAT=stat create_conf,conf_id=e277122bff17f94 conf_name=test123,logging=true,conf_count=1,ch_count=0,v_streams=0
44.413 INFO: [94] org.jitsi.videobridge.Conference.log() CAT=stat create_content,conf_id=e277122bff17f94,content=audio conf_count=1,ch_count=0,v_streams=0
44.434 INFO: [94] org.jitsi.videobridge.RtpChannel.log() CAT=stat create_stream,conf_id=e277122bff17f94,content=audio,ch_id=a6d7bbfac7dcf017,endp_id=null,stream=1968919027
44.434 INFO: [94] org.jitsi.videobridge.Channel.log() CAT=stat transport_connected,conf_id=e277122bff17f94,content=audio,ch_id=a6d7bbfac7dcf017,endp_id=null,stream=1968919027
44.471 FINE: [94] org.jitsi.impl.neomedia.RTPConnectorOutputStream.log() Initialized configuration. Send thread: true. Pool capacity: 100. Queue capacity: 1024. Avg bitrate window: 5000
44.497 FINE: [94] org.jitsi.impl.neomedia.AudioMediaStreamImpl.log() registering format ALAW/rtp, 8000.0 Hz, 8-bit, Mono, Signed with RTPManager
44.498 FINE: [94] org.jitsi.impl.neomedia.AudioMediaStreamImpl.log() registering format g722/rtp, 8000.0 Hz, Mono with RTPManager
44.553 FINE: [94] org.jitsi.impl.neomedia.AudioMediaStreamImpl.log() registering format ALAW/rtp, 8000.0 Hz, 8-bit, Mono, Signed with RTPManager
44.554 FINE: [94] org.jitsi.impl.neomedia.AudioMediaStreamImpl.log() registering format g722/rtp, 8000.0 Hz, Mono with RTPManager
44.554 INFO: [94] org.jitsi.videobridge.Content.log() CAT=stat create_channel,conf_id=e277122bff17f94,content=audio,ch_id=a6d7bbfac7dcf017,endp_id=null,stream=1968919027 transport=octo,bundle=null,initiator=null,media_type=audio,relay_type=null
44.563 FINE: [94] org.jitsi.impl.neomedia.AbstractRTPConnector.log() setDirection sendrecv

Non-working jvb#3

50.720 INFO: [81] org.jitsi.videobridge.Videobridge.log() CAT=stat create_conf,conf_id=12ce13169a073125 conf_name=test123,logging=true,conf_count=1,ch_count=0,v_streams=0
50.720 INFO: [81] org.jitsi.videobridge.Conference.log() CAT=stat create_content,conf_id=12ce13169a073125,content=audio conf_count=1,ch_count=0,v_streams=0
50.720 INFO: [81] org.jitsi.videobridge.RtpChannel.log() CAT=stat create_stream,conf_id=12ce13169a073125,content=audio,ch_id=e1813c487f8f7f8d,endp_id=null,stream=835711465
50.720 INFO: [81] org.jitsi.videobridge.Channel.log() CAT=stat transport_connected,conf_id=12ce13169a073125,content=audio,ch_id=e1813c487f8f7f8d,endp_id=null,stream=835711465
50.721 FINE: [81] org.jitsi.impl.neomedia.AudioMediaStreamImpl.log() registering format ALAW/rtp, 8000.0 Hz, 8-bit, Mono, Signed with RTPManager
50.721 FINE: [81] org.jitsi.impl.neomedia.AudioMediaStreamImpl.log() registering format g722/rtp, 8000.0 Hz, Mono with RTPManager
50.723 FINE: [81] org.jitsi.impl.neomedia.AudioMediaStreamImpl.log() registering format ALAW/rtp, 8000.0 Hz, 8-bit, Mono, Signed with RTPManager
50.723 FINE: [81] org.jitsi.impl.neomedia.AudioMediaStreamImpl.log() registering format g722/rtp, 8000.0 Hz, Mono with RTPManager
50.723 INFO: [81] org.jitsi.videobridge.Content.log() CAT=stat create_channel,conf_id=12ce13169a073125,content=audio,ch_id=e1813c487f8f7f8d,endp_id=null,stream=835711465 transport=octo,bundle=null,initiator=null,media_type=audio,relay_type=null
50.727 FINE: [81] org.jitsi.impl.neomedia.AbstractRTPConnector.log() setDirection sendrecv

I can see that the non-working log is missing a line with RTPConnectorOutputStream.log() Initialized configuration

In any case, I suspect my configuration is wrong, but I just followed the steps in load-balance (to get all jvb connect to the same jicofo) and then octo, I’m not sure what else I need to do.

Thanks,
Jerry

#6

Hi Boris and Damian, any insight for the way I set up the octo?

Thanks,
Jerry

#7

I don’t really have any ideas on this one. You say that 3-region octo works once after restarting jvb3. Does a 2-region conference involving jvb3 work afterwards (which would indicate that jvb3 is in a broken state)?

Can you post the configuration (sip-communicator.properties) of your bridges? Privately is fine if you don’t want to post it here.

Boris

#8

Hi Boris,

Thanks for taking a look.

I can see the nodes working by using RegionBasedBridgeSelectionStrategy and setting the jitsi meet config to any 2 out of 3 jvb. I still get Failed to allocate channels using bridge when trying to add the 3rd.

I did try meet.jit.si and indeed the multi-region works well (I tried with 4). To match as close to that as possible, I installed jicofo 455 and jvb 1106.

Here are my dev env (1 meet, 2 jvb), hopefully with your help, this becomes a sample for the community to build octo easily.

Jicofo (domain tv-dev-1.local)

~# cat /etc/prosody/conf.d/tv-dev-1.local.cfg.lua
VirtualHost "tv-dev-1.local"
        authentication = "anonymous"
        admins = {
                "tvb1.tv-dev-1.local",
                "tvb2.tv-dev-1.local",
                "tvb3.tv-dev-1.local",
        }
        modules_enabled = {
            "bosh";
            "pubsub";
            "ping";
        }
        c2s_require_encryption = false

Component "conference.tv-dev-1.local" "muc"
    storage = "null"
admins = { "focus@auth.tv-dev-1.local" }

Component "tvb1.tv-dev-1.local"
    component_secret = "u5Q1D4Be"
Component "tvb2.tv-dev-1.local"
    component_secret = "I44y6tuO"
Component "tvb3.tv-dev-1.local"
    component_secret = "WOMmiqvv"

VirtualHost "auth.tv-dev-1.local"
    authentication = "internal_plain"

Component "focus.tv-dev-1.local"
    component_secret = "43njFF99"

~# cat /etc/jitsi/jicofo/config
JICOFO_HOST=tv-dev-1.local
JICOFO_HOSTNAME=tv-dev-1.local
JICOFO_SECRET=43njFF99
JICOFO_PORT=5347
JICOFO_AUTH_DOMAIN=auth.tv-dev-1.local
JICOFO_AUTH_USER=focus
JICOFO_AUTH_PASSWORD=XmsHb1KF
JICOFO_OPTS=""
JAVA_SYS_PROPS="..."

~# cat /etc/jitsi/jicofo/sip-communicator.properties
org.jitsi.jicofo.BridgeSelector.BRIDGE_SELECTION_STRATEGY=SplitBridgeSelectionStrategy
org.jitsi.focus.pubsub.ADDRESS=tv-dev-1.local
org.jitsi.jicofo.STATS_PUBSUB_NODE=sharedStatsNode

~# cat /etc/jitsi/videobridge/config
JVB_HOSTNAME=tv-dev-1.local
JVB_HOST=tv-dev-1.local
JVB_PORT=5347
JVB_SECRET=u5Q1D4Be
JVB_OPTS="--apis=rest,xmpp --subdomain=tvb1"
JAVA_SYS_PROPS="..."
AUTHBIND=yes

~# cat /etc/jitsi/videobridge/sip-communicator.properties
org.jitsi.videobridge.AUTHORIZED_SOURCE_REGEXP=focus@auth.tv-dev-1.local/.*
org.jitsi.videobridge.rest.private.jetty.host=127.0.0.1
org.jitsi.videobridge.rest.private.jetty.port=8080
org.jitsi.videobridge.rest.jetty.host=127.0.0.1
org.jitsi.videobridge.rest.jetty.port=8088
org.jitsi.videobridge.octo.BIND_ADDRESS=192.168.100.81
org.jitsi.videobridge.octo.BIND_PORT=9096
org.jitsi.videobridge.REGION=main
org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_TRANSPORT=pubsub
org.jitsi.videobridge.PUBSUB_SERVICE=tv-dev-1.local
org.jitsi.videobridge.PUBSUB_NODE=sharedStatsNode

jvb2 (jvb3 has the same, except for the region name)

~# cat /etc/jitsi/videobridge/config
JVB_HOSTNAME=tv-dev-1.local
JVB_HOST=tv-dev-1.local
JVB_PORT=5347
JVB_SECRET=I44y6tuO
JVB_OPTS="--apis=rest,xmpp --subdomain=tvb2"
JAVA_SYS_PROPS="..."

~# cat /etc/jitsi/videobridge/sip-communicator.properties
org.jitsi.videobridge.AUTHORIZED_SOURCE_REGEXP=focus@auth.tv-dev-1.local/.*
org.jitsi.videobridge.rest.private.jetty.host=127.0.0.1
org.jitsi.videobridge.rest.private.jetty.port=8080
org.jitsi.videobridge.rest.jetty.host=127.0.0.1
org.jitsi.videobridge.rest.jetty.port=8088
org.jitsi.videobridge.octo.BIND_ADDRESS=192.168.100.82
org.jitsi.videobridge.octo.BIND_PORT=9096
org.jitsi.videobridge.REGION=tvb2
org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_TRANSPORT=pubsub
org.jitsi.videobridge.PUBSUB_SERVICE=tv-dev-1.local
org.jitsi.videobridge.PUBSUB_NODE=sharedStatsNode

Thanks,
Jerry

1 Like
#9

Hi Boris and Damian,

When you get a chance, please take a look at the config files. I suspect I missed some settings in there somewhere.

Thanks,
Jerry

#10

@Boris_Grozev when you have time, please take a look at the config files I posted and see if you can find anything obvious.

Thanks,
Jerry

1 Like
#11

@Jerry_Hu. Hi did you get some answers or findings on this configuration?

Thanks

#12

I didn’t get any answers, so I did debug on my own and made a fix. It’s working for me well enough right now - I still don’t understand how meet.jit.si manages to make it work out of the box, I must have missed something.

#13

@Jerry_Hu. Thanks for the update. I have kind of same problem but it probably more related to a network level: I have the same kind of configuration (Prosody/ jifoco / videobridge) on server 1. Trying to add server2 (only videobridge). but prosody log show:
Mar 21 16:49:16 jvb2.<xmppdomain>:component warn Component not connected, bouncing error for: <iq id=‘Vq7ni-111843’ type=‘get’ to='jvb2.<xmppdomain>’ from=‘focus@auth.<xmppdomain>/focus1233799357609’>
@damencho. In which case can we have a failure to register a component videobridge on another server/machine? besides the secret ?
Tcpdump shows both servers request and response on port 5347 on prosody machine…with apparently tcp reset sent by prosody to videobridge in response to a SYN message.
Thanks

#14

you need to add the new jvb component to prosody, while prosody should listen on the non-localhost IP. Or, as I just redid the setup using muc to register jvb, as detailed in Octo with 3 regions which makes adding jvb a much easier process.

#15

Thanks @Jerry_Hu! I did add the JVB component in prosody config. I assume this is why I have this message from prosody log. I am going to recheck network config, IPs:Ports listening.
Regarding “MUC to register jvbs” I did not see anything in this post (which is the one you mentioned).
However, I am going to try to find more details about using MUC to register JVBs. And see if it is compliant with my attempt to deploy cascading briges

#16

oops, I pasted the wrong link should be How to add the secondary jvb to main jitsi-server

#17

Yes @Jerry_Hu. I found this link too. Which is interesting. However I fix the problem of having additional jvb connected to prosody by simply reread your initial response Prosody should listen to a non-local ip.
So in the prosody doc i found the way to do it (as prosody only listen to local ip for components):

component_ports = { 5347, XXXX } -- Listen on 5347 as well as XXXX
component_interfaces = { "192.168.0.1", "::1", "127.0.0.1" } -- Listen only on these interfaces

Now everything is connected … Next step have this working on 2 regions at least

#18

@Jerry_Hu, I now will try the options with MUCs as it seems to be the official way to go. However I am still confused about the JVBs configuration.
So to connect each JVBs to a muc component configured in Prosody I need to add this in /etc/jitsi/config
org.jitsi.videobridge.xmpp.user.jvb2.HOSTNAME=xmpp-server-ip-address
org.jitsi.videobridge.xmpp.user.jvb2.DOMAIN=auth.jitmeet.example.com
org.jitsi.videobridge.xmpp.user.jvb2.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.jvb2.PASSWORD=some_pass
org.jitsi.videobridge.xmpp.user.jvb2.MUC_JIDS=JvbRoom@internal.jitmeet.example.com
org.jitsi.videobridge.xmpp.user.jvb2.MUC=JvbRoom@internal.jitmeet.example.com
org.jitsi.videobridge.xmpp.user.jvb2.MUC_NICKNAME=jvb-2
org.jitsi.videobridge.xmpp.user.jvb2.DISABLE_CERTIFICATE_VERIFICATION=true
and comment everything else as there is no more external component JVB declared, right??
and repeat this in the config file for every shard (prosody/jicofo) I want this JVB to be connect to?

Also each prosody/jicofo (shards) has a different xmpp domain or not?

#19

@Jerry_Hu. I finally made some changes in my sip-communicator.properties for videobridge, but I am struggling to check if jvb is connecting to my internal MUC. If i just have the jvb configured to connect to the internal MUC only and add an internal MUC in prosody but no external components, I got no bridge available when staring a conference. Do you have some hints about how to manage JVB and this internal MUC and JVB? Also my JVB is trying to connect to prosody as an external component. how did you handle it?

Thanks

#20

Can anyone please help me to sortout this issue. Need to test that whether it's working or not after enabling OCTO Thank you.

#21

hello,
can you help me out to setup my octo config for multi region meeting as I want to divide one meeting in 2 different JVB which are in 2 different regions
cheers,
AS

#22

The original problem I faced was addressed long ago in jitsi code and its “how-to” documentation, please refer to official jitsi online doc on setting up octo.

#23

I have done and follwed every step gone through but got no success and so thought to get a help