Trying to deploy Octo in AWS: how to work with different client regions?

Hi all,

I have been struggling for some time with this task. I think I have seen all threads and documentation regarding octo that exist, but Im still not progressing. Goal is to deploy one jitsi meet stack in, lets say AWS eu-central-1 zone. Then place just jitsi bridges using Octo in AWS asia and US zones.

I tried few options how to set this up, but Im still unable to create a setup that will use some of AWSs black box services that will pass me a header or whatever which will I then use in nginx to choose the right config.js

I have read the original documentation https://github.com/jitsi/jitsi-videobridge/blob/master/doc/octo.md but I have to use one domain, not a different domain for each zone.

I run whole stack in docker containers and I have already confirmed its working, because if I use SplitBridgeSelectionStrategy then users are randomly connected to three different bridges that I use for testing.

Thank you all for help, appreciated already.

You need at least 3 HaProxies or nginx fronting your server, every one in the different region and everyone setting a header about the region they are in. Your main nginx read it and set it as a variable which is then replaced by ssi in config.js.
The 3 proxies in the regions are behind by Route53 which will always return the closest one.

Thank you damencho for a suggestion. Ill try that during this week.
Problem was, that I used jitsis nginx instance in each zone and used jitsi meet and prosody as a backend… Which was not working.
Ill post a reply when I test it.
Thanks again!

Its been a while, but thanks to @damencho the test setup with octo is now working :slight_smile:
Having it in docker showed some other problems, but that SSI setup was exactly what worked.

I have another question: what type of AWS region interconnect did jitsi stuff choose? Simplest would be a VPN running where main jitsi stack is located and have octo bridges to connect to it. AWS mentions some options https://aws.amazon.com/answers/networking/aws-multiple-region-multi-vpc-connectivity/
However knowing which and why you have chosen will be a great help and time saver.
Thank you!

For your AWS question @Aaron_K_van_Meerten can confirm, what I know is that we just use a VPC to interconnect bridges (so I suppose it is this multi-regional VPC).

Aaron have not responded to mine private message yet, so Im still testing other parts of jitsi stack and xmpp and octo flow via public internet between main jitsi stack and jvb bridges.
But I have another question.
JVBs in different regions can communicate between each other from time to time (using octo).
When two participants connect and they then initiate P2P connection, everything is working.
But when a third person joins, strange behavior happens. Sometimes everything works, users connect to theyr bridges based on the “user-region” served by config.js, but sometimes new users are not connected to the conference what so ever. Just a refresh of the page and they are connected. And sometimes they ar not connected to the conference what so ever, no matter how many times you refresh the page. I have found nothing in the JVB logs, except this:
JVB 2020-01-23 18:38:45.646 WARNING: [19] org.jitsi.videobridge.EndpointMessageTransport.log() SCTP connection with 5071149c not ready yet.
JVB 2020-01-23 18:38:45.647 WARNING: [19] org.jitsi.videobridge.EndpointMessageTransport.log() No available transport channel, can’t send a message
and
JVB 2020-01-23 18:49:33.281 WARNING: [1562] org.jitsi.videobridge.octo.OctoChannel.log() Received a source without an owner tag.
JVB 2020-01-23 18:49:33.283 WARNING: [1562] org.jitsi.videobridge.xmpp.MediaStreamTrackFactory.log() Unprocessed source groups: Optional[]
JVB 2020-01-23 18:49:33.285 WARNING: [1562] org.jitsi.videobridge.octo.OctoChannel.log() Received a source without an owner tag.
JVB in foreign region is a docker container that binds like this:
org.jitsi.videobridge.octo.BIND_ADDRESS=jvb
org.jitsi.videobridge.octo.PUBLIC_ADDRESS=asia-bridge.secretdomain.com

jvb is name of a docke-compose service name that is translated to a containers private IP when it starts and according to logs, that works
JVB 2020-01-23 18:38:05.113 INFO: [10] org.jitsi.videobridge.octo.OctoRelay.log() Initialized OctoRelay with address jvb/172.20.0.2:4096. Receive buffer size 212992 (asked for 10485760).
JVB 2020-01-23 18:38:05.115 INFO: [1] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() org.jitsi.videobridge.octo.PUBLIC_ADDRESS=asia-bridge.secretdomain.com
JVB 2020-01-23 18:38:05.115 INFO: [1] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() org.jitsi.videobridge.octo.BIND_ADDRESS=jvb
JVB 2020-01-23 18:38:05.116 INFO: [1] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() org.jitsi.videobridge.octo.BIND_PORT=4096

You are testing with Chrome right?
The description seems exactly like all the reports we have for Firefox and we are currently working on fixing those.

1 Like

Via Chrome its working as I wrote earlier. From time to time. Firefox is not working at all (tried maybe 20 different calls and none went through).
@damencho do you think its worth trying an older version of docker jitsi meet? I know that https://github.com/jitsi/docker-jitsi-meet fixed some errors in November that I ran into as well so only commits from past month are usable, probably.

We were having reports for those Firefox issues at least for a year, so I doubt that downgrading will greatly improve the situation. You can see that we enabled a warning on meet.jit.si because of that. We are working on that, we will even be bringing simulcast to Firefox which is a huge gain, as currently one or two Firefox participants can bring down the whole experience by a lot, especially on participants with slightly not so good connection.

That is strange with chrome and octo. Are you using jvb or jvb2? I know few bugs were fixed on jvb2, so my advice is using that.

@roman2 We’re trying to setup a similiar deployment. Could you share more details about your nginx configuration and the topology of the inter VPC setup?

If you have three regions, I’m not quite sure if this setup needs each region to peer with the other. In other words, does it need to be a full mesh? Or do the videobridges only talk to the jicofo instance?

FWIW we’ve got this working:

  • one jitsi meet instance running nginx
  • multiple JVBs in different AWS regions
  • nginx using the geoip2 module and SSI to serve the config js with the appropriate region

To answer my questions:

  • Each JVB must be connectable directly to each other JVB and the prosody instance
  • We used vanilla inter-region VPC peering to achieve this
  • You only need one nginx instance to make this work

We’ve released our full setup using terraform and ansible to deploy to AWS

Major props to the SWITCH CH folks, their ansible role was used as a base.

1 Like

@abelxluck sorry for late response, but Im available now again.
Mine setup was quite different, all was packed in the docker containers. JVBs could not see each other directly.
At the end, this set up was unstable, one time it was working, the second time it was not, nothing changed between the attempts.
Main difference was that I had HA proxy container in each region and I have routed the traffic to specific one based on AWS Route 53 geo service. HA proxy added a right location header and I was able to use it down stream with SSI on the main NGINX.
At the end, the time to set up octo bridges was up so we ditched the unstable octo setup and just used one main jitsi stack, all packed nicely by the docker-compose.
Im glad you was successful and had it running.

1 Like

Hi @roman2,

It is awesome to know you are successfully using docker containers with multi-region JVB deployment. Did you make the container again from the latest Jitsi-meet with JVB2 or from https://github.com/jitsi/docker-jitsi-meet? Do you manage to have Coturn TURN server work in your containerized jitsi above? Love to know and see your step-by-step guide in Jitsi-meet wiki if possible or in this thread.

@abelxluck , you have done a ‘super’ job too.

Thank you both

hi @abelxluck

thanks for the guide! i had a little trouble getting geopip2 installed on my nginx version but an upgrade did the trick. followed the rest of your guide and it works great!

one quick question though, the only part i can’t seem to get working is testing by adding add #config.deploymentInfo.userRegion:“region2” to my meeting url.

however, because of the way quotes are encoded in the URL, when i check with config.deploymentInfo in the console, i get “{userRegion: “useast”, userRegion:%22region2%22: undefined}”

how did you get this way of testing working?

Nevermind, this seems to have fixed itself once I connected my other regions. I guess it tries to “match” the key before passing it.

fyi for anyone who find this thread: the guardianopts project assumes you’ll run one jvb in the same region as the master! you can have as many regions as you want, but there has to be at least one jvb+master in the same region