Octo requirements

@paul-zwarts,

It’s the client side that reports to Jifoco what region it’s in when it connects to a meeting. So your server needs to populate the right “userRegion” in the deploymentInfo of /etc/jitsi/meet/<meet.domain.com>-config.js using the remote host’s ip address.

Then, the client will report this back up to Jicofo. This will be used to assign the right video bridge for that region.

There is an idea here, from @damencho: How to add the secondary jvb to main jitsi-server .

For example, look at the jit.si config file - https://meet.jit.si/config.js - Search for userRegion. I get “us-east-1”, but you may get something else.

You can use Route53 to determine what value should be put in this variable, before returning the response. Make sense?

Thank you for the prompt response. I think this makes sense. So a jitsi meet web must front end and the config is tied to that local jvb which specifies what region that instance is in. The JVB brewery intercommunicates in the background. But if Server B has a jitsi-meet-web there with the config.js specifying us-west, how does someone in that region hit that front end when the dns has the meet domain originally setup in us-east? Or does jicofo push the user from us-east front end over to us-west front end? Also what is the difference between region and userRegion in the config.js? Also I have seen crossRegion as a parameter in a thread somewhere, wondering what that is?

Each VideoBridge has org.jitsi.videobridge.REGION in their respective /etc/jitsi/videobridge/sip-communicator.properties

The jicofo has org.jitsi.jicofo.BridgeSelector.LOCAL_REGION in its /etc/jitsi/jicofo/sip-communicator.properties - I believe this is the region you see you see in the config.js

Maybe others like @damencho, @saghul or @emrah can help with some of your other questions.

Take a look at Understanding OCTO.

Thanks again. I have my own image to sediment if my understanding is correct.

Note the colouring of the different web users. My understanding is that the JVB should return the media from the closest based on what jicofo realizes about the web client location?

Thats my understanding.

Ok, then I guess the jitsi Gods may need to step in because I must be missing something. I have configured as above. In the Youtube videos, documentation and blogs I have read (every google result from “jitsi octo” search) there is always a reference to Route53 which I understand is dynamic DNS shaping. If that is necessary, then it makes sense my setup does not work?

You need some way for your “user in west” to get “us-west-1” for his “userRegion” when connecting to “jitsi-meet”. Is that happening? That is where Route53 would be used.

Alas, no idea. That is where my understanding falters. jitsi-meet is resolving to the domain jitsi.something.com which is in the east. I do not know how it would know the location of the western user to pass since that jitsi-meet config.js specifies its own location as us-east-1 in the deploymentInfo parameter. I would assume that you need an nginx and jitsi meet on both servers and the DNS shaping gets the user to the appropriate front end based on DNS shaping that jitsi really does not determine or is aware of. That makes sense to me based on other applications I’ve built. But then you are really looking at HAProxy for my purposes because Route53 is only an AWS service. In which case you have full jitsi-meet stacks with all the components on either side, and only the JVB brewery knows to pass between each other on the back end.

You can also put the client region in a custom header (for example in haproxy) and when you pass it to nginx, it can make a check and serve different config.js. This way jicofo will know the region of the user, and since it already knows the region(s) of the jvb(s), the user connects to the closest bridge.

It’s simpler to do it with haproxy - also because you receive a lot more with haproxy (balancing, failover of the shards, URL stickiness). But you can serve different config with nginx only - for example if you use geoip module to guess the IP/region of the user. Haproxy approach is much better, but it still doable with nginx only.

@Yasen_Pramatarov1 thank you for that.

I can see that I can serve different config.js from nginx. I understand I could use something like maxmind geolite2 to determine city, but that doesn’t help for creating arbitrary regions based on your JVB placement.

You mention HAProxy being a better method, how would that be? Who is determining the region? I understand using a custom header, I just quite dont understand how a closest server is determined.

@Yasen_Pramatarov1, I guess a better question for an intermediate solution would be how can I pass a querystring into a nginx conf? If I could issue a link: jitsi.meetinstance.com/meetingroom?us-west-1 I could write that manually into the conf to specify the name of the config.js file? Sorry this is not necessarily jitsi related question.

You know where the JVB is (and its region is in its config). What you need to get the full picture is to determine the user’s region. In most (public) clouds you already have different geo regions.

AWS helps here, because in Route53 you have a geo-balancing of CNAMEs, so the users load their closest shards.

If you don’t have such feature, then haproxy also can use geoip - https://www.haproxy.com/blog/use-geoip-database-within-haproxy/ Nginx also can use it directly, then haproxy can be omitted, although for multiple shards this beats the purpose, because haproxy gives you url-stickiness and without it conferences break. But if you have only one nginx (just one shard and no haproxy, no shards failover and HA), you can use geoip directly.

I haven’t used it this way, what I generally do is rely on r53 (or similar) to direct the user to the appropriate (closest) shard, the user ends up on the haproxy, there I add an X-header with the user region and this is passed to Nginx. It does a simple check with if-blocks in the config and aliases the appropriate “regional” config, depending on the x-header.

Something similar should be doable directly with geoip, instead of r53 before it. My idea is that once haproxy knows the region of the user, you pass the header along and serve the right config.js Then Jicofo can make the right connection between user region and jvb region when engaging octo.

Maybe you are thinking about passing config parameters in the url, like jitsi.meetinstance/room#config.deploymentInfo.userRegion=my-region

I don’t think it’s possible to pass the region that way, but I’m not sure, I may be wrong.

But if you have the links to the rooms in a web page, as links, I think that with a little javascript you can make it add http headers to the links, this way you can pass the region info for each different link. But this is out of my reach, I just guess it could be done. :slight_smile:

@Yasen_Pramatarov1, thank you very much. This is a good amount to crunch on, and I have many avenues to look into. Best regards.

Passing the userRegion in the url isn’t exactly what you want even if it was possible. What if someone in the east shares the link with someone in the west?

You need some web server-side logic that maps the incoming IP address to a back-end service or database of regions. Then populate the corresponding userRegion when returning the -config.js file.

I was thinking about implementing this in the front end. Could do a check first and then pass the parameter. Geographic location doesnt always mean something compared to routing and latency. The opposite also is true some times.

There is also nginx http or tcp health checks, but then you need multiple nginx.

An update here. I have a dirty hack which at least allows me to pass a querystring parameter and then load config.js of my choice, or default to a common file if not specified. To @corby’s point you don’t necessarily want to do that but I haven’t been able to figure out the best way to determine server location yet.

So I have 3 config files:
.domain-config.js
us-west-1.domain-config.js
us-east-1.domain-config.js

Each have the deploymentInfo set to the region I want.

In /etc/nginx/sites-enabled/domain.conf I have changed the following:

if ($query_string ~* "server=([a-zA-Z0-9\-]*)([^&]*)") {
    set $refServer $1;
}

location = /config.js {
    alias /etc/jitsi/meet/$refServer.domain-config.js;
}

Even though the .domain.config.js is now hidden, it still works. That way if there is a server= parameter passed it will load the config I want.

Based on my previous post, I think a ping script could be a good idea. Yes it is hacky, but I do not want to use Amazon AWS or Route 53 for reasons I have infrastructure elsewhere already.

@Yasen_Pramatarov1 I found an anycast service which is affordable called cloudns.net. I now have A name records that have region based aliases pointing to them and this works correctly. Now I can get users going to the correct server. However now I do not have users in the same conference so I am quite lost. If you build a full jitsi-meet stack on each server, how do you configure the JVBs together? I can get the user to the right place, but now I want them to be in the same conference and it appears the prosody and jicofo are separate and ignore the fact that the JVB are pooled together.

Is this video you made still relevant to what I want to do?

Yes, the Prosody and Jicofo are separated. In order to have a conference, the users must end on the same shard. The conferences are based on MUC rooms in XMPP and the XMPP server Prosody is not federated throughout the setup, each shard has its own prosody and its own jicofo - so even if the room name and the URL is the same, the MUC room will be different if you load different shards.

After that, on the JVB level, you can have OCTO and you have connection between the bridges etc., but on the signalling level you need to “gather” all the participants in a specific conference on the same signalling node.

Once again, it’s easier to do this with HAProxy, it gives you the stick tables, it stores in memory the URLs and you can use the same backend (the same shard) for an already recently accessed URL.

I would do this with HAProxy, but once again it can be done with Nginx, even though the haproxy has a lot more options, in nginx you have for example the following: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#hash

The idea is to achieve url-stickiness and session persistence, so that once an URL is accessed, all other peers go the same path (to the same backend) and all including the first one keep (re)loading the same backend, until some timeout (session) expires.

N.B.: I’ve never done it with Nginx only (yet), so please take the above with a grain of salt. :slight_smile:

As for the video, you are right, it’s getting old, I really have to make updated version(s). The main concepts there are the same, but the details are different - for example today you would connect the JVBs not as components to XMPP, but as clients in a MUC. So if you follow the principles and not the step-by-step instructions and bash command lines, it’s still relevant as a howto. I’ll try to update it, once I find the time…

2 Likes