OCTO participant routing issue?

Hi Jitsi team,

We have a problem on our self-hosted global jitsi infrastructure with OCTO where some participants are being sent to the wrong regional server. We have US-East, UK-London and Australia-Sydney as the server locations.

We have the DNS routing with our service provider set up correctly for regionalized traffic to be routed to the nearest datacenter, with all jitsi infrastructure on the latest stable and believe we have followed the correct OCTO implementation per these posts…

We currently have it configured for WebSockets only. But noticed on your public meet jitsi service you have both WebSockets and BOSH configured…

https://meet.jit.si/config.js

Should we also configure BOSH? could that potentially help solve the problem?

Thanks
Daryl

What is the behavior you see? Enabling websockets will not change anything for regions.

Thanks for the quick reply.

WebSockets do appear to be causing a problem with what we’re experiencing.

Basically, OCTO and region assignment works perfectly if we enable BOSH and disabled WebSockets.

If WebSockets only is enabled then users gets assigned to wrong region, regardless of the HTTP header or the HAProxy where they’re coming from.

Where do you see that? Is that jicofo giving you jvb from wrong region or?

Yes, Jicofo is assigning the user a wrong JVB, and as a result in user info we see that the user is connected to the wrong region.

Is userRegion value correct in config.js on the client side?

@damencho Yeah it is correct, therefore it works when we switch to BOSH.

If testing will be helpful, then I can share our Dev and PROD URLs with you in private.

Dev currently using BOSH | Octo and Region selection is working fine.
PROD using Websocket only | Octo and Region selection is failing.

Note : both Dev and PROD uses the same HAProxy.

Send it over to check it.

Sent @damencho

Do you have this setting in jicofo? jicofo/reference.conf at 235375f7a11e516bd187d99f37044336be6207f3 · jitsi/jicofo · GitHub

And are all the bridges that jicofo should use with the same version?

Yes, we do have OCTO set to true in jicofo.conf, and below you can find the JVBs versions :

US | dpkg -s jitsi-videobridge2 | grep Version
Version: 2.2-9-g8cded16e-1

UK | dpkg -s jitsi-videobridge2 | grep Version
Version: 2.2-9-g8cded16e-1

Australia | dpkg -s jitsi-videobridge2 | grep Version
Version: 2.2-9-g8cded16e-1

Not that. The correct local region of jicofo is what I’m asking?

No, we’re not using local-region but note that it still work without it when we switch to BOSH.

I can set it and try.

Looks like this helped, but shard keeps on changing, sent you a new screenshot from the console logs.

Something is returning a string with the same region separated by comma.

Hi @damencho, after fixing the shard comma issue you and Mohamed talked about we are still seeing the users being sent to the wrong servers. e.g. 4x UK users are being sent to US server. What else could potentially be causing this issue? Thanks

And you have set the local region in all jicofo instances?

When this happens check the config of that user in the browser js console, what is the user region that you see there?

What happened? You check jicofo settings and you still experiencing the issue? Is it random or you repro in certain scenario everytime?