Scale Videobridge inside kubernetes

I read a lot about scaling the videobridge but I think it’s not possible to scale it inside kubernetes for some reasons I think but please prove me wrong - this would help a lot.

  1. Each instance must run on a single node or have an own node port
  2. Each instance must have a unique --subdomain argument for the node port
  3. Before you shut down an instance you have to wait until nobody is still using it anymore
  4. You can’t use a single deployment because the service will loadbalance them automatically (relates to 1.)

One idea I had while writing this topic is to have one deployment for each videobridge which are scaled to 0 by default. And one container which will scale them to 1 when needed using the Kubernetes API. So maybe an Kubernetes Operator would be a solution.

Are you talking about the jvb argument? This is not needed as using components and scaling has its complexity as you have noticed, that’s why we use muc control rooms (brewery) and this is the default configuration for the docker images.

That’s why there are the graceful shutdown scripts, to handle that.

This single deployment that you mention we call a shard. We run with several shards for high availability and I believe this can be achieved and with Kubernetes. The load balancing happens with the bridges inside a shard.

hi sapkra,

I am looking into the same topic.

I will start with 4)
scaling from 0 will not work as jicofo will not be able to schedule the conference on a JVB. IMO there is no reason to scale from 0.

I have multiple JVBs running and they get their individual ports only (I am counting down from 10000). This works fine. I am not touching the --subdomain option

3 )
semi-correct, this is one reason why I built the jitsi-prom-exporter, see also forum post. My plan is to pump these metrics back into k8s metrics for scaling the jvbs back down again once they are empty.

semi-correct because jicofo is able to reschedule the conferences from broken/exiting bridges to healthy/running ones, which works out of the box in my case and in most cases participants dont even notice it. As damencho said, graceful shutdown will help here (I think the jitsi jvb images do this automatically)

I forgot about the k8s operator topic:
I agree that scaling jvbs automatically is too complex for a simple HPA or something, so an operator is needed. I will look into building the operator once we got our other issues fixed. I can let you know once I start with this so we can toss some ideas around, maybe, if you want :slight_smile:

@damencho @Kroev Thank you for your information. I thought that the the component approach is the only way to scale JVB because I could only find a tutorial for this one.

Sorry but I think we have an misunderstanding. My idea was to have multiple Kubernetes Deployments so that you can define a unique port for each one. So min. one JVB deployment per region is scaled to 1 and no one should ever scale larger than 1.

Ok interesting. How are you telling each videobridge the right port?

Yeah this would be great. Can you estimate when you will start working on it?

No. We are having multiple other issues with our Jitsi deployment, currently mainly with lipsync. But I am facing a jitsi deployment which will be used by potentially 4 figure numbers of users, so I have to find a way to scale out/back in the jvbs (and their nodes) based on the load on the system to be cost effective.

Ehh yeah I have one Deployment per JVB. Otherwise it is not possible to tell each individual bridge their own port, they also need unique JVB_AUTH_USERs which is why one will not get away with a simple HPA.

The region thing you are mentioning sounds a bit suspicous. If you want your users to use bridges geographically close to them based on their location you have to shard your bridge pool with OCTO.

I don’t know if this helps any, but I you don’t need unique JVB_AUTH_USERs.

Multiple bridges can login to XMPP with the same account, each getting a unique ‘resource’ automatically.