Jitsi JVB Load Issues

We were load-testing Jitsi Videobridge on an Azure VM, steadily increasing the load 10 participants at a time. We peaked at 118 concurrent participants (308 channels, 170 streams) spread out across 12 rooms. The breakup was:

  • 10 rooms with 10 participants each, 2 with video, 8 with audio only.
  • 2 more rooms with 4 participants each, 2 with vide, 2 with audio only.

After hitting the peak, a large chunk of participants started dropping out and faced issues reconnecting back to JVB.

Our server hosting JVB is a 16-core CPU with 32 GB RAM and two network cards with 30 Gbps bandwidth each and it isn’t maxing out on either of these specs. We spread out our load across more than 20 systems so we believe our systems / network aren’t maxing out either.

We wanted to know if JVB has some limit or breaking point in terms of the number of concurrent participants / channels it can support before it gives away? And if so, what is the best way to scale? Spinning out a new JVB instance?

Could it be that we should be looking at some config / setting to allow more than 100 participants on the server? We found these settings from scouting the forums:

DefaultLimitNOFILE=65000
DefaultLimitNPROC=65000
DefaultTasksMax=65000

Are there any other settings we should be looking at?

1 Like

When you tested were these sysconfig applied? https://github.com/jitsi/jitsi-videobridge/blob/master/config/20-jvb-udp-buffers.conf
They are set in default jvb install.

It would be interesting to see the results with jvb2 as this is the actively developed and soon we will switch to it as default.

Thanks @damencho. Yes, the sysconfig is applied.

We had downloaded and setup the latest Jitsi last week so it looks like we should already be running JVB2. Is there another way to confirm?

Jvb2 is not default when you install jitsi-meet. You need to apt-get remove jitsi-meet will remove the meta package leaving everything else and then apt-get install jitsi-videobridge2.

Oh okay … then it’s likely we didn’t use jvb2. Is it beta though? Or does it have a stable release at this point? And is it backward compatible with jvb or are there some limitations (any reference doc would help).

Also what’s better in jvb2 in terms of scaling compared to jvb? And circling back to my original query, assuming we were to stick to jvb, is there an estimated concurrent number of participants a single jvb instance can handle (assuming there’s no restriction on servers or system resources).

meet.jit.si has been running on JVB2 for a while now, but we’ve not officially marked it as stable and made it the default yet. JVB2 only supports media over a single port (all streams and RTP and RTCP), whereas JVB1 supported non-bundle and non-rtcpmux. JVB2 works just fine with the stable versions of all the other components (Jicofo, meet, etc.) and that’s what we run on meet.jit.si.

We’re actually in the process of further refining JVB2 to be more performant (we’re doing work on supporting large conferences), so it would be interesting to know what sort of results you saw with it. I don’t have anything with for expected numbers, though–it’s tough as it varies a lot based on the scenario (how many senders spread across how many conferences, etc.)

Thanks for the response. The lack of numbers means we’ll need to come up with some figures using our tests. We’re planning to stick to JVB for a while and stabilize it before we try out JVB2.

We’ve also noticed the channel count keeps fluctuating (ch_count in the JVB logs). Usually it’s 3 per participant (audio, video, data?) but sometimes it’s lesser. We wanted to understand if inactive channels expire after a certain amount since we noticed several expire messages in the logs:

org.jitsi.videobridge.Channel.log() CAT=stat expire_ch,conf_id=d6e27f63509a12cc,content=data,ch_id=f8c4a68bc0819096,endp_id=b3e7e007
org.jitsi.videobridge.Channel.log() CAT=stat expire_ch,conf_id=d6e27f63509a12cc,content=data,ch_id=8c5d56157ac379be,endp_id=e3847bc0
org.jitsi.videobridge.Channel.log() CAT=stat expire_ch,conf_id=d6e27f63509a12cc,content=audio,ch_id=6c770b7d63c7650e,endp_id=e3847bc0,stream=1408690638

If they do expire, is there a default inactive time? And if so, is there any way we can override this default to control how often channels expire? Or do these log entries indicate something else entirely?

Channels do expire after I believe 60 seconds of inactivity. This value can be set in the colibri that Jicofo sends to JVB, but I don’t know if it’s configurable (may require a recompile).

When I try to install videobridge2 it says it cannot find the package? What is the current status on this package ?

It is in unstable repo.

I found it there, but the documentation here says to uninstall jitsi-meet which also uninstalls the web site correct ? at least its not running afterwards anymore

It is assuming that you want to install just the JVB2 on a new instance for load balancing. So, you would uninstall jitsi-meet and keep just the JVB2. Your primary server would still have the jisti-meet and the front-end.

jitsi-meet by itself is a meta package and has only dependencies so we can make deploy/install easier for users, but it depends on jvb1 still.

Removing it just make possible to install jvb2 nothing more.