Jitsi performance issues

Hello here,

I set up a selenium grid platform to benchmark jitsi and I am facing performance issues starting around 12 simultaneous users (4 conferences of 3 users each).

  • VMs running jitsi and selenium are pretty big: 4vCPU, 16Gb RAM, 250Mbps network interfaces
  • I am pushing colibri stats to prometheus, when client number is growing (up to 100) I can not get more than 3 or 4 Mbps from colibri stats as well as on the VMs and Docker network interfaces. On all VMs running multiple Chrome, I am around 1.5Mbps.

I was thinking the issue was coming from Jitsi running in Docker but performances are the same when Jitsi is installed from debian packages directly on VMs.

Looking at the different sources over the Internet, a JVB node is announced to be able to handle 400 Mbps per instance, so I am a bit surprised I can not do more than 3 or 4 Mbps.

Does someone have some ideas where to look at?

Thanks a lot.

Not sure how you are running the selenium, but what we have seen is that this instance can handle 2 or 3 participants before starting to drop bandwidth due to cpu utilisation. This is in case where you just load a browser with pure jitsi-meet in it.
This was the bottleneck we were facing when trying to do load testing.

Selenium grid is running on:

What do you mean by “This instance can handle 2 or 3 participants before starting to drop”? Do you mean that the VM is not big enough? What do you suggest to run things correctly?

The load tests we are doing is simple as opening the same conference on 3 selenium node with Chrome and doing this N times on N different conferences.
We look at the network bandwidth on all the nodes and also on what is returned from the jitsi stats API by doing a selenium driver call like driver.executeScript(‘return APP.conference.getStats();’);

Additional information, lscpu returns this

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             4
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 60
Model name:            Intel Core Processor (Haswell, no TSX)
Stepping:              1
CPU MHz:               2399.996
BogoMIPS:              4799.99
Virtualization:        VT-x
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K
NUMA node0 CPU(s):     0-3

@damencho I created more gafana graphs and here are the results. Can you tell me what do you think about them?

3 attendees:

9 attendees:

12 attendees:

36 attendees:

CPU is around 30 to 40%, and quality highly decrease when attendees size > 10.

I mean one of those instances is able to handle 2, maximum 3 participants. How many are you running?
Also what video are you feeding them with? The default green thing does not produce high framerate and quality to be able to be as close to a real participant. We are using this video FourPeople_1280x720_30.y4m

Jitsi is running:

  • on a single VM with the same characteristics as listed above
  • From a default installation from debian package, nothing has been changed ie we do not scale JVB for example

Selenium Grid is running:

  • In Docker
  • On 7 VMs with the same characteristics as listed above
  • Each docker container is running one and only one Chrome instance
  • Each Chrome instance is configured to use FourPeople_1280x720_30.y4m

Conferences are created with 3 attendees each: I am asking Selenium to open 3 Chrome instance at the same URL. After a delay, I open a new conference with 3 other attendees, etc…

From what I understand:

  1. You say that such VMs are not able to handle more than 2 or 3 participant. This means that it matches the low performance values I have around 12-15 conferences with my 7 VMs dedicated to Selenium nodes.
  2. Jitsi is also having performance issues to handle all the video/audio streams due to the VM it is running on even if CPU is not fully used

Right?

What I plan to do:

  1. Use a bigger VM (bigger CPU) for jitsi
  2. Use another y4m file with lower resolution which better match the real world use case. (Even if from what I understand, there is some negotiation between Chrome and the server to send lower resolution)

Yes, after the load becomes higher Chrome instances stop sending high quality, and at some load it can stop sending at all.

That VM should be fine for running one jvb and handle one conference with 20 people.

For our bridges on meet.jit.si we use c5.xlarge in aws, which is 4cores and 8GB of Ram. And as you are running and the signaling on the same machine 16GB of RAM is a good choice.
Not sure about the network is it symmetric 250 Mbs? If so you will max your server download at around 50 participants, this is in an ideal situation, but normally you should not reach the max.

We are currently working on optimizing the UI which will also help for the testing, you will be able to spin more participants in one of those grid machines. We are currently testing with UI stripped client using just lib-jitsi-meet, so we can just load with fewer machines so we can continue testing and optimizing.