Abrupt fall in server disk usage and spike in load leading to unusable client

Developers
1

Hi people,

I tried setting up a Jitsi server for around 400 participants (with audio and video turned off for everybody except the host).

Specs:

Intel Xeon Gold 6140 (32 threads)
192 GB Memory
700 Mbps bandwidth

My conference was scheduled at 10 AM and for some weird reason, the disk I/O fell to near zero just as people starting coming in.

morning_outage
morning_outage1227×1088 69.4 KB

Even though the CPU utilization hovers around 15% and the bandwidth, memory usage is negligible, the number of processes waiting for execution (Load Average in the attached picture) spikes, and the disk I/O has already started to fall to zero around 10 AM. Participants started coming in around 9:45 AM. I am unable to figure out what the reason might be. I was able to access my meet website from Chrome (but was very very laggy) but even that wasn’t possible from firefox.

Let me know if you need more information. Thanks.

2

Good attempt at serving the most users in one room using one server; competition.
The previous record was 120 people in the same room with audo and video muted.
The previous record of most users served using one server split into different rooms was 260.

On a 32 thread machine 15% CPU equals 4.8 system load thus the load average number is not surprising.
100% cpu on all 32 threads is 32 system load, thus on your beefy machine i would not be alarmed unless load is above 32.

The bottleneck is most likely on the client side, i would suggest profiling the client web-browser and disable things that cause excess redraws and network packets for the clients.

Regarding disk IO, what was the reason you had 34Mb/s in disk IO before the users joined, was it something else than jitsi videobride running on the server?

2 Likes
3

Hi, have you altered nginx setting to serve more than 768 worker connections? :slight_smile: If not, that is your answer likely.

Milan

2 Likes
4

@migo

Thank you for your reply, I did not alter it. We have sadly shifted to Zoom now and it is super smooth. Will try this the next opportunity I get.

5

Thanks for the reply, will check logs and let you know if I find out the reason.

6

Hi @frustrated_admin , we ran into similar problem as you, with around 125 users (video/audio off). The frontend on presenter-side became unusable slow.
I read all the solutions here, in particular from @xranby … did this solve the matter and were you able to ran these bigger conferences with Jitsi?