Server load for many simultaneous meetings

Hi,

We are looking for a solution to run out tutorials in the next semester via video conferencing hosted on our own servers. For this, Jitsi seems like an excellent solution. I was wondering, what the hardware requirements would be to run 20 meetings with 20 people each in parallel. Does this scale linearly in the number of people and / or number of participants in each meeting?

Thanks,
Christian

1 Like

I would say 1 core node and two jvb instances should handle the load. But anyway if you start deploying in this model you can add as much jvbs you want later … For jvb and and core nodes we use c5.xlarge aws instances, you can check those.
Make sure those vms you will run will have enough bandwidth available. If all of them are Chrome (simulcast) and let’s say all participants in a meeting are sending video you need at least 100Mbit upload/download per meeting and those numbers drastically increase if adding just one non-simulcast participant (Firefox at the moment, but we will soon change that :))

3 Likes

what would be the recommendations if we wanted many more meetings…
say 200 meetings * 100 users…some using mobile apps, some connecting from desktops…

what are the precautions we need to take…both server and networkwise…

fantastic tool by the way, thank you…(been testing last few days)

200 meetings with 100 users, make many shards, jvb auto-scale up on hitting some bandwidth threshold and it will autoscale up … probably 30-40 jvbs or more …
Checkout https://jitsi.org/tutorials/

Hi,

We tried out one meeting with Jitsi with 15-20 people and the scaling on the server side seems to be fine, indeed. However, on the client side browsers became unresponsive after a while so that the meeting collapsed because of this. Besides not using Firefox, are there any other ways to improve the stability on the client side? Is it clear what causes the browser behavior?

Thanks,
Christian

Hi,

Today morning we had our first stress test for now jitsi-meet installation. I wanted to know what is our installation capable of. We have jitsi-meet installed on a old dual Xeon E5540@2.53GHz (8C/16T) with 73GB RAM on Debian stable. We created one room and invited our students to join the presentation. 87 student was able to join the room and there were reports that other was not able to join the room. :frowning: Video was shared from teacher (moderator) and all students had cameras disabled only audio was active. Server load was 14 and more. :frowning: All student was advised to use Chrome or Chromium.

This error message was in nginx log few times:
/var/log/nginx/error.log:2020/03/23 10:51:56 [error] 16604#16604: *77285 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 95.102.29.242, server:

How can we find bottleneck of our installation? Server is connected to internet with 1Gb network adapter and not running any other tasks.

May logging tax the cpus so hard? Jitsi is configured to default, besides ldap login for all participants and created lot of log lines. Can setting log level to ERROR help with that cpu load?

Here is screenshot of htop from torture test time:

Are there any things we can adjust or this is expected behavior?

Thank you in advance for any help and suggestions!

Kind regards,

Milan

1 Like

Is this jvb or jvb2?

Hi, VB is installed from stable debian repo. and package in version: jitsi-videobridge_1126-1_amd64.deb

Thank you,

Milan

So you better profile jvb2, we are in the process of pushing it to stable.

1 Like

Hi Damian, thank you for your suggestion. Are there any known problems with jvb2 implementation?

I was wondering about low CPU usage presented here: https://jitsi.org/jitsi-videobridge-performance-evaluation/ Those test were performed on jvb2? I looked at this setup and only big difference is that your CPUs are newer and AVX capable. Can jitsi ecosystem make use of AVX instruction set?

Thank you,

Kind regards,

Milan

Nope, this is jvb1 without simulcast. Very different situation, this was 4-5 years ago … even jvb1 in stable runs with simulcast for 3 years at least …

Hi, thank you for your answer. I was wondering about that conference and 1000 participants, if it is possible. :slight_smile:
OK, I’ve upgraded out test instance to unstable, but now I can see and hear only myself, other side cannot hear or see me. Only chat is working. What other steps are needed after upgrading stable with unstable? Any available documentation for unstable branch?

Thank you!

Kind regards,

Milan

Hi Damian. Upgrading from stable is not good idea. I’ve to clean install jitsi-meet to get it properly configured and working. We have tested it with 3 participants with camera ON and it was fine, and subjectively I would say video call quality a and stability was better. But we have some errors in nginx error log:

5.21.54.8, server: 0.0.0.0:443, upstream: “127.0.0.1:4445”, bytes from/to client:0/0, bytes from/to upstream:0/517
2020/03/29 09:58:39 [error] 16194#16194: *901 recv() failed (104: Connection reset by peer) while proxying and reading from upstream, client: 147.78.168.55, server: 0.0.0.0:443, upstream: “127.0.0.1:4445”, bytes from/to client:0/0, bytes from/to upstream:0/517
2020/03/29 10:10:45 [error] 16194#16194: *1469 recv() failed (104: Connection reset by peer) while proxying and reading from upstream, client: 35.202.2.1, server: 0.0.0.0:443, upstream: “127.0.0.1:4445”, bytes from/to client:0/0, bytes from/to upstream:0/313
2020/03/29 11:01:09 [error] 16194#16194: *1922 recv() failed (104: Connection reset by peer) while proxying and reading from upstream, client: 178.41.174.170, server: 0.0.0.0:443, upstream: “127.0.0.1:4445”, bytes from/to client:0/0, bytes from/to upstream:0/182
2020/03/29 11:01:09 [error] 16194#16194: *1927 connect() failed (111: Connection refused) while connecting to upstream, client: 178.41.174.170, server: 0.0.0.0:443, upstream: “127.0.0.1:4445”, bytes from/to client:0/0, bytes from/to upstream:0/0

Any suggestions on this?

What are your estimations on maximum clients that can connect to one JVB2 instance on single server?

Thank you in advance!

Kind regards,

Milan

Hi Damian. We are testing JVB2 implementation, but today we hit hard java crash and whole jitsi went down. At that time we had 2 big meetings with about 50 participants and 4 smaller rooms with under 15 participants. Only presenters had video enabled, all other participants had their cameras off.

How can we determine cause of that crash?

Here is relevant part from jvb.log

2020-03-31 11:15:45.064 WARNING: [4412] [confId=b242bc0081e23ee2 epId=e9e718ca gid=ffb73a stats_id=Rollin-0IX conf_name=turcani] InterArrival.computeDeltas#220: Packets are
being reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.
2020-03-31 11:15:45.066 WARNING: [4395] [confId=6b985eec610d3d8e epId=40e7b402 gid=ff06cc stats_id=Franz-lRG conf_name=benko] InterArrival.computeDeltas#220: Packets are bei
ng reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.
2020-03-31 11:15:45.073 WARNING: [4409] [confId=b242bc0081e23ee2 epId=05d9c467 gid=ffb73a stats_id=Brisa-luz conf_name=turcani] InterArrival.computeDeltas#220: Packets are b
eing reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.
2020-03-31 11:15:45.088 WARNING: [4397] [confId=b242bc0081e23ee2 epId=5831b0f3 gid=ffb73a stats_id=Amiya-z6j conf_name=turcani] InterArrival.computeDeltas#220: Packets are b
eing reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.
Sctp send error: : Resource temporarily unavailable
2020-03-31 11:15:45.175 WARNING: [4409] [confId=b242bc0081e23ee2 epId=9d8f45a8 gid=ffb73a stats_id=Carolyne-NK8 conf_name=turcani] InterArrival.computeDeltas#220: Packets ar
e being reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.
2020-03-31 11:15:45.202 WARNING: [4402] [confId=6b985eec610d3d8e epId=b6071f57 gid=ff06cc stats_id=Asa-0XY conf_name=benko] InterArrival.computeDeltas#220: Packets are being
reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.
2020-03-31 11:15:45.224 WARNING: [4404] [confId=b242bc0081e23ee2 epId=7a9340fc gid=ffb73a stats_id=Antonietta-wQZ conf_name=turcani] InterArrival.computeDeltas#220: Packets
are being reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.
2020-03-31 11:15:45.266 WARNING: [4400] [confId=b242bc0081e23ee2 epId=a64851a1 gid=ffb73a stats_id=Rusty-zEm conf_name=turcani] InterArrival.computeDeltas#220: Packets are b
eing reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.
2020-03-31 11:15:45.306 WARNING: [4409] [confId=7b7360ae8bd5d85c epId=3c93dfa1 gid=ff10aa stats_id=Murphy-NU8 conf_name=palencikova] InterArrival.computeDeltas#220: Packets
are being reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.
Sctp send error: : Resource temporarily unavailable
2020-03-31 11:15:45.391 WARNING: [4395] [confId=2ec8352e26940191 epId=d27d4465 gid=ffba70 stats_id=Rhea-J9O conf_name=skalka] InterArrival.computeDeltas#220: Packets are bei
ng reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.
2020-03-31 11:15:45.443 WARNING: [4409] [confId=2ec8352e26940191 epId=c09d4209 gid=ffba70 stats_id=Salvatore-39M conf_name=skalka] InterArrival.computeDeltas#220: Packets ar
e being reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.
Sctp send error: : Resource temporarily unavailable
2020-03-31 11:15:45.498 WARNING: [4395] [confId=6b985eec610d3d8e epId=0f1fcb6f gid=ff06cc stats_id=Cade-pD5 conf_name=benko] InterArrival.computeDeltas#220: Packets are bein
g reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.
Sctp send error: : Resource temporarily unavailable
2020-03-31 11:15:45.603 WARNING: [4403] [confId=c3aa6aecb67be634 epId=64aab8e0 gid=ff321d stats_id=Lenna-0vN conf_name=hubinska] InterArrival.computeDeltas#220: Packets are
being reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.
Sctp send error: : Resource temporarily unavailable
Sctp send error: : Resource temporarily unavailable
Sctp send error: : Resource temporarily unavailable
Sctp send error: : Resource temporarily unavailable
Sctp send error: : Resource temporarily unavailable
Sctp send error: : Resource temporarily unavailable
Sctp send error: : Resource temporarily unavailable
Sctp send error: : Resource temporarily unavailable
Sctp send error: : Resource temporarily unavailable
Sctp send error: : Resource temporarily unavailable
Sctp send error: : Resource temporarily unavailable
Sctp send error: : Resource temporarily unavailable
Sctp send error: : Resource temporarily unavailable
Sctp send error: : Resource temporarily unavailable
Sctp send error: : Resource temporarily unavailable
Sctp send error: : Resource temporarily unavailable
2020-03-31 11:15:47.253 WARNING: [4402] [confId=b242bc0081e23ee2 epId=54eb6b9f gid=ffb73a stats_id=Vada-tvs conf_name=turcani] InterArrival.computeDeltas#220: Packets are be
ing reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.
2020-03-31 11:15:47.273 WARNING: [4395] [confId=b242bc0081e23ee2 epId=38deba57 gid=ffb73a stats_id=Ruth-aLQ conf_name=turcani] InterArrival.computeDeltas#220: Packets are be
ing reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.
2020-03-31 11:15:47.305 WARNING: [4399] [confId=b242bc0081e23ee2 epId=95c98e33 gid=ffb73a stats_id=Estel-7f3 conf_name=turcani] InterArrival.computeDeltas#220: Packets are b
eing reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007f7c9d73e723, pid=1030, tid=1454

JRE version: OpenJDK Runtime Environment (11.0.6+10) (build 11.0.6+10-post-Debian-1deb10u1)

Java VM: OpenJDK 64-Bit Server VM (11.0.6+10-post-Debian-1deb10u1, mixed mode, sharing, tiered, compressed oops, concurrent mark sweep gc, linux-amd64)

Problematic frame:

C [libjnisctp.so+0x81723] sctp_report_all_outbound+0xf0

No core dump will be written. Core dumps have been disabled. To enable core dumping, try “ulimit -c unlimited” before starting Java again

An error report file with more information is saved as:

/tmp/hs_err_pid1030.log

2020-03-31 11:15:47.324 WARNING: [4394] [confId=b242bc0081e23ee2 epId=895e0216 gid=ffb73a stats_id=Amani-3pM conf_name=turcani] InterArrival.computeDeltas#220: Packets are b
eing reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.
2020-03-31 11:15:47.325 WARNING: [4401] [confId=b242bc0081e23ee2 epId=fee1f3c5 gid=ffb73a stats_id=Charley-YsT conf_name=turcani] InterArrival.computeDeltas#220: Packets are
being reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.
2020-03-31 11:15:47.356 WARNING: [4402] [confId=b242bc0081e23ee2 epId=05d9c467 gid=ffb73a stats_id=Brisa-luz conf_name=turcani] InterArrival.computeDeltas#220: Packets are b
eing reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.
2020-03-31 11:15:47.377 WARNING: [4413] [confId=6b985eec610d3d8e epId=e6b761d6 gid=ff06cc stats_id=Mina-rP5 conf_name=benko] InterArrival.computeDeltas#220: Packets are bein
g reordered on the path from the socket to the bandwidth estimator. Ignoring this packet for bandwidth estimation.

If you would like to submit a bug report, please visit:

https://bugs.debian.org/openjdk-11

The crash happened outside the Java Virtual Machine in native code.

See problematic frame for where to report the bug.

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

Should I create separate thread for this? I can provide saved java error report if it would help you, or I can give you access to our system if you want.

Thank you for your help!

Kind regards,

Milan

@bbaldino @Boris_Grozev ideas?

Damian, one question to JVB2, can I use this tutorial: https://www.youtube.com/watch?v=LyGV4uW8km8
for configuring fail-over with new JVB2? There is missing jitsi-videobridge component with secret in /prosody/conf.d/ourdomain.cf.lua in JVB2 default installation, I don’t know if adding there new one with second JBV instance as stated in that tutorial has any effect. Thank you for clarifying it.

I want to keep jitsi running when this error triggers again and JVB2 crashes.

Thank you in advance!

Kind regards,

Milan

Are you running with a java version other than 8? 8 is all that we technically support at this point (other versions are untested right now).

Hi, thank you for your answer. Yes, we are using openjdk-11-jre/stable,stable 11.0.6+10-1~deb10u1 amd64 ( OpenJDK Java runtime, using Hotspot JIT) that is default in stable Debian. What do you suggest? What version is preferred OpenJDK 8 or Oracle Java?

Thank you, kind regards,

Milan

OpenJDK 8 should be good.

Thank you for support, I’ll try to install OpenJDK8.

Kind regards,

Milan