Host a meeting with 500 people - ideas?

users
1

Hi! I wonder if I could pick your wonderful brains for a moment? :slight_smile:

We need to host an annual meeting online. (Because of corona, of course.) The problem is that we could be quite a lot of participants - 300-500 at most. So we are looking for ways to do that… The situation:

  • This is a “formal meeting” with a Chairman, speaker list, voting and stuff.

  • People are not allowed to speak freely. They ask to be added to the speaker list. The Chair gives the floor to people in turn. Only then should the speaker be heard/be visible.

  • The voting, speaker list and such are handled in a separate software.

  • We have “developer skills” and can run our own Jitsi server if we have to. And do API calls and stuff. But we would of course like to do as little of that as possible.

I’ve read a lot of the documentation and a number of threads in the forum. The settings for “Everyone starts muted/hidden” and “Everyone follows me” are great! Then the Chair can broadcast the current speaker to everyone. (It would be great if the Chair could prevent others from un-muting, but I think the discipline will be good.)

The problem is that we are so many… Having a “traditional meeting” with everyone sending video would require an enormous server capacity - it won’t work, right? (I saw a recommendation of “max 35 people”.)

Is the server load a lot less if all but the Chair and current speaker stay muted and hidden? All other participants are only receiving audio and video. Or is sending to 500 people way to much, too?

An alternative would be to broadcast to a live stream on YouTube. The people wishing to speak connects to Jitsi (and disconnects/are kicked after the discussion), all others just watch on YouTube. (And vote in the separate software.) There shouldn’t be more than 10 people or so discussing an item at any time. Is that a good idea? Could we host a meeting like that on meet.jit.si - it handles live streaming without lagging too much?

How about access rights? When I tested, any participants could add a password - that seemed strange. We want a couple of users to have moderator rights (the Chair and a helper), other participants should not be able to change any settings or add a password. Possible?

Sorry if I ask stupid questions… :slight_smile: Any thoughts or smart ideas? I’d be very grateful for some input!

/Anders from Sweden

3 Likes
2

Some ideas,

Yeah, broadcasting I would say is your best bet.
Also setup your own server so you can setup Secure Rooms otherwise anyone that enter the room is considered moderator by default.

Secure rooms will prevents guests (co-speakers) to start or stop recordings or streaming.

Also everyone started muted but the moderator would help too.

About the lagging well, you can rent some server on a location close to you just for the event with enough resources, the YouTube streaming will take care of the rest ~490 viewers.

I would say.

3

Yes you can host a meeting that large on meet.jit.si - It will work but load varies from time to time. If you want to checkout the current benchmarks for jitsi and how it compares to zoom, here is a link.

You can easily see how easy it is to host large meetings on small amount of hardware

1 Like
4

Client computer bottleneck
Jitsi meet work out of the box with up to about 15 participants. After that you will start to run into bottlenecks for users with slow computers that the userinterface redraw itself too much and consume 100% cpu.
This can be solved by profiling using the chrome performance monitor and disable all things that consume CPU time.

For example audio level monitoring that render the blue dots when someone talk trigger GUI updates. With many users these GUI updates will cause the cient webbrowser to redraw too frequently that will cause the user to have a bad audio experience due to missed audio updates. The solution is to simply disable the AudioLevels monitoring.
disableAudioLevels: true,

On the left only 10% CPU when audio levels are disabled. On the right 100% CPU time when audio levels are enabled due to fequent audioLevelReport processing and GUI re-layout.

Skärmbild från 2020-04-14 21-09-35
Skärmbild från 2020-04-14 21-09-351114×685 86.4 KB

14 Likes
5

Thanks a lot for digging into this!

1 Like
6

Thanks a lot for your replies! :slight_smile:

I’ve tested livestreaming. It works fine, but there is a delay of five seconds even with the “very low delay” setting on YouTube. And it is a bit cumbersome for people to enter the meeting temporary to speak. (The participants are smart, so I think it would work. :slight_smile: But still a bit unpractical.)

If I understand it correctly, the JVB2 is now stable? And it is “multi-bridge”, to balance the load? If we set up a Jitsi server on good hardware, do you think it would be possible to host a meeting with 500 people, if all but a few were muted-and-without-video? The moderator using the “follow me”-function. Then people can un-mute (and activate their camera, if they have one) temporary, when they speak. That would greatly reduce the number of video/audio streams. Or would it still be way to large?

(We will disable audio level monitoring as @xranby suggested. Good tip, thanks!)

A bit off topic: An alternative is to use Zoom - with the “Large meeting” add-on a meeting can have 1000 participants. According to their webpage… But does it really work? I haven’t been able to find any reviews/comments from people that have hosted really big Zoom meetings. Do you have some input? (And there are of course the security issues with Zoom.)

7

I am sure that 500 users could be possible using jitsi as long as each bottleneck is found, analysed and solved.

on client side i would recommend monitoring-a-lot using the chrome performance analyser

on server side i would recommend instrument the server and generate flame graphs to study server CPU bottlenecks

on server side i would recommend monitor the jvm running the video bridge and check if switching to a pauseless garbagecollector such as shenandoah-gc makes the videobridge run smooth under heavy heap load (the jvb need to garbagecollect a 3Gb heap without pausing).

jitsi send about 125 UDP packets / s to each participant so the server network-stack must be able to handle 62500 packets/s
how to measure: https://blog.cloudflare.com/how-to-receive-a-million-packets/

the server needs to encrypt a lot of data. Verify that hardware encryption work on the server.

tweak network card buffers to make sure that you use the full available bandwidth.
A server on a full gigabit line should be able to serve at least 480p video resolution to all participants (1mb/s * 500 = 500mbit/s)
In order for this to work you may need to increase the networkcards outgoing txqueuelen size to 20000 so that the videobridge can queue up all UDP packets and have the linux kernel send them out as fast as possible.
tweak tips: https://wiki.geant.org/display/public/EK/InterfaceQueueLength
txqueuelen measurements: http://www.hep.ucl.ac.uk/~ytl/tcpip/linux/txqueuelen/datatag-tcp/

Your use-case with only one participant with video enabled. and all rest are muted unless they hold “space” to speak makes it even more likely to succeed.

It is very hard to tell if it will work unless you have 500 clients and test.

here is a test report with 120 users: Maximum number of participants on a meeting on meet.jit.si server

8 Likes
8

Hi Anders,

Maybe we can help. We offer a service that facilitates online events / town hall meetings / congresses etc in a secured platform in combination with professional video streaming / audio / production and live interaction . Let me know if you are interested. Drop me an email: douwe@sqiffer.com

Regards, Douwe

9

Very well explained. Very Useful Insights

1 Like
10

server nginx user bottleneck
In order for the server to handle more than 250 users you need to increase the nginx thread numbers that default to 768

1 Like
11

Additionally you can add this to main section of nginx.conf too:

worker_rlimit_nofile 30000;

Milan

12

Got it @migo and @xranby,

Should we still consider enableLayerSuspension: true ? What is the reason on this? Thanks brothers,

13

read all about the simulcast layer suspension feature here: https://jitsi.org/blog/new-off-stage-layer-suppression-feature/

2 Likes
14

Hi All,

We are looking for a jitsi expert consultant, I would like to ask anyone maybe you would be interested in this opportunity.

Please get in touch and send an email to info@hivelabstech.com

Looking forward to receive replies soon.

Thanks.

15

Reaching new heights: how we added support for 500 participants to Jitsi Meet