Large Number of Participants & Prosody Performance

engesin · September 16, 2021, 3:57pm

Thats true for this load test @Freddie

But although i observed any problem about conference healths, Stress level worries me

Freddie · September 16, 2021, 4:05pm

I vaguely remember something about the stress level being tied to 70% or something like that. It’s in the code. If I find the information, I’ll share.

shawn · September 16, 2021, 4:28pm

The understanding I have (which could be inaccurate) is that the stress level is relative to whatever value you set as the packet-rate load-threshold, so the stress level is only as accurate as the value you have set there. In other words, you kinda need to tweak it to match the expected capacity of your host for it to be useful.

I don’t know the full implication of a high stress level, but from what I’ve gathered so far:

Jicofo would consider a bridge “under stress” if it reports a stress level of >0.8 (configurable). Not sure what this means in practice if octo not in use, but will definitely come into play with Octo (see link below)
JVB has a load-reducer feature (disabled by default) where it starts to lower the last-n value once a stress threshold is breached.

I admit this is all quite waffly. Hopefully someone who knows better will chip in

Edit – Here’s a very insightful post with more details:

Freddie · September 16, 2021, 4:42pm

Yes, that would be the same thread. Here are the code references:

github.com

jitsi/jicofo/blob/ae1b9b8cbc7f3b54900fd05a8d9247d2927b85dc/src/main/resources/reference.conf#L32

    
      
          // The assumed average packet rate per participant.
          average-participant-packet-rate-pps = 500
          // The default assumed average stress per participant. This value is only used when a bridge does not report its
          // own value.
          average-participant-stress = 0.01
          // The assumed time that an endpoint takes to start contributing fully to the load on a bridge. To avoid allocating
          // a burst of endpoints to the same bridge, the bridge stress is adjusted by adding the number of new endpoints
          // in the last [participant-rampup-time] multiplied by [average-participant-stress].
          participant-rampup-interval = 20 seconds
          // The stress level above which a bridge is considered overstressed.
          stress-threshold = 0.8
          // The amount of to wait before retrying using a failed bridge.
          failure-reset-threshold = 1 minute
          // The bridge selection strategy. The built-in strategies are:
          // SingleBridgeSelectionStrategy: Use the least loaded bridge, do not split a conference between bridges (Octo).
          // SplitBridgeSelectionStrategy: Use a separate bridge for each participant (for testing).
          // RegionBasedBridgeSelectionStrategy: Attempt to put each participant in a bridge in their local region (i.e. use
          //    Octo for geo-location).
          // IntraRegionBridgeSelectionStrategy: Use additional bridges when a bridge becomes overloaded (i.e. use Octo for
          //    load balancing).
          //

github.com

jitsi/jitsi-videobridge/blob/ea5116fed2305c972dbb8b8eed2a67ac427b6272/jvb/src/main/resources/reference.conf#L150

    
      
            #   100pps for low-definition and 50pps for audio, this queue is fed
            #   650pps, so its size in terms of millis is 1024/650*1000 ~= 1575ms.
            send-queue-size=1024
          }
          load-management {
            # Whether or not the reducer will be enabled to take actions to mitigate load
            reducer-enabled = false
            load-measurements {
              packet-rate {
                # The packet rate at which we'll consider the bridge overloaded
                load-threshold = 50000
                # The packet rate at which we'll consider the bridge 'underloaded' enough
                # to start recovery
                recovery-threshold = 40000
              }
            }
            load-reducers {
              last-n {
                # The factor by which we'll reduce the current last-n when trying to reduce load
                reduction-scale = .75
                # The factor by which we'll increase the current last-n when trying to recover

engesin · September 16, 2021, 5:49pm

Yep these are calculation parameters of stress level and how jicofo makes decisions by using stress levels

As a result, I understand that , although I did not face with any problem, if stress level above 1.0, call quality is not guaranteed…

damencho · September 16, 2021, 6:32pm

But if you haven’t touched load-threshold = 50000 this means that currently that stress level is based on that number, but it can be that your machine that hosts that bridge is capable of doing more. So basically you need to figure out those numbers based on your VM.
A good metric to look at is the packet delay stat and know the bridge people use that to say is a bridge overloaded or not Bu I don’t have values stored in the back of my head to give you about it

engesin · September 17, 2021, 6:27am

As far as I analyze, when i do load traffic to 1 jvb instance, I saw package loss when stress level is at 4, and package rate is nearly 200000 (upload+download).

so these values are accurate for me, my environment has nearly 4 times better performance

than c5.xlarge.

load-threshold = 200000 is true value for my environment

jbg · September 17, 2021, 11:48am

With bare metal 8C/16T Xeon servers with 32 GiB RAM we use load-threshold = 250000. With that load threshold, a stress level of 1.0 is about where RTP/RTCP packet delay starts to ramp up which is when you start to see an impact on quality.

Tuning the load threshold so that stress < 1.0 in normal operation is important for Jicofo’s bridge selection logic as well as being important if you use the load reducers (which IMO should be a last resort protection if something has really gone wrong with your scaling).

engesin · September 20, 2021, 10:22am

For big room, I see that there is a new configuration at jicofo → source-signaling-delays

Does it enabled at meet.jit.si ? Do you suggest to activate that config?

github.com

bgrozev/jicofo/blob/4bf0e57165db2375bed493b904ad11500e830dac/src/main/resources/reference.conf#L197

    
      
            // that signal support for JSON encoded sources).
            use-json-encoded-sources = true
          
          
  shared-document {
              // If `true` the shared document uses a random name. Otherwise, it uses the conference name.
              use-random-name = false
            }
          
          
  // How much to delay signaling Jingle source-add and source-remove in order to batch them and reduce the number of
            // messages (based on conference size). The values are in milliseconds.
            source-signaling-delays = {
              // Conferences with size <50 will have delay=0.
              // Conferences with size between 50 and 99 will have delay=500 ms.
              #50 = 500
              // Conferences with size >=100 have delay=1000 ms.
              #100 = 1000
            }
          }
          
          
// Configuration for the internal health checks performed by jicofo.
          health {

damencho · September 20, 2021, 1:35pm

This is something we are currently working on … that has still not reached meet.jit.si

engesin · October 6, 2021, 10:15am

Hi,

is following prosody config set at jitsi production environment?

Thank you.

Raghul_J_E · November 9, 2022, 4:38am

You mean 400 participants in a single conference room? or concurrently with multiple conferences? and I have a single server with 8 cores cpu and 16GB RAM and 100 MBPS with this configuration how many participants can i able to scale ?

Raghul_J_E · November 9, 2022, 9:08am

please share the server’s hardware requirements and configurations , It will be very helpful.

damencho · November 9, 2022, 12:08pm

This bandwidth is enough for 40 HD streams if you have 100MB per download and 100 per upload. If its all together this is 20 in HD.

Raghul_J_E · November 15, 2022, 10:38am

Thank you ,yes @damencho ,I have an overall bandwidth of 400 mbps , should I able to have 100+ participants in a single call with 15 participants with video and the rest with only audio.

damencho · November 15, 2022, 2:03pm

So you must check what does 400 means, if its overall and is asymmetric so you have 200 download and 200 upload, which will not be enough.
Cause on the server you will have 15x2.5 ~38 Mbps for download.
And 100*2.5 ~ 250 for upload. If the server has up to 200 Mbps upload this is around 80 and to be safe 60 participants that can see HD stream.

Raghul_J_E · November 16, 2022, 4:37am

Thank you sir.I have an overall of 400 Mbps bandwidth.And I also want to ask you some suggestions , I want to setup a jitsi server through that i have to scale about 350 to 500 active participants in a single conference for that should i have to do any specific configurations in the software side or only required hardware configurations are enough for that?

damencho · November 16, 2022, 12:37pm

Enable websockets instead of bosh and lower the resolution to 360p and make sure you have at least two bridges woth configured relay/octo.

Raghul_J_E · November 17, 2022, 3:55am

Thank you so much. I’ll do this for sure . An overall of 1000 Mbps bandwidth and 16 Core CPU with 32GB ram , will this configuration of hardware and bandwidth is enough ? for scaling 350 to 500 participants in single video conference.