Jitsi performance dropped

[umoove_cbenezech]

Hey Folk,

I’m really new on jitsi (started to work oon it 2 weeks ago) and start to make it working and start to see a couple of issue here and there.

First, I spinned up a single instance with standard installation in GCP (e2-highcpu-16) and successfully made it working with ~35 user in a single room, everything went pretty well and I thought I have some room.

Then I tried with ~60 people and here everything became really bad. CPU and bandwith are ok according to monitoring (stackdriver) and me logged into the instance.

By bad I mean people have audio and video dropped from time to time and the conference became impossible.

I checked the log and I see a lot of SEVERE log (jvb.log):
2020-04-20 19:24:54.106 SEVERE: [542] [confId=a04b97589fc3ab7e epId=c555bb9a gid=fff0b7 stats_id=Grace-oEN conf_name=umeet-eu3-130-87444] DataChannelStack.onIncomingDataChannelPacket#81: Could not find data channel for sid 1

and a lot of WARNING log (jvb.log):
2020-04-20 19:11:12.480 WARNING: [387] [confId=a04b97589fc3ab7e epId=3a2c3e92 gid=fff0b7 stats_id=xxxxxxxx conf_name=xxxxxxx TransportCcEngine.tccReceived#157: TCC packet contained received sequence numbers: 2006, 2009, 2012, 2016, 2018, 2020, 2022, 2025, 2028, 2030, 2032, 2034, 2042, 2044, 2047, 2052, 2057, 2062, 2066, 2070, 2073, 2076, 2078, 2080, 2087, 2093, 2103, 2110, 2112, 2116, 2122, 2124, 2129, 2132, 2135, 2138, 2140, 2142, 2144, 2147, 2154, 2157, 2172, 2174, 2178, 2184-2185, 2189, 2191, 2196, 2199, 2203-2204, 2207, 2212-2213, 2215-2216, 2218, 2222, 2224, 2227-2229. Couldn’t find packet detail for the seq nums: 2006, 2009, 2012, 2016, 2018, 2020, 2022, 2025, 2028, 2030, 2032, 2034, 2042, 2044, 2047, 2052, 2057, 2062, 2066, 2070, 2073. Latest seqNum was 3077, size is 1000. Latest RTT is 262.819214 ms.

in jicofo.log I can find:
SEVERE: [37] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Participant has rejected our transport offer

I have a couple of idea regarding what is happening… Can the main problem can be on particpant where they are overloaded by the number of flux received, if yes can we aggregate them in a single stream, or at least reduce the number of stream sent, a lot concurrent stream can cause a lot of context switch and decrease performance… That’s my feeling, but I can be wrong…

Can someone help me investigate what can be done in order to host more people in a single room?

Anyway jitsi looks to be a great solution and I really appreciate the work done on installation part !

many thanks and enjoy the rest of your day

[masteryoda]

you should seriously look at adding more JVBs for that many users…

[umoove_cbenezech]

I don’t see how adding more JVB will solve this issue, from my understanding on how it works it will create 3 upload stream per User and send n2 stream back… meaning form 50 user you will have 150 stream to the server and 2500 out from the server…

I’m not an expert on streaming but for this test I setup a “massive” instance and with I made running application for way more trafic…

Can you explain to me what will be the benefit of adding more jvb how it could solve my problem please?

PS: From my understanding again I understood that 1 conference can only live in one JVB, am I wrong?

[masteryoda]

i’ve understood (maybe i’m wrong & need to re-read) that jvb load balancing cuts both ways. either more users on a single conf OR multiple conf. with few users.

maybe i misunderstood by the benchmark profile cited in this document

Jitsi Videobridge Performance Evaluation | Performance Testing

also the answer to this question

Maximum number of participants on a meeting on meet.jit.si server

Currently you cannot create a meeting with 200 people. We have a hard limit of 75 participants, but even more than 35, the experience will suffer. But we are working on adding big meetings with more participants (more than 100). For such big audiences we recommend livestreaming the meeting and let the rest of the participants watch it on youtube where you can have even millions participants.

think kinda gives some clues for your scenario

[umoove_cbenezech]

ok ok, I alread saw this topic and thought it was due to meet.jit.si configuration… I will re-read it to see if I miss something, probably I did.

In the meantime, if you or someone else have any idea on how to help me to leverage the number of participants I will really appreciate

I also had a look at those optimisation (didn’t had chance to test it yet)

High CPU utilization on client end

Same here, my laptop shoot to 100% as soon as jitsi starts. And coworkers complain that jitsi eat away the battery of their laptop too. I tried to list every options that can influence the CPU usage of end-client, is this accurate ? is there anything else ? Size of the local window Set LastN to a low value, less stream to decode Avoid using firefox. Firefox cannot do simulcast so if no constraints, it will only send the HD stream, even if others need just thumbail video, big burden to decode …

[rasos]

We are faced with similar issues. Running Jitsi with 4 videobridges, each on a dedicated bare metal server. When we host a conference with 50+ users (>2000 streams), video and audio gets distorted and delayed. But the CPU on the bridge hosting that large conference is bored, bandwidth usage is not more than 1/3 of available uplink and we did all the TCP/UDP kernel parameter tweaks. What we see very often is:

2020-04-24 14:29:02.006 WARNING: [131] [confId=fd5bc9533e8433bb epId=29d40b57 gid=ff44c0 stats_id=Don-dir conf_name=illu_skills_business] TransportCcEngine.tccReceived#157: TCC packet contained received sequence numbers: 53765-53815. Couldn't find packet detail for the seq nums: 53765-53815. Latest seqNum was 55660, size is 1000. Latest RTT is 3078.146541 ms.

Do we need a faster kernel with 1000 Hz to handle so many TCP packets, similar to when gaming PCs had been boosted in former times?

[eos]

IIRC, multiple videobridges help when running multiple conversations (rooms).
A given video conference - regardless of number of participants - always uses just one videobridge…