How Jibri memory leaks depends on participants count in the room?

Hi!
I have a problem with a Jibri memory leak. It comes very fast when about 25 participants in the room.

If I have the room with less participant (for example 7) the memory on Jibri server will not leak, I can recording one or two hours without problems… memory consumption remains at the same level.

This is stable issue.

How does the number of participants affect the Jibri?
What should I do to fix the problem?

server components:
Jibri dedicated server has 8G RAM.
Ubuntu 16.04
jibri: Installed: 8.0-61-g99288dc-1
ffmpeg version 2.8.17-0ubuntu0.1
google-chrome-stable: Installed: 87.0.4280.88-1

First of all I would recommend using ubuntu 20 , and then you are able to run the up-to-date packages within the operating system.

1 Like

@jzs175 from my experience:

if 5 people on the room, jibri will receive 5 video stream,
if 10 people on the room, jibri will receive 10 video stream,
if 25 people on the room, jibri will receive 25 video stream
and so on…

Jibri will have to decode the video all at once, record them, and then save them to a file.

That decode job is done by the chrome. It using much cpu & ram. This is actually happened to us too. Our browser will consume more CPU & RAM when we have more participant in the room.

The record & save to a file job is done by ffmpeg, this process using much cpu, but for some reason it makes the memory leaking if the cpu is not strong enough.

So, in your case, the cpu is already busy to handle the decoding, and there is not much left to handle the recording, that is why the memory is leaking. the solution is one of these 2:

  1. downgrade the jibri, your version of jibri record in FHD resolution. that consume much cpu, I use jibri version 8.0-14-g0ccc3f6-1. by default it recording in HD resolution. It will reduce the CPU consumptions when recording. Or you can build your own jibri from source code and change the recording resolution by your self.

  2. give jibri more cpu to handle the recording. how much? sorry but i don’t know either, you have to test it yourself.

please cmiiw

3 Likes

Someone posted some really good findings on this recently, but basically the “leak” comes from ffmpeg’s buffers when it can’t keep up. If ffmpeg can’t keep up, then it buffers data and if it can’t keep up for too long it can crash. If you attach your ffmpeg logs we can probably see this: you’ll see factors of < 1 in there probably.

2 Likes

Yes, its good idea for the future. I installed jibri according this manual GitHub - jitsi/jibri: Jitsi BRoadcasting Infrastructure where using Ubuntu 16. For a start I want to try fix the problem on this OS, if possible :slight_smile:

I already rebuild jibri for HD resolution.

Thanks, I will try it.
I have 4 CPU of Intel(R) Xeon(R) CPU E5640 @ 2.67GHz

What “factors” do you mean?
file “ffmpeg.1.txt” contains three aborted records. I parsed timings for your comfort:

record 1 - from 2021-06-10T09:01:49 to 2021-06-10T09:04:11
record 2 - from 2021-06-10T09:31:07 to 2021-06-10T09:38:17
record 3 - from 2021-06-10T09:41:21 to 2021-06-10T09:43:53

ffmpeg.1.txt (883.5 KB)

Ffmpeg is frequently running at < 30fps there, so it’s having trouble keeping up.

Can it affect memory consumption?

Yes, it will buffer data when it can’t keep up.

1 Like