Recording stops, consumes all RAM exponentially

Hi,

I installed jisti meet with jibri and the system was working as expected, my team could make a 4 hours meeting and record it all successfully

But, time passed (1, 2 months) and now that we want to record again, the records crashes quickly. I made tests and I see the ram is being consumed more and more, little by little, even if I set a 4 GB swap, the recording consumes it all and crashes. We haven’t made any changes so it’s weird that the server is performing like that, since we made long recordings time ago

I don’t have any idea of how to check it or repair it. So any help is appreciated

Regards

Hello,

what’s this output of

cat /var/log/jitsi/jibri/log.0.txt

and

cat /var/log/jitsi/jibri/ffmpeg.0.txt

Thank you for your suggestion; but hte output is huge for both files. Should I upload the file or paste how many lines, or look for something in special?

Yes,

you can’t understand the problem without the log,

RAM issue mostly occurs due to weak CPU power. How many cores do you have on Jibri server?

log.0.txt (117.5 KB)
ffmpeg.0.txt (681.5 KB)

I attached the log.0.txt and ffmpeg.0.txt but this one was too big to upload it all (more than 7 MB) so I cut it to only show the logs of today

Hi, and thank you for joining. I have 4 CPU cores

@emrah

I think it’s not a problem that can be ignored ? when you dont have a Finalize script

SEVERE: [1749] JibriServiceFinalizeCommandRunner.doFinalize#63: Failed to run finalize script
java.io.IOException: Cannot run program “/path/to/finalize”: error=2, No such file or directory

2022-08-04 18:04:25.972 INFO: [1776] LoggingUtils$Companion$OutputLogger$1$1.call#42: frame=    7 fps=0.0 q=25.0 size=       0kB time=00:00:00.20 bitrate=   1.8kbits/s speed=0.394x    
2022-08-04 18:04:26.972 INFO: [1776] LoggingUtils$Companion$OutputLogger$1$1.call#42: frame=   16 fps= 15 q=20.0 size=       0kB time=00:00:00.51 bitrate=   0.8kbits/s speed=0.475x    
2022-08-04 18:04:26.972 INFO: [1776] LoggingUtils$Companion$OutputLogger$1$1.call#42: frame=   26 fps= 16 q=21.0 size=     256kB time=00:00:00.83 bitrate=2509.3kbits/s speed=0.53x    
2022-08-04 18:04:27.973 INFO: [1776] LoggingUtils$Companion$OutputLogger$1$1.call#42: frame=   37 fps= 17 q=18.0 size=     256kB time=00:00:01.20 bitrate=1737.2kbits/s speed=0.567x    
2022-08-04 18:04:27.973 INFO: [1776] LoggingUtils$Companion$OutputLogger$1$1.call#42: frame=   48 fps= 18 q=19.0 size=     256kB time=00:00:01.57 bitrate=1328.4kbits/s speed=0.599x    
2022-08-04 18:04:28.973 INFO: [1776] LoggingUtils$Companion$OutputLogger$1$1.call#42: frame=   57 fps= 18 q=20.0 size=     256kB time=00:00:01.86 bitrate=1123.6kbits/s speed=0.595x    
2022-08-04 18:04:28.974 INFO: [1776] LoggingUtils$Companion$OutputLogger$1$1.call#42: frame=   65 fps= 18 q=22.0 size=     512kB time=00:00:02.13 bitrate=1963.6kbits/s speed=0.584x    
2022-08-04 18:04:29.974 INFO: [1776] LoggingUtils$Companion$OutputLogger$1$1.call#42: frame=   74 fps= 18 q=21.0 size=     512kB time=00:00:02.46 bitrate=1704.2kbits/s speed=0.591x

The CPU on your VM is not powerful enough to encode at required speed, so ffmpeg starts eating into your memory till it crashes. You need a more powerful CPU (CPU quality matters).

Also, what resolution are you recording at?

That error can be ignored. You don’t need to define a finalize.sh script if you don’t plan to use it.

Thank you for joining. The /etc/jitsi/jibri/jibri.conf file is set as follows

ffmpeg {
resolution = "1920x1080"
// The audio source that will be used to capture audio on Linux
audio-source = "alsa"
// The audio device that will be used to capture audio on Linux
audio-device = "plug:bsnoop"
 }

So, as you say, I have two options, or I record in lower resolution or I get more CPUs

If the boss decides to keep the resolution, how many CPUs are required?

I also found that in the file xorg-video-dummy.conf I have
Virtual 1920 1080
Should I change it too there or only in jibri.conf?

It depends on the quality of the cores. I’d say, aim for 8CPU for peace of mind.

Yes, you need to change this too. Then restart both services.

Thank you for your advice. I will meet with the boss and talk about it. And make tests after that. I will update if nothing improves and if it does, what we did

If this is a shared host (which has not dedicated CPUs), your recordings may fail sometimes depending on the other VM’s actions.

Thank you, that’s also a good thing to consider

Well, I updated the video resolution to 1280x720 in

/etc/jitsi/jibri/jibri.conf

and

/etc/jitsi/jibri/xorg-video-dummy.conf

After two test of long recording (around 2 hours) the server stood still and everything was flowing as expected. So I consider this topic closed and I appreciate all the advices, they will be considered for further situations like this

2 Likes