Jibri / Chrome runs out of memory and stops - both recording and streaming

We are running jibri on Ubuntu 16.04 + all patches in an ESXi VM, JAVA 1.8 & jibri installed as per https://github.com/jitsi/jibri/blob/master/README.md

openjdk version “1.8.0_252”
OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1~16.04-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

lsmod | grep snd_aloop
snd_aloop 24576 0
snd_pcm 106496 4 snd_aloop,snd_hda_codec,snd_hda_intel,snd_hda_core
snd 86016 7 snd_aloop,snd_hwdep,snd_timer,snd_pcm,snd_hda_codec_generic,snd_hda_codec,snd_hda_intel

After 10 to 15 minutes of either recording or streaming it just stops. If you watch the RAM in top it just decreases & then hits 0 - jibri stops

There is always this line in ffmpeg.0.txt

2020-07-17 10:35:07.476 INFO: [51] ffmpeg.call() [alsa @ 0xc8af20] Thread message queue blocking; consider raising the thread_queue_size option (current value: 4096)

Followed later by -

2020-07-17 10:37:13.821 INFO: [51] ffmpeg.call() frame= 8336 fps= 24 q=24.0 size= 37112kB time=00:04:37.86 bitrate=1094.1kbits/s
2020-07-17 10:37:35.388 INFO: [51] ffmpeg.call() [alsa @ 0xc8af20] ALSA buffer xrun.
2020-07-17 10:37:39.227 INFO: [51] ffmpeg.call() Last message repeated 3 times
2020-07-17 10:37:40.117 INFO: [51] ffmpeg.call() frame= 8338 fps= 23 q=26.0 size= 37121kB time=00:04:37.93 bitrate=1094.1kbits/s
2020-07-17 10:37:40.411 INFO: [51] ffmpeg.call() [alsa @ 0xc8af20] ALSA buffer xrun.
2020-07-17 10:37:40.461 INFO: [51] ffmpeg.call() frame= 8338 fps= 23 q=26.0 size= 37125kB time=00:04:37.94 bitrate=1094.2kbits/s
2020-07-17 10:37:45.568 INFO: [51] ffmpeg.call() frame= 8339 fps= 22 q=25.0 size= 37125kB time=00:04:37.96 bitrate=1094.1kbits/s
2020-07-17 10:37:47.822 INFO: [51] ffmpeg.call() [alsa @ 0xc8af20] ALSA buffer xrun.
2020-07-17 10:37:47.827 INFO: [51] ffmpeg.call() frame= 8339 fps= 22 q=25.0 size= 37125kB time=00:04:37.96 bitrate=1094.1kbits/s

At which point it dies.

Any ideas ?

Same as this reported error … Jibri stop stremaing

Out of memory: Kill process 1992 (chrome) score 353 or sacrifice child

@childsafevpn. what is the server specification

Huge. Hardware spec means nothing - gave the VM 4GB/12GB/36GB RAM and it just records until it runs out. Definitely looking like a Chrome issue. Just building a new VM with an older Chrome version.

OK so I have resolved this by downgrading both Chrome and the Chrome driver.

Using 83.0.4103.116 of chrome - from here https://www.slimjet.com/chrome/download-chrome.php
and the chrome driver - https://chromedriver.storage.googleapis.com/83.0.4103.39/chromedriver_linux64.zip

So far in an hours recording only use 2GB of 16GB of RAM, whereas latest version just memory leaked like a sieve!!

Hi! I have Jibri in an ESXi VM too and was facing the same problems of yours. In my case, the problem was solved by editing the Jibri VM and setting 4 or more VCPUs and some CPU reservation. With 4 VCPUs and the speed of 2 cores as CPU reservation, the 4GB of memory doesn’t run out anymore. Maybe this configuration can help you too @childsafevpn.

Thanks. But I’d rather not have to change standard settings for VM or allocate more resources than necessary. As it stands it runs now happily on 2GB RAM and 2 vCPU with the downgrade for chrome & chrome driver.

Sorry. I didn’t see your Ubuntu version. The settings I’ve posted are for Ubuntu 18.04.

I had the same issues of memory leak with 16.04, 18.04 and 20.04. Went with 20.04 and the downgrade of chrome driver and chrome which works well.

I always use chromium, chromium-driver and nvidia-openjdk-8-jre from the official repo of the distro

I think the memory issues that you had are related to the ffmpeg @childsafevpn. Check this jibri issue in github for more information. By the way, in Ubuntu 18.04 is easy to check this. Each line of the ffmpeg log has an indicator of the performance (speed). If it’s close to 1x there are no memory leaks. When it came down to 0.900x (or less), the errors start to appear and if it keeps dropping, the ffmpeg will eventually be closed by the kernel. When this occur the recordings won’t play. You will have to recover the end of file.

I didn’t know that chromium works well with jibri. Thanks for the tip. I’ll give a try. Are you using which distro @emrah? Ubuntu 18.04?

Debian Buster (version 10.4)

How do you get rid of “chrome is being controlled by automated test software” if you use chromium ? I agree it seems to be more stable with jibri but with chrome you set echo ‘{ “CommandLineFlagSecurityWarningsEnabled”: false }’ >>/etc/opt/chrome/policies/managed/managed_policies.json - whats the correct way to do this with chromium ?

Using Ubuntu 18.04

Thanks

Policy configuration files live under /etc/chromium for Chromium:

cp -ar /etc/opt/chrome /etc/chromium

There’s no opt for chromium.

1 Like

Thanks! Logical its different …!