Ffmpeg eats all the memory and crash within a minute - recording or streaming

Hi.

I installed jibri by following the instruction here https://github.com/jitsi/jibri
My server is a Ubuntu 18, quad-core, 8Gb of memory

On jitsi when I start a recording or a streaming session, if I keep it less than 10 seconds, then it’s fine, I get my file/stream. But above 10 seconds the recording/stream will stop and my whole server become slow and unresponsive.

With top, I could pin the culprit: ffmpeg. It eats away all the memory very quickly. In less than a minute my 8GB are filled.

You can find attached the log of jibri when I tried a streaming session. Nothing stands out to me. I stopped the streaming after 15 seconds and ffmpeg was already at 40% memory.

Any ideas what is going on ??

browser.0.txt (634.2 KB) ffmpeg.0.txt (15.5 KB) log.0.txt (12.4 KB)

Even if I stop completely prosody, jicofo, jvb and jibri, if I log as a jibri user and starts ffmpeg by myself, using the command I found in log.0.txt, I get the same issue, the CPU shoot to 150% and the memory keeps growing. I have to kill ffmpeg before it saturates the memory…

$ ffmpeg -y -v info -f x11grab -draw_mouse 0 -r 30 -s 1280x720 -thread_queue_size 4096 -i :0.0+0,0 -f alsa -thread_queue_size 4096 -i plug:bsnoop -acodec aac -strict -2 -ar 44100 -c:v libx264 -preset veryfast -maxrate 2976k -bufsize 5952k -pix_fmt yuv420p -r 30 -crf 25 -g 60 -tune zerolatency -f flv rtmp://a.rtmp.youtube.com/live2/aaa
truc
-rtbufsize 702000k -y -v info -f x11grab -draw_mouse 0 -r 30 -s 1280x720 -thread_queue_size 4096 -i :0.0+0,0 -f alsa -thread_queue_size 4096 -i plug:bsnoop -acodec aac -strict -2 -ar 44100 -c:v libx264 -preset veryfast -maxrate 2976k -bufsize 5952k -pix_fmt yuv420p -r 30 -crf 25 -g 60 -tune zerolatency -f flv rtmp://a.rtmp.youtube.com/live2/aaa
ffmpeg version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
  configuration: --prefix=/usr --extra-version=0ubuntu0.18.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
[x11grab @ 0x55a193a963a0] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, x11grab, from ':0.0+0,0':
  Duration: N/A, start: 1586929106.243604, bitrate: N/A
    Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1280x720, 30 fps, 1000k tbr, 1000k tbn, 1000k tbc
Guessed Channel Layout for Input Stream #1.0 : stereo
Input #1, alsa, from 'plug:bsnoop':
  Duration: N/A, start: 1586929105.915025, bitrate: 1536 kb/s
    Stream #1:0: Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
  Stream #1:0 -> #0:1 (pcm_s16le (native) -> aac (native))
Press [q] to stop, [?] for help
[libx264 @ 0x55a193acb840] using cpu capabilities: MMX2 SSE Cache64
[libx264 @ 0x55a193acb840] profile High, level 3.1
[libx264 @ 0x55a193acb840] 264 - core 152 r2854 e9a5903 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=1 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=2 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=4 lookahead_threads=4 sliced_threads=1 slices=4 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=1 keyint=60 keyint_min=6 scenecut=40 intra_refresh=0 rc_lookahead=0 rc=crf mbtree=0 crf=25.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 vbv_maxrate=2976 vbv_bufsize=5952 crf_max=0.0 nal_hrd=none filler=0 ip_ratio=1.40 aq=1:1.00
Output #0, flv, to 'rtmp://a.rtmp.youtube.com/live2/aaa':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: h264 (libx264) ([7][0][0][0] / 0x0007), yuv420p(progressive), 1280x720, q=-1--1, 30 fps, 1k tbn, 30 tbc
    Metadata:
      encoder         : Lavc57.107.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 2976000/0/0 buffer size: 5952000 vbv_delay: -1
    Stream #0:1: Audio: aac (LC) ([10][0][0][0] / 0x000A), 44100 Hz, stereo, fltp, 128 kb/s
    Metadata:
      encoder         : Lavc57.107.100 aac
[aac @ 0x55a193ad00c0] Queue input is backward in time
[flv @ 0x55a193ac9de0] Non-monotonous DTS in output stream 0:1; previous: 223, current: 182; changing to 223. This may result in incorrect timestamps in the output file.
[aac @ 0x55a193ad00c0] Queue input is backward in time
[flv @ 0x55a193ac9de0] Non-monotonous DTS in output stream 0:1; previous: 223, current: 206; changing to 223. This may result in incorrect timestamps in the output file.
[flv @ 0x55a193ac9de0] Non-monotonous DTS in output stream 0:1; previous: 229, current: 210; changing to 229. This may result in incorrect timestamps in the output file.
[flv @ 0x55a193ac9de0] Failed to update header with correct duration. 36.1kbits/s speed=0.476x
[flv @ 0x55a193ac9de0] Failed to update header with correct filesize.
frame= 1016 fps= 14 q=21.0 Lsize=     149kB time=00:00:33.87 bitrate=  36.1kbits/s speed=0.476x
video:93kB audio:9kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 46.605049%
[libx264 @ 0x55a193acb840] frame I:17    Avg QP: 3.24  size:   393
[libx264 @ 0x55a193acb840] frame P:999   Avg QP: 6.03  size:    88
[libx264 @ 0x55a193acb840] mb I  I16..4: 99.9%  0.0%  0.1%
[libx264 @ 0x55a193acb840] mb P  I16..4:  0.0%  0.0%  0.0%  P16..4:  0.0%  0.0%  0.0%  0.0%  0.0%    skip:100.0%
[libx264 @ 0x55a193acb840] 8x8 transform intra:0.0%
[libx264 @ 0x55a193acb840] coded y,uvDC,uvAC intra: 0.0% 0.0% 0.0% inter: 0.0% 0.0% 0.0%
[libx264 @ 0x55a193acb840] i16 v,h,dc,p: 91%  0%  9%  0%
[libx264 @ 0x55a193acb840] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu:  0%  0% 100%  0%  0%  0%  0%  0%  0%
[libx264 @ 0x55a193acb840] i8c dc,h,v,p: 100%  0%  0%  0%
[libx264 @ 0x55a193acb840] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x55a193acb840] kb/s:22.38
[aac @ 0x55a193ad00c0] Qavg: 65536.000
Exiting normally, received signal 15.

If I remove every parameters related to sound in this ffmpeg command line, so removing -f alsa -thread_queue_size 4096 -i plug:cloop -acodec aac, then the memory saturation issue goes away. Memory usage is stable. So It clearly seems to be related to the soud. How can I debug this kind of issue ?

lsmod snd_aloop is ok, my .asoundrc file is also ok. Any help ?

lsmod | grep snd_aloop
snd_aloop              24576  0
snd_pcm               106496  1 snd_aloop
snd                    81920  3 snd_aloop,snd_timer,snd_pcm


cat /home/jibri/.asoundrc
# playback PCM device: using loopback subdevice 0,0
pcm.amix {
  type dmix
  ipc_key 219345
  slave.pcm "hw:Loopback,0,0"
}

# capture PCM device: using loopback subdevice 0,1
pcm.asnoop {
  type dsnoop
  ipc_key 219346
  slave.pcm "hw:Loopback,0,1"
}

# duplex device combining our PCM devices defined above
pcm.aduplex {
  type asym
  playback.pcm "amix"
  capture.pcm "asnoop"
}

# ------------------------------------------------------
# for jack alsa_in and alsa_out: looped-back signal at other ends
pcm.ploop {
  type plug
  slave.pcm "hw:Loopback,1,1"
}

pcm.cloop {
  type dsnoop
  ipc_key 219348
  slave.pcm "hw:Loopback,1,0"
}

# ------------------------------------------------------
# default device

pcm.!default {
  type plug
  slave.pcm "aduplex"
1 Like

Hi Arzar,
Were you able to solve this? I have the same issue.

Thanks, Nav

Hi Kumar,

No, I couldn’t solve it. :frowning_face: I opened an issue on jibri github.

Please add a comment saying you got the issue too, maybe it will catch the attention of the developers.

In the meantime, I’m trying to modify JVB to catch the stream I want to record directly in the videobridge, but no luck so far.

I have the same situation.

Our server is independent, just jibri (and chrome, chromedriver, ffmpeg and all the needed dependencies) is installed.
Recordings are OK, but if i use htop I can clearly see that ffmpeg memory usage keeps increasing until all the physical memory is eaten. Then CPU load goes down (I suppose that it’s affected by the overhead) and it keeps eating swap. When both physical and swap memory is taken, ffmpeg crashes and the recording stops abruptly.

Our system specifications:

  • Debian Buster.
  • 2 Virtual CPUs.
  • 8GB RAM

Anyone had success with this?
I’m stuck and I don’t know what to do?
An y other version of debian to test?

I cannot go with Ubuntu, must be Debian…

Not sure if I can help… had a similar problem about 3 weeks ago.
I started with 4GB RAM - it crashed. 8GB RAM - again. Tried with 12GB RAM
(for no reason, tried everything before) - it worked. Didn’t even have time
to check if it’s normal but yeah, that’s what helped me.

I was just in the assumption that jibri needed 16 GB anyhow. Our docker-jibri instance stops after a few seconds (latest a minute) with 8 GB of memory. Keeps going for at least a few minutes with 16 GB and assumed that it would work longer, but haven’t tested yet.

Has there been any update on this issue?

I spun up a VPS with 2 CPUs and 8GB RAM and I tried recording an hour long conference and I experienced the same crashes everyone has described here.

Hi All,

Using Debian 9 and a static build of ffmpeg (4.2.2) on a 2-core, 16GB n2 on GCP , we’ve been able to record for 2 hours without a crash - but for some reason the live streaming to YouTube doesn’t work with the static build, so we’re debugging that at the moment…

On Ubuntu 18.04LTS and the static build, the memory leak is still there but instead of crashing after 2-3 minutes, we get 10mins on an n2 with 8GB of memory. The FFMPEG logs indicate the memory leak is related to the alsa buffer xrun as indicated in posts above - previous posts about the issue do exist (https://stackoverflow.com/questions/28359855/alsa-buffer-xrun-induced-by-low-quality-source-in-ffmpeg-capture) but the thread queue size is specified above in one of the posts and is higher… will run some tests and report back if we find a solution, but for now, Debian 9 on a 2core/16GB n2 on GCP does allow 2+hour recordings…

UPDATE: Using a 4-core/16GB n2 instance running Ubuntu 18.04LTS on GCP with an nginx reverse proxy to divert rtmp streams to any streaming service, we have Jibri and ffmpeg running stably and both recording and live streaming for more than 2+hrs - I don’t think the ffmpeg memory leak has gone away, but that with 16GB memory there’s sufficient overhead so it doesn’t crash - at least within the timespans that we’ve tested it with so far…

1 Like

Hi

I have this exactly same problem… My jitsi server simply crasher after 10-20 seconds of recording… I have 4GB of ram, and upload it to 6gb but still the same…

I had the same problem (ffmpeg eating up all RAM) when I installed jibri in Oracle virtualbox running a freshly installed Ubuntu 18.04 LTS. I then made the identical Ubuntu und jibri installation on a real computer and the problem was gone.

I want to report that I can not produce this problem on my system. I recorded the 40 minutes video perfectly fine. I could not reach to 2 GB used memory, mostly ~1.6 GB and no swap use.

I tried some things to crash recording, anyway everything is OK:

  • diffrent resolution: 480, 720
  • 2, 3, 4 participants
  • desktop sharing, shared Youtube video, participants’ screen etc
  • disableAudioLevels with false and true

I installed a new system today to only test this issue. The system specifications:

  • Droplet from Digital Ocean
  • 4 CPUs, 8GB RAM, 160 GB SSD disk
  • Debian Buster 10.3
  • Standart kernel from Debian Buster repo (linux-image-4.19.0-9-amd64), not cloud kernel (linux-image-4.19.0-8-cloud-amd64) coming with the droplet
  • Jitsi/Jibri installed using my own installer

The installation is a bit different from the official instructions. You can check my installation steps from jitsi and jibri scripts.

hello guy, any uptade? I have to disable this feature, because its crashing my server…

I can confirm out problems are gone when we increased the memory.
12GB of RAM and my own compilation of ffmpeg fixed it.

Did someone try the usage of ffmpeg without transcoding in that case? I suppose it could be decrease using cpu.