Multiple jibri instances on docker, the second instance unable to record video

Hai guys, I am deploying multiple Jibri instances on docker using https://github.com/jitsi/docker-jitsi-meet and already follow the official guide properly but still faced an issue when recording in the second instance, for the first instance is working fine. I deployed docker Jibri in google cloud with a compute engine ubuntu server 16.04.
I will explain to you what I have done besides what is in the official guide. I updated jibri.yml to make it standalone. I just removed depends_on Jicofo and networks meet.jitsi. Here is my jibri.yml :

version: '3'

services:
    jibri:
        image: jitsi/jibri
        restart: ${RESTART_POLICY}
        volumes:
            - ${CONFIG}/jibri:/config
            - /dev/shm:/dev/shm
        cap_add:
            - SYS_ADMIN
            - NET_BIND_SERVICE
        devices:
            - /dev/snd:/dev/snd
        environment:
            - XMPP_AUTH_DOMAIN
            - XMPP_INTERNAL_MUC_DOMAIN
            - XMPP_RECORDER_DOMAIN
            - XMPP_SERVER
            - XMPP_DOMAIN
            - JIBRI_XMPP_USER
            - JIBRI_XMPP_PASSWORD
            - JIBRI_BREWERY_MUC
            - JIBRI_RECORDER_USER
            - JIBRI_RECORDER_PASSWORD
            - JIBRI_RECORDING_DIR
            - JIBRI_FINALIZE_RECORDING_SCRIPT_PATH
            - JIBRI_STRIP_DOMAIN_JID
            - JIBRI_LOGS_DIR
            - DISPLAY=:0
            - TZ

I run the first instance using this command :

docker-compose -f jibri.yml up -d

After the first instance goes up, I change /home/jibri/.asoundrc to setup the second instance :

...
slave.pcm "hw:Loopback_1,0,0"
...
slave.pcm "hw:Loopback_1,0,1"
...
slave.pcm "hw:Loopback_1,1,1"
...
slave.pcm "hw:Loopback_1,1,0"
...

And then run the second instance using this command :

docker-compose -f jibri.yml up -d --scale jibri=2

Like I stated before, the first instance is working fine. The video has shown in ~/.jitsi-meet-cfg/jibri/recordings . but as for the second instance got this error on log.0.txt.1 :

2020-04-27 06:11:18.607 INFO: [59] org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.onFfmpegProcessUpdate() Ffmpeg quit abruptly. Last output line: plug:cloop: Input/output error
2020-04-27 06:11:18.609 INFO: [59] org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.onFfmpegStateMachineStateChange() Ffmpeg capturer transitioning from state Starting up to Error: SESSION Ffmpeg failed to start
2020-04-27 06:11:18.611 INFO: [59] org.jitsi.jibri.service.impl.FileRecordingJibriService.onServiceStateChange() File recording service transitioning from state Starting up to Error: SESSION Ffmpeg failed to start
2020-04-27 06:11:18.612 INFO: [59] org.jitsi.jibri.api.xmpp.XmppApi.invoke() Current service had an error, sending error iq

and here is ffmpeg.0.txt.1 error log :

2020-04-27 06:11:13.769 INFO: [59] ffmpeg.call() Input #0, x11grab, from ‘:0.0+0,0’:
2020-04-27 06:11:13.769 INFO: [59] ffmpeg.call() Duration: N/A, start: 1587960673.724562, bitrate: N/A
2020-04-27 06:11:13.769 INFO: [59] ffmpeg.call() Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1280x720, 30 fps, 1000k tbr, 1000k tbn, 1000k tbc
2020-04-27 06:11:13.769 INFO: [59] ffmpeg.call() ALSA lib pcm_dsnoop.c:618:(snd_pcm_dsnoop_open) unable to open slave
2020-04-27 06:11:13.769 INFO: [59] ffmpeg.call() [alsa @ 0x55f311051e80] cannot open audio device plug:cloop (Device or resource busy)
2020-04-27 06:11:13.769 INFO: [59] ffmpeg.call() plug:cloop: Input/output error

I suspect the Jibri instance doesn’t read .asoundrc file. So all instances still use the same loopback therefore the second instance got the error. I have updated with the wrong one and renamed it to .bsoundrc to make an error. But the first instance is working fine.
Any suggestions to fix this issue?
Thanks in advance.

Hi,

I am also working on running multiple Jibri on single host itself. What I have done after starting first Jibri container, copied its /root/config/.asoundrc on host.

Then I create a separate yaml having 2 other Jibri services. Each of which mounts different .asoundrc from host. After which I tried to record two simultaneous conference, which worked fine for me.Multiple_Jibri.txt (2.1 KB)

asoundrc2.txt (814 Bytes) asoundrc3.txt (814 Bytes)

Hope this will help you.

Can you give more details about this solution in docker. Nobody has achieved this. Thanks in advance!

I have attached my docker-compose file having two JIbri instances and the asoundrc files which they will mount. I thought this will be enough.

Let me know what more details you require.

Thanks for answering! At this point I have tried everything and I’m still unable to get secondary instances to record properly. This is what I did…

  • Set different .asoundrc files to mount on each service asoundrc1.txt (773 Bytes) asoundrc2.txt (773 Bytes)

  • Edited your copy of compose .yml file to add minor changes. In my case a bind mount to get all the recordings to a single directory in the FS. Multiple_Jibri(1).txt (2.0 KB)

  • Got running 1 instance up of each service
    docker-compose -f Multiple_Jibri.txt up -d --scale jibri2=1
    docker-compose -f Multiple_Jibri.txt up -d --scale jibri3=1
    docker-compose -f Multiple_Jibri.txt ps --all

    Name Command State Ports
    docker-jitsi-meet_jibri2_1 /init Up
    docker-jitsi-meet_jibri3_1 /init Up

But once I try to record I get …

log.0.txt

2020-07-28 16:01:33.397 INFO: [91] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.launch() Starting ffmpeg with command ffmpeg -y -v info -f x11grab -draw_mouse 0 -r 30 -s 1280x720 -thread_queue_size 4096 -i :0.0+0,0 -f alsa -thread_queue_size 4096 -i plug:cloop -acodec aac -strict -2 -ar 44100 -c:v libx264 -preset veryfast -profile:v main -level 3.1 -pix_fmt yuv420p -r 30 -crf 25 -g 60 -tune zerolatency -f mp4 [spoiler]/home/user/recordings/vbunxehdivlhmuoa[/spoiler]/testvideorecord12345_2020-07-28-16-01-30.mp4 ([ffmpeg, -y, -v, info, -f, x11grab, -draw_mouse, 0, -r, 30, -s, 1280x720, -thread_queue_size, 4096, -i, :0.0+0,0, -f, alsa, -thread_queue_size, 4096, -i, plug:cloop, -acodec, aac, -strict, -2, -ar, 44100, -c:v, libx264, -preset, veryfast, -profile:v, main, -level, 3.1, -pix_fmt, yuv420p, -r, 30, -crf, 25, -g, 60, -tune, zerolatency, -f, mp4, /home/user/recordings/vbunxehdivlhmuoa/testvideorecord12345_2020-07-28-16-01-30.mp4])
2020-07-28 16:01:38.407 FINE: [92] org.jitsi.jibri.util.ProcessStatePublisher.ffmpeg.invoke() Process ffmpeg hasn't written in 2 seconds, publishing periodic update
2020-07-28 16:01:38.408 INFO: [81] org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.onFfmpegProcessUpdate() Ffmpeg quit abruptly.  Last output line: plug:cloop: Input/output error
2020-07-28 16:01:38.409 INFO: [81] org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.onFfmpegStateMachineStateChange() Ffmpeg capturer transitioning from state Starting up to Error: SESSION Ffmpeg failed to start
2020-07-28 16:01:38.409 INFO: [81] org.jitsi.jibri.service.impl.FileRecordingJibriService.onServiceStateChange() File recording service transitioning from state Starting up to Error: SESSION Ffmpeg failed to start
2020-07-28 16:01:38.410 INFO: [81] org.jitsi.jibri.api.xmpp.XmppApi.invoke() Current service had an error, sending error iq <iq to='jibribrewery@[spoiler]internal.auth.my.domain.com[/spoiler]/focus' id='odUXW-116' type='set'><jibri xmlns='http://jitsi.org/protocol/jibri' status='off' failure_reason='error'/></iq>
2020-07-28 16:01:38.410 FINE: [81] org.jitsi.jibri.statsd.JibriStatsDClient.incrementCounter() Incrementing statsd counter: stop:recording
2020-07-28 16:01:38.411 INFO: [81] org.jitsi.jibri.JibriManager.stopService() Stopping the current service
2020-07-28 16:01:38.411 INFO: [81] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Stopping capturer
2020-07-28 16:01:38.411 INFO: [81] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() Stopping ffmpeg process
2020-07-28 16:01:38.423 INFO: [81] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() ffmpeg exited with value 1

ffmpeg.0.txt

2020-07-28 16:01:33.551 INFO: [96] ffmpeg.call() Input #0, x11grab, from ‘:0.0+0,0’:
2020-07-28 16:01:33.551 INFO: [96] ffmpeg.call() Duration: N/A, start: 1595952093.509353, bitrate: N/A
2020-07-28 16:01:33.552 INFO: [96] ffmpeg.call() Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1280x720, 30 fps, 1000k tbr, 1000k tbn, 1000k tbc
2020-07-28 16:01:33.552 INFO: [96] ffmpeg.call() ALSA lib pcm_direct.c:1605:(_snd_pcm_direct_get_slave_ipc_offset) Invalid value for card
2020-07-28 16:01:33.552 INFO: [96] ffmpeg.call() [alsa @ 0x5591070f04a0] cannot open audio device plug:cloop (No such device)
2020-07-28 16:01:33.554 INFO: [96] ffmpeg.call() plug:cloop: Input/output error

I’m ussing Ubuntu 18 on a VM of 4vCPU / 8Gb RAM
I have also noted that the only container not giving this error and recording properly is the one I let with .asounrc untouched (slave.pcm “hw:Loopback,1,0”…)

I really don’t know whats happening. If you could provide me with details about the OS, Docker version, and jibri image version you use to make this done, I guess that would help too.

What is the output?

ls -alh /proc/asound

Hi @emrah ! Thanks for taking the time to help me.

ls -alh /proc/asound

total 0
dr-xr-xr-x 9 root root 0 Jul 28 21:41 .
dr-xr-xr-x 154 root root 0 Jul 28 21:41 …
lrwxrwxrwx 1 root root 5 Jul 28 21:48 Loopback -> card0
lrwxrwxrwx 1 root root 5 Jul 28 21:48 Loopback_1 -> card1
lrwxrwxrwx 1 root root 5 Jul 28 21:48 Loopback_2 -> card2
lrwxrwxrwx 1 root root 5 Jul 28 21:48 Loopback_3 -> card3
lrwxrwxrwx 1 root root 5 Jul 28 21:48 Loopback_4 -> card4
dr-xr-xr-x 6 root root 0 Jul 28 21:48 card0
dr-xr-xr-x 6 root root 0 Jul 28 21:48 card1
dr-xr-xr-x 6 root root 0 Jul 28 21:48 card2
dr-xr-xr-x 6 root root 0 Jul 28 21:48 card3
dr-xr-xr-x 6 root root 0 Jul 28 21:48 card4
-r–r--r-- 1 root root 0 Jul 28 21:48 cards
-r–r--r-- 1 root root 0 Jul 28 21:48 devices
-r–r--r-- 1 root root 0 Jul 28 21:48 modules
dr-xr-xr-x 2 root root 0 Jul 28 21:48 oss
-r–r--r-- 1 root root 0 Jul 28 21:48 pcm
dr-xr-xr-x 2 root root 0 Jul 28 21:48 seq
-r–r--r-- 1 root root 0 Jul 28 21:48 timers
-r–r--r-- 1 root root 0 Jul 28 21:48 version

It seems that the snd-aloop module and the loopback devices are OK.

Are you sure the container can read the content of asoundrc (maybe a permission problem)?

Could you try to read .asoundrc inside the container as the jibri user

Hi,

I dont know if below info will help you, but I faced some issue like yours, so sharing below info.

I have two setups.

  1. Docker setup where I have tested multiple recordings. This setup uses plug:cloop. I have shared the asoundrc files to you from this setup.
    “jitsi/jibri@sha256:092b097af2b0b8ff2e419c6b3a106e6f39514369642b002add641a3b7d32f638”
    Host OS: Ubuntu 18.04.4 LTS
    Docker version 19.03.11, build 42e35e61f3

  2. K8s setup where I was facing similar issue like yours, but there ffmpeg is using plug:bsnoop. “jitsi/jibri@sha256:87530f48db37fc297a3def8f201fe5a27999ab01b1cac18100ade8abd6562d72”

Working(1st setup)
ffmpeg -y -v info -f x11grab -draw_mouse 0 -r 30 -s 1280x720 -thread_queue_size 4096 -i :0.0+0,0 -f alsa -thread_queue_size 4096 -i plug:cloop -acodec aac -strict -2 -ar 44100 -c:v libx264 -preset veryfast -profile:v main -level 3.1 -pix_fmt yuv420p -r 30 -crf 25 -g 60 -tune zerolatency -f mp4 /config/recordings/wmbzvsooqxorijxu/ongoingeastsinvitesharply_2020-07-21-14-16-36.mp4 ([ffmpeg, -y, -v, info, -f, x11grab, -draw_mouse, 0, -r, 30, -s, 1280x720, -thread_queue_size, 4096, -i, :0.0+0,0, -f, alsa, -thread_queue_size, 4096, -i, plug:cloop, -acodec, aac, -strict, -2, -ar, 44100, -c:v, libx264, -preset, veryfast, -profile:v, main, -level, 3.1, -pix_fmt, yuv420p, -r, 30, -crf, 25, -g, 60, -tune, zerolatency, -f, mp4, /config/recordings/wmbzvsooqxorijxu/ongoingeastsinvitesharply_2020-07-21-14-16-36.mp4])

Non-Working(2nd setup)
ffmpeg -y -v info -f x11grab -draw_mouse 0 -r 30 -s 1280x720 -thread_queue_size 4096 -i :0.0+0,0 -f alsa -thread_queue_size 4096 -i plug:bsnoop -acodec aac -strict -2 -ar 44100 -b:a 128k -af aresample=async=1 -c:v libx264 -preset veryfast -profile:v main -level 3.1 -pix_fmt yuv420p -r 30 -crf 25 -g 60 -tune zerolatency -f mp4 /config/recordings/pkavmirneakhqtyf/classicalentitiesexpressstrictly_2020-07-28-17-37-14.mp4

The logs which you have shared uses cloop as plug. So the asoundrc which I have shared, should not create problem.

For troubleshooting, I went inside container and executed the ffmpeg command which I got in my logs. I changed the plug and it showed me capturing data. You may try same in your container.
e.g. ffmpeg -y -v info -f x11grab -draw_mouse 0 -r 30 -s 1280x720 -thread_queue_size 4096 -i :0.0+0,0 -f alsa -thread_queue_size 4096 -i plug:cloop -acodec aac -strict -2 -ar 44100 -c:v libx264 -preset veryfast -profile:v main -level 3.1 -pix_fmt yuv420p -r 30 -crf 25 -g 60 -tune zerolatency -f mp4 /home/user/recordings/vbunxehdivlhmuoa/testvideorecord12345_2020-07-28-16-01-30.mp4

See if there is some error in it.

For solving my problem on new setup I found here to change the asoundrc file. Direct link is this

After changing this file into my new setup(which uses plug:bsnoop, recording started working. I still need to check multiple recordings in this setup.

Also in new .asoundrc, you will find two capture.pcm, asnoop and bsnoop. May be the new config support both plug and the older supported only asnoop.

If you dont find any solution, I would suggest you to use latest image and change the asoundrc file and then check(As old image may not support the asoundrc file). Also check the permissions of asoundrc file.
You can also try to find the difference in the ffmpeg command which comes in logs of your working(First container) and that on your second container, non working container.

Hey @emrah! Thanks… actually something happened between version tests and it was a problem with the Kernel loading the snd-aloop module… after loading it again it worked.

Odd stuff, because lsmod shown that it was loaded.
Thanks for your help!

Hi @Anshul_Sehgal… thank you for everything. Actually it was a kernel issue… thanks for the detailed instructions anyway. I noticed the change to plug:bsnoop in newer versions of jibri, but looking for an answer to the previous problem I ended with image stable:4627 wich still doesnt have the change.

If anybody struggles with this same problem (cloop or bsnoop) and uses DigitalOcean… double check your module is loaded with ls -alh /proc/asound , as @emrah sugested… and also be sure to install a compatible Kernel correctly before anything else.

hi @Anshul_Sehgal: I have used 2 instances of jibri in docker. When trying to record parallel using 2 instances, 2nd one is stopped after 2 minutes. I did not apply your suggestion yet. Could please help me whether copying the .asoundrc to the host .asoundrc will solve the issue?
Also I have checked the container size, it is 2.3 mb. Is this issue happening for the container size or for the .asoundrc ?
Also I have checked with ls -alh /proc/asound command the result is showing correct values.
dr-xr-xr-x 9 root root 0 Aug 28 22:29 .
dr-xr-xr-x 182 root root 0 Aug 30 03:02 …
dr-xr-xr-x 6 root root 0 Aug 31 16:24 card0
dr-xr-xr-x 6 root root 0 Aug 31 16:24 card1
dr-xr-xr-x 6 root root 0 Aug 31 16:24 card2
dr-xr-xr-x 6 root root 0 Aug 31 16:24 card3
dr-xr-xr-x 6 root root 0 Aug 31 16:24 card4
-r–r--r-- 1 root root 0 Aug 31 16:24 cards
-r–r--r-- 1 root root 0 Aug 31 16:24 devices
lrwxrwxrwx 1 root root 5 Aug 31 16:24 Loopback -> card0
lrwxrwxrwx 1 root root 5 Aug 31 16:24 Loopback_1 -> card1
lrwxrwxrwx 1 root root 5 Aug 31 16:24 Loopback_2 -> card2
lrwxrwxrwx 1 root root 5 Aug 31 16:24 Loopback_3 -> card3
lrwxrwxrwx 1 root root 5 Aug 31 16:24 Loopback_4 -> card4

Thanks

Hi @Anshul_Sehgal I have tried to use your jibri.yml but when trying to start the docker it is giving below error.
Command to start the docker
docker-compose -f jibri.yml up -d --scale jibri_instance_1_jibri1_1=2
It is giving below error.
Starting jibri_instance_1_jibri1_1 … error

ERROR: for jibri_instance_1_jibri1_1 Cannot start service jibri1: OCI runtime create failed: container_linux.go:349: starting container process caused “process_linux.go:449: container init caused “rootfs_linux.go:58: mounting \”/root/config/.asoundrc2\” to rootfs \"/var/lib/docker/overlay2/bc9ef835c324bfe0aaa025488ebf94e1259bebcc775029c7bc5ca6d2c9cd590e/merged\" at \"/var/lib/docker/overlay2/bc9ef835c324bfe0aaa025488ebf94e1259bebcc775029c7bc5ca6d2c9cd590e/merged/home/jibri/.asoundrc\" caused \“not a directory\”"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

ERROR: for jibri1 Cannot start service jibri1: OCI runtime create failed: container_linux.go:349: starting container process caused “process_linux.go:449: container init caused “rootfs_linux.go:58: mounting \”/root/config/.asoundrc2\” to rootfs \"/var/lib/docker/overlay2/bc9ef835c324bfe0aaa025488ebf94e1259bebcc775029c7bc5ca6d2c9cd590e/merged\" at \"/var/lib/docker/overlay2/bc9ef835c324bfe0aaa025488ebf94e1259bebcc775029c7bc5ca6d2c9cd590e/merged/home/jibri/.asoundrc\" caused \“not a directory\”"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
ERROR: Encountered errors while bringing up the project.

Could you please help me in this regards?

This is my jibri.yml file
version: ‘3’

services:
jibri1:
image: jitsi/jibri
volumes:
- ./config:/config
- /dev/shm:/dev/shm
# - /root/config/.asoundrc2:/home/jibri/.asoundrc
extra_hosts:
- “xmppdemo.tcgdigital.com:15.206.177.251”
cap_add:
- SYS_ADMIN
- NET_BIND_SERVICE
devices:
- /dev/snd:/dev/snd
environment:
- XMPP_AUTH_DOMAIN
- XMPP_INTERNAL_MUC_DOMAIN
- XMPP_RECORDER_DOMAIN
- XMPP_SERVER
- XMPP_DOMAIN
- JIBRI_XMPP_USER
- JIBRI_XMPP_PASSWORD
- JIBRI_BREWERY_MUC
- JIBRI_RECORDER_USER
- JIBRI_RECORDER_PASSWORD
- JIBRI_RECORDING_DIR
- JIBRI_FINALIZE_RECORDING_SCRIPT_PATH
- JIBRI_STRIP_DOMAIN_JID
- JIBRI_LOGS_DIR
- DISPLAY=:0
- TZ

Hi,

Scaling will cause error till you find a way to change the asoundrc at the time of scaling. Thats why I used separate mount file for each container. The error you are getting is related to mount.(“Are you trying to mount a directory onto a file (or vice-versa”)

As you suggested that your recording stops after 2 mins, do you get recording file of that 2 mins or it remains in some kind of stuck state for that span.

Are the resources(CPU, RAM) enough on the server where you are initiating multi Jibris. Because running multi jibri will put extra load.

If you are getting recording for initial time, most probably resource constraint could cause recording to stop. If recording is not there, then you should check Prosody and Jibri logs to identify that your 2nd Jibri instance is connected well with other components or not.

Thanks @Anshul_Sehgal for you reply.

In this case to start the docker if I use “docker-compose -f jibri.yml up -d” command then it will work?

The file is there, but when I am trying to play that file getting file corrupted error. 2nd instance of Jibri automatically disconnected and recording got stopped.

I am using CPU Intel® Xeon® CPU E5-2620 v2 @ 2.10GHz and 8GB RAM.

Please suggest how can I resolve that issue.
Thanks,

Hi,

Yes. As you will get option to load dedicated asoundrc file to each container.

Jibri standard requirement is 4GB RAM. Not sure if you can test all on mentioned server.

Ideally you should check logs of jibri/jicofo when you face the issue. It may give some insight. Based on the logs output you can search for any similar reported issue.

It’s a good idea to test your jibri docker setup on a bigger box. I tried a DigitalOcean 4vCPUs with 8Gb of RAM and barely got running 4 Jibri docker instances per box. Understand than Jibri is a very CPU intensive task, so do not expect to expect to get too much of your server.

Also try to check your CPU and RAM consumption when your 2nd jibri crashes.

Thanks @hypnotic-frog, @Anshul_Sehgal could you please let me know what is your chrome version for jibri.

Thanks in advance, I am still stuck with that issue.