Multiple jibri instances on docker, the second instance unable to record video

Hai guys, I am deploying multiple Jibri instances on docker using https://github.com/jitsi/docker-jitsi-meet and already follow the official guide properly but still faced an issue when recording in the second instance, for the first instance is working fine. I deployed docker Jibri in google cloud with a compute engine ubuntu server 16.04.
I will explain to you what I have done besides what is in the official guide. I updated jibri.yml to make it standalone. I just removed depends_on Jicofo and networks meet.jitsi. Here is my jibri.yml :

version: '3'

services:
    jibri:
        image: jitsi/jibri
        restart: ${RESTART_POLICY}
        volumes:
            - ${CONFIG}/jibri:/config
            - /dev/shm:/dev/shm
        cap_add:
            - SYS_ADMIN
            - NET_BIND_SERVICE
        devices:
            - /dev/snd:/dev/snd
        environment:
            - XMPP_AUTH_DOMAIN
            - XMPP_INTERNAL_MUC_DOMAIN
            - XMPP_RECORDER_DOMAIN
            - XMPP_SERVER
            - XMPP_DOMAIN
            - JIBRI_XMPP_USER
            - JIBRI_XMPP_PASSWORD
            - JIBRI_BREWERY_MUC
            - JIBRI_RECORDER_USER
            - JIBRI_RECORDER_PASSWORD
            - JIBRI_RECORDING_DIR
            - JIBRI_FINALIZE_RECORDING_SCRIPT_PATH
            - JIBRI_STRIP_DOMAIN_JID
            - JIBRI_LOGS_DIR
            - DISPLAY=:0
            - TZ

I run the first instance using this command :

docker-compose -f jibri.yml up -d

After the first instance goes up, I change /home/jibri/.asoundrc to setup the second instance :

...
slave.pcm "hw:Loopback_1,0,0"
...
slave.pcm "hw:Loopback_1,0,1"
...
slave.pcm "hw:Loopback_1,1,1"
...
slave.pcm "hw:Loopback_1,1,0"
...

And then run the second instance using this command :

docker-compose -f jibri.yml up -d --scale jibri=2

Like I stated before, the first instance is working fine. The video has shown in ~/.jitsi-meet-cfg/jibri/recordings . but as for the second instance got this error on log.0.txt.1 :

2020-04-27 06:11:18.607 INFO: [59] org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.onFfmpegProcessUpdate() Ffmpeg quit abruptly. Last output line: plug:cloop: Input/output error
2020-04-27 06:11:18.609 INFO: [59] org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.onFfmpegStateMachineStateChange() Ffmpeg capturer transitioning from state Starting up to Error: SESSION Ffmpeg failed to start
2020-04-27 06:11:18.611 INFO: [59] org.jitsi.jibri.service.impl.FileRecordingJibriService.onServiceStateChange() File recording service transitioning from state Starting up to Error: SESSION Ffmpeg failed to start
2020-04-27 06:11:18.612 INFO: [59] org.jitsi.jibri.api.xmpp.XmppApi.invoke() Current service had an error, sending error iq

and here is ffmpeg.0.txt.1 error log :

2020-04-27 06:11:13.769 INFO: [59] ffmpeg.call() Input #0, x11grab, from ‘:0.0+0,0’:
2020-04-27 06:11:13.769 INFO: [59] ffmpeg.call() Duration: N/A, start: 1587960673.724562, bitrate: N/A
2020-04-27 06:11:13.769 INFO: [59] ffmpeg.call() Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1280x720, 30 fps, 1000k tbr, 1000k tbn, 1000k tbc
2020-04-27 06:11:13.769 INFO: [59] ffmpeg.call() ALSA lib pcm_dsnoop.c:618:(snd_pcm_dsnoop_open) unable to open slave
2020-04-27 06:11:13.769 INFO: [59] ffmpeg.call() [alsa @ 0x55f311051e80] cannot open audio device plug:cloop (Device or resource busy)
2020-04-27 06:11:13.769 INFO: [59] ffmpeg.call() plug:cloop: Input/output error

I suspect the Jibri instance doesn’t read .asoundrc file. So all instances still use the same loopback therefore the second instance got the error. I have updated with the wrong one and renamed it to .bsoundrc to make an error. But the first instance is working fine.
Any suggestions to fix this issue?
Thanks in advance.

Hi,

I am also working on running multiple Jibri on single host itself. What I have done after starting first Jibri container, copied its /root/config/.asoundrc on host.

Then I create a separate yaml having 2 other Jibri services. Each of which mounts different .asoundrc from host. After which I tried to record two simultaneous conference, which worked fine for me.Multiple_Jibri.txt (2.1 KB)

asoundrc2.txt (814 Bytes) asoundrc3.txt (814 Bytes)

Hope this will help you.

Can you give more details about this solution in docker. Nobody has achieved this. Thanks in advance!

I have attached my docker-compose file having two JIbri instances and the asoundrc files which they will mount. I thought this will be enough.

Let me know what more details you require.

Thanks for answering! At this point I have tried everything and I’m still unable to get secondary instances to record properly. This is what I did…

  • Set different .asoundrc files to mount on each service asoundrc1.txt (773 Bytes) asoundrc2.txt (773 Bytes)

  • Edited your copy of compose .yml file to add minor changes. In my case a bind mount to get all the recordings to a single directory in the FS. Multiple_Jibri(1).txt (2.0 KB)

  • Got running 1 instance up of each service
    docker-compose -f Multiple_Jibri.txt up -d --scale jibri2=1
    docker-compose -f Multiple_Jibri.txt up -d --scale jibri3=1
    docker-compose -f Multiple_Jibri.txt ps --all

    Name Command State Ports
    docker-jitsi-meet_jibri2_1 /init Up
    docker-jitsi-meet_jibri3_1 /init Up

But once I try to record I get …

log.0.txt

2020-07-28 16:01:33.397 INFO: [91] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.launch() Starting ffmpeg with command ffmpeg -y -v info -f x11grab -draw_mouse 0 -r 30 -s 1280x720 -thread_queue_size 4096 -i :0.0+0,0 -f alsa -thread_queue_size 4096 -i plug:cloop -acodec aac -strict -2 -ar 44100 -c:v libx264 -preset veryfast -profile:v main -level 3.1 -pix_fmt yuv420p -r 30 -crf 25 -g 60 -tune zerolatency -f mp4 [spoiler]/home/user/recordings/vbunxehdivlhmuoa[/spoiler]/testvideorecord12345_2020-07-28-16-01-30.mp4 ([ffmpeg, -y, -v, info, -f, x11grab, -draw_mouse, 0, -r, 30, -s, 1280x720, -thread_queue_size, 4096, -i, :0.0+0,0, -f, alsa, -thread_queue_size, 4096, -i, plug:cloop, -acodec, aac, -strict, -2, -ar, 44100, -c:v, libx264, -preset, veryfast, -profile:v, main, -level, 3.1, -pix_fmt, yuv420p, -r, 30, -crf, 25, -g, 60, -tune, zerolatency, -f, mp4, /home/user/recordings/vbunxehdivlhmuoa/testvideorecord12345_2020-07-28-16-01-30.mp4])
2020-07-28 16:01:38.407 FINE: [92] org.jitsi.jibri.util.ProcessStatePublisher.ffmpeg.invoke() Process ffmpeg hasn't written in 2 seconds, publishing periodic update
2020-07-28 16:01:38.408 INFO: [81] org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.onFfmpegProcessUpdate() Ffmpeg quit abruptly.  Last output line: plug:cloop: Input/output error
2020-07-28 16:01:38.409 INFO: [81] org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.onFfmpegStateMachineStateChange() Ffmpeg capturer transitioning from state Starting up to Error: SESSION Ffmpeg failed to start
2020-07-28 16:01:38.409 INFO: [81] org.jitsi.jibri.service.impl.FileRecordingJibriService.onServiceStateChange() File recording service transitioning from state Starting up to Error: SESSION Ffmpeg failed to start
2020-07-28 16:01:38.410 INFO: [81] org.jitsi.jibri.api.xmpp.XmppApi.invoke() Current service had an error, sending error iq <iq to='jibribrewery@[spoiler]internal.auth.my.domain.com[/spoiler]/focus' id='odUXW-116' type='set'><jibri xmlns='http://jitsi.org/protocol/jibri' status='off' failure_reason='error'/></iq>
2020-07-28 16:01:38.410 FINE: [81] org.jitsi.jibri.statsd.JibriStatsDClient.incrementCounter() Incrementing statsd counter: stop:recording
2020-07-28 16:01:38.411 INFO: [81] org.jitsi.jibri.JibriManager.stopService() Stopping the current service
2020-07-28 16:01:38.411 INFO: [81] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Stopping capturer
2020-07-28 16:01:38.411 INFO: [81] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() Stopping ffmpeg process
2020-07-28 16:01:38.423 INFO: [81] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() ffmpeg exited with value 1

ffmpeg.0.txt

2020-07-28 16:01:33.551 INFO: [96] ffmpeg.call() Input #0, x11grab, from ‘:0.0+0,0’:
2020-07-28 16:01:33.551 INFO: [96] ffmpeg.call() Duration: N/A, start: 1595952093.509353, bitrate: N/A
2020-07-28 16:01:33.552 INFO: [96] ffmpeg.call() Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1280x720, 30 fps, 1000k tbr, 1000k tbn, 1000k tbc
2020-07-28 16:01:33.552 INFO: [96] ffmpeg.call() ALSA lib pcm_direct.c:1605:(_snd_pcm_direct_get_slave_ipc_offset) Invalid value for card
2020-07-28 16:01:33.552 INFO: [96] ffmpeg.call() [alsa @ 0x5591070f04a0] cannot open audio device plug:cloop (No such device)
2020-07-28 16:01:33.554 INFO: [96] ffmpeg.call() plug:cloop: Input/output error

I’m ussing Ubuntu 18 on a VM of 4vCPU / 8Gb RAM
I have also noted that the only container not giving this error and recording properly is the one I let with .asounrc untouched (slave.pcm “hw:Loopback,1,0”…)

I really don’t know whats happening. If you could provide me with details about the OS, Docker version, and jibri image version you use to make this done, I guess that would help too.

What is the output?

ls -alh /proc/asound

Hi @emrah ! Thanks for taking the time to help me.

ls -alh /proc/asound

total 0
dr-xr-xr-x 9 root root 0 Jul 28 21:41 .
dr-xr-xr-x 154 root root 0 Jul 28 21:41 …
lrwxrwxrwx 1 root root 5 Jul 28 21:48 Loopback -> card0
lrwxrwxrwx 1 root root 5 Jul 28 21:48 Loopback_1 -> card1
lrwxrwxrwx 1 root root 5 Jul 28 21:48 Loopback_2 -> card2
lrwxrwxrwx 1 root root 5 Jul 28 21:48 Loopback_3 -> card3
lrwxrwxrwx 1 root root 5 Jul 28 21:48 Loopback_4 -> card4
dr-xr-xr-x 6 root root 0 Jul 28 21:48 card0
dr-xr-xr-x 6 root root 0 Jul 28 21:48 card1
dr-xr-xr-x 6 root root 0 Jul 28 21:48 card2
dr-xr-xr-x 6 root root 0 Jul 28 21:48 card3
dr-xr-xr-x 6 root root 0 Jul 28 21:48 card4
-r–r--r-- 1 root root 0 Jul 28 21:48 cards
-r–r--r-- 1 root root 0 Jul 28 21:48 devices
-r–r--r-- 1 root root 0 Jul 28 21:48 modules
dr-xr-xr-x 2 root root 0 Jul 28 21:48 oss
-r–r--r-- 1 root root 0 Jul 28 21:48 pcm
dr-xr-xr-x 2 root root 0 Jul 28 21:48 seq
-r–r--r-- 1 root root 0 Jul 28 21:48 timers
-r–r--r-- 1 root root 0 Jul 28 21:48 version

It seems that the snd-aloop module and the loopback devices are OK.

Are you sure the container can read the content of asoundrc (maybe a permission problem)?

Could you try to read .asoundrc inside the container as the jibri user

Hi,

I dont know if below info will help you, but I faced some issue like yours, so sharing below info.

I have two setups.

  1. Docker setup where I have tested multiple recordings. This setup uses plug:cloop. I have shared the asoundrc files to you from this setup.
    “jitsi/jibri@sha256:092b097af2b0b8ff2e419c6b3a106e6f39514369642b002add641a3b7d32f638”
    Host OS: Ubuntu 18.04.4 LTS
    Docker version 19.03.11, build 42e35e61f3

  2. K8s setup where I was facing similar issue like yours, but there ffmpeg is using plug:bsnoop. “jitsi/jibri@sha256:87530f48db37fc297a3def8f201fe5a27999ab01b1cac18100ade8abd6562d72”

Working(1st setup)
ffmpeg -y -v info -f x11grab -draw_mouse 0 -r 30 -s 1280x720 -thread_queue_size 4096 -i :0.0+0,0 -f alsa -thread_queue_size 4096 -i plug:cloop -acodec aac -strict -2 -ar 44100 -c:v libx264 -preset veryfast -profile:v main -level 3.1 -pix_fmt yuv420p -r 30 -crf 25 -g 60 -tune zerolatency -f mp4 /config/recordings/wmbzvsooqxorijxu/ongoingeastsinvitesharply_2020-07-21-14-16-36.mp4 ([ffmpeg, -y, -v, info, -f, x11grab, -draw_mouse, 0, -r, 30, -s, 1280x720, -thread_queue_size, 4096, -i, :0.0+0,0, -f, alsa, -thread_queue_size, 4096, -i, plug:cloop, -acodec, aac, -strict, -2, -ar, 44100, -c:v, libx264, -preset, veryfast, -profile:v, main, -level, 3.1, -pix_fmt, yuv420p, -r, 30, -crf, 25, -g, 60, -tune, zerolatency, -f, mp4, /config/recordings/wmbzvsooqxorijxu/ongoingeastsinvitesharply_2020-07-21-14-16-36.mp4])

Non-Working(2nd setup)
ffmpeg -y -v info -f x11grab -draw_mouse 0 -r 30 -s 1280x720 -thread_queue_size 4096 -i :0.0+0,0 -f alsa -thread_queue_size 4096 -i plug:bsnoop -acodec aac -strict -2 -ar 44100 -b:a 128k -af aresample=async=1 -c:v libx264 -preset veryfast -profile:v main -level 3.1 -pix_fmt yuv420p -r 30 -crf 25 -g 60 -tune zerolatency -f mp4 /config/recordings/pkavmirneakhqtyf/classicalentitiesexpressstrictly_2020-07-28-17-37-14.mp4

The logs which you have shared uses cloop as plug. So the asoundrc which I have shared, should not create problem.

For troubleshooting, I went inside container and executed the ffmpeg command which I got in my logs. I changed the plug and it showed me capturing data. You may try same in your container.
e.g. ffmpeg -y -v info -f x11grab -draw_mouse 0 -r 30 -s 1280x720 -thread_queue_size 4096 -i :0.0+0,0 -f alsa -thread_queue_size 4096 -i plug:cloop -acodec aac -strict -2 -ar 44100 -c:v libx264 -preset veryfast -profile:v main -level 3.1 -pix_fmt yuv420p -r 30 -crf 25 -g 60 -tune zerolatency -f mp4 /home/user/recordings/vbunxehdivlhmuoa/testvideorecord12345_2020-07-28-16-01-30.mp4

See if there is some error in it.

For solving my problem on new setup I found here to change the asoundrc file. Direct link is this

After changing this file into my new setup(which uses plug:bsnoop, recording started working. I still need to check multiple recordings in this setup.

Also in new .asoundrc, you will find two capture.pcm, asnoop and bsnoop. May be the new config support both plug and the older supported only asnoop.

If you dont find any solution, I would suggest you to use latest image and change the asoundrc file and then check(As old image may not support the asoundrc file). Also check the permissions of asoundrc file.
You can also try to find the difference in the ffmpeg command which comes in logs of your working(First container) and that on your second container, non working container.

Hey @emrah! Thanks… actually something happened between version tests and it was a problem with the Kernel loading the snd-aloop module… after loading it again it worked.

Odd stuff, because lsmod shown that it was loaded.
Thanks for your help!

Hi @Anshul_Sehgal… thank you for everything. Actually it was a kernel issue… thanks for the detailed instructions anyway. I noticed the change to plug:bsnoop in newer versions of jibri, but looking for an answer to the previous problem I ended with image stable:4627 wich still doesnt have the change.

If anybody struggles with this same problem (cloop or bsnoop) and uses DigitalOcean… double check your module is loaded with ls -alh /proc/asound , as @emrah sugested… and also be sure to install a compatible Kernel correctly before anything else.