Failed to start a Jibri session, no Jibris available

Hello,

I setup jibri in its own server to record jitsi meetings but I haven’t been able to make it work still. I have read some threads about this topic but I haven’t been able to solve this.

Jibri starts and connects to Jitsi as you can see in these log registers:

# Jibri log (server 2)
2021-02-11 10:34:53.936 INFO: [51] org.jitsi.xmpp.mucclient.MucClient.log() Connected.
2021-02-11 10:34:54.011 INFO: [51] org.jitsi.xmpp.mucclient.MucClient.log() Leaving a MUC we already occupy.
2021-02-11 10:34:54.040 INFO: [51] org.jitsi.xmpp.mucclient.MucClient.log() Joined MUC: jibribrewery@internal.auth.call.mydomain.com
2021-02-11 10:35:48.679 FINE: [18] org.jitsi.jibri.webhooks.v1.WebhookClient.invokeSuspend() Updating 0 subscribers of status
../..

# prosody log (server 1)
Feb 11 11:52:50 c2s55ce9534f200 info    Client connected
Feb 11 11:52:50 c2s55ce9534f200 info    Stream encrypted (TLSv1.3 with TLS_AES_256_GCM_SHA384)
Feb 11 11:52:50 c2s55ce9534f200 info    Authenticated as jibri@auth.call.mydomain.com

# jicofo log (server 1)
Jicofo 2021-02-11 11:55:04.305 SEVERE: [148] org.jitsi.jicofo.recording.jibri.JibriSession.log() Unable to find an available Jibri, can't start
Jicofo 2021-02-11 11:55:04.305 INFO: [148] org.jitsi.jicofo.recording.jibri.JibriRecorder.log() Failed to start a Jibri session, no Jibris available

In jibri there are two users: jibri itself and recorder. Whereas the jibri user is logged in prosody, there is no trace of recorder doing the same thing. I don’t know whether this can be a problem. I don’t know what to do, so can I pick your brains to solve this issue?

Regards,

Hello @emrah,

I have double checked the jibri.conf configuration file. I had some mistakes, because I copied the xmpp environments section from an example which used the old syntax from config.json, but I have fixed them all. I had another mistake in the jicofo sip-comunicator.properties file, the line org.jitsi.jicofo.jibri.BREWERY=JibriBrewery@internal.call.mydomain.com must be org.jitsi.jicofo.jibri.BREWERY=JibriBrewery@internal.auth.call.mydomain.com instead (note the missing auth word before the XMPP server name).

On fixing the errors brought up above, the recording started, but it stopped a few seconds later. There was a problem because the jibri server couldn’t connect to the jitsi server over the HTTPS port. It was related to my setup, which uses linux containers. The jibri container was trying to connect directly to the jitsi one, but the last one expected a proxy protocol connection.

Now I’m facing another error with ffmpeg:

2021-02-11 16:15:50.221 INFO: [58] ffmpeg.call() ALSA lib pcm_direct.c:1824:(_snd_pcm_direct_get_slave_ipc_offset) Invalid value for card
2021-02-11 16:15:50.221 INFO: [58] ffmpeg.call() [alsa @ 0x5643a759d9c0] cannot open audio device plug:bsnoop (No such device)
2021-02-11 16:15:50.221 INFO: [58] ffmpeg.call() plug:bsnoop: Input/output error

Jibri is running in a linux container, but I loaded the snd-aloop kernel module in the host:

snd_aloop              28672  0
snd_pcm               114688  1 snd_aloop
snd_timer              36864  1 snd_pcm
sunrpc                425984  1
snd                    94208  3 snd_timer,snd_aloop,snd_pcm
soundcore              16384  1 snd

jibri user in the container belongs to the audio group:

jibri : jibri adm audio video plugdev jitsi

Regards,

Is there a .asoundrc file in the /home/jibri folder?

Yeah, here it is the file:

root@recorder:~# cat /home/jibri/.asoundrc
pcm.amix {
  type dmix
  ipc_key 219345
  slave.pcm "hw:Loopback,0,0"
}

pcm.asnoop {
  type dsnoop
  ipc_key 219346
  slave.pcm "hw:Loopback_1,1,0"
}

pcm.aduplex {
  type asym
  playback.pcm "amix"
  capture.pcm "asnoop"
}

pcm.bmix {
  type dmix
  ipc_key 219347
  slave.pcm "hw:Loopback_1,0,0"
}

pcm.bsnoop {
  type dsnoop
  ipc_key 219348
  slave.pcm "hw:Loopback,1,0"
}

pcm.bduplex {
  type asym
  playback.pcm "bmix"
  capture.pcm "bsnoop"
}

pcm.pjsua {
  type plug
  slave.pcm "bduplex"
}

pcm.!default {
  type plug
  slave.pcm "aduplex"
}

If I run aplay -l inside the container I get no devices:

root@recorder:~# su - jibri
$ whoami
jibri
$ aplay -l
aplay: device_list:272: no soundcards found...

Running the same command in the host:

$ aplay -l
**** List of PLAYBACK Hardware Devices ****
card 0: Loopback [Loopback], device 0: Loopback PCM [Loopback PCM]
  Subdevices: 8/8
  Subdevice #0: subdevice #0
  Subdevice #1: subdevice #1
  Subdevice #2: subdevice #2
  Subdevice #3: subdevice #3
  Subdevice #4: subdevice #4
  Subdevice #5: subdevice #5
  Subdevice #6: subdevice #6
  Subdevice #7: subdevice #7
card 0: Loopback [Loopback], device 1: Loopback PCM [Loopback PCM]
  Subdevices: 8/8
  Subdevice #0: subdevice #0
  Subdevice #1: subdevice #1
  Subdevice #2: subdevice #2
  Subdevice #3: subdevice #3
  Subdevice #4: subdevice #4
  Subdevice #5: subdevice #5
  Subdevice #6: subdevice #6
  Subdevice #7: subdevice #7

If this is an LXC then add the following line in the container config

lxc.cgroup.devices.allow = c 116:* rwm
lxc.mount.entry = /dev/snd dev/snd none bind,optional,create=dir

Hello @emrah ,

Thank you very much, I owe you one!

I added the lines you mentioned to the container using these commands:

lxc config device add mycontainer dev_snd disk source=/dev/snd raw.mount.options="bind,optional,create=dir" path="/dev/snd"
lxc config set mycontainer raw.lxc="lxc.cgroup.devices.allow = c 116:* rwm"

The /dev/snd/ char devices where mounted in the container with read and write permissions just for nobody.nogroup, so the jibri user did not have permissión to read/write from/to them. Despite the fact that I added jibri user to the nogroup group, he could not access to the sound devices, so I had to change the permissions for this devices in the host:

user@host:~$ sudo chmod 666 /dev/snd/pcm* /dev/snd/seq /dev/snd/timer
user@host:~$ ls -l /dev/snd/
total 0
total 0
drwxr-xr-x 2 root root       60 Feb 11 17:45 by-path
crw-rw-rw- 1 root audio 116,  6 Feb 11 17:45 controlC0
crw-rw-rw- 1 root audio 116,  3 Feb 11 17:45 pcmC0D0c
crw-rw-rw- 1 root audio 116,  2 Feb 12 10:18 pcmC0D0p
crw-rw-rw- 1 root audio 116,  5 Feb 12 10:17 pcmC0D1c
crw-rw-rw- 1 root audio 116,  4 Feb 11 17:45 pcmC0D1p
crw-rw-rw- 1 root audio 116,  1 Feb 11 17:45 seq
crw-rw-rw- 1 root audio 116, 33 Feb 11 17:45 timer

In the container they are shown like this:

root@mycontainer:~# ls -l /dev/snd/
total 0
drwxr-xr-x 2 nobody nogroup      60 Feb 11 17:45 by-path
crw-rw-rw- 1 nobody nogroup 116,  6 Feb 11 17:45 controlC0
crw-rw-rw- 1 nobody nogroup 116,  3 Feb 11 17:45 pcmC0D0c
crw-rw-rw- 1 nobody nogroup 116,  2 Feb 12 10:18 pcmC0D0p
crw-rw-rw- 1 nobody nogroup 116,  5 Feb 12 10:17 pcmC0D1c
crw-rw-rw- 1 nobody nogroup 116,  4 Feb 11 17:45 pcmC0D1p
crw-rw-rw- 1 nobody nogroup 116,  1 Feb 11 17:45 seq
crw-rw-rw- 1 nobody nogroup 116, 33 Feb 11 17:45 timer

I would prefer not to allow anybody to read and write to these devices, but I have not found any other way to allow the jibri user to access the sound devices. I would be grateful if anybody gives me some hint to fix this. Anyway, the recording is working now!

2021-02-12 10:17:55.603 INFO: [76] ffmpeg.call() Input #0, x11grab, from ':0.0+0,0':
2021-02-12 10:17:55.603 INFO: [76] ffmpeg.call()   Duration: N/A, start: 1613121475.548416, bitrate: N/A
2021-02-12 10:17:55.603 INFO: [76] ffmpeg.call()     Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1280x720, 30 fps, 1000k tbr, 1000k tbn, 1000k tbc
2021-02-12 10:17:55.603 INFO: [76] ffmpeg.call() Guessed Channel Layout for Input Stream #1.0 : stereo
2021-02-12 10:17:55.682 INFO: [76] ffmpeg.call() Input #1, alsa, from 'plug:bsnoop':
2021-02-12 10:17:55.683 INFO: [76] ffmpeg.call()   Duration: N/A, start: 1613121475.213969, bitrate: 1536 kb/s
2021-02-12 10:17:55.683 INFO: [76] ffmpeg.call()     Stream #1:0: Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s
2021-02-12 10:17:55.683 INFO: [76] ffmpeg.call() Stream mapping:
2021-02-12 10:17:55.683 INFO: [76] ffmpeg.call()   Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
2021-02-12 10:17:55.683 INFO: [76] ffmpeg.call()   Stream #1:0 -> #0:1 (pcm_s16le (native) -> aac (native))

Hope this helps.

Regards,

Dont change the devices permissions. Add the jibri user to audio group

usermod -aG adm,audio,video,plugdev jibri

Hello,

I have done it already, but it did not work:

root@mycontainer:/var/log/jitsi/jibri# groups jibri
jibri : jibri adm audio video plugdev nogroup jitsi

The sound devices in the container are owned by nobody.nogroup and I could not change it to root.audio like in the host.

root@mycontainer:/var/log/jitsi/jibri# ls -l /dev/snd/
total 0
drwxr-xr-x 2 nobody nogroup      60 Feb 11 17:45 by-path
crw-rw-rw- 1 nobody nogroup 116,  6 Feb 11 17:45 controlC0
crw-rw-rw- 1 nobody nogroup 116,  3 Feb 11 17:45 pcmC0D0c
crw-rw-rw- 1 nobody nogroup 116,  2 Feb 12 11:37 pcmC0D0p
crw-rw-rw- 1 nobody nogroup 116,  5 Feb 12 10:17 pcmC0D1c
crw-rw-rw- 1 nobody nogroup 116,  4 Feb 11 17:45 pcmC0D1p
crw-rw-rw- 1 nobody nogroup 116,  1 Feb 11 17:45 seq
crw-rw-rw- 1 nobody nogroup 116, 33 Feb 11 17:45 timer

The only option I found to allow jibri user to access these devices were to make them readable/writable by everyone.

By the way, in spite of the fact that I was able to make a recording, now it has stopped working again. I am reading the logs to find what is wrong.

Regards,

Is your container ephemeral?

No, as far as I know it is not ephemeral. I created it using lxc launch images:debian/10 mycontainer -p jibri.

then if you restarted the Jibri service after the usermod command, it should work

Hi!

Ok, I think I made it. I just needed to map the audio gid on the host to the container, so this made the trick:

 printf "gid 29 29"  | lxc config set recorder raw.idmap -

With that command, the guid 29 (audio) is directly mapped between the host and the container, so guid 29 in the container is also guid 29 in the host. I’m not sure if this can pose a security risk though.

root@container:~# ls -l /dev/snd/
total 0
drwxr-xr-x 2 nobody nogroup      60 Feb 12 19:01 by-path
crw-rw---- 1 nobody audio   116,  6 Feb 12 19:01 controlC0
crw-rw---- 1 nobody audio   116,  3 Feb 12 19:01 pcmC0D0c
crw-rw---- 1 nobody audio   116,  2 Feb 17 13:59 pcmC0D0p
crw-rw---- 1 nobody audio   116,  5 Feb 17 13:59 pcmC0D1c
crw-rw---- 1 nobody audio   116,  4 Feb 12 19:01 pcmC0D1p
crw-rw---- 1 nobody audio   116,  1 Feb 12 19:01 seq
crw-rw---- 1 nobody audio   116, 33 Feb 12 19:01 timer

Here is the documentation on this issue: Idmaps for user namespace - LXD - system container manager