JIbri on Kubernetes | Multiple Recording

Hello

I have setup Jibri on Kubernetes. One container is doing just okay.
If I add multiple containers and record only one container is doing fine.
Other container can’t record multiple sessions

The logs I get is in another container are

020-05-11 11:24:37.319 INFO: [57] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.launch() Starting ffmpeg with command ffmpeg -y -v info -f x11grab -draw_mouse 0 -r 30 -s 1280x720 -thread_queue_size 4096 -i :0.0+0,0 -f alsa -thread_queue_size 4096 -i plug:cloop -acodec aac -strict -2 -ar 44100 -c:v libx264 -preset veryfast -profile:v main -level 3.1 -pix_fmt yuv420p -r 30 -crf 25 -g 60 -tune zerolatency -f mp4 /config/recordings/domscmwyypttlrsl/ck_2020-05-11-11-24-33.mp4 ([ffmpeg, -y, -v, info, -f, x11grab, -draw_mouse, 0, -r, 30, -s, 1280x720, -thread_queue_size, 4096, -i, :0.0+0,0, -f, alsa, -thread_queue_size, 4096, -i, plug:cloop, -acodec, aac, -strict, -2, -ar, 44100, -c:v, libx264, -preset, veryfast, -profile:v, main, -level, 3.1, -pix_fmt, yuv420p, -r, 30, -crf, 25, -g, 60, -tune, zerolatency, -f, mp4, /config/recordings/domscmwyypttlrsl/ck_2020-05-11-11-24-33.mp4])
2020-05-11 11:24:42.328 FINE: [58] org.jitsi.jibri.util.ProcessStatePublisher.ffmpeg.invoke() Process ffmpeg hasn’t written in 2 seconds, publishing periodic update
2020-05-11 11:24:42.330 INFO: [61] org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.onFfmpegProcessUpdate() Ffmpeg quit abruptly. Last output line: plug:cloop: Input/output error
2020-05-11 11:24:42.330 INFO: [61] org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.onFfmpegStateMachineStateChange() Ffmpeg capturer transitioning from state Starting up to Error: SESSION Ffmpeg failed to start
2020-05-11 11:24:42.331 INFO: [61] org.jitsi.jibri.service.impl.FileRecordingJibriService.onServiceStateChange() File recording service transitioning from state Starting up to Error: SESSION Ffmpeg failed to start
2020-05-11 11:24:42.332 INFO: [61] org.jitsi.jibri.api.xmpp.XmppApi.invoke() Current service had an error, sending error iq
2020-05-11 11:24:42.333 FINE: [61] org.jitsi.jibri.statsd.JibriStatsDClient.incrementCounter() Incrementing statsd counter: stop:recording
2020-05-11 11:24:42.333 INFO: [61] org.jitsi.jibri.JibriManager.stopService() Stopping the current service
2020-05-11 11:24:42.334 INFO: [61] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Stopping capturer
2020-05-11 11:24:42.334 INFO: [61] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() Stopping ffmpeg process
2020-05-11 11:24:42.338 INFO: [61] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() ffmpeg exited with value 1
2020-05-11 11:24:42.338 INFO: [61] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Quitting selenium
2020-05-11 11:24:42.370 INFO: [61] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Participants in this recording:
2020-05-11 11:24:42.448 INFO: [61] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 16 log entries for type browser
2020-05-11 11:24:42.525 INFO: [61] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 624 log entries for type driver
2020-05-11 11:24:42.719 INFO: [61] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 0 log entries for type client
2020-05-11 11:24:42.720 INFO: [61] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Leaving web call
2020-05-11 11:24:42.883 INFO: [61] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Quitting chrome driver
2020-05-11 11:24:42.959 INFO: [61] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Chrome driver quit
2020-05-11 11:24:42.960 INFO: [61] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Finalizing the recording
2020-05-11 11:24:42.963 INFO: [61] org.jitsi.jibri.service.impl.FileRecordingJibriService.finalize() Recording finalize script finished with exit value 0
2020-05-11 11:24:42.964 INFO: [61] org.jitsi.jibri.status.JibriStatusManager.log() Busy status has changed: BUSY -> IDLE

Solved the issue by

Just change the Loopback in /home/jibri/.asoundrc to anyname you like
for example

sed -i ‘s/Loopback/Loopback-1/g’ /home/jibri/.asoundrc`

Make sure you also restart the instance after this
and it’s working good

2 Likes

Is it possible you make a guide explaining how to create this multiple recording using kubernets?

This is an amazing solution, im planning to create several virtual machines on my environment for extra jibri servers but this seems to be much better.

Sure
I will I am too working on this project.
WIll post when completes

Thanks
Manish

1 Like

Hi,
while i am deploying jibri on kubernetes getting errors
ERROR: Please load snd-aloop module on the docker host.
ERROR: Binding /dev/snd is not found. Please check that you run docker-compose with -f jibri.yml.

Please help me to resolve this problem.And also please share jibri-deployment.yaml file
Thanks.

Use the link
use a1-statefulset.yaml and a2-statefulset.yaml files for pods deployment.

1 Like

Hi,
I have deployed the same statefulsets as you mentioned abovein hit repo.The only thing I have changed is instead of using nfs persistant volume Iam using aws EBS pv, but still iam getting errors. below are the logs of pod

root@kubectl:~/k8s-jibri# kubectl logs -f jibri-one-0 init-set
Copying files
0
root@kubectl:~/k8s-jibri# kubectl logs -f jibri-one-0 jibri
[s6-init] making user provided files available at /var/run/s6/etc…exited 0.
[s6-init] ensuring user provided files have correct perms…exited 0.
[fix-attrs.d] applying ownership & permissions fixes…
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts…
[cont-init.d] 01-set-timezone: executing…
[cont-init.d] 01-set-timezone: exited 0.
[cont-init.d] 10-config: executing…
ERROR: Please load snd-aloop module on the docker host.
ERROR: Binding /dev/snd is not found. Please check that you run docker-compose with -f jibri.yml.
Usage: usermod [options] LOGIN

Options:
-c, --comment COMMENT new value of the GECOS field
-d, --home HOME_DIR new home directory for the user account
-e, --expiredate EXPIRE_DATE set account expiration date to EXPIRE_DATE
-f, --inactive INACTIVE set password inactive after expiration
to INACTIVE
-g, --gid GROUP force use GROUP as new primary group
-G, --groups GROUPS new list of supplementary GROUPS
-a, --append append the user to the supplemental GROUPS
mentioned by the -G option without removing
him/her from other groups
-h, --help display this help message and exit
-l, --login NEW_LOGIN new value of the login name
-L, --lock lock the user account
-m, --move-home move contents of the home directory to the
new location (use only with -d)
-o, --non-unique allow using duplicate (non-unique) UID
-p, --password PASSWORD use encrypted password for the new password
-R, --root CHROOT_DIR directory to chroot into
-s, --shell SHELL new login shell for the user account
-u, --uid UID new UID for the user account
-U, --unlock unlock the user account
-v, --add-subuids FIRST-LAST add range of subordinate uids
-V, --del-subuids FIRST-LAST remove range of subordinate uids
-w, --add-subgids FIRST-LAST add range of subordinate gids
-W, --del-subgids FIRST-LAST remove range of subordinate gids
-Z, --selinux-user SEUSER new SELinux user mapping for the user account

[cont-finish.d] executing container finish scripts…
[cont-finish.d] done.
[s6-finish] waiting for services.
[cont-init.d] 10-config: exited 0.
[cont-init.d] done.
[services.d] starting services
s6-svscanctl: fatal: unable to control /var/run/s6/services: supervisor not listening
s6-svscanctl: fatal: unable to control /var/run/s6/services: supervisor not listening
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.

Please help me to resolve this error.
Thanks

Which os-release verison you are using?
And what is tha output of

aplay -l

Hi,
Kubernetes node OS version
[ec2-user@ip-172-31-46-32 ~]$ uname -a
Linux ip-172-31-46-32.ap-south-1.compute.internal 4.14.177-139.254.amzn2.x86_64 #1 SMP Thu May 7 18:48:23 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

JIBRI pod details
jibri@jibri-8bd4d7f96-4n5jt:/ aplay -l aplay: device_list:268: no soundcards found... jibri@jibri-8bd4d7f96-4n5jt:/ arecord -l
arecord: device_list:268: no soundcards found…
jibri@jibri-8bd4d7f96-4n5jt:/ uname -a Linux jibri-8bd4d7f96-4n5jt 4.14.177-139.254.amzn2.x86_64 #1 SMP Thu May 7 18:48:23 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux jibri@jibri-8bd4d7f96-4n5jt:/

I have some doubts to ask you. Please let me know,

  1. Installation of linux-image-extra-virtual & alsa-utils is necessary.
  2. Instead of installing those packages in node, can we install in pod

jibri@jibri-8bd4d7f96-4n5jt:/$ lsmod | grep snd_aloop
output is empty

When I start the recording after a while recording is stopping with below logs of jibri pod

2020-05-28 11:35:35.482 INFO: [19] org.jitsi.xmpp.mucclient.MucClient.log() [MucClient id=10.100.198.14 hostname=10.100.198.14] connected
2020-05-28 11:35:35.684 INFO: [19] org.jitsi.xmpp.mucclient.MucClient.log() Joined MUC: jibribrewery@internal.auth.jitsi.devopsk8s.cf

2020-05-28 13:23:00.818 INFO: [245] org.jitsi.jibri.api.xmpp.XmppApi.handleJibriIq() Received JibriIq from environment [MucClient id=10.100.198.14 hostname=10.100.198.14]
2020-05-28 13:23:00.819 INFO: [245] org.jitsi.jibri.api.xmpp.XmppApi.handleStartJibriIq() Received start request
2020-05-28 13:23:00.825 INFO: [245] org.jitsi.jibri.api.xmpp.XmppApi.handleStartJibriIq() Sending ‘pending’ response to start IQ
2020-05-28 13:23:00.825 INFO: [246] org.jitsi.jibri.api.xmpp.XmppApi.run() Starting service
2020-05-28 13:23:00.862 INFO: [246] org.jitsi.jibri.api.xmpp.XmppApi.handleStartService() Parsed call url info: CallUrlInfo(baseUrl=https://jitsi.devopsk8s.cf, callName=jitsi, urlParams=)
2020-05-28 13:23:00.863 INFO: [246] org.jitsi.jibri.JibriManager.startFileRecording() Starting a file recording with params: FileRecordingRequestParams(callParams=CallParams(callUrlInfo=CallUrlInfo(baseUrl=https://jitsi.devopsk8s.cf, callName=jitsi, urlParams=)), sessionId=xzcbmjspqtebqrcb, callLoginParams=XmppCredentials(domain=recorder.jitsi.devopsk8s.cf, username=recorder, password=password)) finalize script path: /opt/jibri/resources/finalize_recording.sh and recordings directory: /tmp/recordings
2020-05-28 13:23:03.183 INFO: [246] org.openqa.selenium.remote.ProtocolHandshake.createSession() Detected dialect: OSS
2020-05-28 13:23:03.251 INFO: [246] org.jitsi.jibri.selenium.JibriSelenium.() Starting empty call check with a timeout of PT30S
2020-05-28 13:23:03.268 FINE: [246] org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.() Detected os as OS: LINUX
2020-05-28 13:23:03.273 INFO: [246] org.jitsi.jibri.service.impl.FileRecordingJibriService.() Writing recording to /tmp/recordings/xzcbmjspqtebqrcb
2020-05-28 13:23:03.275 FINE: [246] org.jitsi.jibri.statsd.JibriStatsDClient.incrementCounter() Incrementing statsd counter: start:recording
2020-05-28 13:23:03.276 INFO: [246] org.jitsi.jibri.status.JibriStatusManager.log() Busy status has changed: IDLE -> BUSY
2020-05-28 13:23:03.276 INFO: [246] org.jitsi.jibri.api.xmpp.XmppApi.updatePresence() Jibri reports its status is now JibriStatus(busyStatus=BUSY, health=OverallHealth(healthStatus=HEALTHY, details={})), publishing presence to connections
2020-05-28 13:23:03.277 INFO: [246] org.jitsi.xmpp.mucclient.MucClientManager.log() Setting a presence extension: org.jitsi.xmpp.extensions.jibri.JibriStatusPacketExt@4ca8e0a5
2020-05-28 13:23:04.855 FINE: [256] org.jitsi.jibri.selenium.pageobjects.CallPage.visit() Visiting url https://jitsi.devopsk8s.cf/jitsi#config.iAmRecorder=true&config.externalConnectUrl=null&config.startWithAudioMuted=true&config.startWithVideoMuted=true&interfaceConfig.APP_NAME=“Jibri”&config.analytics.disabled=true&config.p2p.enabled=false
2020-05-28 13:23:07.213 INFO: [256] org.jitsi.jibri.selenium.pageobjects.CallPage.visit() Waited 1161 milliseconds for call page to load
2020-05-28 13:23:07.270 INFO: [256] org.jitsi.jibri.selenium.JibriSelenium.onSeleniumStateChange() Transitioning from state Starting up to Running
2020-05-28 13:23:07.271 INFO: [256] org.jitsi.jibri.service.impl.FileRecordingJibriService.invoke() Selenium joined the call, starting the capturer
2020-05-28 13:23:07.373 INFO: [256] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.launch() Starting ffmpeg with command ffmpeg -y -v info -f x11grab -draw_mouse 0 -r 30 -s 1280x720 -thread_queue_size 4096 -i :0.0+0,0 -f alsa -thread_queue_size 4096 -i plug:cloop -acodec aac -strict -2 -ar 44100 -c:v libx264 -preset veryfast -profile:v main -level 3.1 -pix_fmt yuv420p -r 30 -crf 25 -g 60 -tune zerolatency -f mp4 /tmp/recordings/xzcbmjspqtebqrcb/jitsi_2020-05-28-13-23-03.mp4 ([ffmpeg, -y, -v, info, -f, x11grab, -draw_mouse, 0, -r, 30, -s, 1280x720, -thread_queue_size, 4096, -i, :0.0+0,0, -f, alsa, -thread_queue_size, 4096, -i, plug:cloop, -acodec, aac, -strict, -2, -ar, 44100, -c:v, libx264, -preset, veryfast, -profile:v, main, -level, 3.1, -pix_fmt, yuv420p, -r, 30, -crf, 25, -g, 60, -tune, zerolatency, -f, mp4, /tmp/recordings/xzcbmjspqtebqrcb/jitsi_2020-05-28-13-23-03.mp4])
2020-05-28 13:23:12.445 FINE: [257] org.jitsi.jibri.util.ProcessStatePublisher.ffmpeg.invoke() Process ffmpeg hasn’t written in 2 seconds, publishing periodic update
2020-05-28 13:23:12.447 INFO: [261] org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.onFfmpegProcessUpdate() Ffmpeg quit abruptly. Last output line: plug:cloop: Input/output error
2020-05-28 13:23:12.448 INFO: [261] org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.onFfmpegStateMachineStateChange() Ffmpeg capturer transitioning from state Starting up to Error: SESSION Ffmpeg failed to start
2020-05-28 13:23:12.449 INFO: [261] org.jitsi.jibri.service.impl.FileRecordingJibriService.onServiceStateChange() File recording service transitioning from state Starting up to Error: SESSION Ffmpeg failed to start
2020-05-28 13:23:12.449 INFO: [261] org.jitsi.jibri.api.xmpp.XmppApi.invoke() Current service had an error, sending error iq
2020-05-28 13:23:12.450 FINE: [261] org.jitsi.jibri.statsd.JibriStatsDClient.incrementCounter() Incrementing statsd counter: stop:recording
2020-05-28 13:23:12.450 INFO: [261] org.jitsi.jibri.JibriManager.stopService() Stopping the current service
2020-05-28 13:23:12.451 INFO: [261] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Stopping capturer
2020-05-28 13:23:12.451 INFO: [261] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() Stopping ffmpeg process
2020-05-28 13:23:12.478 INFO: [261] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() ffmpeg exited with value 1
2020-05-28 13:23:12.478 INFO: [261] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Quitting selenium
2020-05-28 13:23:12.571 INFO: [261] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Participants in this recording:
2020-05-28 13:23:13.088 INFO: [261] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 15 log entries for type browser
2020-05-28 13:23:13.655 INFO: [261] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 630 log entries for type driver
2020-05-28 13:23:14.458 INFO: [261] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 0 log entries for type client
2020-05-28 13:23:14.459 INFO: [261] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Leaving web call
2020-05-28 13:23:14.884 INFO: [261] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Quitting chrome driver
2020-05-28 13:23:15.010 INFO: [261] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Chrome driver quit
2020-05-28 13:23:15.010 INFO: [261] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Finalizing the recording
2020-05-28 13:23:15.014 INFO: [261] org.jitsi.jibri.service.impl.FileRecordingJibriService.finalize() Recording finalize script finished with exit value 0
2020-05-28 13:23:15.014 INFO: [261] org.jitsi.jibri.status.JibriStatusManager.log() Busy status has changed: BUSY -> IDLE
2020-05-28 13:23:15.014 INFO: [261] org.jitsi.jibri.api.xmpp.XmppApi.updatePresence() Jibri reports its status is now JibriStatus(busyStatus=IDLE, health=OverallHealth(healthStatus=HEALTHY, details={})), publishing presence to connections
2020-05-28 13:23:15.015 INFO: [261] org.jitsi.xmpp.mucclient.MucClientManager.log() Setting a presence extension: org.jitsi.xmpp.extensions.jibri.JibriStatusPacketExt@425674bd

What I have to do. Please give me any suggestion.
Thanks.

Your sound module is not loaded properly. Guess you have used AWS services
You can add me on skype: itsmaniche18
We can have a conversation there

Thanks
Manish

Hello again @itsmaniche18,

Have you done succesfully autoscaling videobridge JVB2 with Kubernetes too?

Regards,
Harijanto

Hi,
Yes, I am using Aws eks for Kubernetes. I successfully deployed jitsi meet containers(web,prosody,jicofo,jvb) including jibri on ubuntu machine by setting up snd-aloop module in it. But in case of kubernetes aws providing only one AMI for nodes that is Amazon linux 2. In eks there is no option to use our own custom ami. So Iam unable to setup snd-aloop module in amazon linux 2 machine.
Any way now I got the answer that, why I am getting those errors in jibri logs.
Thank you very much for your quick reply…

@ramalingam_varkala I installed jibri on kubernetes locally behind a NAT and I have exactly the same error as you. Do you find a solution?

It’s good for me, jibri works. The problem was that not all audio settings were charged. The solution for me was to add the following to in /home/jibri/.asoundrc

pcm.cloop {

type dsnoop

ipc_key 219348

slave.pcm “hw:Loopback,1,0”

}

Hello! I’m new here. I need help installing Jibri with Kubernetes. Would anyone have the step by step how to install using Kubernetes?

Use the link
use a1-statefulset.yaml and a2-statefulset.yaml files for pods deployment.

Hey! Do you mind to extend the README a litte to those of us who are kind of noobs using K8s? Thanks!

I had the same problem in several installation attempts (docker, docker swarm, Kubernetes and directly through debian packages).

I solved the problem by changing the kernel, what happens is that Amazon images are not supported by ALSA.

Procedure (Ubuntu 20.04 on AWS)

wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.7.10/amd64/linux-headers-5.7.10-050710_5.7.10-050710.202007220834_all.deb
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.7.10/amd64/linux-image-unsigned-5.7.10-050710-generic_5.7.10-050710.202007220834_amd64.deb
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.7.10/amd64/linux-modules-5.7.10-050710-generic_5.7.10-050710.202007220834_amd64.deb

sudo dpkg -i * .deb

sudo reboot now

Then just follow the procedures for installing the jibri.

Red ‘Leave Room’ Button Result

Wat is about