Recording stops after 2-3 mins

I have Jitsi and Jibri Docker configured on two different AWS EC2 instances. I tested scaling up Jibri Docker to 6 instances and run parallel recordings but recordings stop after 2-3 mins. Jitsi server’s memory utilisation was around 19% and each Jibri instance was also running at about 20%. As recording is not faily instantly but stopping only after 2-3 mins, could the reason be underpowered server ? If so, which server should we try to upgrade - Jitsi or Jibri ? Appreciate your inputs.

Did you check recorded MP4 files? Is the audio missing in it?

I had just set up the multiple instances. When you asked about audio, I checked the recordings and this is what I found - in about 2mins itself there were multiple .mp4 filed of 25-30secs each and there was no video or audio. Each video is blank except for the first character of logged in person’s name in a circle.

Looks like Jibri cannot communicate with JVB. Probably JVB’s UDP/10000 is not accessible for Jibri.

Thanks, let me check

These ports are open to all IPs on the server hosting Jitsi. UDP - 3478, 10000
TCP - 80, 443, 5222, 5349.

UDP 10000 is open.

This is the log from Jibri server. The “permission denied” error for finalize.sh is strange because I have given 777 permission to that file. Please check the log and let me know what might be wrong.

Jibri 2022-11-14 15:40:11.564 INFO: [763] [session_id=zpweduufqqlyhoam] JibriSelenium.onSeleniumStateChange#215: Transitioning from state Starting up to Running
Jibri 2022-11-14 15:40:11.564 INFO: [763] [session_id=zpweduufqqlyhoam] FileRecordingJibriService$start$1.invoke#168: Selenium joined the call, starting the capturer
Jibri 2022-11-14 15:40:11.578 INFO: [763] [session_id=zpweduufqqlyhoam] JibriSubprocess.launch#42: Starting ffmpeg with command ffmpeg -y -v info -f x11grab -draw_mouse 0 -r 30 -s 1280x720 -thread_queue_size 4096 -i :0.0+0,0 -f alsa -thread_queue_size 4096 -i plug:bsnoop -acodec aac -strict -2 -ar 44100 -b:a 128k -af aresample=async=1 -c:v libx264 -preset veryfast -profile:v main -level 3.1 -pix_fmt yuv420p -r 30 -crf 25 -g 60 -tune zerolatency -f mp4 /config/recordings/zpweduufqqlyhoam/testmicro1_2022-11-14-15-40-06.mp4 ([ffmpeg, -y, -v, info, -f, x11grab, -draw_mouse, 0, -r, 30, -s, 1280x720, -thread_queue_size, 4096, -i, :0.0+0,0, -f, alsa, -thread_queue_size, 4096, -i, plug:bsnoop, -acodec, aac, -strict, -2, -ar, 44100, -b:a, 128k, -af, aresample=async=1, -c:v, libx264, -preset, veryfast, -profile:v, main, -level, 3.1, -pix_fmt, yuv420p, -r, 30, -crf, 25, -g, 60, -tune, zerolatency, -f, mp4, /config/recordings/zpweduufqqlyhoam/testmicro1_2022-11-14-15-40-06.mp4])
Jibri 2022-11-14 15:40:12.769 INFO: [767] [session_id=zpweduufqqlyhoam] FfmpegCapturer.onFfmpegStateMachineStateChange#130: Ffmpeg capturer transitioning from state Starting up to Running
Jibri 2022-11-14 15:40:12.769 INFO: [767] [session_id=zpweduufqqlyhoam] StatefulJibriService.onServiceStateChange#39: File recording service transitioning from state Starting up to Running
Jibri 2022-11-14 15:40:12.769 INFO: [767] XmppApi$createServiceStatusHandler$1.invoke#328: Current service started up successfully, sending on iq
Jibri 2022-11-14 15:40:26.593 INFO: [18] [session_id=zpweduufqqlyhoam] MediaReceivedStatusCheck.run#40: Jibri client receive bitrates: {audio={download=0, upload=0}, download=0, upload=0, video={download=0, upload=0}}, num participants: 1, numMutedParticipants: 0, numJigasis: 0, all clients muted? false
Jibri 2022-11-14 15:40:41.610 INFO: [18] [session_id=zpweduufqqlyhoam] MediaReceivedStatusCheck.run#40: Jibri client receive bitrates: {audio={download=0, upload=0}, download=0, upload=0, video={download=0, upload=0}}, num participants: 1, numMutedParticipants: 0, numJigasis: 0, all clients muted? false
Jibri 2022-11-14 15:40:41.610 INFO: [18] [session_id=zpweduufqqlyhoam] JibriSelenium$startRecurringCallStatusChecks$1.invoke#250: Recurring call status checks generated event org.jitsi.jibri.selenium.SeleniumEvent$NoMediaReceived@2e2cd6f5
Jibri 2022-11-14 15:40:41.611 INFO: [763] [session_id=zpweduufqqlyhoam] JibriSelenium.onSeleniumStateChange#215: Transitioning from state Running to Error: NoMediaReceived SESSION No media received
Jibri 2022-11-14 15:40:41.611 INFO: [763] [session_id=zpweduufqqlyhoam] StatefulJibriService.onServiceStateChange#39: File recording service transitioning from state Running to Error: NoMediaReceived SESSION No media received
Jibri 2022-11-14 15:40:41.611 INFO: [763] XmppApi$createServiceStatusHandler$1.invoke#311: Current service had an error Error: NoMediaReceived SESSION No media received, sending error iq
Jibri 2022-11-14 15:40:41.611 FINE: [763] JibriStatsDClient.incrementCounter#38: Incrementing statsd counter: stop:recording
Jibri 2022-11-14 15:40:41.612 INFO: [763] JibriManager.stopService#260: Stopping the current service
Jibri 2022-11-14 15:40:41.612 INFO: [763] [session_id=zpweduufqqlyhoam] FileRecordingJibriService.stop#182: Stopping capturer
Jibri 2022-11-14 15:40:41.612 INFO: [763] [session_id=zpweduufqqlyhoam] JibriSubprocess.stop#75: Stopping ffmpeg process
Jibri 2022-11-14 15:40:41.634 INFO: [767] [session_id=zpweduufqqlyhoam] FfmpegCapturer.onFfmpegStateMachineStateChange#130: Ffmpeg capturer transitioning from state Running to Finished
Jibri 2022-11-14 15:40:41.635 INFO: [763] [session_id=zpweduufqqlyhoam] JibriSubprocess.stop#89: ffmpeg exited with value 255
Jibri 2022-11-14 15:40:41.636 INFO: [763] [session_id=zpweduufqqlyhoam] FileRecordingJibriService.stop#184: Quitting selenium
Jibri 2022-11-14 15:40:41.641 INFO: [763] [session_id=zpweduufqqlyhoam] FileRecordingJibriService.stop#211: Participants in this recording:
Jibri 2022-11-14 15:40:41.653 INFO: [763] [session_id=zpweduufqqlyhoam] JibriSelenium.leaveCallAndQuitBrowser#336: Leaving call and quitting browser
Jibri 2022-11-14 15:40:41.653 INFO: [763] [session_id=zpweduufqqlyhoam] JibriSelenium.leaveCallAndQuitBrowser#339: Recurring call status checks cancelled
Jibri 2022-11-14 15:40:41.666 INFO: [763] [session_id=zpweduufqqlyhoam] JibriSelenium.leaveCallAndQuitBrowser#345: Got 9 log entries for type browser
Jibri 2022-11-14 15:40:41.809 INFO: [763] [session_id=zpweduufqqlyhoam] JibriSelenium.leaveCallAndQuitBrowser#345: Got 2184 log entries for type driver
Jibri 2022-11-14 15:40:41.886 INFO: [763] [session_id=zpweduufqqlyhoam] JibriSelenium.leaveCallAndQuitBrowser#345: Got 0 log entries for type client
Jibri 2022-11-14 15:40:41.886 INFO: [763] [session_id=zpweduufqqlyhoam] JibriSelenium.leaveCallAndQuitBrowser#354: Leaving web call
Jibri 2022-11-14 15:40:41.998 INFO: [763] [session_id=zpweduufqqlyhoam] JibriSelenium.leaveCallAndQuitBrowser#361: Quitting chrome driver
Jibri 2022-11-14 15:40:42.081 INFO: [763] [session_id=zpweduufqqlyhoam] JibriSelenium.leaveCallAndQuitBrowser#363: Chrome driver quit
Jibri 2022-11-14 15:40:42.081 INFO: [763] [session_id=zpweduufqqlyhoam] FileRecordingJibriService.stop#232: Finalizing the recording
Jibri 2022-11-14 15:40:42.081 INFO: [763] JibriServiceFinalizeCommandRunner.doFinalize#44: Finalizing the jibri service operation using command [/config/finalize.sh, /config/recordings/zpweduufqqlyhoam]
Jibri 2022-11-14 15:40:42.082 SEVERE: [763] JibriServiceFinalizeCommandRunner.doFinalize#63: Failed to run finalize script
java.io.IOException: Cannot run program “/config/finalize.sh”: error=13, Permission denied
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
at org.jitsi.jibri.util.ProcessWrapper.start(ProcessWrapper.kt:88)
at org.jitsi.jibri.service.impl.JibriServiceFinalizeCommandRunner.doFinalize(JibriServiceFinalizeCommandRunner.kt:47)
at org.jitsi.jibri.service.impl.FileRecordingJibriService.stop(FileRecordingJibriService.kt:233)
at org.jitsi.jibri.JibriManager.stopService(JibriManager.kt:263)
at org.jitsi.jibri.JibriManager$startService$1.invoke(JibriManager.kt:211)
at org.jitsi.jibri.JibriManager$startService$1.invoke(JibriManager.kt:85)
at org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:37)
at org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:29)
Jibri 2022-11-14 15:40:42.083 INFO: [763] JibriStatusManager$$special$$inlined$observable$1.afterChange#72: Busy status has changed: BUSY → IDLE
Jibri 2022-11-14 15:40:42.084 FINE: [763] WebhookClient$updateStatus$1.invokeSuspend#107: Updating 0 subscribers of status

This is the main problem.

What could be the reason ? I have the ports open to all IPs on the Jitsi server.

This is your problem. Does your deployment work if you open 3 tabs in the browser to the same room?

Are those correctly forwarded to the JVB machine? Is jvb reporting its correct public address

Also chiming at @emrah and @damencho I had the same issue with multiple recordings being produced. What I learned what issue was CPU related and ffmpeg was stopping and starting. What fixed it was:

  1. Reducing the video quality of the ffmpeg command. This can be done with reducing the resolution, bitrate and presets as the three areas to focus on.
  2. Uping the server size of my Jibri instances. I went from a c3.xlarge to a c3.2xlarge on AWS because I wanted the higher quality videos.

And my recordings stopped splitting into multiple files.

Recently I moved to a GPU instance which worked wonders, tutorial here: Tutorial: How To Install FFMPEG GPU with Jibri On Ubuntu 20.04

It seems to be a problem with the Jitsi EC2 instance that I cloned from an EC2 instance (using “Launch more like this” feature of AWS) on which I had set up Jitsi from scratch. When I stopped the cloned instance, started the original instance and did a recording, it worked fine. I tried then with 6 instances, one of them stopped after a few secs but the remaining 5 recordings ran till I manually stopped after 4 mins.

I faced an issue with AWS “Launch more like this” with Jibri server earlier. The docker folder did not even get copied when I launched 6 instances. I then had to create a template and launch from that to get them properly created. Probably a point to note to not use “Launch more like this” feature of AWS EC2 instance.