TUTORIAL: Jibri Overview, Troubleshooting Tips & Tricks - Solve your Jibri Problems, Quickly!

It seems that most people find Jibri implementation to be the most challenging of the suite of solutions. I am by no measure an expert on this, but I’ve done several installs of Jibri, purged it a number of times and reinstalled, so I thought I should share some of what I’ve gained with anyone who may find it useful. I’ll try to explain this in layman terms, so the information is not overwhelming. This is a long read, but it will save you a lot of time and a lot of headache trying to figure out what’s wrong or waiting for help in the forum. If you like, you can just jump to the Troubleshooting and Tips section.

QUICK OVERVIEW:

1. What is Jibri?
Jibri is the recording and streaming instrument for Jitsi Meet conferences. Its capabilities include:

  1. Record a Jitsi meeting, capturing both audio and video
  2. Livestream a Jitsi meeting (through any RTMP)
  3. Do a local (audio) recording of a Jitsi meeting

2. How does it work?
Understanding how Jibri works will help you to know what to look for when you run into problems with it.

Jibri operates just like any other participant in a meeting. It enters the meeting as a hidden user (not sending audio or video of its own) and it records the meeting (screencasting), pretty much the same way as you would use a screen recorder to record events on your screen. In fact, you can confirm the presence of Jibri if you activate “Local Recording” and then look in the speaker stats - you will see a ‘name’ (mix of characters) that shows an extra participant in your meeting. For instance, in the example below, the Participant with the name “8ae9f547” is Jibri.

Screen Shot 2020-08-31 at 12.46.39 AM

Jibri does screencasting by joining the meeting just like any other participant, spinning its own ‘monitor’ (which you don’t see), and then recording the meeting from its own screen ('monitor). This is important to note, because, amongst other things, it means that just like one participant can have problems in a meeting while everyone else is fine, Jibri can also have problems, yet a Jitsi Meet conference will run successfully and other participants will not be affected.

One important thing to note: Jibri can only record one meeting at a time. This means you need an instance of Jibri for every meeting you need to record.

TROUBLESHOOTING

Now, this is the main purpose of this tutorial and probably the reason why you clicked this link. There are two messages you’re likely to get when your recording does not start as expected:

  1. All recorders are currently busy - This happens when all available Jibris are being used. Remember, Jibri can only record one meeting at a time, so if your available pool of Jibris is used up (in use), you will not be able to record until one becomes available. And that’s when you get this message.
  2. Recording unavailable - This is an error message. It is the generic message you get when something is actually wrong with Jibri - usually telling you Jibri is not properly installed or configured. It’s important to note that you will get this same error message on your screen when Jibri doesn’t work for any reason other than that the available recorders are busy. So the error message itself is not very specific. However, the timing of the error message could be very instructive.

Jibri errors (“Recording Unavailable”)

Screen Shot 2020-11-21 at 5.46.30 PM
There are a few things that could go wrong, but as complicated as Jibri seems, the beautiful thing is, there are just a handful of places where those things could go wrong. This makes troubleshooting relatively not as painful.

TIPS

Recording fails immediately

Remember that Jibri is a meeting participant as well, right? So that means, the first problem Jibri could have could be with getting into the meeting room. In fact, one BIG clue that suggests this is the cause of your Jibri problem is that when you click the “Start recording” command, you INSTANTLY get that error message. If recording fails immediately, check the following:

Jibri.conf

  • Take a careful look at your Jibri.conf file (/etc/jitsi/jibri/jibri.conf). This is your main configuration file for Jibri and the area where most mistakes happen. This file will pretty much be empty when you first install Jibri, so you have to populate it. If it’s not populated, Jibri will fail instantly.
  • Use the sample/reference file to populate your jibri.conf. Make sure you complete the xmpp section using the example from example_xmpp. Here’s a full example of Jibri.conf you can just copy and paste, then make the necessary edits to reflect your domain and registered passwords [click arrow to reveal]:
Jibri,conf
> jibri {
>   // A unique identifier for this Jibri
>   // TODO: eventually this will be required with no default
>   id = ""
>   // Whether or not Jibri should return to idle state after handling
>   // (successfully or unsuccessfully) a request.  A value of 'true'
>   // here means that a Jibri will NOT return back to the IDLE state
>   // and will need to be restarted in order to be used again.
>   single-use-mode = false
>   api {
>     http {
>       external-api-port = 2222
>       internal-api-port = 3333
>     }
>     xmpp {
>       // See example_xmpp_envs.conf for an example of what is expected here
>       environments = [
> 	      {
>                 name = "prod environment"
>                 xmpp-server-hosts = ["your.domain.com"]
>                 xmpp-domain = "your.domain.com"
> 
>                 control-muc {
>                     domain = "internal.auth.your.domain.com"
>                     room-name = "JibriBrewery"
>                     nickname = "jibri-nickname"
>                 }
> 
>                 control-login {
>                     domain = "auth.your.domain.com"
>                     username = "jibri"
>                     password = "JPwd"
>                 }
> 
>                 call-login {
>                     domain = "recorder.your.domain.com"
>                     username = "recorder"
>                     password = "RPwd"
>                 }
> 
>                 strip-from-room-domain = "conference."
>                 usage-timeout = 0
>                 trust-all-xmpp-certs = true
>             }]
>     }
>   }
>   recording {
>     recordings-directory = "/srv/recordings"
>     # TODO: make this an optional param and remove the default
>     finalize-script = "/path/to/finalize_recording.sh"
>   }
>   streaming {
>     // A list of regex patterns for allowed RTMP URLs.  The RTMP URL used
>     // when starting a stream must match at least one of the patterns in
>     // this list.
>     rtmp-allow-list = [
>       // By default, all services are allowed
>       ".*"
>     ]
>   }
>   chrome {
>     // The flags which will be passed to chromium when launching
>     flags = [
>       "--use-fake-ui-for-media-stream",
>       "--start-maximized",
>       "--kiosk",
>       "--enabled",
>       "--disable-infobars",
>       "--autoplay-policy=no-user-gesture-required"
>     ]
>   }
>   stats {
>     enable-stats-d = true
>   }
>   webhook {
>     // A list of subscribers interested in receiving webhook events
>     subscribers = []
>   }
>   jwt-info {
>     // The path to a .pem file which will be used to sign JWT tokens used in webhook
>     // requests.  If not set, no JWT will be added to webhook requests.
>     # signing-key-path = "/path/to/key.pem"
> 
>     // The kid to use as part of the JWT
>     # kid = "key-id"
> 
>     // The issuer of the JWT
>     # issuer = "issuer"
> 
>     // The audience of the JWT
>     # audience = "audience"
> 
>     // The TTL of each generated JWT.  Can't be less than 10 minutes.
>     # ttl = 1 hour
>   }
>   call-status-checks {
>     // If all clients have their audio and video muted and if Jibri does not
>     // detect any data stream (audio or video) comming in, it will stop
>     // recording after NO_MEDIA_TIMEOUT expires.
>     no-media-timeout = 30 seconds
> 
>     // If all clients have their audio and video muted, Jibri consideres this
>     // as an empty call and stops the recording after ALL_MUTED_TIMEOUT expires.
>     all-muted-timeout = 10 minutes
> 
>     // When detecting if a call is empty, Jibri takes into consideration for how
>     // long the call has been empty already. If it has been empty for more than
>     // DEFAULT_CALL_EMPTY_TIMEOUT, it will consider it empty and stop the recording.
>     default-call-empty-timeout = 30 seconds
>   }
> }
  • In the xmpp section, make sure you enter the names and passwords you chose when creating the relevant accounts for Jibri in Prosody (“jibri” and “recorder”) accurately. Verify and re-verify that you actually entered the usernames and passwords correctly! To verify:

Check the registered recorder name and password

grep password /var/lib/prosody/recorder*/accounts/recorder.dat
sed -n ‘/call.login/, /}/p’ /etc/jitsi/jibri/config.json | grep password
sed -n ‘/call.login/, /}/p’ /etc/jitsi/jibri/jibri.conf | grep password

Check the registered jibri username and password

grep password /var/lib/prosody/auth*/accounts/jibri.dat
sed -n ‘/control.login/, /}/p’ /etc/jitsi/jibri/config.json | grep password
sed -n ‘/control.login/, /}/p’ /etc/jitsi/jibri/jibri.conf | grep password

  • If you previously created a config.json file (previous version of Jibri), don’t just copy and paste the contents of that file into your jibri.conf; you need to extract the actual values as the field names are slightly - but significantly - different. [Note that config.json is being deprecated, so it’s best to move your configurations to jibri.conf, if you haven’t done so already. Once you do, delete the config.json file].

Jibri log folder

This is an unusual one, but it’s worth noting. I’ve only seen it happen when I uninstalled and then re-installed Jibri, but who knows? Anyway, if you’ve confirmed that everything is fine with your jibri.conf, go to “/var/log/jitsi” and confirm that you have a directory named “jibri” there. If you don’t, create one and make sure Jibri can write to it (just make jibri the owner). If this directory is missing or if it’s present but Jibri does not have permissions to access it, Jibri will fail instantly. Also. make sure that jibri is part of all the necessary groups:

usermod -aG adm,audio,video,plugdev jibri

Recording fails after a few seconds

This differs significantly from an instant error message. In this case, you get a pop-up message advising that Jibri is “preparing to record the meeting”

Screen Shot 2020-11-21 at 5.50.26 PM

But then a few seconds or several seconds later, that pop-up is replaced by the error pop-up.

Screen Shot 2020-11-21 at 5.46.30 PM

In this case, it often suggests that Jibri may have been able to enter the meeting, but it wasn’t configured properly, so it’s unable to start recording. In instances like this, it’s not even unusual to have a folder created in the recording directory, but the folder will only contain a metadata.json without an actual mp4 file. When this happens, check the following:

Jibri.conf

  • Yes, again, jibri.conf. If you in fact populated your jibri.conf file, but have the wrong values or made a mistake, this could be the reason. What happens is that the configuration is being read (hence the pause) and when it’s discovered that the credentials are wrong, then the error message gets kicked out. Again, do all the checks listed above under Jibri.conf and correct any errors discovered.

Sip Communicator Properties (Jicofo)

  • Check your Jicofo configurations for jibri to make sure you’ve entered them correctly:

sudo nano /etc/jitsi/jicofo/sip-communicator.properties

  • Confirm that you’ve added the following lines:

org.jitsi.jicofo.jibri.BREWERY=JibriBrewery@internal.auth.my.domain.com
org.jitsi.jicofo.jibri.PENDING_TIMEOUT=90

  • BE SURE TO CHANGE “my.domain.com” TO YOUR JITSI SUBDOMAIN NAME!!!

cfg.lua file (Prosody)

  • Check your prosody (lua) configuration

nano /etc/prosody/conf.avail/my.domain.com.cfg.lua

Make sure you’ve added this block at the end of the file:

Screen Shot 2020-11-22 at 4.40.41 AM

  • Again, be sure to remember to edit it to reflect your Jitsi subdomain.

Recording is successful but mp4 is corrupt (cannot be opened)

This is usually due to a java version error. Jibri requires Java 8 to work properly.

  • Check to make sure you’ve installed Java 8 on your system.

java -version

  • If you have another (more current) version of Java installed, that’s what will be listed when you check your java version.
  • To check for multiple versions of java on your system:

update-alternatives --config java

  • This should list all the versions. Make sure you see the following line as one of the options:

/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/bin/java

  • Now, make sure you set it as the correct java home environment variable. Open launch.sh

nano /opt/jitsi/jibri/launch.sh

replace “java” with the full path “/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/bin/java”

Finally, ALWAYS restart Jibri when you make any configuration change. This is more often the cause of errors than one would believe.

CONCLUSION
Hopefully one of these tips has helped you to solve your Jibri problem. Please, feel free to add anything else you may have discovered that could prove helpful to others. I will try to add more information as I run across them.

2 Likes

Some other possible issues:

  • the resource deficiencies
    at least 4 cores / 4 GB RAM for a seperate Jibri server
    at least 8 cores / 8 GB RAM if JMS and Jibri are on the same server

  • the missing snd_aloop module
    Some cloud kernel has no snd_aloop support. It’s needed to switch to the standard Linux kernel in this case.

  • self-signed TLS certificate for JMS
    Add --ignore-certificate-errors to the Chrome flags if this is the case

    chrome {
        flags = [
            "--use-fake-ui-for-media-stream",
            "--start-maximized",
            "--kiosk",
            "--enabled",
            "--disable-infobars",
            "--autoplay-policy=no-user-gesture-required",
            "--ignore-certificate-errors"
        ]
    }
  • Unaccessable TCP/5222
    If jibri is on a seperate server, TCP/5222 must be open on JMS

  • Unresolvable JMS address
    jibri must resolv the JMS host address correctly

  • Nonexistant recording folder
    The recording folder must exist

  • Nonwritable recording folder
    The recording folder must be writable for the jibri user

  • Already installed desktop environment
    Jibri has its own desktop environment. If there is already installed one, this may cause some problems.

  • Not working JVB
    Don’t start installing Jibri before a fully working JMS. Don’t forget to test it with 3 participants.

  • Docker installation
    God help you

2 Likes

:joy: :joy: :joy: :joy: :joy: :joy: :joy:

Any idea on JWT?
I’m having a hard time to get jibri along lobby with JWT.

Regards

@Freddie @emrah I’m running latest jibri image(yes it is docker :frowning: ) on 8vCPU and 8GB Memory. After 20 minutes first it eats up all the Memory and then vCPU and finally crashed. Is there any link which I can refer to setup a jibri with optimal cofiguration which can stop this crash?

@emrah I have seen your Jitsi_Cluster template. Lets say if I have only one machine with 8core/16 GB with JMS and jibri running on same machine. can JMS utilize 8 cores if Jibri is not recording?
Also lets say if I increase this machine to 12core /16GB then Jibri is running can JMS use 8 cores?
I want to run a jitsi+jibri one one machine only and one room only, but want to accommodate as many users as possible, most of the time with only 2 or 3 videos, all other users off wont use audio and video. What do you think roughly number of users can be achieved.

Hello @metadata,

I heard this issue before but never happened to me. I checked some friend servers which have this issue but I couldn’t found the root cause. At a random time, ffmpeg starts to eat memory and it crashs.

I have no idea for now

Theoretically, yes… But some services can not use multi-processors due to their design. For example Prosody…

No reservation for CPU power. It is shared for all running processes. If the load is bigger than the capacity then all proceses wait in a queue and run slowly

I didn’t test it but officially it’s recommended maximum 100 participants in a single room

@metadata yes, I’ve had this issue too - actually happened to me again a few days ago during a meeting. It’s really random and I’ve not been able to find a cause or pattern. I’ve looked through logs and haven’t found anything. I did save all the logs from the last time it happened and I plan to scour through them carefully. To be frank, it’s perhaps the one fear I have when I’m having an important meeting and need to record. I hope we can find a definitive fix for it soon.

Does it always happen when you record?

@kkd my configuration is actually the same (8 core/16GB) and I run both Jitsi and Jibri on the same machine (bare metal though). JMS alone uses little to no resources, its main dependency is bandwidth. When I run only Jitsi, all 8 cores get engaged, but they barely do ANY work (I always run with videos for all participants). So I know when it comes to just JMS, my only constraint would be bandwidth.

With Jibri, it’s different - but not by that much. Jibri spikes the CPU, but still to negligible limits considering the resources I throw at it. When Jibri doesn’t act up and start groveling my memory, everything stays at less than (or just about) 1% of my available memory.

All this to say, in your case, you might be able to squeeze in up to 150 users with just 2 or 3 showing video (don’t quote me on this, of course… lol).

Right now I’m using jitsi/jibri:latest image and meeting(2 participants only) is in progress ( 23 minutes so far) and Memory consumption is at 1.09 Gb Memory and around 20% VCPU consumption. It seems stable for now.

if I build my own docker image using the Dockerfile provided in docker-jitsi-meet/jibri directory and use this image then after 20 minutes it started eating memory.

I guess I have to check whether the jitsi/base-java image is built using stable or unstable build.

I’ve seen some people say disabling calls to third party apps/services helps. I can’t swear on it as I’ve done that and I’ve still seen it spike occasionally. And it’s basically 2 services that go crazy - ffmpeg and xorg.

The memory issue may be related in CPU bottleneck. IIRC I always saw it on low end machines or shared virtual hosts

@emrah @Freddie So I have created a new docker image using base-java stable build( I think earlier I was using unstable build to add the streaming to any server feature) and now I’m not seeing any memory and cpu issue BUT there is an exception in jibri logs. This is happening with both latest jitsi/jibri image and my own custom image. Moreover the final video isn’t playable at all.(size of the video is abount 166Mb and when I’m trying to play it then nothing is happening)

2020-12-16 22:45:37.941 FINE: [146] org.jitsi.jibri.statsd.JibriStatsDClient.incrementCounter() Incrementing statsd counter: stop:recording
2020-12-16 22:45:37.941 INFO: [146] org.jitsi.jibri.JibriManager.stopService() Stopping the current service
2020-12-16 22:45:37.941 INFO: [146] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Stopping capturer
2020-12-16 22:45:37.941 INFO: [146] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() Stopping ffmpeg process
2020-12-16 22:45:37.943 SEVERE: [146] org.jitsi.jibri.util.ProcessWrapper.stopAndWaitFor() Error stopping process: java.io.IOException: Cannot run program "kill": error=2, No such file or directory with stack:
java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
java.lang.Runtime.exec(Runtime.java:621)
java.lang.Runtime.exec(Runtime.java:451)
java.lang.Runtime.exec(Runtime.java:348)
org.jitsi.jibri.util.ProcessWrapper.stop(ProcessWrapper.kt:101)
org.jitsi.jibri.util.ProcessWrapper.stopAndWaitFor(ProcessWrapper.kt:112)
org.jitsi.jibri.util.JibriSubprocess.stop(JibriSubprocess.kt:77)
org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.stop(FfmpegCapturer.kt:125)
org.jitsi.jibri.service.impl.FileRecordingJibriService.stop(FileRecordingJibriService.kt:166)
org.jitsi.jibri.JibriManager.stopService(JibriManager.kt:262)
org.jitsi.jibri.api.xmpp.XmppApi.handleStopJibriIq(XmppApi.kt:263)
org.jitsi.jibri.api.xmpp.XmppApi.handleJibriIq(XmppApi.kt:169)
org.jitsi.jibri.api.xmpp.XmppApi.handleIq(XmppApi.kt:150)
org.jitsi.xmpp.mucclient.MucClient.handleIq(MucClient.java:569)
org.jitsi.xmpp.mucclient.MucClient.access$800(MucClient.java:50)
org.jitsi.xmpp.mucclient.MucClient$2.handleIQRequest(MucClient.java:533)
org.jivesoftware.smack.AbstractXMPPConnection$4.run(AbstractXMPPConnection.java:1188)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

2020-12-16 22:45:37.943 SEVERE: [146] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() Error trying to gracefully stop ffmpeg, destroying forcibly
2020-12-16 22:45:37.993 INFO: [146] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() ffmpeg exited with value 137
2020-12-16 22:45:37.993 INFO: [146] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Quitting selenium
2020-12-16 22:45:38.001 INFO: [146] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Participants in this recording: []
2020-12-16 22:45:38.013 INFO: [146] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Leaving call and quitting browser
2020-12-16 22:45:38.014 INFO: [146] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Recurring call status checks cancelled
2020-12-16 22:45:38.019 INFO: [146] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 17 log entries for type browser
2020-12-16 22:45:38.082 INFO: [146] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 1079 log entries for type driver
2020-12-16 22:45:38.186 INFO: [146] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 0 log entries for type client
2020-12-16 22:45:38.186 INFO: [146] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Leaving web call
2020-12-16 22:45:38.445 INFO: [146] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Quitting chrome driver
2020-12-16 22:45:38.521 INFO: [146] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Chrome driver quit
2020-12-16 22:45:38.521 INFO: [146] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Finalizing the recording
2020-12-16 22:45:40.010 INFO: [146] org.jitsi.jibri.status.JibriStatusManager.log() Busy status has changed: BUSY -> IDLE
2020-12-16 22:45:40.011 FINE: [146] org.jitsi.jibri.webhooks.v1.WebhookClient.invokeSuspend() Updating 0 subscribers of status
2020-12-16 22:45:40.012 INFO: [146] org.jitsi.jibri.api.xmpp.XmppApi.updatePresence() Jibri reports its status is now JibriStatus(busyStatus=IDLE, health=OverallHealth(healthStatus=HEALTHY, details={})), publishing presence to connections

I’m not really that big on docker, but what version of java are you running?

Yeah, there’s been some thoughts in that direction. But it doesn’t seem like it’s really that, because it happens even with more-than-sufficient cores.

openjdk version “1.8.0_275”
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_275-b01)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.275-b01, mixed mode)

@Freddie There’s a bug in docker-jitsi-meet repo where users reported that the recording is corrupt.

Phew! This is why I don’t care for docker. I know it seems easier to manage, but once you start digging in and need to get granular, it’s a headache. :confused:

Nm, got it sorted.
All working great