Jibri install from git - Strange behaviour since 1 day

Hi everyone,

I’m here because I’m using the debian version of Jibri. It was working until some hours ago when it started to have errors when starting.
I tried two simultaneous clean installs on two fresh new Ubuntu 18.04 servers and i had that error in both the servers (also on server where jibri was working ok).

When the process finishes (installing jibri.deb via dpkg) jibri starts correctly; then i have to edit the config.json in order to make it work with my jitsi installation on a server. Until some hours ago, everything was working by reloading jibri with sh /opt/jitsi/jibri/reload.sh.
Now, when i try to reload, i receive:
curl: (7) Failed to connect to 127.0.0.1 port 3333: Connection refused
So i know that jibri has not started.
Then i try to:
sh /opt/jitsi/jibri/launch.sh
and it produces the following:
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation

Ports 3333 and 2222 are listening after the faulty launch command but jitsi cannot start recording or live streaming.

Here the jibri log.0.txt:
2020-04-06 18:29:45.734 INFO: [1] org.jitsi.jibri.Main.main() Jibri run with args [--config, /etc/jitsi/jibri/config.json]
2020-04-06 18:29:45.736 INFO: [1] org.jitsi.jibri.Main.main() Using config file /etc/jitsi/jibri/config.json
2020-04-06 18:29:45.736 INFO: [1] org.jitsi.jibri.Main.main() Using port 3333 for internal HTTP API
2020-04-06 18:29:45.736 INFO: [1] org.jitsi.jibri.Main.main() Using port 2222 for the HTTP API
2020-04-06 18:29:46.155 INFO: [1] org.jitsi.jibri.Main.loadConfig() Parsed config:
JibriConfig(recordingDirectory=/tmp/recordings, singleUseMode=false, enabledStatsD=true, finalizeRecordingScriptPath=/path/to/finalize_recording.sh, xmppEnvironments=[XmppEnvironmentConfig(name=prod environment, xmppServerHosts=[X.Y.Z], xmppDomain=X.Y.Z, controlLogin=XmppCredentials(domain=auth.X.Y.Z, username=jibri, password=****), controlMuc=XmppMuc(domain=internal.auth.X.Y.Z, roomName=JibriBrewery, nickname=jus1), sipControlMuc=null, callLogin=XmppCredentials(domain=recorder.X.Y.Z, username=recorder, password=****), stripFromRoomDomain=conference., usageTimeoutMins=0, trustAllXmppCerts=true)])
2020-04-06 18:29:46.523 WARNING: [1] org.glassfish.jersey.internal.inject.Providers.checkProviderRuntime() A provider org.jitsi.jibri.api.http.internal.InternalHttpApi registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.jitsi.jibri.api.http.internal.InternalHttpApi will be ignored.
2020-04-06 18:29:46.710 INFO: [1] org.jitsi.jibri.api.xmpp.XmppApi.updatePresence() Jibri reports its status is now JibriStatus(busyStatus=IDLE, health=OverallHealth(healthStatus=HEALTHY, details={})), publishing presence to connections
2020-04-06 18:29:46.716 INFO: [1] org.jitsi.jibri.api.xmpp.XmppApi.start() Connecting to xmpp environment on X.Y.Zwith config XmppEnvironmentConfig(name=prod environment, xmppServerHosts=[X.Y.Z], xmppDomain=X.Y.Z, controlLogin=XmppCredentials(domain=auth.X.Y.Z, username=X, password=Y), controlMuc=XmppMuc(domain=internal.auth.X.Y.Z, roomName=JibriBrewery, nickname=jus1), sipControlMuc=null, callLogin=XmppCredentials(domain=recorder.X.Y.Z, username=recorder, password=1234), stripFromRoomDomain=conference., usageTimeoutMins=0, trustAllXmppCerts=true)
2020-04-06 18:29:46.716 INFO: [1] org.jitsi.jibri.api.xmpp.XmppApi.start() The trustAllXmppCerts config is enabled for this domain, all XMPP server provided certificates will be accepted
2020-04-06 18:29:46.734 WARNING: [20] org.jitsi.xmpp.mucclient.MucClient.log() Disabling certificate verification!
2020-04-06 18:29:46.796 WARNING: [1] org.glassfish.jersey.internal.inject.Providers.checkProviderRuntime() A provider org.jitsi.jibri.api.http.HttpApi registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.jitsi.jibri.api.http.HttpApi will be ignored.
2020-04-06 18:29:47.030 INFO: [20] org.jitsi.xmpp.mucclient.MucClient.log() Connected.
2020-04-06 18:29:47.187 INFO: [20] org.jitsi.xmpp.mucclient.MucClient.log() Joined MUC: jibribrewery@internal.auth.X.Y.Z

What is causing these? Thank you!

Don’t worry about these, they aren’t a problem.

I don’t see anything in your logs that looks like an issue with it starting up, I think your problem may be elsewhere (like Jicofo not seeing the Jibri). Can you attach Jibri and Jicofo logs from when the error happens?

Thank you Bbaldino for your answer!

Here you can find the two logs files that describes jibri starting via sh, a conference start in jitsi-meet (name: jibri) and the jicofo log (starting at 22:46 when i tried to start a recording).

P.S. Jibri and Jicofo servers are not at setted on the same time, so you will see 21:46 instead of 22:46 in jibri_LOG file.

Please, let me know if I can help in some way, thank you!
jibri_LAUNCH_SH_6_5_22_46.txt (1.8 KB)
jibri_LOG_6_4_22_46.txt (37.0 KB) jicofo_LOG_6_4_22_46.txt (16.8 KB)

From the Jibri logs:

2020-04-06 21:46:54.445 SEVERE: [39] org.jitsi.jibri.api.xmpp.XmppApi.run() Error starting Jibri service : org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: exited abnormally.
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
  (Driver info: chromedriver=80.0.3987.106 (f68069574609230cf9b635cd784cfb1bf81bb53a-refs/branch-heads/3987@{#882}),platform=Linux 4.15.0-91-generic x86_64) (WARNING: The server did not provide any stacktrace information)

So something is up with chromedriver/selenium. Googling around for that error message should give some clues.

1 Like

Thank you!

I was searchin for this error and i will continue to do until i will find a solution.

The strange thing is that, before this afternoon, everything was working good and i was creating several jibri instances on several VM, and i had no problems at all. From 5PM (circa) any new istance i created inside any new server or VM started to have this problem. Is it possible that could be something related to some updates or similiar? Because anytime i tried to create a jibri instance, i follow this exact procedure:

  • create VM with same OS version (ubu18.04 server; from same ISO file i’m using from one month or online images on cloud VPS);
  • clone jibri from git;
  • install all dependencies and software;
  • compile and install deb package.
    and it seems really strange that i have same results on different VM running something that was not updated from ten days from now.

I wouldn’t think so, but I haven’t run into that error so I don’t know what the cause is–maybe it could be related. (Did chromedriver or chrome update, for example?)

Following the readme.md in jibri git, it allowed me to donwload and install chrome-stable 80.0 and chrome-driver 80.0 (and this is ok following this) but i’ve seen that the latest version of both is 81.0… .
From regular apt update and upgrade it seems that nothing has to upgrade.

I’m going to try to install latest chrome and chromedriver version and see if this change something.

I tried to use chrome and chromedriver 81.etc and same issue.
I tried to install jibri on an Ubuntu 16.04 fresh new server (without anything) and same issue.
I followed, as usual, every step inside the jibri readme and debian_package.md and this is really just incredible.
Same error: chrome failed to start. Now i’m trying to install chrome and chromedriver version 79.

No way of using jibri also with version 79.
Could someone please try to clean install jibri deb on a new ubuntu server (i will take you one if you need) and try if it works for you?
I’m guessing that not, it will not.

UPDATE

Today Jibri started to work. These are the differencies in the environment i tested:

  • google-chrome-stable latest release changed to 81.0…
  • chromedriver latest release changed to 81.0…
  • The only way i had it working is to firstly install Jibri by apt install, then purge it and do the manual installation via deb package.

It worked on Ubuntu 18.04 server. I will try to make it work also on others VM.

Hi Marco

When you mean “started to work” are you able to record any jitsi meeting sessions?

I have the same setup & for the life of me I’m unable to get the Recording OR Streaming feature to work.

I tried purging & installing via deb, but no luck for me. Jibri starts & registers in jicofo.

Hello there,

Yes, but remember that you can only do 1 recording OR 1 live stream with an instance one Jibri. This means that, for example, if you want to be able to record 2 rooms and live stream another room, you will need 3 instances of Jibri (I read that meet.jit.si is using octo and autoscaling groups on AWS in order to make this work for large numbers).

There is some work you have to do here: firs read logs (jibri one and jicofo) and search for any error (usually SEVERE). Please check this:

  1. Try to start with a clean installation of Jibri via apt install (from here).
  2. Be sure to properly configure /etc/jitsi/jibri/config.json by adding your Jitsi instance url and set the same login and recorder username and psswd you set on prosody (everything is written in the linked readme);
  3. Be sure to give to Jibri user right permissions (linked readme) and to create the recording folder (/tmp/recordongs by default) and give it jibri read and write permissions;
  4. Restart Jibri (systemctl restart).
    At this point you could be able to start/stop recording or livestream (note that if you are using the apt install Jibri, livestream is only possible to YouTube by pasting YB’s stream key).
    Then, if you want to install via deb package, apt purge Jibri (only Jibri) and follow this doc after you edited what you have in Jibri folder. Also, running the last command (resources/jenkins/release.sh Minor) may give you some dependencies error: just instal those 2-3 packages with apt.

Also - very important: in order to use Jibri, verify that your current java version is something like 8.0… and not 11 (in order to solve this, do apt install openjdk-8-jdk and then select java 8 by running
update -alternatives --config java

Hope this helps but, if needed, ask me more and I will try to help you in figure it out.

I have detailed my woes in this thread.

i have configured everything to the T as in readme/guide. this darned error is the only SEVERE error that stops me from recording or streaming

2020-04-10 12:07:15.208 FINE: [46] org.jitsi.jibri.selenium.pageobjects.CallPage.visit() Visiting url https://meet.mydomain.com/testing#config.iAmRecorder=true&config.externalConnectUrl=null&config.startWithAudioMuted=true&config.startWithVideoMuted=true&interfaceConfig.APP_NAME="Jibri"&config.analytics.disabled=true&config.p2p.enabled=false
2020-04-10 12:07:46.321 SEVERE: [46] org.jitsi.jibri.selenium.pageobjects.CallPage.visit() Timed out waiting for call page to load
2020-04-10 12:07:46.323 INFO: [46] org.jitsi.jibri.selenium.JibriSelenium.onSeleniumStateChange() Transitioning from state Starting up to Error: SESSION Failed to join call

I’m now going to try this deb route you suggested… :crossed_fingers:

Before installing the deb, make sure that also all of the domain ifo are correct (i saw in the other thread that you are using meet.domain.com both in jitsi and in jibri, but you have to use “meet” only if your FQDN really is meet.domain.com; for example, if my jitsi server is on abc.def.com, i don’t have to use meet.abc.def.com).

After this, another thing to try is to upgrade google-chrome-stable and chromedriver to new latest release (81.0…) if previous steps don’t work.

May i ask you for your prosody and jibri configuration with real domain infos? Just delete passwords so i can’t view them.

EDIT
Sorry, i just read that you are in esxi, so i think you are using right domains info

thanks

please see pm.

meet.mydomain.com does resolve to my server correctly

chrome is now 81

So, i was seeing your prosody cfg and mine. In my jitsi server, i added the virtualhos recorder and internal.auth to /etc/prosody/prosody.cfg.lua and not in /etc/prosody/conf.d/my.domain.cfg.lua or /etc/prosody/conf.avail/my.domain.cfg.lua

Is it the same for you? If not, just try to edit the proper prosody.cfg.lua, clean added lines in others .cfg.lua and then restart prosody (also jicofo and videobridge, who knows).

Hi Marco

I did what you said & still no luck, same problem

thank you anyway

Hi masteryoda
i install jibri/jitsi but when i want to statrt jibri, i face with this error:failed to connect to 127.0.0.1 port 3333 refused .
it causes not able to start recording. please give me some tips if you solve it…