Jibri unavailable after few days

Install & Config
#1

Hi, I have setup jitsi in one server, and Jibri in another one, they work perfectly together. The problem is that the system is not used everyday, so after a few days, the Jibri stops working, and becomes unavailable.
Log in jitsi server:

Jicofo 2021-05-03 08:55:36.443 SEVERE: [4684] org.jitsi.jicofo.recording.jibri.JibriSession.log() Unable to find an available Jibri, can't start
Jicofo 2021-05-03 08:55:36.443 INFO: [4684] org.jitsi.jicofo.recording.jibri.JibriRecorder.log() Failed to start a Jibri session, no Jibris available

Jibri server log:

2021-04-28 15:06:31.952 INFO: [41] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Leaving web call
2021-04-28 15:06:32.020 INFO: [41] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Quitting chrome driver
2021-04-28 15:06:32.089 INFO: [41] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Chrome driver quit
2021-04-28 15:06:32.090 INFO: [41] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Finalizing the recording
2021-04-28 15:06:35.917 INFO: [41] org.jitsi.jibri.service.impl.FileRecordingJibriService.finalize() Recording finalize script finished with exit value 0
2021-04-28 15:06:35.918 INFO: [41] org.jitsi.jibri.status.JibriStatusManager.log() Busy status has changed: BUSY -> IDLE
2021-04-28 15:06:35.919 FINE: [41] org.jitsi.jibri.webhooks.v1.WebhookClient.invokeSuspend() Updating 0 subscribers of status
2021-04-28 15:06:35.919 INFO: [41] org.jitsi.jibri.api.xmpp.XmppApi.updatePresence() Jibri reports its status is now JibriStatus(busyStatus=IDLE, health=OverallHealth(healthStatus=HEALTHY, details={})), publishing presence to connections
2021-04-28 15:06:35.919 INFO: [55] org.jitsi.jibri.JibriManager.stopService() No service active, ignoring stop
2021-04-28 15:06:57.677 FINE: [18] org.jitsi.jibri.webhooks.v1.WebhookClient.invokeSuspend() Updating 0 subscribers of status
2021-04-28 15:07:57.677 FINE: [18] org.jitsi.jibri.webhooks.v1.WebhookClient.invokeSuspend() Updating 0 subscribers of status
2021-04-28 15:08:57.677 FINE: [18] org.jitsi.jibri.webhooks.v1.WebhookClient.invokeSuspend() Updating 0 subscribers of status
2021-04-28 15:09:57.677 FINE: [18] org.jitsi.jibri.webhooks.v1.WebhookClient.invokeSuspend() Updating 0 subscribers of status
2021-04-28 15:10:57.677 FINE: [18] org.jitsi.jibri.webhooks.v1.WebhookClient.invokeSuspend() Updating 0 subscribers of status

then comes more than 10 hours of this same message:

2021-04-29 01:55:57.677 FINE: [18] org.jitsi.jibri.webhooks.v1.WebhookClient.invokeSuspend() Updating 0 subscribers of status

and then (and last):

2021-04-29 01:56:57.676 FINE: [18] org.jitsi.jibri.webhooks.v1.WebhookClient.invokeSuspend() Updating 0 subscribers of status
2021-04-29 01:57:57.676 FINE: [18] org.jitsi.jibri.webhooks.v1.WebhookClient.invokeSuspend() Updating 0 subscribers of status
2021-04-29 01:58:57.677 FINE: [18] org.jitsi.jibri.webhooks.v1.WebhookClient.invokeSuspend() Updating 0 subscribers of status
2021-04-29 01:59:57.676 FINE: [18] org.jitsi.jibri.webhooks.v1.WebhookClient.invokeSuspend() Updating 0 subscribers of status
2021-04-29 02:00:01.253 INFO: [1364] org.jitsi.jibri.api.http.internal.InternalHttpApi.invokeSuspend() Jibri gracefully shutting down
2021-04-29 02:00:01.273 INFO: [1366] org.jitsi.jibri.Main.invoke() Jibri has been told to graceful shutdown, waiting to be idle before exiting
2021-04-29 02:00:01.273 INFO: [1366] org.jitsi.jibri.Main.invoke() Jibri is idle and has been told to gracefully shutdown, exiting

$ service jibri status
● jibri.service - Jibri Process
Loaded: loaded (/etc/systemd/system/jibri.service; disabled; vendor preset: enabled)
Active: inactive (dead)

and if I do:
$ service jibri restart

everything works fine.

What causing this?

Thank you all

#2

What are your server specs?

#3

is a Digital Ocean VPS with Ubuntu 18.04.4 LTS

#4

What are the specs - CPU, RAM, Network?

#5

Jibri is being told to shutdown by something hitting the graceful shutdown REST endpoint

#6

16GB RAM
8vCPUs

Don’t know about network

#7

Any idea on how to debug this? AFAIK There is nothing setup to do so…

#8

It may be that there is something that can trigger this in the scripts included with the default install. @Aaron_K_van_Meerten do you know?

#9

@Woodworker_Life @damencho any idea?

#10

@gbelvedere As @bbaldino already mentioned (and trust me, he knows), something is triggering a shutdown. What else do you have on the server hosting Jibri? Your logs show that not only does Jibri shutdown, it remains in an inactive state. So, some script on the server is triggering this. What else do you have running on the server?

#11

what is the value of single-use-mode in jibri.conf?

#12

my /etc/jitsi/jibri/jibri.conf file is empty :frowning:

jibri {
}
#13

:flushed:

So how are you getting any recording done at all?

Oh, you’re probably using config.json. Yeah, bad idea. Check out this thread - TUTORIAL - How to Install the NEW JIBRI

#14

yes, there it is : /etc/jitsi/jibri/config.json
but there is no single-use-mode setting .

There is this setting:

    // The amount of time, in minutes, a service is allowed to continue.
    //  Once a service has been running for this long, it will be
    //  stopped (cleanly).  A value of 0 means an indefinite amount
    //  of time is allowed
    "usage_timeout": "0"
#15

may be, but how can I tell?
We have a dedicated server for Jibri and another server for Jitsi

#16

If this was from the single-use-mode setting, there should be a log.