Jibri disconnected after prosody restart

We are running latest jitsi in kubernetes using docker-jitsi-meet. We have several jibri instances joined the jitsi.

When prosody is restarted, jibri’s are trying to reconnect to prosody, but some of them receive during reconnecting the “not-authorized” response from prosody and stop trying to reconnect and will stay in this state indefinitely, while still reporting jibri health status as HEALTHY and IDLE. Querying jibri health status thus doesn’t make it visible. All we can do is going through every jibri log and verify, if there is some “Jibri connected” message. And even then jicofo sometimes doesn’t see the jibri in brewery, but this might be some other issue.
Out of 8 jibris, we have like 5 ok and 3 not joined, on other shard out of 4 jibris, 3 are ok and 1 is not. This happens almost all the time. Restarting jibri will make it to join, no problem.

We now have an alert set up, that counts HEALTHY jibri instances and compares it with jicofo’s jibri_detector_count metric and in case this number is different, alerts person to manually find the jibris that are not connected and manually restart them, which is painfull.

I think the problem is, that somewhere in prosody startup, there is a momemnt, when prosody is receiving requests, but it’s not probably fully started yet (like user db is not loaded, or something like that?) and sends “not-authorized” responses. Again, we are using docker-meet-jitsi, where the init scripts setup all accounts and then start prosody, so everything should be prepared.

My question is, is there a way how to verify on jibri side, whether it is connected to prosody? We would than make a regular check and if not connected, than restart jibri. Of course, the best solution would be to fix the “not-authorized” thing and remove the problem at all.

What is the version of jibri that you use?

We are running on commit fix: remove xserver-xorg-input-void dependency · jitsi/jibri@ddca7a2 · GitHub from Oct 4, 2021.

We custom compile jibri because we need to patch some things related to GPU accelerated encoding (ffmpeg switches), but that is unrelated to the problem described here, i think.

Use the latest, there are a few fixes about reconnecting after that change.

1 Like

I think I am facing same/similar issue running Jitsi on Kubernetes cluster.

When Jibri is starting it randomly succeeds or fails with ‘not-allowed’ error and as a result can not record the video. While /jibri/api/v1.0/health says HEALTHY, IDLE.

On Jitsi version stable-6433 the error is this:

2021-12-19 15:45:42.994 FINE: [27] org.jitsi.xmpp.mucclient.MucClient.log() About to join MUCs: [jibribrewery@internal-muc.meet.jitsi]
2021-12-19 15:45:43.024 SEVERE: [27] org.jivesoftware.smack.AbstractXMPPConnection.callConnectionAuthenticatedListener() Exception in authenticated listener
java.lang.RuntimeException: org.jivesoftware.smack.XMPPException$XMPPErrorException: XMPP error reply received from jibribrewery@internal-muc.meet.jitsi/jibri-609054856: XMPPError: not-allowed - cancel

After upgrading Jitsi to the latest - greatest stable-6726-1 as suggested in this thread - I am getting a bit different error:
2021-12-19 15:55:17.615 WARNING: [29] org.jitsi.xmpp.mucclient.MucClient.log() Failed to join the MUCs.
org.jivesoftware.smack.XMPPException$XMPPErrorException: XMPP error reply received from jibribrewery@internal-muc.meet.jitsi/jibri-954632620: XMPPError: not-allowed - cancel [Room creation is restricted]

I would really appreciate your help on this.

Stop jibri, restart jicofo and then start jibri.

Or another option is to add jibri to the admin users here: jitsi-meet/prosody.cfg.lua-jvb.example at 12bc054386c7af20b2abd1051924c38df8167d72 · jitsi/jitsi-meet · GitHub

I’ve made it to work by adding jibri to admin users, as suggested above. I was restarting jibri several times already and each time it connects successfully, while previously it was failing every second time. So I assume it is fine now. Thanks for suggestion.
I am running Jitsi on Kubernetes so I was modifying prosody docker image:

were I simply added a line for jibri:
“{{ .Env.JIBRI_XMPP_USER }}@{{ .Env.XMPP_AUTH_DOMAIN }}”
And I was forced to create my custom prosody image with the change. It is not a big deal but is there any way, some docker Env property I am missing, that will allow me to add Jibri to admins without the need to build my custom prosody image? Or should I just create PR against jitsi/docker-jitsi-meet proposing the fix?