JVB Health check failed - but only on one videobridge

Having issues with one of our two JVBs

Jicofo 2021-09-27 12:26:52.305 WARNING: [164] JvbDoctor$HealthCheckTask.doHealthCheck#284: Health check failed for: jvbbrewery@internal.auth.domain.tld/jvb2: <error type='cancel'><service-unavailable xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'/></error

As soon as JVB2 is in use (either dictated by Octo or if JVB1 is unavailable) the meeting crashes with the “Unfortunantely, something went wrong”-error (SINGLE_PORT_HARVESTER_PORT is reachable).

We have two identically configured bare-metal servers running Debian 10. With the exception of ports (org.jitsi.videobridge.SINGLE_PORT_HARVESTER_PORT and org.jitsi.videobridge.rest.jetty.port) and IPs in use by the JVBs themselves.

When the Jicofo service is restarted both bridges are picked up:
Jicofo 2021-09-27 10:09:32.268 INFO: [29] [type=bridge brewery=jvbbrewery] BaseBrewery.addInstance#358: Added brewery instance: jvbbrewery@internal.auth.domain.tld/jvb2 Jicofo 2021-09-27 10:09:32.291 INFO: [17] [type=bridge brewery=jvbbrewery] BaseBrewery.start#171: Joined the room. Jicofo 2021-09-27 10:09:32.303 INFO: [29] BridgeSelector.addJvbAddress#118: Added new videobridge: Bridge[jid=jvbbrewery@internal.auth.domain.tld/hvb2, relayId=192.168.200.146:4096, region=region, stress=0.00] Jicofo 2021-09-27 10:09:32.308 INFO: [29] JvbDoctor.addBridge#140: Scheduled health-check task for: jvbbrewery@internal.auth.domain.tld/jvb2 Jicofo 2021-09-27 10:09:32.316 INFO: [17] [type=jibri brewery=jibribrewery] BaseBrewery.start#171: Joined the room. Jicofo 2021-09-27 10:09:33.330 INFO: [29] [type=bridge brewery=jvbbrewery] BaseBrewery.addInstance#358: Added brewery instance: jvbbrewery@internal.auth.domain.tld/jvb1 Jicofo 2021-09-27 10:09:33.331 INFO: [29] BridgeSelector.addJvbAddress#118: Added new videobridge: Bridge[jid=jvbbrewery@internal.auth.domain.tld/jvb1, relayId=192.168.200.31:4096, region=region, stress=0.00] Jicofo 2021-09-27 10:09:33.331 INFO: [29] JvbDoctor.addBridge#140: Scheduled health-check task for: jvbbrewery@internal.auth.domain.tld/jvb1

And then we’re back to the healt check failed for JVB2

JVB videobridgeversion:

dpkg -l | grep jitsi
ii jitsi-videobridge2 2.1-551-g2ad6eb0b-1 all WebRTC compatible Selective Forwarding Unit (SFU)

JMS package version:

dpkg -l | grep jitsi
ii jitsi-meet 2.0.6293-1 all WebRTC JavaScript video conferences
ii jitsi-meet-prosody 1.0.5307-1 all Prosody configuration for Jitsi Meet
ii jitsi-meet-turnserver 1.0.5307-1 all Configures coturn to be used with Jitsi Meet
ii jitsi-meet-web 1.0.5307-1 all WebRTC JavaScript video conferences
ii jitsi-meet-web-config 1.0.5307-1 all Configuration for web serving of Jitsi Meet
dpkg -l | grep prosody
ii jitsi-meet-prosody 1.0.5307-1 all Prosody configuration for Jitsi Meet
ii prosody 0.11.2-1+deb10u2 amd64 Lightweight Jabber/XMPP server
ii prosody-modules 0.0~hg20190203.b54e98d5c4a1+dfsg-1+deb10u1 all Selection of community modules for Prosody

The bridge is able to join the MUC (confirmed both in prosody.log and in jvb.log). So I’m frankly out of ideas where to even look next.

Any ideas?

I’ve noticed one thing that does differ between the working and the non-working JVB2. In prosody.log the following appears just before the “something went wrong”-error.

Sep 28 09:54:09 speakerstats.domain.tld:speakerstats_component warn A module has been configured that triggers external events.
Sep 28 09:54:09 speakerstats.domain.tld:speakerstats_component warn Implement this lib to trigger external events.

But as far as I understand these errors cand be ignored? Still curious as to why they show up on one bridge but not the other.

I checked the fine manual and sadly it’s outdated (there is a completely obsolete reference to port 5347 ‘for jicofo’ that is not necessary anymore). You should be better off trusting the source and even this forum itself

Interesting, looks like it has been updated since I last used it to get websockets going. My config is lacking this line:

proxy_set_header Host alpha.jitsi.net;

I’ll have to do some testing.

I’m sad to say I still haven’t figured out what’s wrong here, even tried wiping the bridge (jvb2) clean and reconfigure it from scratch. Currently running the latest stable (released 2021-12-10).

Hi, did you resolve this problem ? I have the same issue…

@noje

I ended up completely re-installing Debian (10) then re-adding my configs. I only did a purge of the packages and config files last time.

Initially I had the exact same behaviour, meeting crashed when JVB2 was being used. But then I noticed I had accidentally set the exact same org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME= (in sip-communicator.properties) value as on JVB1 (which is something I’m absolutely sure I did not have before the re-install as it’s not present in the backup config files). I decided to completely change the names of both of my videobridges and voila… it just started working…

Still getting the same health check failed (in jicofo) for the bridge with the older name. As far as I can tell (grep -rnw ‘/’ -e ‘old-jvb-name’) there’s no mention of it on any of the three servers.

So honestly I’m not quite sure what fixed it… I guess we’ll just have to wait and see if the problem eventually re-appears.

Thanks for reply. Yesterday, I changed on my problematic videobridge hostname from “jitsi-jvb08b” to “jitsi-jvb10b and apply it in configuration on all Jitsi machines. It has resolved my problem but I have also health check failed with the older name as you :slight_smile:

could it be that the MUC_NICKNAME was identical on 2 bridges ?

No. I have different MUC_NICKNAME on all of my bridges.