Shutting down a JVB crashes a conference on another JVB

Hi @gpolitis, thanks for having a look at this. We were also assuming the misaligned default settings regarding octo could be the reason for this, but we already verified that explicitely disabling octo in both jicofo and the jvbs does not resolve this issue:

Hey @Kordian_Kowalski, you also tried to explicitly disable octo in jicofo and nothing changed, didn’t you? I think @gpolitis would be interested in the config files from that attempt as well…

Yeah, we already posted that it failed a couple of comments above.

I don’t have the exact files any more - reverted the changes after they didn’t solve the issue, but here’s what I ended up with before:
jicofo.conf:

jicofo {
    octo {
        enabled = false
    }
}

jvb.conf:

videobridge {
    http-servers {
        public {
            port = 9090
        }
    }
    websockets {
        enabled = true
        domain = "vanilla.OUR-DOMAIN:443"
        tls = true
        server-id = jvb-01
    }
    octo {
        enabled = false
    }
}

Just to be sure, I re-tested this with the configuration I pasted above and the result is the same.

1 Like

looking at Jicofo code, it’s a bit disturbing that this octo-enabling flag is used only one time, to be passed to the select() method of the BridgeSelectionStrategy class, when it’s from the caller pow it’s called OctoConfig.config.getEnabled() and in the callee it’s called allowMultiBridge. This is suggesting that somehow there is an mental equation like:

octo === multibridge => !octo == one bridge.

Maybe you could try to enable debug logging in jicofo
I think that it should be

.level=DEBUG (or FINE ?)
in logging.properties.

jicofo.log (602.2 KB) - Same config as above (octo enabled = false)

Seeing lots of logs containing “octo”

That’s really strange!!! could you please grab a heapdump from a failed bridge & jicofo? You can use the scripts in /usr/share/{jicofo, jitsi-videobridge}/collect-dump.sh and share with me somewhere? Feel free to send me a dropbox link or anything else you use.

this is the one I was waiting for:
Jicofo 2021-03-25 14:59:48.942 FINE: [30] org.jitsi.jicofo.bridge.BridgeSelectionStrategy.log() Existing bridge does not have a relay, will not consider other bridges.
here:

that’s a bit suspicious. May be a false trail of course.

@gpatel-fr I’ll look into that. In the meantime, do you guys have the option to run a custom baked Jicofo and try this fix: Don't propagate sources if octo is disabled. by gpolitis · Pull Request #713 · jitsi/jicofo · GitHub ?

I just built and ran code from @gpolitis 's PR and the issue seems to be fixed.

There are still octo logs in jicofo though:
jicofo.log (602.2 KB)
jvb-01.log (211.3 KB)
jvb-02.log (63.7 KB)

And octo should be disabled:

jicofo {
    octo {
        enabled = false
    }
    xmpp: {
      client: {
        client-proxy: focus.vanilla.OUR-DOMAIN
      }
    }
}
2 Likes

Do you still need the dump?

No, thanks!

I didn’t spot anything functional, most matches are initialization and stats, which are harmless but let me know if I missed anything.

let me know if I missed anything.

I didn’t really look into it, just saw “octo” in there - if you say it’s fine then it’s fine :slight_smile:

I think we can consider this issue resolved then? I assume the fix will be available in a future release?

Yes, the issue is fixed. The PR is already merged and there should already be a nightly that has it. I’m not sure when it’s going to hit stable, I’ll let @damencho comment on that.

Soon, I hope. This or next week.

1 Like

I am using jitsi-meet version 2.0.5870, prosody version 1.0.4985, jitsi-meet-web version 1.0.4985, jitsi-videobridge2 version 2.1-492 and jicofo version 1.0-747.
A load balancing platform with 4 servers is set up where the first server is loaded with all the above components and the rest 3 servers with only videobridge.
Since the video quality was poor when the meeting is hosted on all bridges except the first one, all these videobridges were stopped for the time being. Now only the first server is running with all packages. But, the meetings crashes at many times with the message ‘Something went wrong’ and rejoins. Sometimes it crashes continuously.
Jicofo logs include
“JitsiMeetConferenceImpl.selectBridge#673: Can not invite participant, no bridge available”.

Sounds like websockets are not properly configured in your deployment, for one. ALso seems like some kind of misconfiguration is tripping off the local JVB. What do you see in your browser console?

Now only the first videobridge is active. All the packages like jicofo, prosody and nginx are also running in the same server.The other 3 video bridges are in shutdown state. As per my understanding, by default, web socket is enabled for first bridge. Do we need to add any configuration for web socket?
On the browsers we get the message ‘Unfortunately Something went wrong’ and then rejoins. Will there be any problem if users with both Mozilla and Chrome are participating?

If JMS and JVB are on the same server then the websocket should work. No need to configure anything. Could you check the JVB status?

systemctl status jitsi-videobridge2.service

Status shows active (running).