Websocket channel closes when connection interrupted

I noticed that in our product, when there is a connection interruption (I can recreate this by simply switching my PC from one wifi network to another), the bridge channel websocket connection disconnects and doesn’t re-establish the connection, so any messages we were passing between clients using the websocket channel will no longer go through.

I then tested this in Jitsi’s production deployment and am seeing the same issue and error messages in my console after I switch wifi networks. The video and audio reconnects just fine, but the websocket connection does not.

Is there a way to re-establish the websocket connection after an interruption?

Attached is a screenshot of the console errors (the ones inside the blue box are the ones that started appearing after I switched to another wifi network). Also attaching my full console log.

meet.jit.si-1613409116403.log (160.2 KB)

any update on this issue? @damencho
if we are going to handle it in our application which event should be triggered?

There are retries in lib-jitsi-meet already.

It seems that whatever retries are happening are failing for us in these cases and a full refresh / rejoin is required to re-establish the websocket connection.

Is there some configuration in lib-jitsi-meet we can change to make the retries more successful?

Check the meet.jit.si config we use. I don’t think there is anything special …

I did some more testing and I’m finding that I can consistently recreate the bug when there are two participants in the conference, but not when there are three. When there are three, websockets seem to reconnect successfully.

I first figured it might be related to P2P, but then we disabled P2P in config.js and the above still holds true. With two people, it won’t reconnect, but with three it will.

Here’s what the console log looks like when it doesn’t successfully reconnect. We can see that it initiates the retry, but then it just keeps showing the ‘No opened channel’ error over and over again and never re-establishes the connection

Anything else that might be different between when there are only two people vs more? Or maybe there is something else we need to do to disable P2P in addition to setting ‘P2P enabled’ to false?

Are you reproducing the same on meet.jit.si?
How are you testing it?

I’m testing it on our own product which has a forked version of Jitsi

Within our product, we use websockets to send quite a few messages between clients, so easy to see when they fail to send

I could also test it and try to reproduce on meet.jit.si, just not sure what would be the best way to generate the websocket messages. Thoughts?

But how do you test the disconnect? Asking for the steps so I can test it on meet.jit.si?

Well that part is a bit funny cause I have a unique ability to reproduce this bug consistently with my specific network setup. We also know this bug happens in other instances (ie it also happens to other users), but it’s more intermittent for them. With my setup, I can reproduce it consistently.

I have my home network set up with two routers (one is connected to the other), so what I’ll do is simply switch my wifi from one router’s network to the other. When I make that switch, that’s when the websocket connection drops. And then as I mentioned above, if there are only two people in the conference the connection does not recover.

Happy to help you test on meet.jit.si with my setup, just need a way to generate websocket messages within Jitsi. Are there any features which use websockets that I can just spam (for example, raising your hand or muting / unmuting yourself, etc)?

Nope, its the stats and dominant speaker events going through the jvb websocket. Why you need to spam it?

Maybe ‘spam’ was the wrong term, but at least generate messages over websockets at a certain rate so that I can test if they are working or not.

I’ll test on meet.jit.si in a bit by alternating dominant speaker and seeing if those messages are going through

Okay, just tested it. I was not able to recreate the issue on meet.jit.si. Even with only two people in the conference, every time I would switch wifi networks, the web socket connection would close and then re-open correctly right after.

So either there is something in our product’s configuration that we need to change, or might be a timing of the code (we are a bit behind in merging the latest Jitsi code, working off of mid-January code)

Here is what I would see in the console log. One thing I noticed is that when the websocket channel re-opens, it would also say that a data channel is open. Does that mean that Jitsi’s configuration is to use both websockets and SCTP data channels? Or does the data channel in this case refer to the websocket channel?

Correct. Its either websocket or sctp.

Got it - thanks

We’ll first check our configs to make sure they match the Jitsi repo.

If so, then we’ll merge the latest Jitsi code into our product and see if that fixes it.

Will report back (the latter might take a bit of time)

Hi @damencho, in our case we simulate disconnect by putting the network in “Offline”

then here’s the console log

after several seconds there was a session termination in jicofo and in jvb

so when I turn on back the network the WS is trying to reconnect but the endpoint is not existing anymore since it was terminated.
image

jvb log client WS retries

that’s why we still need to click refresh to create a new session and WS…
any suggestion on how to resolve this?

@Bobi_Tena do you repro the same on meet.jit.si? What happens there?

no , I didn’t report this

@damencho could you help us on this?

I didn’t get your answer @Bobi_Tena. You were showing a case about the websockets … and I was asking when you test the same steps on meet.jit.si what do you see?