Random disconnection mid meeting on scalable setup

We created a scalable setup recently which seems to be working fine, except there are some random disconnections during long meetings. This could be a network issue but has been reported too often so far. During our testing not all clients were disconnected it seems. The client is disconnected with an “Unfortunately, something went wrong” message and then Jitsi tries to reconnect.

The following are the logs when the disconnection happened. I can’t say for sure but these were the logs around the disconnection event time.

Jicofo logs:

Jicofo 2020-07-08 11:01:51.572 INFO: [1123] org.jitsi.jicofo.discovery.DiscoveryUtil.log() Successfully discovered features for testmeeting@conference.meet.example.com/993372bc in 1
Jicofo 2020-07-08 11:01:51.572 INFO: [1123] org.jitsi.jicofo.AbstractChannelAllocator.log() Using jvbbrewery@internal.auth.meet.example.com/882db71d-0beb-40c9-9501-0ba38ce82d56 to allocate channels for: Participant[testmeeting@conference.meet.example.com/993372bc]@680567209
Jicofo 2020-07-08 11:01:51.584 SEVERE: [1123] org.jitsi.jicofo.AbstractChannelAllocator.log() jvbbrewery@internal.auth.meet.example.com/882db71d-0beb-40c9-9501-0ba38ce82d56 - failed to allocate channels, will consider the bridge faulty: XMPP error:
org.jitsi.protocol.xmpp.colibri.exception.ColibriException: XMPP error:
at org.jitsi.impl.protocol.xmpp.colibri.ColibriConferenceImpl.maybeThrowOperationFailed(ColibriConferenceImpl.java:373)
at org.jitsi.impl.protocol.xmpp.colibri.ColibriConferenceImpl.createColibriChannels(ColibriConferenceImpl.java:277)
at org.jitsi.protocol.xmpp.colibri.ColibriConference.createColibriChannels(ColibriConference.java:112)
at org.jitsi.jicofo.ParticipantChannelAllocator.doAllocateChannels(ParticipantChannelAllocator.java:146)
at org.jitsi.jicofo.AbstractChannelAllocator.allocateChannels(AbstractChannelAllocator.java:271)
at org.jitsi.jicofo.AbstractChannelAllocator.doRun(AbstractChannelAllocator.java:190)
at org.jitsi.jicofo.AbstractChannelAllocator.run(AbstractChannelAllocator.java:150)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Jicofo 2020-07-08 11:01:51.585 SEVERE: [1123] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() One of our bridges failed: jvbbrewery@internal.auth.meet.example.com/882db71d-0beb-40c9-9501-0ba38ce82d56
Jicofo 2020-07-08 11:01:51.585 INFO: [1123] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=ffb29d octo_enabled= false: [[null, null, null, null]]
Jicofo 2020-07-08 11:01:51.585 INFO: [1123] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=ffb29d octo_enabled= false: [[null, null, null]]

JVB logs:

2020-07-08 11:01:50.357 INFO: [9580] [confId=1618b2f93399f0a8 gid=ffb29d stats_id=Ashleigh-3lP conf_name=testmeeting ufrag=cd5at1ecn1b5cu epId=993372bc local_ufrag=cd5at1ecn1b5cu] ConnectivityCheckClient.processTimeout#857: timeout for pair: [64:ff9c:0:0:0:0:3fa:1b93]:10000/udp/prflx -> 49.15.13.113:17115/udp/prflx (stream-993372bc.RTP), failing.
2020-07-08 11:01:51.576 INFO: [9625] [confId=1618b2f93399f0a8 epId=993372bc gid=ffb29d stats_id=Ashleigh-3lP conf_name=testmeeting] AbstractEndpoint.expire#303: Expiring.
2020-07-08 11:01:51.576 INFO: [9625] [confId=1618b2f93399f0a8 epId=993372bc gid=ffb29d stats_id=Ashleigh-3lP conf_name=testmeeting] Transceiver.teardown#315: Tearing down
2020-07-08 11:01:51.576 INFO: [9625] [confId=1618b2f93399f0a8 epId=993372bc gid=ffb29d stats_id=Ashleigh-3lP conf_name=testmeeting] RtpReceiverImpl.tearDown#287: Tearing down
2020-07-08 11:01:51.576 SEVERE: [9607] XmppCommon.handleIQRequest#251: Exception handling IQ request
java.lang.NullPointerException
at org.jitsi.videobridge.shim.ChannelShim.(ChannelShim.java:163)
at org.jitsi.videobridge.shim.ContentShim.createRtpChannel(ContentShim.java:135)
at org.jitsi.videobridge.shim.ContentShim.getOrCreateChannelShim(ContentShim.java:276)
at org.jitsi.videobridge.shim.VideobridgeShim.processChannels(VideobridgeShim.java:143)
at org.jitsi.videobridge.shim.VideobridgeShim.handleColibriConferenceIQ(VideobridgeShim.java:345)
at org.jitsi.videobridge.Videobridge.handleColibriConferenceIQ(Videobridge.java:600)
at org.jitsi.videobridge.Videobridge.handleColibriConferenceIQ(Videobridge.java:585)
at org.jitsi.videobridge.xmpp.XmppCommon.handleIQRequest(XmppCommon.java:228)
at org.jitsi.videobridge.xmpp.XmppCommon.handleIQInternal(XmppCommon.java:179)
at org.jitsi.videobridge.xmpp.XmppCommon.handleIQ(XmppCommon.java:150)
at org.jitsi.videobridge.xmpp.ClientConnectionImpl.handleIq(ClientConnectionImpl.java:108)
at org.jitsi.xmpp.mucclient.IQListener.handleIq(IQListener.java:50)
at org.jitsi.xmpp.mucclient.MucClient.handleIq(MucClient.java:547)
at org.jitsi.xmpp.mucclient.MucClient.access$500(MucClient.java:50)
at org.jitsi.xmpp.mucclient.MucClient$2.handleIQRequest(MucClient.java:511)
at org.jivesoftware.smack.AbstractXMPPConnection$4.run(AbstractXMPPConnection.java:1188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-07-08 11:01:51.576 INFO: [9625] [confId=1618b2f93399f0a8 epId=993372bc gid=ffb29d stats_id=Ashleigh-3lP conf_name=testmeeting] RtpSenderImpl.tearDown#263: Tearing down
2020-07-08 11:01:51.577 INFO: [9625] [confId=1618b2f93399f0a8 epId=993372bc gid=ffb29d stats_id=Ashleigh-3lP conf_name=testmeeting] DtlsTransport.stop#180: Stopping
2020-07-08 11:01:51.577 INFO: [9625] [confId=1618b2f93399f0a8 epId=993372bc local_ufrag=cd5at1ecn1b5cu gid=ffb29d stats_id=Ashleigh-3lP conf_name=testmeeting] IceTransport.stop#235: Stopping
2020-07-08 11:01:51.577 INFO: [9618] [confId=1618b2f93399f0a8 gid=ffb29d stats_id=Ashleigh-3lP componentId=1 conf_name=testmeeting ufrag=cd5at1ecn1b5cu name=stream-993372bc epId=993372bc local_ufrag=cd5at1ecn1b5cu] MergingDatagramSocket$SocketContainer.runInReaderThread#770: Failed to receive: java.net.SocketException: Socket closed
2020-07-08 11:01:51.577 WARNING: [9618] [confId=1618b2f93399f0a8 gid=ffb29d stats_id=Ashleigh-3lP componentId=1 conf_name=testmeeting ufrag=cd5at1ecn1b5cu name=stream-993372bc epId=993372bc local_ufrag=cd5at1ecn1b5cu] MergingDatagramSocket.doRemove#349: Removing the active socket. Won’t be able to send until a new one is elected.
2020-07-08 11:01:51.579 INFO: [9625] [confId=1618b2f93399f0a8 gid=ffb29d stats_id=Ashleigh-3lP componentId=1 conf_name=testmeeting ufrag=cd5at1ecn1b5cu name=stream-993372bc epId=993372bc local_ufrag=cd5at1ecn1b5cu] MergingDatagramSocket.close#142: Closing.
2020-07-08 11:01:51.581 INFO: [9614] [confId=1618b2f93399f0a8 epId=993372bc local_ufrag=cd5at1ecn1b5cu gid=ffb29d stats_id=Ashleigh-3lP conf_name=testmeeting] IceTransport.startReadingData#201: Socket closed, stopping reader

Are these any helpful? Thanks in advance

1 Like

We are facing the same issue. The relevant part of the jvb logs are below.

SEVERE: Exception handling IQ request
java.lang.NullPointerException
	at org.jitsi.videobridge.shim.ChannelShim.<init>(ChannelShim.java:163)
	at org.jitsi.videobridge.shim.ContentShim.createRtpChannel(ContentShim.java:135)
	at org.jitsi.videobridge.shim.ContentShim.getOrCreateChannelShim(ContentShim.java:276)
	at org.jitsi.videobridge.shim.VideobridgeShim.processChannels(VideobridgeShim.java:144)
	at org.jitsi.videobridge.shim.VideobridgeShim.handleColibriConferenceIQ(VideobridgeShim.java:336)
	at org.jitsi.videobridge.Videobridge.handleColibriConferenceIQ(Videobridge.java:424)
	at org.jitsi.videobridge.xmpp.XmppCommon.handleIQRequest(XmppCommon.java:228)
	at org.jitsi.videobridge.xmpp.XmppCommon.handleIQInternal(XmppCommon.java:179)
	at org.jitsi.videobridge.xmpp.XmppCommon.handleIQ(XmppCommon.java:150)
	at org.jitsi.videobridge.xmpp.ClientConnectionImpl.handleIq(ClientConnectionImpl.java:110)
	at org.jitsi.xmpp.mucclient.IQListener.handleIq(IQListener.java:50)
	at org.jitsi.xmpp.mucclient.MucClient.handleIq(MucClient.java:566)
	at org.jitsi.xmpp.mucclient.MucClient.access$700(MucClient.java:50)
	at org.jitsi.xmpp.mucclient.MucClient$2.handleIQRequest(MucClient.java:530)
	at org.jivesoftware.smack.AbstractXMPPConnection$4.run(AbstractXMPPConnection.java:1188)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

@damencho this seems like a bug in jitsi-videobridge. We are on latest jitsi-videobridge2/stable,now 2.1-273-g072dd44b-1

Another things about this error: the client does not automatically connect back to the jvb it gets stuck in the “We are trying to fix this. Reconnecting in 0 seconds…” message.

Probably related to Jitsi-Meet's iceStream returns null when doing handleColibriConferenceIQ

Did you guys ever solve this? We’re experiencing the same periodic disconnects on our scalable setup. It just started a few weeks ago for us.

Hi,
Anyone has a fix for this error?

actually, my team may have solved this…But first, a caveat: There are many potential reasons for disconnects, so we all might be experiencing different root problems. But in our case, the logs showed that all the disconnects were related to the recent web sockets upgrade a month or so ago, I forget the timing. My team tells me they disabled the web sockets code path and we haven’t experienced any disconnects since. Now, I’m not the developer… I’m the project manager. So if you want me to get more detailed info about what my team did, let me know. But that’s what they told me.

Dear Omatic,

could you please check? We are also experiencing sudden user disconnects

Sure, here’s more info from my developer:

“Frequent disconnections started to happen a month ago when we enabled websockets and it was fixed as soon as we disabled websockets and moved back to bosh only. (BOSH/Websockets are enabled/disabled in /etc/jitsi/meet/site-config.js)”

Thank you!

On our side only BOSH is active in mentioned file, so seems, we have some different issue