JVBs disconnecting under load

HI,
We have three Videobridges on three ec2 using SplitBridgeSelectionStrategy
when we load test this setup somewhere around 70-80 participants I’am being disconnected from the meeting. The instances are using about 40% CPU of and 30% of RAM
In logs, it appears that Videobridges disconnect randomly from prosody.

prosody_1  | c2s55d48d53bc00                                                         info	Client disconnected: connection closed
web_1      | 18.184.104.82 - - [20/Jul/2021:08:38:54 +0000] "GET /xmpp-websocket?room=poc HTTP/1.1" 101 483371 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/91.0.4472.164 Safari/537.36"
jicofo_1   | Jul 20, 2021 8:38:55 AM org.jitsi.utils.logging2.LoggerImpl log
jicofo_1   | INFO: Terminating 2faac577, reason: general-error, send session-terminate: true
jicofo_1   | Jul 20, 2021 8:38:55 AM org.jitsi.utils.logging2.LoggerImpl log
jicofo_1   | INFO: Removed participant 2faac577 removed=true
jicofo_1   | Jul 20, 2021 8:38:55 AM org.jitsi.utils.logging2.LoggerImpl log
jicofo_1   | SEVERE: Channel allocator failed: 
jicofo_1   | java.lang.NullPointerException
jicofo_1   | 	at org.jitsi.jicofo.ParticipantChannelAllocator.invite(ParticipantChannelAllocator.java:171)
jicofo_1   | 	at org.jitsi.jicofo.AbstractChannelAllocator.doRun(AbstractChannelAllocator.java:200)
jicofo_1   | 	at org.jitsi.jicofo.AbstractChannelAllocator.run(AbstractChannelAllocator.java:133)
jicofo_1   | 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
jicofo_1   | 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
jicofo_1   | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
jicofo_1   | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
jicofo_1   | 	at java.lang.Thread.run(Thread.java:748)
jicofo_1   | 
web_1      | 18.184.104.82 - - [20/Jul/2021:08:38:55 +0000] "GET /xmpp-websocket?room=poc HTTP/1.1" 101 380388 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/91.0.4472.164 Safari/537.36"
prosody_1  | c2s55d48ccbe650                                                         info	Client disconnected: connection closed
prosody_1  | c2s55d4913563a0                                                         info	Client disconnected: connection closed
web_1      | 18.184.104.82 - - [20/Jul/2021:08:38:55 +0000] "GET /xmpp-websocket?room=poc HTTP/1.1" 101 296203 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/91.0.4472.164 Safari/537.36"
web_1      | 2021/07/20 08:38:56 [error] 500#500: *377 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 18.184.104.82, server: _, request: "GET /xmpp-websocket?room=poc HTTP/1.1", upstream: "http://172.19.0.2:5280/xmpp-websocket?room=poc", host: "jitsi.kingsch.at"
web_1      | 18.184.104.82 - - [20/Jul/2021:08:38:56 +0000] "GET /xmpp-websocket?room=poc HTTP/1.1" 101 399776 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/91.0.4472.164 Safari/537.36"
prosody_1  | c2s55d48cc96120                                                         info	Client disconnected: connection closed
prosody_1  | c2s55d490c14260                                                         info	Client disconnected: connection closed
web_1      | 18.184.104.82 - - [20/Jul/2021:08:38:56 +0000] "GET /xmpp-websocket?room=poc HTTP/1.1" 101 386430 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/91.0.4472.164 Safari/537.36"
prosody_1  | c2s55d48c7905b0                                                         info	Client disconnected: connection closed
web_1      | 18.184.104.82 - - [20/Jul/2021:08:38:56 +0000] "GET /xmpp-websocket?room=poc HTTP/1.1" 101 489148 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/91.0.4472.164 Safari/537.36"
jicofo_1   | Jul 20, 2021 8:38:56 AM org.jitsi.utils.logging2.LoggerImpl log
jicofo_1   | INFO: Chat room event Left member=ChatMember[poc@muc.jitsi.kingsch.at/5e58cae1, jid: null]@343369732
jicofo_1   | Jul 20, 2021 8:38:56 AM org.jitsi.utils.logging2.LoggerImpl log
jicofo_1   | INFO: Member left:5e58cae1
jicofo_1   | Jul 20, 2021 8:38:56 AM org.jitsi.utils.logging2.LoggerImpl log
jicofo_1   | INFO: Terminating 5e58cae1, reason: gone, send session-terminate: false

Do you guys know what might be the cause?

Regards.

Since prosody can use only one core, there may be an issue related with the prosody load. Try to stop JVB which is on the JMS server.

And check if limits is enabled for prosody. If so then you may need to increase the rate limit.

egrep "(limits|rate)" /etc/prosody -R

Thanks for your reply. JVB was already removed from the main machine. Unfortunately, after increasing limits 10 times to

  c2s = {
    rate = "100kb/s";
  };
  s2sin = {
    rate = "300kb/s";
  };
}

it crashes as well. throwing

jicofo_1   | WARNING: Smack: XMPPConnection closed due to an exception (XMPPTCPConnection[focus@auth.jitsi.somedomain/focus] (0))
jicofo_1   | org.jivesoftware.smack.XMPPException$StreamErrorException: not-well-formed You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
jicofo_1   | <stream:error><not-well-formed xmlns='urn:ietf:params:xml:ns:xmpp-streams'/></stream:error>
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1064)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
jicofo_1   | 	at java.lang.Thread.run(Thread.java:748)
jicofo_1   | 
jicofo_1   | Jul 21, 2021 7:54:26 AM org.jitsi.utils.logging2.LoggerImpl log
jicofo_1   | SEVERE: Channel allocator failed: 
jicofo_1   | java.lang.NullPointerException
jicofo_1   | 	at org.jitsi.jicofo.ParticipantChannelAllocator.invite(ParticipantChannelAllocator.java:171)
jicofo_1   | 	at org.jitsi.jicofo.AbstractChannelAllocator.doRun(AbstractChannelAllocator.java:200)
jicofo_1   | 	at org.jitsi.jicofo.AbstractChannelAllocator.run(AbstractChannelAllocator.java:133)
jicofo_1   | 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
jicofo_1   | 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
jicofo_1   | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
jicofo_1   | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
jicofo_1   | 	at java.lang.Thread.run(Thread.java:748)
jicofo_1   | 
jicofo_1   | INFO: Terminate session: poc@muc.jitsi.somedomain/7b9827ca, reason: gone, send terminate: true
jicofo_1   | Jul 21, 2021 7:54:26 AM org.jitsi.utils.logging2.LoggerImpl log
jicofo_1   | SEVERE: No connection - unable to send packet: <iq to='poc@muc.jitsi.somedomain/7b9827ca' from='focus@auth.jitsi.somedomain/focus' id='EiGqD-50002' type='set'><jingle xmlns='urn:xmpp:jingle:1' action='session-terminate' sid='dag37p4u5gbct'><reason><gone/></reason></jingle></iq>
jicofo_1   | org.jivesoftware.smack.SmackException$NotConnectedException: The connection XMPPTCPConnection[focus@auth.jitsi.somedomain/focus] (0) is no longer connected. done=true smResumptionPossible=false
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketWriter.throwNotConnectedExceptionIfDoneAndResumptionNotPossible(XMPPTCPConnection.java:1327)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection.throwNotConnectedExceptionIfAppropriate(XMPPTCPConnection.java:358)
jicofo_1   | 	at org.jivesoftware.smack.AbstractXMPPConnection.sendStanza(AbstractXMPPConnection.java:670)
jicofo_1   | 	at org.jitsi.jicofo.xmpp.UtilKt.tryToSendStanza(Util.kt:71)
jicofo_1   | 	at org.jitsi.protocol.xmpp.AbstractOperationSetJingle.terminateSession(AbstractOperationSetJingle.java:525)
jicofo_1   | 	at org.jitsi.protocol.xmpp.AbstractOperationSetJingle.terminateHandlersSessions(AbstractOperationSetJingle.java:486)
jicofo_1   | 	at org.jitsi.jicofo.JitsiMeetConferenceImpl.stop(JitsiMeetConferenceImpl.java:402)
jicofo_1   | 	at org.jitsi.jicofo.JitsiMeetConferenceImpl.registrationChanged(JitsiMeetConferenceImpl.java:1198)
jicofo_1   | 	at org.jitsi.impl.protocol.xmpp.AbstractXmppProvider.fireRegistrationStateChanged(AbstractXmppProvider.java:88)
jicofo_1   | 	at org.jitsi.impl.protocol.xmpp.XmppProviderImpl.fireRegistrationStateChanged(XmppProviderImpl.java:322)
jicofo_1   | 	at org.jitsi.impl.protocol.xmpp.AbstractXmppProvider.setRegistered(AbstractXmppProvider.java:119)
jicofo_1   | 	at org.jitsi.impl.protocol.xmpp.XmppProviderImpl$XmppConnectionListener.connectionClosedOnError(XmppProviderImpl.java:400)
jicofo_1   | 	at org.jivesoftware.smack.AbstractXMPPConnection.callConnectionClosedOnErrorListener(AbstractXMPPConnection.java:1335)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection.notifyConnectionError(XMPPTCPConnection.java:931)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection.access$3200(XMPPTCPConnection.java:151)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1258)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
jicofo_1   | 	at java.lang.Thread.run(Thread.java:748)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketWriter.throwNotConnectedExceptionIfDoneAndResumptionNotPossible(XMPPTCPConnection.java:1327)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection.throwNotConnectedExceptionIfAppropriate(XMPPTCPConnection.java:358)
jicofo_1   | 	at org.jivesoftware.smack.AbstractXMPPConnection.sendStanza(AbstractXMPPConnection.java:670)
jicofo_1   | 	at org.jitsi.jicofo.xmpp.UtilKt.tryToSendStanza(Util.kt:71)
jicofo_1   | 	at org.jitsi.protocol.xmpp.AbstractOperationSetJingle.terminateSession(AbstractOperationSetJingle.java:525)
jicofo_1   | 	at org.jitsi.protocol.xmpp.AbstractOperationSetJingle.terminateHandlersSessions(AbstractOperationSetJingle.java:486)
jicofo_1   | 	at org.jitsi.jicofo.JitsiMeetConferenceImpl.stop(JitsiMeetConferenceImpl.java:402)
jicofo_1   | 	at org.jitsi.jicofo.JitsiMeetConferenceImpl.registrationChanged(JitsiMeetConferenceImpl.java:1198)
jicofo_1   | 	at org.jitsi.impl.protocol.xmpp.AbstractXmppProvider.fireRegistrationStateChanged(AbstractXmppProvider.java:88)
jicofo_1   | 	at org.jitsi.impl.protocol.xmpp.XmppProviderImpl.fireRegistrationStateChanged(XmppProviderImpl.java:322)
jicofo_1   | 	at org.jitsi.impl.protocol.xmpp.AbstractXmppProvider.setRegistered(AbstractXmppProvider.java:119)
jicofo_1   | 	at org.jitsi.impl.protocol.xmpp.XmppProviderImpl$XmppConnectionListener.connectionClosedOnError(XmppProviderImpl.java:400)
jicofo_1   | 	at org.jivesoftware.smack.AbstractXMPPConnection.callConnectionClosedOnErrorListener(AbstractXMPPConnection.java:1335)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection.notifyConnectionError(XMPPTCPConnection.java:931)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection.access$3200(XMPPTCPConnection.java:151)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1258)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
jicofo_1   | 	at java.lang.Thread.run(Thread.java:748)
jicofo_1   | 
jicofo_1   | SEVERE: No connection - unable to send packet: <iq to='poc@muc.jitsi.somedomain/728d27e8' from='focus@auth.jitsi.somedomain/focus' id='EiGqD-49999' type='set'><jingle xmlns='urn:xmpp:jingle:1' action='session-terminate' sid='c33bnpncek13q'><reason><gone/></reason></jingle></iq>
jicofo_1   | org.jivesoftware.smack.SmackException$NotConnectedException: The connection XMPPTCPConnection[focus@auth.jitsi.somedomain/focus] (0) is no longer connected. done=true smResumptionPossible=false
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketWriter.throwNotConnectedExceptionIfDoneAndResumptionNotPossible(XMPPTCPConnection.java:1327)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection.throwNotConnectedExceptionIfAppropriate(XMPPTCPConnection.java:358)
jicofo_1   | 	at org.jivesoftware.smack.AbstractXMPPConnection.sendStanza(AbstractXMPPConnection.java:670)
jicofo_1   | 	at org.jitsi.jicofo.xmpp.UtilKt.tryToSendStanza(Util.kt:71)
jicofo_1   | 	at org.jitsi.protocol.xmpp.AbstractOperationSetJingle.terminateSession(AbstractOperationSetJingle.java:525)
jicofo_1   | 	at org.jitsi.protocol.xmpp.AbstractOperationSetJingle.terminateHandlersSessions(AbstractOperationSetJingle.java:486)
jicofo_1   | 	at org.jitsi.jicofo.JitsiMeetConferenceImpl.stop(JitsiMeetConferenceImpl.java:402)
jicofo_1   | 	at org.jitsi.jicofo.JitsiMeetConferenceImpl.registrationChanged(JitsiMeetConferenceImpl.java:1198)
jicofo_1   | 	at org.jitsi.impl.protocol.xmpp.AbstractXmppProvider.fireRegistrationStateChanged(AbstractXmppProvider.java:88)
jicofo_1   | 	at org.jitsi.impl.protocol.xmpp.XmppProviderImpl.fireRegistrationStateChanged(XmppProviderImpl.java:322)
jicofo_1   | 	at org.jitsi.impl.protocol.xmpp.AbstractXmppProvider.setRegistered(AbstractXmppProvider.java:119)
jicofo_1   | 	at org.jitsi.impl.protocol.xmpp.XmppProviderImpl$XmppConnectionListener.connectionClosedOnError(XmppProviderImpl.java:400)
jicofo_1   | 	at org.jivesoftware.smack.AbstractXMPPConnection.callConnectionClosedOnErrorListener(AbstractXMPPConnection.java:1335)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection.notifyConnectionError(XMPPTCPConnection.java:931)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection.access$3200(XMPPTCPConnection.java:151)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1258)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
jicofo_1   | 	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
jicofo_1   | 	at java.lang.Thread.run(Thread.java:748)
jicofo_1   | 
jicofo_1   | Jul 21, 2021 7:54:26 AM org.jitsi.utils.logging2.LoggerImpl log
jicofo_1   | INFO: Terminate session: poc@muc.jitsi.somedomain/db7ddeeb, reason: gone, send terminate: true

Also, i noticed JVBs are reporting lots of these from the very beginning of the conference and only 1/3 of enabled webcams are visible

jvb_1  | Jul 21, 2021 8:15:57 AM org.jitsi.utils.logging2.LoggerImpl log
jvb_1  | WARNING: Error sending data
jvb_1  | java.lang.IllegalArgumentException: unresolved address
jvb_1  | 	at java.net.DatagramPacket.setSocketAddress(DatagramPacket.java:316)
jvb_1  | 	at java.net.DatagramPacket.<init>(DatagramPacket.java:144)
jvb_1  | 	at org.jitsi.videobridge.transport.udp.UdpTransport.send(UdpTransport.kt:123)
jvb_1  | 	at org.jitsi.videobridge.transport.udp.UdpTransport.send(UdpTransport.kt:135)
jvb_1  | 	at org.jitsi.videobridge.octo.OctoRelayService$2.sendData(OctoRelayService.kt:85)
jvb_1  | 	at org.jitsi.videobridge.transport.octo.BridgeOctoTransport.sendData(BridgeOctoTransport.kt:231)
jvb_1  | 	at org.jitsi.videobridge.transport.octo.BridgeOctoTransport.sendMediaData(BridgeOctoTransport.kt:180)
jvb_1  | 	at org.jitsi.videobridge.octo.ConfOctoTransport.doSend(ConfOctoTransport.java:292)
jvb_1  | 	at org.jitsi.nlj.util.PacketInfoQueue$sam$org_jitsi_utils_queue_PacketQueue_PacketHandler$0.handlePacket(PacketInfoQueue.kt)
jvb_1  | 	at org.jitsi.utils.queue.PacketQueue$HandlerAdapter.handleItem(PacketQueue.java:380)
jvb_1  | 	at org.jitsi.utils.queue.AsyncQueueHandler$1.run(AsyncQueueHandler.java:133)
jvb_1  | 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
jvb_1  | 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
jvb_1  | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
jvb_1  | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
jvb_1  | 	at java.lang.Thread.run(Thread.java:748)

It has something to do with Octo and might be related.

Maybe you should increase more