Jitsi torture test, slow connections and at moments disconnects

Hello, I’ve set up a jitsi meet server on production and I am testing it just before pushing it live.

I have tested it beforehand with 5 of my colleagues and it looked ok, but I am expecting about 300-500 users to join on a one-to-one conversation. So 150 conferences with 2 people each.

I started using jitsi meet torture, but when I request for 30 participants to join immediately it seems to degrade the performance of the server. Whenever the participants join I try to join in one of the rooms and it takes about 50 seconds to join and when I am inside the videos come really slow.

The other strange thing is that sometimes when the participants (selenium nodes) leave the “Disconnected” screen is shown and it waits for 20 seconds to reconnect.

The jitsi meet server is on a “c5.xlarge” server and the jitsi videobridge on a c5n.xlarge. Would it be able to give me some insights on what might the problem be?

I’ve attached a tarball zip with the logs thank you!

https://openvidu-test2.s3.amazonaws.com/jicofo-and-prosod.tar.gz
https://openvidu-test2.s3.amazonaws.com/jvb.log

Go through this thread there are several recommendations and tweaks

Yeah, I had seen the thread, and pretty much have done everything in it. But the problem is that the problem there was reported with 600 users, mine is about 10-20.

I’ve pretty much increased the limit of everything and it still disconnects and has slow join time.

The other thing I am noticing in prosody is this:

Jun 24 08:03:12 mod_posix   warn    Received SIGTERM
Jun 24 08:03:12 startup info    Shutting down: Received SIGTERM

and in jitsi videobridge.

[confId=b513c0ae12c52e44 gid=43617 stats_id=Cali-Qaz componentId=1 conf_name=fast0@conference.my-domain.com ufrag=544fq1f8v2ntro name=stream-2efef895 epId=2efef895 local_ufrag=544fq1f8v2ntro] MergingDatagramSocket.doRemove#349: Removing the active socket. Won't be able to send until a new one is elected.
JVB 2021-06-24 14:06:51.254 WARNING: [26] org.jivesoftware.smack.AbstractXMPPConnection.callConnectionClosedOnErrorListener: Connection XMPPTCPConnection[jvb@auth.my-domain.com/6785jE1b] (0) closed with error
org.jivesoftware.smack.XMPPException$StreamErrorException: system-shutdown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
<stream:error><system-shutdown xmlns='urn:ietf:params:xml:ns:xmpp-streams'/><text>Received SIGTERM</text></stream:error>
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1064)
                at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
                        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
                                at java.lang.Thread.run(Thread.java:748)
                                JVB 2021-06-24 14:06:51.254 WARNING: [26] [hostname=my-domain.com id=shard] MucClient$1.connectionClosedOnError#277: Closed on error:
                                org.jivesoftware.smack.XMPPException$StreamErrorException: system-shutdown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
                                <stream:error><system-shutdown xmlns='urn:ietf:params:xml:ns:xmpp-streams'/><text>Received SIGTERM</text></stream:error>
                                        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1064)
                                                at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
                                                        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
                                                                at java.lang.Thread.run(Thread.java:748)
                                                                JVB 2021-06-24 14:06:54.464 WARNING: [24] [hostname=my-domain.com id=shard] MucClient.setPresenceExtensions#420: Cannot set presence extension: not connected.
                                                                JVB 2021-06-24 14:07:41.585 WARNING: [141] org.jivesoftware.smack.AbstractXMPPConnection.callConnectionClosedOnErrorListener: Connection XMPPTCPConnection[jvb@auth.my-domain.com/sefe9bkL] (0) closed with error
                                                                org.jivesoftware.smack.XMPPException$StreamErrorException: system-shutdown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
                                                                <stream:error><system-shutdown xmlns='urn:ietf:params:xml:ns:xmpp-streams'/><text>Received SIGTERM</text></stream:error>
                                                                        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1064)
                                                                                at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
                                                                                        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
                                                                                                at java.lang.Thread.run(Thread.java:748)
                                                                                                JVB 2021-06-24 14:07:41.585 WARNING: [141] [hostname=my-domain.com id=shard] MucClient$1.connectionClosedOnError#277: Closed on error:
                                                                                                org.jivesoftware.smack.XMPPException$StreamErrorException: system-shutdown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
                                                                                                <stream:error><system-shutdown xmlns='urn:ietf:params:xml:ns:xmpp-streams'/><text>Received SIGTERM</text></stream:error>
                                                                                                        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1064)
                                                                                                                at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
                                                                                                                        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
                                                                                                                                at java.lang.Thread.run(Thread.java:748)

@damencho And here is the jicofo

Jicofo 2021-06-24 14:07:41.642 WARNING: [15] org.jivesoftware.smack.AbstractXMPPConnection.callConnectionClosedOnErrorListener: Connection XMPPTCPConnection[focus@auth.my-domain.com/focus] (0) closed with error
org.jivesoftware.smack.XMPPException$StreamErrorException: system-shutdown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
<stream:error><system-shutdown xmlns='urn:ietf:params:xml:ns:xmpp-streams'/><text>Received SIGTERM</text></stream:error>
    at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1064)
    at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
    at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
    at java.base/java.lang.Thread.run(Thread.java:829)
Jicofo 2021-06-24 14:07:41.647 SEVERE: [15] [xmpp_connection=client] XmppProviderImpl$XmppConnectionListener.connectionClosedOnError#366: XMPP connection closed on error: system-shutdown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
<stream:error><system-shutdown xmlns='urn:ietf:params:xml:ns:xmpp-streams'/><text>Received SIGTERM</text></stream:error>
Jicofo 2021-06-24 14:07:41.653 INFO: [15] [type=bridge brewery=jvbbrewery] BaseBrewery.stop#199: Left the room.
Jicofo 2021-06-24 14:07:41.664 INFO: [15] [type=bridge brewery=jvbbrewery] BaseBrewery.removeInstance#350: Removed brewery instance: jvbbrewery@internal.auth.my-domain.com/12a4db1d-5513-44b3-8b5e-759a8b346d4f
Jicofo 2021-06-24 14:07:41.664 INFO: [15] BridgeSelector.removeJvbAddress#135: Removing JVB: jvbbrewery@internal.auth.my-domain.com/12a4db1d-5513-44b3-8b5e-759a8b346d4f
Jicofo 2021-06-24 14:07:41.665 FINE: [15] BridgeSelector.notifyBridgeDown#279: Propagating bridge went down event: jvbbrewery@internal.auth.my-domain.com/12a4db1d-5513-44b3-8b5e-759a8b346d4f
Jicofo 2021-06-24 14:07:41.682 INFO: [15] JvbDoctor.removeBridge#152: Stopping health-check task for: jvbbrewery@internal.auth.my-domain.com/12a4db1d-5513-44b3-8b5e-759a8b346d4f
Jicofo 2021-06-24 14:07:41.683 INFO: [15] [room=constantchangesdemandmore@conference.my-domain.com] JitsiMeetConferenceImpl.registrationChanged#1209: XMPP disconnected.
Jicofo 2021-06-24 14:07:41.691 SEVERE: [15] UtilKt.tryToSendStanza#73: No connection - unable to send packet: <presence to='constantchangesdemandmore@conference.my-domain.com/focus' id='prfGb-60'><etherpad xmlns='http://jitsi.org/jitmeet/etherpad'>constantchangesdemandmore</etherpad><versions xmlns='http://jitsi.org/jitmeet'><component xmlns='http://jitsi.org/jitmeet' name='focus'>1.0.747</component></versions><c xmlns='http://jabber.org/protocol/caps' hash='sha-1' node='http://jitsi.org/jicofo' ver='Lg0vhCNhxjoeKJi2/hukdsizNWA='/><conference-properties xmlns='http://jitsi.org/protocol/focus'><property xmlns='http://jitsi.org/protocol/focus' key='support-terminate-restart' value='true'/><property xmlns='http://jitsi.org/protocol/focus' key='bridge-count' value='0'/></conference-properties></presence>
org.jivesoftware.smack.SmackException$NotConnectedException: The connection XMPPTCPConnection[focus@auth.my-domain.com/focus] (0) is no longer connected. done=true smResumptionPossible=false
    at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketWriter.throwNotConnectedExceptionIfDoneAndResumptionNotPossible(XMPPTCPConnection.java:1327)
    at org.jivesoftware.smack.tcp.XMPPTCPConnection.throwNotConnectedExceptionIfAppropriate(XMPPTCPConnection.java:358)
    at org.jivesoftware.smack.AbstractXMPPConnection.sendStanza(AbstractXMPPConnection.java:670)
    at org.jitsi.jicofo.xmpp.UtilKt.tryToSendStanza(Util.kt:71)
    at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.sendLastPresence(ChatRoomImpl.java:645)
    at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.setPresenceExtension(ChatRoomImpl.java:589)
    at org.jitsi.jicofo.JitsiMeetConferenceImpl.setConferenceProperty(JitsiMeetConferenceImpl.java:516)
    at org.jitsi.jicofo.JitsiMeetConferenceImpl.setConferenceProperty(JitsiMeetConferenceImpl.java:498)
    at org.jitsi.jicofo.JitsiMeetConferenceImpl.disposeConference(JitsiMeetConferenceImpl.java:1058)
    at org.jitsi.jicofo.JitsiMeetConferenceImpl.stop(JitsiMeetConferenceImpl.java:400)
    at org.jitsi.jicofo.JitsiMeetConferenceImpl.registrationChanged(JitsiMeetConferenceImpl.java:1210)
    at org.jitsi.impl.protocol.xmpp.AbstractXmppProvider.fireRegistrationStateChanged(AbstractXmppProvider.java:88)
    at org.jitsi.impl.protocol.xmpp.XmppProviderImpl.fireRegistrationStateChanged(XmppProviderImpl.java:319)
    at org.jitsi.impl.protocol.xmpp.AbstractXmppProvider.setRegistered(AbstractXmppProvider.java:119)
    at org.jitsi.impl.protocol.xmpp.XmppProviderImpl$XmppConnectionListener.connectionClosedOnError(XmppProviderImpl.java:372)
    at org.jivesoftware.smack.AbstractXMPPConnection.callConnectionClosedOnErrorListener(AbstractXMPPConnection.java:1335)
    at org.jivesoftware.smack.tcp.XMPPTCPConnection.notifyConnectionError(XMPPTCPConnection.java:931)
    at org.jivesoftware.smack.tcp.XMPPTCPConnection.access$3200(XMPPTCPConnection.java:151)
    at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1258)
    at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
    at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
    at java.base/java.lang.Thread.run(Thread.java:829)

Something is shutting down prosody, this what jvb and jicofo and prosody logs show. Check syslog or something to figure out why

I got it to work better by updating jitsi to the latest versions.

I tried to use jitsi with everything in one place with a lower-tier server than the jitsi videobridge one c5x.large and it was more stable(better connection, better video quality and better join time) than having jitsi videobridge outside. Do you have any idea what it might be?
@damencho

My open ports on jitsi video bridge(to the world) are 443, 4443, and 10000. I am connecting jitsi videobridge and jitsi meet through the VPC and disabled all internal ports between jitsi and the videobridge to the outside world.

From this information, could you see any difference of the videobridge being in the same place and being outside of jitsi meet?

Nope.

On all out deployments the bridges are not on the shard. The only thing I would guess you have forgotten to fix is the websocket to the bridge.
If you move the bridge from the shard you need to fix its IP address here jitsi-meet/jitsi-meet.example at 38b14c5d6254996c049fafae1a3896b3fef3585e · jitsi/jitsi-meet · GitHub so websocket connections to the bridge can be established.
If you have multiple bridges you can make that nginx rule dynamic by server-id and have the server-id to be the internal ip-address. This is how docker images work, you can get the rules from there.

Yeah, I have this setup.

Might have missed another config, but since I updated the version I don’t get those frequent disconnects.

Thank you for your time again and I wish you a great day!

Oh, by the way, I’ve forgotten to mention this error in jvb. EndpointMessageTransport still not connected.

Any clue what does that mean @damencho ?

Same :slight_smile: websocket connection to the bridge is not open.
You can execute in the js console APP.conference._room.rtc._channel.isOpen() to check is it open.

1 Like

Strange it says true.

Could this help?

org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=<local.ip>
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=<public.ip>

Hum … then that’s strange …

If that is wrong you will not see any media with 3 participants and more.

Do you see any errors in js console?

No particular errors that could be a concert for the JVB.

Here are there:

[Error] Unhandled Promise Rejection: NotAllowedError: The request is not allowed by the user agent or the platform in the current context, possibly because the user denied permission.
	(anonymous function)
	rejectPromise
	play
	play (app.bundle.min.js:159:113545)
	(anonymous function) (app.bundle.min.js:425:870198)
	(anonymous function) (app.bundle.min.js:425:870305)
	(anonymous function) (app.bundle.min.js:425:868856)
	(anonymous function) (app.bundle.min.js:425:868038)
	(anonymous function) (app.bundle.min.js:425:867608)
	(anonymous function) (app.bundle.min.js:425:862904)
	(anonymous function) (app.bundle.min.js:425:854488)
	(anonymous function) (app.bundle.min.js:425:854036)
	(anonymous function) (app.bundle.min.js:425:845514)
	(anonymous function) (app.bundle.min.js:425:843253)
	asyncFunctionResume
	asyncFunctionResume
	(anonymous function) (app.bundle.min.js:425:838722)
	(anonymous function) (app.bundle.min.js:425:837206)
	(anonymous function) (app.bundle.min.js:425:835932)
	asyncFunctionResume
	(anonymous function) (app.bundle.min.js:425:834377)
	(anonymous function) (app.bundle.min.js:425:832754)
	An (app.bundle.min.js:425:863835)
	(anonymous function) (app.bundle.min.js:425:866035)
	(anonymous function) (app.bundle.min.js:425:862904)
	(anonymous function) (app.bundle.min.js:425:854488)
	(anonymous function) (app.bundle.min.js:425:854036)
	(anonymous function) (app.bundle.min.js:425:845514)
	(anonymous function) (app.bundle.min.js:425:843253)
	asyncFunctionResume
	asyncFunctionResume
	(anonymous function) (app.bundle.min.js:425:838722)
	(anonymous function) (app.bundle.min.js:425:837206)
	(anonymous function) (app.bundle.min.js:425:835932)
	asyncFunctionResume
	(anonymous function) (app.bundle.min.js:425:834377)
	(anonymous function) (app.bundle.min.js:425:832754)
	(anonymous function) (app.bundle.min.js:425:924244)
	(anonymous function) (app.bundle.min.js:425:923682)
	(anonymous function) (app.bundle.min.js:425:920392)
	(anonymous function) (app.bundle.min.js:425:919995)
	(anonymous function) (app.bundle.min.js:425:898370)
	(anonymous function) (app.bundle.min.js:425:892623)
	(anonymous function) (app.bundle.min.js:425:891331)
	(anonymous function) (app.bundle.min.js:425:888046)
	(anonymous function) (app.bundle.min.js:425:885282)
	(anonymous function) (app.bundle.min.js:425:878783)
	asyncFunctionResume
	(anonymous function) (app.bundle.min.js:425:868856)
	(anonymous function) (app.bundle.min.js:425:868038)
	(anonymous function) (app.bundle.min.js:425:867608)
	(anonymous function) (app.bundle.min.js:425:862904)
	(anonymous function) (app.bundle.min.js:425:854488)
	(anonymous function) (app.bundle.min.js:425:854036)
	(anonymous function) (app.bundle.min.js:425:845514)
	(anonymous function) (app.bundle.min.js:425:843253)
	asyncFunctionResume
	asyncFunctionResume
	(anonymous function) (app.bundle.min.js:425:838722)
	(anonymous function) (app.bundle.min.js:425:837206)
	(anonymous function) (app.bundle.min.js:425:835932)
	asyncFunctionResume
	(anonymous function) (app.bundle.min.js:425:834377)
	(anonymous function) (app.bundle.min.js:425:832754)
	p (app.bundle.min.js:145:138258)
	(anonymous function) (app.bundle.min.js:200:20014)
	(anonymous function) (lib-jitsi-meet.min.js:1:119622)
	(anonymous function) (lib-jitsi-meet.min.js:10:50467)
	(anonymous function)
	(anonymous function) (lib-jitsi-meet.min.js:1:119565)
	onPresence (lib-jitsi-meet.min.js:10:145941)
	onPresence (lib-jitsi-meet.min.js:10:138001)
	onPresence
	run (lib-jitsi-meet.min.js:1:27379)
	(anonymous function) (lib-jitsi-meet.min.js:1:35813)
	forEachChild (lib-jitsi-meet.min.js:1:19035)
	_dataRecv (lib-jitsi-meet.min.js:1:35671)
	_onMessage (lib-jitsi-meet.min.js:1:64906)
	_onMessage
[Error] 2021-06-25T12:31:37.542Z – "[JitsiMeetJS.js]" – "UnhandledError: The operation was aborted." – "Script: null" – "Line: null" – "Column: null" – "StackTrace: " – undefined
	error
	o (lib-jitsi-meet.min.js:10:20815)
	o
	getGlobalOnErrorHandler (lib-jitsi-meet.min.js:17:143178)
	(anonymous function) (app.bundle.min.js:425:856065)
[Error] Unhandled Promise Rejection: AbortError: The operation was aborted.
	(anonymous function) (app.bundle.min.js:150:216818)
	promiseReactionJob