IntraRegionBridge SelectionStrategy seems not spreading the conference between jvbs

Install & Config
1

Hi,

We setup Octo and we are using “IntraRegionBridgeSelectionStrategy”.

Our current architecture is to have several shards but in different close region

ex: eu-west1 , eu-west2 …

We have several JVB in one region but it seems only 1 JVB is taking a big conference on specific shard/

As far as i understand, IntraRegionBridgeSelectionStrategy should spread people on different jvbs inside the same region ? We didnt configured “config.js” deployment info or the web component for octo since we dont use geo ip. (jitsi-videobridge/octo.md at master · jitsi/jitsi-videobridge · GitHub)

How can we check Octo is well working ?

Thank you

2

In order Jicofo to chose a bridge the client reports its deploymentInfo.userRegion to it. You need to make sure the clients report their correct region.

1 Like
3

Thank you , so basically IntraRegionBridgeSelectionStrategy doesnt work if i dont setup config.js with deploymentInfo if i understand correctly. (Its not only for geo ip but also inside a same region)

4

What do you mean?

Correct, the clients need to report their region in order for a selection to be correct.

1 Like
5

I mean, after reading the documentation i understood it was only needed in case of geo ip with different people in different region and not needed for spreading inside a same region.

After reading your explanation, i understand i was wrong. i will test it

6

Aaaa now understand, yes that is needed when people are in different regions and you want to select a bridge closer to the participant.

If you want to spread a conference between bridges in same region, normally that is done when the bridge is stressed. There are settings about that in the bridge what to consider as stressed based on packet rate.
Another option you have is jicofo/reference.conf at d0f4a61f40f7f99f580bde1553697e9281f65d97 · jitsi/jicofo · GitHub
You can set like 2 participants in a conference per bridge … and you can spread your participants like that in one conference …

7

Thank you so i assume its not required to setup deployment info for clients in the same regions.

I tried to stress jicofo (torture on a same room) and get kicked from the room with that kind of log:


INFO: Selected bridge Bridge[jid=jvbbrewery@internal-muc.meet.jitsi/shard-jvb-2, relayId=10.2.2.9:4096, region=region1, stress=0.26] with stress=0.0 for participantRegion=null
SEVERE: jvbbrewery@internal-muc.meet.jitsi/shard-jvb-2 - failed to allocate channels, will consider the bridge faulty: Creator thread has failed to allocate channels: Timed out waiting for a response.
org.jitsi.protocol.xmpp.colibri.exception.TimeoutException: Creator thread has failed to allocate channels: Timed out waiting for a response.
        at org.jitsi.protocol.xmpp.colibri.exception.TimeoutException.clone(TimeoutException.java:39)
        at org.jitsi.impl.protocol.xmpp.colibri.ColibriConferenceImpl$ConferenceCreationSemaphore.acquire(ColibriConferenceImpl.java:891)
        at org.jitsi.impl.protocol.xmpp.colibri.ColibriConferenceImpl.acquireCreateConferenceSemaphore(ColibriConferenceImpl.java:385)
        at org.jitsi.impl.protocol.xmpp.colibri.ColibriConferenceImpl.createColibriChannels(ColibriConferenceImpl.java:222)
        at org.jitsi.protocol.xmpp.colibri.ColibriConference.createColibriChannels(ColibriConference.java:97)
        at org.jitsi.jicofo.ParticipantChannelAllocator.doAllocateChannels(ParticipantChannelAllocator.java:122)
        at org.jitsi.jicofo.AbstractChannelAllocator.allocateChannels(AbstractChannelAllocator.java:253)
        at org.jitsi.jicofo.AbstractChannelAllocator.doRun(AbstractChannelAllocator.java:172)
        at org.jitsi.jicofo.AbstractChannelAllocator.run(AbstractChannelAllocator.java:133)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Sep 28, 2021 6:31:05 PM org.jitsi.utils.logging2.LoggerImpl log
SEVERE: jvbbrewery@internal-muc.meet.jitsi/shard-jvb-2 - failed to allocate channels, will consider the bridge faulty: Creator thread has failed to allocate channels: Timed out waiting for a response.
org.jitsi.protocol.xmpp.colibri.exception.TimeoutException: Creator thread has failed to allocate channels: Timed out waiting for a response.
        at org.jitsi.protocol.xmpp.colibri.exception.TimeoutException.clone(TimeoutException.java:39)
        at org.jitsi.impl.protocol.xmpp.colibri.ColibriConferenceImpl$ConferenceCreationSemaphore.acquire(ColibriConferenceImpl.java:891)
        at org.jitsi.impl.protocol.xmpp.colibri.ColibriConferenceImpl.acquireCreateConferenceSemaphore(ColibriConferenceImpl.java:385)
        at org.jitsi.impl.protocol.xmpp.colibri.ColibriConferenceImpl.createColibriChannels(ColibriConferenceImpl.java:222)
        at org.jitsi.protocol.xmpp.colibri.ColibriConference.createColibriChannels(ColibriConference.java:97)
        at org.jitsi.jicofo.ParticipantChannelAllocator.doAllocateChannels(ParticipantChannelAllocator.java:122)
        at org.jitsi.jicofo.AbstractChannelAllocator.allocateChannels(AbstractChannelAllocator.java:253)
        at org.jitsi.jicofo.AbstractChannelAllocator.doRun(AbstractChannelAllocator.java:172)
        at org.jitsi.jicofo.AbstractChannelAllocator.run(AbstractChannelAllocator.java:133)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Sep 28, 2021 6:32:37 PM org.jitsi.utils.logging2.LoggerImpl log
INFO: Terminate session: torture-test0@muc.meet.jitsi/932928ba, reason: gone, send terminate: true
Sep 28, 2021 6:32:37 PM org.jitsi.utils.logging2.LoggerImpl log
SEVERE: No connection - unable to send packet: <iq to='torture-test0@muc.meet.jitsi/932928ba' from='focus@auth.meet.jitsi/focus' id='7eVCF-103033' type='set'><jingle xmlns='urn:xmpp:jingle:1' action='session-terminate' sid='ebl8h66drobr5'><reason><gone/></reason></jingle></iq>
org.jivesoftware.smack.SmackException$NotConnectedException: The connection XMPPTCPConnection[focus@auth.meet.jitsi/focus] (0) is no longer connected. done=true smResumptionPossible=false
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketWriter.throwNotConnectedExceptionIfDoneAndResumptionNotPossible(XMPPTCPConnection.java:1327)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection.throwNotConnectedExceptionIfAppropriate(XMPPTCPConnection.java:358)
        at org.jivesoftware.smack.AbstractXMPPConnection.sendStanza(AbstractXMPPConnection.java:670)
        at org.jitsi.jicofo.xmpp.UtilKt.tryToSendStanza(Util.kt:71)
        at org.jitsi.protocol.xmpp.AbstractOperationSetJingle.terminateSession(AbstractOperationSetJingle.java:346)
        at org.jitsi.protocol.xmpp.AbstractOperationSetJingle.terminateHandlersSessions(AbstractOperationSetJingle.java:307)
        at org.jitsi.jicofo.JitsiMeetConferenceImpl.stop(JitsiMeetConferenceImpl.java:416)
        at org.jitsi.jicofo.JitsiMeetConferenceImpl.registrationChanged(JitsiMeetConferenceImpl.java:1175)
        at org.jitsi.impl.protocol.xmpp.AbstractXmppProvider.fireRegistrationStateChanged(AbstractXmppProvider.java:88)
        at org.jitsi.impl.protocol.xmpp.XmppProviderImpl.fireRegistrationStateChanged(XmppProviderImpl.java:322)
        at org.jitsi.impl.protocol.xmpp.AbstractXmppProvider.setRegistered(AbstractXmppProvider.java:119)
        at org.jitsi.impl.protocol.xmpp.XmppProviderImpl$XmppConnectionListener.connectionClosedOnError(XmppProviderImpl.java:400)
        at org.jivesoftware.smack.AbstractXMPPConnection.callConnectionClosedOnErrorListener(AbstractXMPPConnection.java:1335)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection.notifyConnectionError(XMPPTCPConnection.java:931)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection.access$3200(XMPPTCPConnection.java:151)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1258)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
        at java.lang.Thread.run(Thread.java:748)

Sep 28, 2021 6:32:37 PM org.jitsi.utils.logging2.LoggerImpl log
INFO: Terminate session: torture-test0@muc.meet.jitsi/366d3b66, reason: gone, send terminate: true
Sep 28, 2021 6:32:37 PM org.jitsi.utils.logging2.LoggerImpl log
SEVERE: No connection - unable to send packet: <iq to='torture-test0@muc.meet.jitsi/366d3b66' from='focus@auth.meet.jitsi/focus' id='7eVCF-103034' type='set'><jingle xmlns='urn:xmpp:jingle:1' action='session-terminate' sid='54qs1b7mdd85e'><reason><gone/></reason></jingle></iq>
org.jivesoftware.smack.SmackException$NotConnectedException: The connection XMPPTCPConnection[focus@auth.meet.jitsi/focus] (0) is no longer connected. done=true smResumptionPossible=false
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketWriter.throwNotConnectedExceptionIfDoneAndResumptionNotPossible(XMPPTCPConnection.java:1327)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection.throwNotConnectedExceptionIfAppropriate(XMPPTCPConnection.java:358)
        at org.jivesoftware.smack.AbstractXMPPConnection.sendStanza(AbstractXMPPConnection.java:670)
        at org.jitsi.jicofo.xmpp.UtilKt.tryToSendStanza(Util.kt:71)
        at org.jitsi.protocol.xmpp.AbstractOperationSetJingle.terminateSession(AbstractOperationSetJingle.java:346)
        at org.jitsi.protocol.xmpp.AbstractOperationSetJingle.terminateHandlersSessions(AbstractOperationSetJingle.java:307)
        at org.jitsi.jicofo.JitsiMeetConferenceImpl.stop(JitsiMeetConferenceImpl.java:416)
        at org.jitsi.jicofo.JitsiMeetConferenceImpl.registrationChanged(JitsiMeetConferenceImpl.java:1175)
        at org.jitsi.impl.protocol.xmpp.AbstractXmppProvider.fireRegistrationStateChanged(AbstractXmppProvider.java:88)
        at org.jitsi.impl.protocol.xmpp.XmppProviderImpl.fireRegistrationStateChanged(XmppProviderImpl.java:322)
        at org.jitsi.impl.protocol.xmpp.AbstractXmppProvider.setRegistered(AbstractXmppProvider.java:119)
        at org.jitsi.impl.protocol.xmpp.XmppProviderImpl$XmppConnectionListener.connectionClosedOnError(XmppProviderImpl.java:400)
        at org.jivesoftware.smack.AbstractXMPPConnection.callConnectionClosedOnErrorListener(AbstractXMPPConnection.java:1335)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection.notifyConnectionError(XMPPTCPConnection.java:931)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection.access$3200(XMPPTCPConnection.java:151)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1258)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
        at java.lang.Thread.run(Thread.java:748)

Sep 28, 2021 6:32:37 PM org.jitsi.utils.logging2.LoggerImpl log
INFO: Terminate session: torture-test0@muc.meet.jitsi/c81ed8e7, reason: gone, send terminate: true
Sep 28, 2021 6:32:37 PM org.jitsi.utils.logging2.LoggerImpl log
SEVERE: No connection - unable to send packet: <iq to='torture-test0@muc.meet.jitsi/c81ed8e7' from='focus@auth.meet.jitsi/focus' id='7eVCF-103035' type='set'><jingle xmlns='urn:xmpp:jingle:1' action='session-terminate' sid='3dmuqmeq050o5'><reason><gone/></reason></jingle></iq>
org.jivesoftware.smack.SmackException$NotConnectedException: The connection XMPPTCPConnection[focus@auth.meet.jitsi/focus] (0) is no longer connected. done=true smResumptionPossible=false
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketWriter.throwNotConnectedExceptionIfDoneAndResumptionNotPossible(XMPPTCPConnection.java:1327)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection.throwNotConnectedExceptionIfAppropriate(XMPPTCPConnection.java:358)
        at org.jivesoftware.smack.AbstractXMPPConnection.sendStanza(AbstractXMPPConnection.java:670)
        at org.jitsi.jicofo.xmpp.UtilKt.tryToSendStanza(Util.kt:71)
        at org.jitsi.protocol.xmpp.AbstractOperationSetJingle.terminateSession(AbstractOperationSetJingle.java:346)
        at org.jitsi.protocol.xmpp.AbstractOperationSetJingle.terminateHandlersSessions(AbstractOperationSetJingle.java:307)
        at org.jitsi.jicofo.JitsiMeetConferenceImpl.stop(JitsiMeetConferenceImpl.java:416)
        at org.jitsi.jicofo.JitsiMeetConferenceImpl.registrationChanged(JitsiMeetConferenceImpl.java:1175)
        at org.jitsi.impl.protocol.xmpp.AbstractXmppProvider.fireRegistrationStateChanged(AbstractXmppProvider.java:88)
        at org.jitsi.impl.protocol.xmpp.XmppProviderImpl.fireRegistrationStateChanged(XmppProviderImpl.java:322)
        at org.jitsi.impl.protocol.xmpp.AbstractXmppProvider.setRegistered(AbstractXmppProvider.java:119)
        at org.jitsi.impl.protocol.xmpp.XmppProviderImpl$XmppConnectionListener.connectionClosedOnError(XmppProviderImpl.java:400)
        at org.jivesoftware.smack.AbstractXMPPConnection.callConnectionClosedOnErrorListener(AbstractXMPPConnection.java:1335)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection.notifyConnectionError(XMPPTCPConnection.java:931)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection.access$3200(XMPPTCPConnection.java:151)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1258)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
        at java.lang.Thread.run(Thread.java:748)

Sep 28, 2021 6:32:37 PM org.jitsi.utils.logging2.LoggerImpl log
INFO: Stopped.
Sep 28, 2021 6:32:37 PM org.jitsi.utils.logging2.LoggerImpl log
WARNING: Smack: XMPPConnection closed due to an exception (XMPPTCPConnection[focus@auth.meet.jitsi/focus] (0))
java.io.EOFException: no more data available - expected end tag </stream:stream> to close start tag <stream:stream> from line 1, parser stopped on END_TAG seen ...</presence>... @1:25469067
        at org.xmlpull.mxp1.MXParser.fillBuf(MXParser.java:3035)
        at org.xmlpull.mxp1.MXParser.more(MXParser.java:3046)
        at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1144)
        at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1248)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
        at java.lang.Thread.run(Thread.java:748)

WARNING: Smack: Reconnection failed due to an exception (XMPPTCPConnection[focus@auth.meet.jitsi/focus] (0))
org.jivesoftware.smack.SmackException$NoResponseException: No response received within reply timeout. Timeout was 15000ms (~15s). While waiting for establishing TLS
        at org.jivesoftware.smack.SmackException$NoResponseException.newWith(SmackException.java:93)
        at org.jivesoftware.smack.SynchronizationPoint.checkForResponse(SynchronizationPoint.java:272)
        at org.jivesoftware.smack.SynchronizationPoint.checkIfSuccessOrWait(SynchronizationPoint.java:157)
        at org.jivesoftware.smack.SynchronizationPoint.checkIfSuccessOrWaitOrThrow(SynchronizationPoint.java:128)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection.connectInternal(XMPPTCPConnection.java:908)
        at org.jivesoftware.smack.AbstractXMPPConnection.connect(AbstractXMPPConnection.java:383)
        at org.jivesoftware.smack.ReconnectionManager$2.run(ReconnectionManager.java:289)
        at java.lang.Thread.run(Thread.java:748)

Sep 28, 2021 6:36:42 PM org.jitsi.utils.logging2.LoggerImpl log
SEVERE: XMPP reconnection failed: No response received within reply timeout. Timeout was 15000ms (~15s). While waiting for establishing TLS
Sep 28, 2021 6:36:42 PM org.jivesoftware.smack.AbstractXMPPConnection callConnectionClosedOnErrorListener
WARNING: Connection XMPPTCPConnection[focus@auth.meet.jitsi/focus] (0) closed with error
org.jivesoftware.smack.SmackException$NoResponseException: No response received within reply timeout. Timeout was 15000ms (~15s). While waiting for initial open stream element send to server
        at org.jivesoftware.smack.SmackException$NoResponseException.newWith(SmackException.java:93)
        at org.jivesoftware.smack.SynchronizationPoint.checkForResponse(SynchronizationPoint.java:272)
        at org.jivesoftware.smack.SynchronizationPoint.checkIfSuccessOrWait(SynchronizationPoint.java:157)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1033)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
        at java.lang.Thread.run(Thread.java:748)

Sep 28, 2021 6:36:42 PM org.jitsi.utils.logging2.LoggerImpl log
SEVERE: XMPP connection closed on error: No response received within reply timeout. Timeout was 15000ms (~15s). While waiting for initial open stream element send to server
Sep 28, 2021 6:36:42 PM org.jitsi.utils.logging2.LoggerImpl log
WARNING: Smack: XMPPConnection closed due to an exception (XMPPTCPConnection[focus@auth.meet.jitsi/focus] (0))
org.jivesoftware.smack.SmackException$NoResponseException: No response received within reply timeout. Timeout was 15000ms (~15s). While waiting for initial open stream element send to server
        at org.jivesoftware.smack.SmackException$NoResponseException.newWith(SmackException.java:93)
        at org.jivesoftware.smack.SynchronizationPoint.checkForResponse(SynchronizationPoint.java:272)
        at org.jivesoftware.smack.SynchronizationPoint.checkIfSuccessOrWait(SynchronizationPoint.java:157)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1033)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
        at java.lang.Thread.run(Thread.java:748)

Sep 28, 2021 6:36:43 PM org.jitsi.utils.logging2.LoggerImpl log
INFO: XMPP reconnecting in: 419

i can read: INFO: Selected bridge Bridge[jid=jvbbrewery@internal-muc.meet.jitsi/shard-jvb-2, relayId=10.2.2.9:4096, region=region1, stress=0.26] with stress=0.0 for participantRegion=null
is it a problem to have that value as null ?

8

Are you monitoring your prosody process cpu usage, not the machine but that process? How does that look like?

1 Like
9

ok interesting i will dig into it and monitoring jicofo + prosody process (thats a good track)
btw, basically my question is what are the prerequisites to make octo works in a same region just for spreading participants on different jvb and finally supporting big conference (100 participants+)

  • octo set (jvb+jicofo)
  • no need to setup config.js since its not geoip (nothing at jitsi meet config)
  • IntraRegionBridgeSelectionStrategy
  • Default jicofo stress parameters are enough (no need to update that, normally default will spread)
  • jicofo + prosody process with enough CPU/RAM (what i miss here)
  • any other missing ?
10

so after tuning ressources CPU/RAM i can support more users in a conference but it seems only one JVB is taking all the traffic instead of beeing spread (based on stats participants from colibri stats)

only jvb 6 is mentionning participants and not the other jvb (config.js still not tuned)

any idea about what i miss to allow spreading on a single region with OCTO. (if i changed strategy to split, i get participants on each jvb). should i consider participants stats not reliable in octo with same region.

image

11

Yes.

This really depends on the machine of your choice for hosting jvb. Check out this thread Large Number of Participants & Prosody Performance - #19 by engesin

There are multiple posts in the forum about optimizing this.
The weakest point is prosody as it is single-threaded, so you need to make room for it and optimize settings … search for epoll network backend we had mentioned it several times.

What do you mean, what settings did you change and you expected the participants to spread?

1 Like
12

thank you, yes if i use SplitBridgeSelectionStrategy settings i can see participants on each jvb that is not the case with IntraRegionBridgeSelectionStrategy.

i will read your link about stress threshold, i thought it had spread with default settings but maybe in fact i have to change load threshold etc… to allow spreading participants over bridges.

Maybe indeed, i dont reach enough stress with current setup to trigger spread.
22. // The stress level above which a bridge is considered overstressed.
23. stress-threshold = 0.8

13

After changing stress-threshold , it started to spread

1 Like
14

@tanandy So you’ve been successful in spreading meet participants across more than one JVB?
I’m trying this configuration, but when running torture all participants are only in one.