JVB Load balancing, MUC etc

I am really confused with exactly which ports are used by each jitsi services, what talks to what exactly and how to do load balancing properly (using containers).

I am trying to have a containerized deployment with autoscaled JVBs (AWS ECS Fargate). I am using latest containers from https://github.com/jitsi/docker-jitsi-meet

I read several conflicting posts, docs, diagrams etc. All I can find are tutorials that explain how to setup certain config but don’t provide details on how it actually really works. There doesn’t seem to be a clear explanation of each params and what exactly needs to talk to what, which IPs / hosts are used internally and can be on a private subnet, which IPs / Hosts need to be reachable over the internet etc…

Maybe I dont know where to look exactly. The whole XMPP vs MUC is also unclear and confusing.

I am currently running 2 Tasks / Services on AWS ECS Fargate

1/ What I call the “Base” service with Web / Prosody / Jicofo, which i understood cant really be scaled and load balanced on a given cluster as JICOFO orchestrates all the meetings / JVBs etc.

2/ And the JVB (which I intend to scale as needed)

  • Each ECS Task has its own Public IP
  • The base service is behind an Application Load Balancer port 443 with SSL / cert etc.

I seem to understand that in MUC mode the JVBs needs to connect to Prodosy and i am having it connect over the private IPs on the same subnet.

I found this diagram which seems to confirm this…

However my config is not working and it is difficult to troubleshoot. I can get to the web portal but starting the conf is not working.

Update - Adding logs:

**Jicofo error:**
2020-06-05T11:23:09.366-07:00
Jicofo 2020-06-05 11:23:09.366 WARNING: [83] org.jitsi.jicofo.xmpp.FocusComponent.processIQ() (serving component 'Jitsi Meet Focus') Unexpected exception while processing IQ stanza: <iq id="20d2a59e-6161-4a78-9037-445bddf14d08:sendIQ" from="yl40i-rqhotdvl9p@jitsi-base.local/nivGBNbB" type="set" to="focus.jitsi-base.local"><conference xmlns="http://jitsi.org/protocol/focus" room="test@" machine-uid="cdaa3062179b28ae983c2e849a05042b"><property value="-1" name="channelLastN"/><property value="false" name="disableRtx"/><property value="false" name="enableLipSync"/><property value="true" name="openSctp"/></conference></iq>

2020-06-05T11:23:09.366-07:00
org.jxmpp.stringprep.XmppStringprepException: Argument can't be the empty string

2020-06-05T11:23:09.366-07:00
at org.jxmpp.stringprep.XmppStringPrepUtil.throwIfEmptyString(XmppStringPrepUtil.java:131)

2020-06-05T11:23:09.366-07:00
at org.jxmpp.stringprep.XmppStringPrepUtil.domainprep(XmppStringPrepUtil.java:79)

2020-06-05T11:23:09.366-07:00
at org.jxmpp.jid.parts.Domainpart.from(Domainpart.java:76)

2020-06-05T11:23:09.366-07:00
at org.jitsi.xmpp.extensions.jitsimeet.ConferenceIqProvider.getRoomJid(ConferenceIqProvider.java:201)

2020-06-05T11:23:09.366-07:00
at org.jitsi.xmpp.extensions.jitsimeet.ConferenceIqProvider.parse(ConferenceIqProvider.java:71)

2020-06-05T11:23:09.366-07:00
at org.jitsi.xmpp.extensions.jitsimeet.ConferenceIqProvider.parse(ConferenceIqProvider.java:36)

2020-06-05T11:23:09.366-07:00
at org.jivesoftware.smack.provider.Provider.parse(Provider.java:43)

2020-06-05T11:23:09.366-07:00
at org.jitsi.xmpp.util.IQUtils.convert(IQUtils.java:162)

2020-06-05T11:23:09.366-07:00
at org.jitsi.jicofo.xmpp.FocusComponent.handleIQSetImpl(FocusComponent.java:249)

2020-06-05T11:23:09.366-07:00
at org.jitsi.xmpp.component.ComponentBase.handleIQSet(ComponentBase.java:362)

2020-06-05T11:23:09.366-07:00
at org.xmpp.component.AbstractComponent.processIQRequest(AbstractComponent.java:515)

2020-06-05T11:23:09.366-07:00
at org.xmpp.component.AbstractComponent.processIQ(AbstractComponent.java:289)

2020-06-05T11:23:09.366-07:00
at org.xmpp.component.AbstractComponent.processQueuedPacket(AbstractComponent.java:239)

2020-06-05T11:23:09.366-07:00
at org.xmpp.component.AbstractComponent.access$100(AbstractComponent.java:81)

2020-06-05T11:23:09.366-07:00
at org.xmpp.component.AbstractComponent$PacketProcessor.run(AbstractComponent.java:1051)

2020-06-05T11:23:09.366-07:00
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

2020-06-05T11:23:09.366-07:00
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

2020-06-05T11:23:09.366-07:00
at java.lang.Thread.run(Thread.java:748)

No apparent errors in Prosody logs.

JVB is unable to connect to Prosody (which runs on a separate instance):
2020-06-05T11:36:52.509-07:00
WARNING: [MucClient id=shard hostname=jitsi2.opus1dev.io] error connecting

2020-06-05T11:36:52.509-07:00
org.jivesoftware.smack.SmackException$ConnectionException: The following addresses failed: 'jitsi2.opus1dev.io:5222' failed because: jitsi2.opus1dev.io/54.183.187.28 exception: java.net.SocketTimeoutException: connect timed out, jitsi2.opus1dev.io/52.9.117.174 exception: java.net.SocketTimeoutException: connect timed out

2020-06-05T11:36:52.509-07:00
at org.jivesoftware.smack.SmackException$ConnectionException.from(SmackException.java:278)

2020-06-05T11:36:52.509-07:00
at org.jivesoftware.smack.tcp.XMPPTCPConnection.connectUsingConfiguration(XMPPTCPConnection.java:619)

2020-06-05T11:36:52.509-07:00
at org.jivesoftware.smack.tcp.XMPPTCPConnection.connectInternal(XMPPTCPConnection.java:902)

2020-06-05T11:36:52.509-07:00
at org.jivesoftware.smack.AbstractXMPPConnection.connect(AbstractXMPPConnection.java:383)

2020-06-05T11:36:52.509-07:00
at org.jitsi.xmpp.mucclient.MucClient.lambda$getConnectAndLoginCallable$7(MucClient.java:638)

2020-06-05T11:36:52.509-07:00
at org.jitsi.retry.RetryStrategy$TaskRunner.run(RetryStrategy.java:193)

2020-06-05T11:36:52.509-07:00
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

2020-06-05T11:36:52.509-07:00
at java.util.concurrent.FutureTask.run(FutureTask.java:266)

2020-06-05T11:36:52.509-07:00
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

2020-06-05T11:36:52.509-07:00
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

2020-06-05T11:36:52.509-07:00
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

2020-06-05T11:36:52.509-07:00
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

2020-06-05T11:36:52.509-07:00

And finally the Web browser console log seem to indicate that the private hostname is exposed to the web client to connect to the JVB.

2020-06-05T18:44:20.400Z [modules/xmpp/xmpp.js] <A.connectionHandler>:  (TIME) Strophe connected:	 1267.3899999936111
Logger.js:154 2020-06-05T18:44:20.400Z [modules/xmpp/xmpp.js] <A.connectionHandler>:  My Jabber ID: pgqnvmr08dxornky@jitsi-base.local/0VkKvKRA
Logger.js:154 2020-06-05T18:44:20.405Z [conference.js] <Object.startConference>:  Initialized with 2 local tracks
Logger.js:154 2020-06-05T18:44:20.410Z [modules/xmpp/xmpp.js] <A.createRoom>:  JID pgqnvmr08dxornky@jitsi-base.local/0VkKvKRA using MUC nickname cfc069a7
Logger.js:154 2020-06-05T18:44:20.410Z [modules/xmpp/ChatRoom.js] <new _>:  Joined MUC as test@/cfc069a7
Logger.js:154 2020-06-05T18:44:20.411Z [modules/e2eping/e2eping.js] <new h>:  Initializing e2e ping; pingInterval=10000, analyticsInterval=60000.
Logger.js:154 2020-06-05T18:44:20.411Z [modules/connectivity/ParticipantConnectionStatus.js] <new S>:  RtcMuteTimeout set to: 10000
Logger.js:154 2020-06-05T18:44:20.413Z [modules/statistics/AvgRTPStatsReporter.js] <new g>:  Avg RTP stats will be calculated every 15 samples
Logger.js:154 2020-06-05T18:44:20.413Z [JitsiConference.js] <new se>:  backToP2PDelay: 5
Logger.js:154 2020-06-05T18:44:20.424Z [JitsiConference.js] <se._doReplaceTrack>:  _doReplaceTrack - no JVB JingleSession
Logger.js:154 2020-06-05T18:44:20.424Z [JitsiConference.js] <se._doReplaceTrack>:  _doReplaceTrack - no P2P JingleSession
Logger.js:154 2020-06-05T18:44:20.425Z [JitsiConference.js] <se._doReplaceTrack>:  _doReplaceTrack - no JVB JingleSession
Logger.js:154 2020-06-05T18:44:20.425Z [JitsiConference.js] <se._doReplaceTrack>:  _doReplaceTrack - no P2P JingleSession
Logger.js:154 2020-06-05T18:44:20.427Z [modules/remotecontrol/RemoteControl.js] <d.init>:  Initializing remote control.
Logger.js:154 2020-06-05T18:44:20.427Z [modules/xmpp/moderator.js] <u.setFocusUserJid>:  Focus jid set to:  focus@auth.jitsi-base.local
Logger.js:154 2020-06-05T18:44:20.427Z [modules/xmpp/moderator.js] <u.createConferenceIq>:  Session ID: null machine UID: cdaa3062179b28ae983c2e849a05042b
Logger.js:154 2020-06-05T18:44:20.430Z [features/base/tracks] Replace audio track - unmuted
Logger.js:154 2020-06-05T18:44:20.433Z [features/base/tracks] Replace video track - unmuted
Logger.js:154 2020-06-05T18:44:20.460Z [modules/xmpp/strophe.ping.js] <p.startInterval>:  XMPP pings will be sent every 10000 ms
Logger.js:154 2020-06-05T18:44:20.483Z [modules/RTC/RTCUtils.js] <z.newObtainAudioAndVideoPermissions>:  Using the new gUM flow
Logger.js:154 2020-06-05T18:44:20.483Z [modules/RTC/RTCUtils.js] <z.<anonymous>>:  Got media constraints:  {video: false, audio: {…}}
Logger.js:154 2020-06-05T18:44:20.497Z [modules/RTC/RTCUtils.js] onUserMediaSuccess
react_devtools_backend.js:6 2020-06-05T18:44:20.544Z [JitsiMeetJS.js] <Object.getGlobalOnErrorHandler>:  UnhandledError: null Script: null Line: null Column: null StackTrace:  Error: Focus error, retry after 1000
    at u._allocateConferenceFocusError (moderator.js:446)
    at moderator.js:373
    at I.Handler.handler (strophe.umd.js:2716)
    at I.Handler.run (strophe.umd.js:1875)
    at strophe.umd.js:3157
    at Object.forEachChild (strophe.umd.js:830)
    at I.Connection._dataRecv (strophe.umd.js:3146)
    at D.Bosh._onRequestStateChange (strophe.umd.js:5012)
r @ react_devtools_backend.js:6
s @ md5.js:1
[...]
react_devtools_backend.js:6 2020-06-05T18:44:20.546Z [modules/xmpp/moderator.js] <u._allocateConferenceFocusError>:  Focus error, retry after 1000 <iq xmlns=​"jabber:​client" id=​"6485665e-89a2-471f-a391-be1d3b82e904:​sendIQ" from=​"focus.jitsi-base.local" type=​"error" to=​"pgqnvmr08dxornky@jitsi-base.local/​0VkKvKRA">​…​</iq>​
[...]
Logger.js:154 2020-06-05T18:44:20.558Z [features/overlay] <Object.componentDidMount>:  The conference will be reloaded after 21 seconds.
react_devtools_backend.js:6 2020-06-05T18:44:20.561Z [conference.js] <ee._onConferenceFailed>:  CONFERENCE FAILED: conference.focusDisconnected focus.jitsi-base.local 1
[...;
Logger.js:154 2020-06-05T18:44:20.722Z [modules/UI/videolayout/LargeVideoManager.js] hover in local
Logger.js:154 2020-06-05T18:44:21.564Z [modules/xmpp/moderator.js] <u.createConferenceIq>:  Session ID: null machine UID: cdaa3062179b28ae983c2e849a05042b
react_devtools_backend.js:6 2020-06-05T18:44:21.578Z [JitsiMeetJS.js] <Object.getGlobalOnErrorHandler>:  UnhandledError: null Script: null Line: null Column: null StackTra

So all in all my settings are wrong and I would love if someone could clearly explain which hosts are used for which connection, which are exposed over the internet and need to be accessible publicly, how the different containers (web/jicofo/prosody & JVB) need to connect to each other (over private Ip or over public ip / fqdns etc.)

Thanks for the help!