When more than 2 video senders join, only 1 video can be seen, that of the moderator. And when user3 leaves crashes room

Install & Config
#1

Issue: When more than 2 video senders join, only 1 video can be seen, that of the moderator. And when user3 leaves crashes room

This is with one instance hosting nginx, jitsi-meet, prosody1, and jicofo, but configured to use 2 separate instances for JVBs (jvb1a and jvb1b).

User1 is connecting with cable (moderator), dsl (user2), and cellphone via web client (user3).

With user1 and user2 connected both videos visible to each other simultaneously.
When user3 joins only user1’s video is visible (to all 3 users). Did I error with the configuration, miss opening a port or is this something else? Thanks for any suggestions.

Log on lmjit1 /var/log/jitsi/jicofo.log shows around time of user3 connecting (and dropping to just 1 video sender (moderator):

Jicofo 2021-07-04 03:41:12.012 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] JitsiMeetConferenceImpl.onSessionTerminate#1373: Received session-terminate from Participant[loadtest0@conference.lmjit1.dev2dev.net/70fbb15c]@1498250624, bridge session: BridgeSession[id=25882_78ed32, bridge=Bridge[jid=jvbbrewery@internal.auth.lmjit1.dev2dev.net/jitsi-videobridge1a, relayId=null, region=null, stress=0.02]]@552759880, restart: true
Jicofo 2021-07-04 03:41:12.012 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] JitsiMeetConferenceImpl.terminateParticipant#1118: Terminating 70fbb15c, reason: null, send session-terminate: false
Jicofo 2021-07-04 03:41:12.012 INFO: [28] AbstractOperationSetJingle.terminateSession#509: Terminate session: loadtest0@conference.lmjit1.dev2dev.net/70fbb15c, reason: null, send terminate: false
Jicofo 2021-07-04 03:41:12.013 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] JitsiMeetConferenceImpl.removeSources#1823: Removing sources from loadtest0@conference.lmjit1.dev2dev.net/70fbb15c: Sources{ video: [ssrc=3980409337 ssrc=1998524865 ssrc=3096652020 ssrc=2202790829 ssrc=1130353905 ssrc=4289115263 ] }@851501921
Jicofo 2021-07-04 03:41:12.013 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] JitsiMeetConferenceImpl.terminateParticipant#1143: Removed participant 70fbb15c removed=true
Jicofo 2021-07-04 03:41:12.013 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] JitsiMeetConferenceImpl$BridgeSession.terminate#2758: Expiring channels for: loadtest0@conference.lmjit1.dev2dev.net/70fbb15c on: Bridge[jid=jvbbrewery@internal.auth.lmjit1.dev2dev.net/jitsi-videobridge1a, relayId=null, region=null, stress=0.02]
Jicofo 2021-07-04 03:41:12.013 WARNING: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] JitsiMeetConferenceImpl.onSessionTerminate#1397: Rate limiting Participant[loadtest0@conference.lmjit1.dev2dev.net/70fbb15c]@1498250624 for restart requests
Jicofo 2021-07-04 03:41:12.180 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] ChatRoomRoleAndPresence.memberPresenceChanged#130: Chat room event Joined member=ChatMember[loadtest0@conference.lmjit1.dev2dev.net/62fa335a, jid: null]@1073878205
Jicofo 2021-07-04 03:41:12.180 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] JitsiMeetConferenceImpl.onMemberJoined#547: Member joined:62fa335a
Jicofo 2021-07-04 03:41:12.180 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] JitsiMeetConferenceImpl.inviteParticipant#771: Added participant id= 62fa335a, bridge=jvbbrewery@internal.auth.lmjit1.dev2dev.net/jitsi-videobridge1a
Jicofo 2021-07-04 03:41:12.180 INFO: [139] DiscoveryUtil.discoverParticipantFeatures#152: Doing feature discovery for loadtest0@conference.lmjit1.dev2dev.net/62fa335a
Jicofo 2021-07-04 03:41:12.181 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] ChatRoomRoleAndPresence.memberPresenceChanged#130: Chat room event PresenceUpdated member=ChatMember[loadtest0@conference.lmjit1.dev2dev.net/62fa335a, jid: null]@1073878205
Jicofo 2021-07-04 03:41:12.181 INFO: [139] DiscoveryUtil.discoverParticipantFeatures#192: Successfully discovered features for loadtest0@conference.lmjit1.dev2dev.net/62fa335a in 1
Jicofo 2021-07-04 03:41:12.181 INFO: [139] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] AbstractChannelAllocator.allocateChannels#248: Using jvbbrewery@internal.auth.lmjit1.dev2dev.net/jitsi-videobridge1a to allocate channels for: Participant[loadtest0@conference.lmjit1.dev2dev.net/62fa335a]@1468696793
Jicofo 2021-07-04 03:41:12.188 INFO: [139] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b participant=62fa335a] ParticipantChannelAllocator.doInviteOrReinvite#229: Sending session-initiate to: loadtest0@conference.lmjit1.dev2dev.net/62fa335a
Jicofo 2021-07-04 03:41:12.248 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] JitsiMeetConferenceImpl.onSessionTerminate#1373: Received session-terminate from Participant[loadtest0@conference.lmjit1.dev2dev.net/d2911153]@730581393, bridge session: BridgeSession[id=25882_78ed32, bridge=Bridge[jid=jvbbrewery@internal.auth.lmjit1.dev2dev.net/jitsi-videobridge1a, relayId=null, region=null, stress=0.03]]@552759880, restart: true
Jicofo 2021-07-04 03:41:12.248 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] JitsiMeetConferenceImpl.terminateParticipant#1118: Terminating d2911153, reason: null, send session-terminate: false
Jicofo 2021-07-04 03:41:12.248 INFO: [28] AbstractOperationSetJingle.terminateSession#509: Terminate session: loadtest0@conference.lmjit1.dev2dev.net/d2911153, reason: null, send terminate: false
Jicofo 2021-07-04 03:41:12.248 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] JitsiMeetConferenceImpl.removeSources#1823: Removing sources from loadtest0@conference.lmjit1.dev2dev.net/d2911153: Sources{ video: [ssrc=1167719291 ssrc=335861398 ssrc=3605497330 ssrc=1712597491 ssrc=1801341367 ssrc=2626527523 ] }@1772290311
Jicofo 2021-07-04 03:41:12.248 WARNING: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] JitsiMeetConferenceImpl.lambda$removeSources$13#1841: Remove source: no jingle session for loadtest0@conference.lmjit1.dev2dev.net/d2911153
Jicofo 2021-07-04 03:41:12.249 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] JitsiMeetConferenceImpl.terminateParticipant#1143: Removed participant d2911153 removed=true
Jicofo 2021-07-04 03:41:12.249 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] JitsiMeetConferenceImpl$BridgeSession.terminate#2758: Expiring channels for: loadtest0@conference.lmjit1.dev2dev.net/d2911153 on: Bridge[jid=jvbbrewery@internal.auth.lmjit1.dev2dev.net/jitsi-videobridge1a, relayId=null, region=null, stress=0.03]
Jicofo 2021-07-04 03:41:12.249 WARNING: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] JitsiMeetConferenceImpl.onSessionTerminate#1397: Rate limiting Participant[loadtest0@conference.lmjit1.dev2dev.net/d2911153]@730581393 for restart requests

Jicofo 2021-07-04 03:41:16.426 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] JitsiMeetConferenceImpl.onSessionAccept#1265: Receive session-accept from loadtest0@conference.lmjit1.dev2dev.net/62fa335a
Jicofo 2021-07-04 03:41:16.426 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=57815835-d918-4c3b-aee6-b1c3d217dc0b] JitsiMeetConferenceImpl.onSessionAcceptInternal#1681: Received session-accept from 62fa335a with accepted sources:Sources{ video: [ssrc=1461595676 ssrc=661870674 ssrc=2386127773 ssrc=410706242 ssrc=3507385998 ssrc=1954310522 ] audio: [ssrc=3377073270 ] }@1548561593

Then when user3 leaves, the room crashes

same log file shows as user3 leaves:

Jicofo 2021-07-04 03:44:01.557 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=58948597-4dd2-4929-82d5-260feeae6c0b] ChatRoomRoleAndPresence.memberPresenceChanged#130: Chat room event Left member=ChatMember[loadtest0@conference.lmjit1.dev2dev.net/a0c8cf81, jid: null]@1774151728
Jicofo 2021-07-04 03:44:01.557 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=58948597-4dd2-4929-82d5-260feeae6c0b] JitsiMeetConferenceImpl.onMemberLeft#1081: Member left:a0c8cf81
Jicofo 2021-07-04 03:44:01.557 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=58948597-4dd2-4929-82d5-260feeae6c0b] JitsiMeetConferenceImpl.terminateParticipant#1118: Terminating a0c8cf81, reason: gone, send session-terminate: false
Jicofo 2021-07-04 03:44:01.557 INFO: [28] AbstractOperationSetJingle.terminateSession#509: Terminate session: loadtest0@conference.lmjit1.dev2dev.net/a0c8cf81, reason: gone, send terminate: false
Jicofo 2021-07-04 03:44:01.558 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=58948597-4dd2-4929-82d5-260feeae6c0b] JitsiMeetConferenceImpl.removeSources#1823: Removing sources from loadtest0@conference.lmjit1.dev2dev.net/a0c8cf81: Sources{ video: [ssrc=4094522931 ssrc=775736914 ssrc=2535151146 ssrc=891371972 ssrc=2244688464 ssrc=1562060252 ] audio: [ssrc=3449364601 ] }@296823627
Jicofo 2021-07-04 03:44:01.558 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=58948597-4dd2-4929-82d5-260feeae6c0b] JitsiMeetConferenceImpl.terminateParticipant#1143: Removed participant a0c8cf81 removed=true
Jicofo 2021-07-04 03:44:01.558 INFO: [28] [room=loadtest0@conference.lmjit1.dev2dev.net meeting_id=58948597-4dd2-4929-82d5-260feeae6c0b] JitsiMeetConferenceImpl$BridgeSession.terminate#2758: Expiring channels for: loadtest0@conference.lmjit1.dev2dev.net/a0c8cf81 on: Bridge[jid=jvbbrewery@internal.auth.lmjit1.dev2dev.net/jitsi-videobridge1a, relayId=null, region=null, stress=0.03]

Thanks for suggestions on what I misconfigured.

#2

For one, your jvb websockets are not configured properly.

#3

Could you be a bit more specific? There are a lot of settings with those does the error message tell you something specific? Or do I need to post their config files to drill down?
Btw, I did just stop the crashing on leaving by opening up more ports, it appears someone else was monkeying around with the AWS security rules and had very different port numbers. I’m walking through each of those, but at least the crash on leave has stopped. Still have to resolve the video senders issue.
Appreciate pointers what area to be looking at on the JVB config for that issue.
Thanks kindly.

#4

;Here is log from jvb1a jvb log when user3 joins:
JVB 2021-07-04 04:20:11.997 INFO: [19] HealthChecker.run#171: Performed a successful health check in PT0.000004S. Sticky failure: false
JVB 2021-07-04 04:20:20.570 INFO: [32] [confId=c3b934cdba832d35 gid=25248 conf_name=loadtest0@conference.lmjit1.dev2dev.net] EndpointConnectionStatusMonitor.start#58: Starting connection status monitor
JVB 2021-07-04 04:20:20.570 INFO: [32] Videobridge.createConference#242: create_conf, id=c3b934cdba832d35 gid=25248
JVB 2021-07-04 04:20:20.571 INFO: [98] [confId=c3b934cdba832d35 epId=fc031fc3 local_ufrag=9irp81f9nrtqer gid=25248 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9irp81f9nrtqer] Agent.gatherCandidates#622: Gathering candidates for component stream-fc031fc3.RTP.
JVB 2021-07-04 04:20:20.573 INFO: [100] [confId=c3b934cdba832d35 gid=25248 conf_name=loadtest0@conference.lmjit1.dev2dev.net] Conference.dominantSpeakerChanged#422: ds_change ds_id=fc031fc3
JVB 2021-07-04 04:20:20.574 INFO: [98] [confId=c3b934cdba832d35 epId=fc031fc3 gid=25248 conf_name=loadtest0@conference.lmjit1.dev2dev.net] Endpoint.setTransportInfo#679: Ignoring empty DtlsFingerprint extension:
JVB 2021-07-04 04:20:20.578 INFO: [98] [confId=c3b934cdba832d35 epId=c8086ca2 local_ufrag=9pvi51f9nrtqf2 gid=25248 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9pvi51f9nrtqf2] Agent.gatherCandidates#622: Gathering candidates for component stream-c8086ca2.RTP.
JVB 2021-07-04 04:20:20.580 INFO: [98] [confId=c3b934cdba832d35 epId=c8086ca2 gid=25248 conf_name=loadtest0@conference.lmjit1.dev2dev.net] Endpoint.setTransportInfo#679: Ignoring empty DtlsFingerprint extension:
JVB 2021-07-04 04:20:20.581 INFO: [98] [confId=c3b934cdba832d35 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5 gid=25248 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=4i8nr1f9nrtqf5] Agent.gatherCandidates#622: Gathering candidates for component stream-7ba1d6a8.RTP.
JVB 2021-07-04 04:20:20.583 INFO: [98] [confId=c3b934cdba832d35 epId=7ba1d6a8 gid=25248 conf_name=loadtest0@conference.lmjit1.dev2dev.net] Endpoint.setTransportInfo#679: Ignoring empty DtlsFingerprint extension:
JVB 2021-07-04 04:20:21.078 INFO: [98] [confId=c3b934cdba832d35 epId=7ba1d6a8 gid=25248 stats_id=Neil-slb conf_name=loadtest0@conference.lmjit1.dev2dev.net] DtlsTransport.setSetupAttribute#120: The remote side is acting as DTLS client, we’ll act as server
JVB 2021-07-04 04:20:21.078 INFO: [98] [confId=c3b934cdba832d35 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5 gid=25248 stats_id=Neil-slb conf_name=loadtest0@conference.lmjit1.dev2dev.net] IceTransport.startConnectivityEstablishment#184: Starting the Agent without remote candidates.
JVB 2021-07-04 04:20:21.078 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Neil-slb conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=4i8nr1f9nrtqf5 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5] Agent.startConnectivityEstablishment#713: Start ICE connectivity establishment.
JVB 2021-07-04 04:20:21.078 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Neil-slb conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=4i8nr1f9nrtqf5 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5] Agent.initCheckLists#949: Init checklist for stream stream-7ba1d6a8
JVB 2021-07-04 04:20:21.078 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Neil-slb conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=4i8nr1f9nrtqf5 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5] Agent.setState#923: ICE state changed from Waiting to Running.
JVB 2021-07-04 04:20:21.078 INFO: [98] [confId=c3b934cdba832d35 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5 gid=25248 stats_id=Neil-slb conf_name=loadtest0@conference.lmjit1.dev2dev.net] IceTransport.iceStateChanged#323: ICE state changed old=Waiting new=Running
JVB 2021-07-04 04:20:21.078 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Neil-slb conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=4i8nr1f9nrtqf5 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5] ConnectivityCheckClient.startChecks#142: Start connectivity checks.
JVB 2021-07-04 04:20:21.212 INFO: [98] [confId=c3b934cdba832d35 epId=fc031fc3 gid=25248 stats_id=Irwin-GBx conf_name=loadtest0@conference.lmjit1.dev2dev.net] DtlsTransport.setSetupAttribute#120: The remote side is acting as DTLS client, we’ll act as server
JVB 2021-07-04 04:20:21.212 INFO: [98] [confId=c3b934cdba832d35 epId=fc031fc3 local_ufrag=9irp81f9nrtqer gid=25248 stats_id=Irwin-GBx conf_name=loadtest0@conference.lmjit1.dev2dev.net] IceTransport.startConnectivityEstablishment#184: Starting the Agent without remote candidates.
JVB 2021-07-04 04:20:21.212 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Irwin-GBx conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9irp81f9nrtqer epId=fc031fc3 local_ufrag=9irp81f9nrtqer] Agent.startConnectivityEstablishment#713: Start ICE connectivity establishment.
JVB 2021-07-04 04:20:21.212 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Irwin-GBx conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9irp81f9nrtqer epId=fc031fc3 local_ufrag=9irp81f9nrtqer] Agent.initCheckLists#949: Init checklist for stream stream-fc031fc3
JVB 2021-07-04 04:20:21.212 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Irwin-GBx conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9irp81f9nrtqer epId=fc031fc3 local_ufrag=9irp81f9nrtqer] Agent.setState#923: ICE state changed from Waiting to Running.
JVB 2021-07-04 04:20:21.212 INFO: [98] [confId=c3b934cdba832d35 epId=fc031fc3 local_ufrag=9irp81f9nrtqer gid=25248 stats_id=Irwin-GBx conf_name=loadtest0@conference.lmjit1.dev2dev.net] IceTransport.iceStateChanged#323: ICE state changed old=Waiting new=Running
JVB 2021-07-04 04:20:21.212 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Irwin-GBx conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9irp81f9nrtqer epId=fc031fc3 local_ufrag=9irp81f9nrtqer] ConnectivityCheckClient.startChecks#142: Start connectivity checks.
JVB 2021-07-04 04:20:21.274 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Neil-slb componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=4i8nr1f9nrtqf5 name=stream-7ba1d6a8 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5] Component.addUpdateRemoteCandidates#347: Update remote candidate for stream-7ba1d6a8.RTP: 192.168.104.53:45466/udp
JVB 2021-07-04 04:20:21.274 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Neil-slb componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=4i8nr1f9nrtqf5 name=stream-7ba1d6a8 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5] Component.addUpdateRemoteCandidates#347: Update remote candidate for stream-7ba1d6a8.RTP: 192.168.0.130:40073/udp
JVB 2021-07-04 04:20:21.274 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Neil-slb componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=4i8nr1f9nrtqf5 name=stream-7ba1d6a8 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5] Component.updateRemoteCandidates#481: new Pair added: 172.31.253.34:10000/udp/host → 192.168.104.53:45466/udp/host (stream-7ba1d6a8.RTP).
JVB 2021-07-04 04:20:21.274 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Neil-slb componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=4i8nr1f9nrtqf5 name=stream-7ba1d6a8 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5] Component.updateRemoteCandidates#481: new Pair added: 172.31.253.34:10000/udp/host → 192.168.0.130:40073/udp/host (stream-7ba1d6a8.RTP).
JVB 2021-07-04 04:20:21.280 INFO: [102] [confId=c3b934cdba832d35 gid=25248 stats_id=Neil-slb conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=4i8nr1f9nrtqf5 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5] ConnectivityCheckClient$PaceMaker.run#922: Pair failed: 172.31.253.34:10000/udp/host → 192.168.104.53:45466/udp/host (stream-7ba1d6a8.RTP)
JVB 2021-07-04 04:20:21.301 INFO: [102] [confId=c3b934cdba832d35 gid=25248 stats_id=Neil-slb conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=4i8nr1f9nrtqf5 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5] ConnectivityCheckClient$PaceMaker.run#922: Pair failed: 172.31.253.34:10000/udp/host → 192.168.0.130:40073/udp/host (stream-7ba1d6a8.RTP)
JVB 2021-07-04 04:20:21.318 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Irwin-GBx componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9irp81f9nrtqer name=stream-fc031fc3 epId=fc031fc3 local_ufrag=9irp81f9nrtqer] Component.addUpdateRemoteCandidates#347: Update remote candidate for stream-fc031fc3.RTP: 192.168.0.111:54966/udp
JVB 2021-07-04 04:20:21.318 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Irwin-GBx componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9irp81f9nrtqer name=stream-fc031fc3 epId=fc031fc3 local_ufrag=9irp81f9nrtqer] Component.updateRemoteCandidates#481: new Pair added: 172.31.253.34:10000/udp/host → 192.168.0.111:54966/udp/host (stream-fc031fc3.RTP).
JVB 2021-07-04 04:20:21.333 INFO: [102] [confId=c3b934cdba832d35 gid=25248 stats_id=Irwin-GBx conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9irp81f9nrtqer epId=fc031fc3 local_ufrag=9irp81f9nrtqer] ConnectivityCheckClient$PaceMaker.run#922: Pair failed: 172.31.253.34:10000/udp/host → 192.168.0.111:54966/udp/host (stream-fc031fc3.RTP)
JVB 2021-07-04 04:20:21.662 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Neil-slb componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=4i8nr1f9nrtqf5 name=stream-7ba1d6a8 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5] Component.addUpdateRemoteCandidates#347: Update remote candidate for stream-7ba1d6a8.RTP: 192.168.104.53:45466/udp
JVB 2021-07-04 04:20:21.662 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Neil-slb componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=4i8nr1f9nrtqf5 name=stream-7ba1d6a8 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5] Component.addUpdateRemoteCandidates#369: Not adding duplicate remote candidate: 192.168.104.53:45466/udp
JVB 2021-07-04 04:20:21.662 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Neil-slb componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=4i8nr1f9nrtqf5 name=stream-7ba1d6a8 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5] Component.addUpdateRemoteCandidates#347: Update remote candidate for stream-7ba1d6a8.RTP: 192.168.0.130:40073/udp
JVB 2021-07-04 04:20:21.662 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Neil-slb componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=4i8nr1f9nrtqf5 name=stream-7ba1d6a8 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5] Component.addUpdateRemoteCandidates#369: Not adding duplicate remote candidate: 192.168.0.130:40073/udp
JVB 2021-07-04 04:20:21.662 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Neil-slb componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=4i8nr1f9nrtqf5 name=stream-7ba1d6a8 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5] Component.addUpdateRemoteCandidates#347: Update remote candidate for stream-7ba1d6a8.RTP: 172.31.253.82:64481/udp
JVB 2021-07-04 04:20:21.662 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Neil-slb componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=4i8nr1f9nrtqf5 name=stream-7ba1d6a8 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5] Component.updateRemoteCandidates#481: new Pair added: 172.31.253.34:10000/udp/host → 172.31.253.82:64481/udp/relay (stream-7ba1d6a8.RTP).
JVB 2021-07-04 04:20:21.663 INFO: [102] [confId=c3b934cdba832d35 gid=25248 stats_id=Neil-slb conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=4i8nr1f9nrtqf5 epId=7ba1d6a8 local_ufrag=4i8nr1f9nrtqf5] ConnectivityCheckClient$PaceMaker.run#922: Pair failed: 172.31.253.34:10000/udp/host → 172.31.253.82:64481/udp/relay (stream-7ba1d6a8.RTP)
JVB 2021-07-04 04:20:21.795 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Irwin-GBx componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9irp81f9nrtqer name=stream-fc031fc3 epId=fc031fc3 local_ufrag=9irp81f9nrtqer] Component.addUpdateRemoteCandidates#347: Update remote candidate for stream-fc031fc3.RTP: 192.168.0.111:54966/udp
JVB 2021-07-04 04:20:21.795 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Irwin-GBx componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9irp81f9nrtqer name=stream-fc031fc3 epId=fc031fc3 local_ufrag=9irp81f9nrtqer] Component.addUpdateRemoteCandidates#369: Not adding duplicate remote candidate: 192.168.0.111:54966/udp
JVB 2021-07-04 04:20:21.795 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Irwin-GBx componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9irp81f9nrtqer name=stream-fc031fc3 epId=fc031fc3 local_ufrag=9irp81f9nrtqer] Component.addUpdateRemoteCandidates#347: Update remote candidate for stream-fc031fc3.RTP: 172.31.253.82:60423/udp
JVB 2021-07-04 04:20:21.795 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=Irwin-GBx componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9irp81f9nrtqer name=stream-fc031fc3 epId=fc031fc3 local_ufrag=9irp81f9nrtqer] Component.updateRemoteCandidates#481: new Pair added: 172.31.253.34:10000/udp/host → 172.31.253.82:60423/udp/relay (stream-fc031fc3.RTP).
JVB 2021-07-04 04:20:21.796 INFO: [102] [confId=c3b934cdba832d35 gid=25248 stats_id=Irwin-GBx conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9irp81f9nrtqer epId=fc031fc3 local_ufrag=9irp81f9nrtqer] ConnectivityCheckClient$PaceMaker.run#922: Pair failed: 172.31.253.34:10000/udp/host → 172.31.253.82:60423/udp/relay (stream-fc031fc3.RTP)
JVB 2021-07-04 04:20:21.996 INFO: [19] HealthChecker.run#171: Performed a successful health check in PT0.000005S. Sticky failure: false
JVB 2021-07-04 04:20:24.494 INFO: [98] [confId=c3b934cdba832d35 epId=c8086ca2 gid=25248 stats_id=German-RsJ conf_name=loadtest0@conference.lmjit1.dev2dev.net] DtlsTransport.setSetupAttribute#120: The remote side is acting as DTLS client, we’ll act as server
JVB 2021-07-04 04:20:24.494 INFO: [98] [confId=c3b934cdba832d35 epId=c8086ca2 local_ufrag=9pvi51f9nrtqf2 gid=25248 stats_id=German-RsJ conf_name=loadtest0@conference.lmjit1.dev2dev.net] IceTransport.startConnectivityEstablishment#184: Starting the Agent without remote candidates.
JVB 2021-07-04 04:20:24.494 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=German-RsJ conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9pvi51f9nrtqf2 epId=c8086ca2 local_ufrag=9pvi51f9nrtqf2] Agent.startConnectivityEstablishment#713: Start ICE connectivity establishment.
JVB 2021-07-04 04:20:24.494 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=German-RsJ conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9pvi51f9nrtqf2 epId=c8086ca2 local_ufrag=9pvi51f9nrtqf2] Agent.initCheckLists#949: Init checklist for stream stream-c8086ca2
JVB 2021-07-04 04:20:24.494 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=German-RsJ conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9pvi51f9nrtqf2 epId=c8086ca2 local_ufrag=9pvi51f9nrtqf2] Agent.setState#923: ICE state changed from Waiting to Running.
JVB 2021-07-04 04:20:24.494 INFO: [98] [confId=c3b934cdba832d35 epId=c8086ca2 local_ufrag=9pvi51f9nrtqf2 gid=25248 stats_id=German-RsJ conf_name=loadtest0@conference.lmjit1.dev2dev.net] IceTransport.iceStateChanged#323: ICE state changed old=Waiting new=Running
JVB 2021-07-04 04:20:24.494 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=German-RsJ conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9pvi51f9nrtqf2 epId=c8086ca2 local_ufrag=9pvi51f9nrtqf2] ConnectivityCheckClient.startChecks#142: Start connectivity checks.
JVB 2021-07-04 04:20:25.204 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=German-RsJ componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9pvi51f9nrtqf2 name=stream-c8086ca2 epId=c8086ca2 local_ufrag=9pvi51f9nrtqf2] Component.addUpdateRemoteCandidates#347: Update remote candidate for stream-c8086ca2.RTP: 192.0.0.4:41964/udp
JVB 2021-07-04 04:20:25.204 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=German-RsJ componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9pvi51f9nrtqf2 name=stream-c8086ca2 epId=c8086ca2 local_ufrag=9pvi51f9nrtqf2] Component.updateRemoteCandidates#481: new Pair added: 172.31.253.34:10000/udp/host → 192.0.0.4:41964/udp/host (stream-c8086ca2.RTP).
JVB 2021-07-04 04:20:25.218 INFO: [102] [confId=c3b934cdba832d35 gid=25248 stats_id=German-RsJ conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9pvi51f9nrtqf2 epId=c8086ca2 local_ufrag=9pvi51f9nrtqf2] ConnectivityCheckClient$PaceMaker.run#922: Pair failed: 172.31.253.34:10000/udp/host → 192.0.0.4:41964/udp/host (stream-c8086ca2.RTP)
JVB 2021-07-04 04:20:25.834 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=German-RsJ componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9pvi51f9nrtqf2 name=stream-c8086ca2 epId=c8086ca2 local_ufrag=9pvi51f9nrtqf2] Component.addUpdateRemoteCandidates#347: Update remote candidate for stream-c8086ca2.RTP: 192.0.0.4:41964/udp
JVB 2021-07-04 04:20:25.834 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=German-RsJ componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9pvi51f9nrtqf2 name=stream-c8086ca2 epId=c8086ca2 local_ufrag=9pvi51f9nrtqf2] Component.addUpdateRemoteCandidates#369: Not adding duplicate remote candidate: 192.0.0.4:41964/udp
JVB 2021-07-04 04:20:25.834 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=German-RsJ componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9pvi51f9nrtqf2 name=stream-c8086ca2 epId=c8086ca2 local_ufrag=9pvi51f9nrtqf2] Component.addUpdateRemoteCandidates#347: Update remote candidate for stream-c8086ca2.RTP: 172.31.253.82:50609/udp
JVB 2021-07-04 04:20:25.835 INFO: [98] [confId=c3b934cdba832d35 gid=25248 stats_id=German-RsJ componentId=1 conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9pvi51f9nrtqf2 name=stream-c8086ca2 epId=c8086ca2 local_ufrag=9pvi51f9nrtqf2] Component.updateRemoteCandidates#481: new Pair added: 172.31.253.34:10000/udp/host → 172.31.253.82:50609/udp/relay (stream-c8086ca2.RTP).
JVB 2021-07-04 04:20:25.842 INFO: [102] [confId=c3b934cdba832d35 gid=25248 stats_id=German-RsJ conf_name=loadtest0@conference.lmjit1.dev2dev.net ufrag=9pvi51f9nrtqf2 epId=c8086ca2 local_ufrag=9pvi51f9nrtqf2] ConnectivityCheckClient$PaceMaker.run#922: Pair failed: 172.31.253.34:10000/udp/host → 172.31.253.82:50609/udp/relay (stream-c8086ca2.RTP)
^C

#5

Hmm, spoke too soon. Took a few minutes but when user3 left, about the fifth time crashed room briefly again. Hmmm.

#6

It was about a month ago I last had a chance to dig into these servers, so trying to get back around that headspace. I think I was using this as a guide for this setup:DevOps Guide (scalable setup) · Jitsi Meet Handbook

on the jvb server, /etc/jitsi/videobridge, should the JVB_HOST= be default blank, or do I need to set them to the same as the prosody server? (lmjit1.dev2dev.net (or auth.lmjit1.dev2dev.net (or something else)))?
Minuse the JVB_SECRETS, here is what the two JVB /etc/jist/videobridge/config files look like:

jvb1a:

Jitsi Videobridge settings

sets the XMPP domain (default: none)

JVB_HOSTNAME=jitsi-videobridge.lmjit1.dev2dev.net

sets the hostname of the XMPP server (default: domain if set, localhost otherwise)

JVB_HOST=

sets the port of the XMPP server (default: 5275)

JVB_PORT=5347

extra options to pass to the JVB daemon

JVB_OPTS="–apis=,"

adds java system props that are passed to jvb (default are for home and logging config file)

JAVA_SYS_PROPS="-Dconfig.file=/etc/jitsi/videobridge/jvb.conf -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/videobridge/logging.properties"

jvb1b:

Jitsi Videobridge settings

sets the XMPP domain (default: none)

JVB_HOSTNAME=jvb2.lmjit1.dev2dev.net

sets the hostname of the XMPP server (default: domain if set, localhost otherwise)

JVB_HOST=

sets the port of the XMPP server (default: 5275)

JVB_PORT=5347

extra options to pass to the JVB daemon

JVB_OPTS="–apis=,"

adds java system props that are passed to jvb (default are for home and logging config file)

JAVA_SYS_PROPS="-Dconfig.file=/etc/jitsi/videobridge/jvb.conf -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/videobridge/logging.properties"

jvb.conf:
videobridge {
http-servers {
public {
port = 9090
}
}
websockets {
enabled = true
domain = “lmjit1.dev2dev.net:443”
tls = true
}
}

jvb1a sip-communicator.properties (minus password):
org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=true
#org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=meet-jit-si-turnrelay.jitsi.net:443
org.jitsi.videobridge.ENABLE_STATISTICS=true
#changed by hawke to pubsub org.jitsi.videobridge.STATISTICS_TRANSPORT=muc
org.jitsi.videobridge.xmpp.user.shard.HOSTNAME=lmjit1.dev2dev.net
org.jitsi.videobridge.xmpp.user.shard.DOMAIN=auth.lmjit1.dev2dev.net
org.jitsi.videobridge.xmpp.user.shard.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS=JvbBrewery@internal.auth.lmjit1.dev2dev.net
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME=jitsi-videobridge1a
org.jitsi.videobridge.xmpp.user.shard.DISABLE_CERTIFICATE_VERIFICATION=true
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=18.218.168.149

added by Hawke:

org.jitsi.videobridge.STATISTICS_TRANSPORT=pubsub
org.jitsi.videobridge.PUBSUB_SERVICE=lmjit1.dev2dev.net
org.jitsi.videobridge.PUBSUB_NODE=sharedStatsNode

Anything obvious jump out?

#7

WebSocket connection to 'wss://lmjit1.dev2dev.net/colibri-ws/default-id/825c3e131fa7c591/fc5cca78?pwd=6a8dtn2q1gld95r9euubkt9hks' failed: Error during WebSocket handshake: Unexpected response code: 502

That suggests you only have an nginx config for a local jvb (jvb on the same server). You have to create a block for remote jvbs in your nginx config.

Take a look at this guide:

#8

I don’t have any JVBs running on the web server (lmjit1). The only JVBs runnnig on are on jvb1a and jvb1b. I’ll check those settings you linked though. Thanks!

#9

That’s my point. Your nginx is configured by default to support a JVB on the web server; this will work only if jvb is on the same machine as nginx, so nginx forwards the connection to 127.0.0.1. if you’re hosting JVBs on separate servers, you have to configure nginx for that.

#10

Okay. Thanks.
So far the settings match those recommendations for nginx, however the turnserver settings might be off?

jitsi-meet coturn config. Do not modify this line

use-auth-secret
keep-address-family
realm=lmjit1.dev2dev.net
cert=/etc/coturn/certs/lmjit1.dev2dev.net.fullchain.pem
pkey=/etc/coturn/certs/lmjit1.dev2dev.net.privkey.pem
no-multicast-peers
no-cli
no-loopback-peers
no-tcp-relay
no-tcp
listening-port=3478
tls-listening-port=5349
no-tlsv1
no-tlsv1_1

Do I change the listening-port to also be 5349 or just rem the non-tls one out so it only has the tls-listening-port=5349 entry?

I’ll try remming out the former and just leaving the latter.

The prosody turns listing looks like this already:
{ type = “turns”, host = “lmjit1.dev2dev.net”, port = 5349, transport = “tcp”, secret = true, ttl = 86400, algorithm = “turn” }

I see have matching in nginx:
# colibri (JVB) websockets for jvb1
location ~ ^/colibri-ws/default-id/(.*) {
proxy_pass http://127.0.0.1:9090/colibri-ws/default-id/$1$is_args$args;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection “upgrade”;
tcp_nodelay on;
}

Both jvbs jvb.conf match:
cat jvb.conf
videobridge {
http-servers {
public {
port = 9090
}
}
websockets {
enabled = true
domain = “lmjit1.dev2dev.net:443”
tls = true
}
}

So maybe it is the turnserver config that I missed… checking…
restarting services… hmm, still didn’t improve. Hmmmm. I must have missed something along the way… rechecking the steps again…

lmjit1:/etc/nginx/sites-available# cat lmjit1.dev2dev.net.conf
server_names_hash_bucket_size 64;

types {

nginx’s default mime.types doesn’t include a mapping for wasm

application/wasm     wasm;

}
server {
listen 80;
listen [::]:80;
server_name lmjit1.dev2dev.net;

location ^~ /.well-known/acme-challenge/ {
    default_type "text/plain";
    root         /usr/share/jitsi-meet;
}
location = /.well-known/acme-challenge/ {
    return 404;
}
location / {
    return 301 https://$host$request_uri;
}

}
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name lmjit1.dev2dev.net;

# Mozilla Guideline v5.4, nginx 1.17.7, OpenSSL 1.1.1d, intermediate configuration
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;

ssl_session_timeout 1d;
ssl_session_cache shared:SSL:10m;  # about 40000 sessions
ssl_session_tickets off;

add_header Strict-Transport-Security "max-age=63072000" always;

ssl_certificate /etc/letsencrypt/live/lmjit1.dev2dev.net/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/lmjit1.dev2dev.net/privkey.pem;

root /usr/share/jitsi-meet;

# ssi on with javascript for multidomain variables in config.js
ssi on;
ssi_types application/x-javascript application/javascript;

index index.html index.htm;
error_page 404 /static/404.html;

gzip on;
gzip_types text/plain text/css application/javascript application/json image/x-icon application/octet-stream application/wasm;
gzip_vary on;
gzip_proxied no-cache no-store private expired auth;
gzip_min_length 512;

location = /config.js {
    alias /etc/jitsi/meet/lmjit1.dev2dev.net-config.js;
}

location = /external_api.js {
    alias /usr/share/jitsi-meet/libs/external_api.min.js;
}

# ensure all static content can always be found first
location ~ ^/(libs|css|static|images|fonts|lang|sounds|connection_optimization|.well-known)/(.*)$
{
    add_header 'Access-Control-Allow-Origin' '*';
    alias /usr/share/jitsi-meet/$1/$2;

    # cache all versioned files
    if ($arg_v) {
        expires 1y;
    }
}

# BOSH
location = /http-bind {
    proxy_pass       http://localhost:5280/http-bind;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header Host $http_host;
}

# xmpp websockets
location = /xmpp-websocket {
    proxy_pass http://127.0.0.1:5280/xmpp-websocket?prefix=$prefix&$args;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
    tcp_nodelay on;
}

# colibri (JVB) websockets for jvb1 (does there need to be another one for jvb2?)
location ~ ^/colibri-ws/default-id/(.*) {
    proxy_pass http://127.0.0.1:9090/colibri-ws/default-id/$1$is_args$args;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    tcp_nodelay on;
}

# load test minimal client, uncomment when used
#location ~ ^/_load-test/([^/?&:'"]+)$ {
#    rewrite ^/_load-test/(.*)$ /load-test/index.html break;
#}
#location ~ ^/_load-test/libs/(.*)$ {
#    add_header 'Access-Control-Allow-Origin' '*';
#    alias /usr/share/jitsi-meet/load-test/libs/$1;
#}

location ~ ^/([^/?&:'"]+)$ {
    try_files $uri @root_path;
}

location @root_path {
    rewrite ^/(.*)$ / break;
}

location ~ ^/([^/?&:'"]+)/config.js$
{
    set $subdomain "$1.";
    set $subdir "$1/";

    alias /etc/jitsi/meet/lmjit1.dev2dev.net-config.js;
}

# Anything that didn't match above, and isn't a real file, assume it's a room name and redirect to /
location ~ ^/([^/?&:'"]+)/(.*)$ {
    set $subdomain "$1.";
    set $subdir "$1/";
    rewrite ^/([^/?&:'"]+)/(.*)$ /$2;
}

# BOSH for subdomains
location ~ ^/([^/?&:'"]+)/http-bind {
    set $subdomain "$1.";
    set $subdir "$1/";
    set $prefix "$1";

    rewrite ^/(.*)$ /http-bind;
}

# websockets for subdomains
location ~ ^/([^/?&:'"]+)/xmpp-websocket {
    set $subdomain "$1.";
    set $subdir "$1/";
    set $prefix "$1";

    rewrite ^/(.*)$ /xmpp-websocket;
}

}

lmjit1:/etc# cat turnserver.conf

jitsi-meet coturn config. Do not modify this line

use-auth-secret
keep-address-family
realm=lmjit1.dev2dev.net
no-multicast-peers
no-cli
no-loopback-peers
no-tcp-relay
no-tcp
#listening-port=3478
tls-listening-port=5349
no-tlsv1
no-tlsv1_1

Mozilla SSL Configuration Generator

cipher-list=ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384

jitsi-meet coturn relay disable config. Do not modify this line

denied-peer-ip=0.0.0.0-0.255.255.255
denied-peer-ip=10.0.0.0-10.255.255.255
denied-peer-ip=100.64.0.0-100.127.255.255
denied-peer-ip=127.0.0.0-127.255.255.255
denied-peer-ip=169.254.0.0-169.254.255.255
denied-peer-ip=127.0.0.0-127.255.255.255
denied-peer-ip=172.16.0.0-172.31.255.255
denied-peer-ip=192.0.0.0-192.0.0.255
denied-peer-ip=192.0.2.0-192.0.2.255
denied-peer-ip=192.88.99.0-192.88.99.255
denied-peer-ip=192.168.0.0-192.168.255.255
denied-peer-ip=198.18.0.0-198.19.255.255
denied-peer-ip=198.51.100.0-198.51.100.255
denied-peer-ip=203.0.113.0-203.0.113.255
denied-peer-ip=240.0.0.0-255.255.255.255
syslog

lmjit1:/etc/prosody/conf.avail# cat lmjit1.dev2dev.net.cfg.lua
unlimited_jids = { “focus@auth.lmjit1.dev2dev.net”, “jvb@auth.lmjit1.dev2dev.net” }
plugin_paths = { “/usr/share/jitsi-meet/prosody-plugins/” }

– domain mapper options, must at least have domain base set to use the mapper
muc_mapper_domain_base = “lmjit1.dev2dev.net”;

external_services = {
{ type = “stun”, host = “lmjit1.dev2dev.net”, port = 3478 },
{ type = “turn”, host = “lmjit1.dev2dev.net”, port = 3478, transport = “udp”, secret = true, ttl = 86400, algorithm = “turn” },
{ type = “turns”, host = “lmjit1.dev2dev.net”, port = 5349, transport = “tcp”, secret = true, ttl = 86400, algorithm = “turn” }
};

cross_domain_bosh = false;
consider_bosh_secure = true;
– https_ports = { }; – Remove this line to prevent listening on port 5284

– Mozilla SSL Configuration Generator
ssl = {
protocol = “tlsv1_2+”;
}

VirtualHost “lmjit1.dev2dev.net”
– enabled = false – Remove this line to enable this host
authentication = “anonymous”
– Properties below are modified by jitsi-meet-tokens package config
– and authentication above is switched to “token”
–app_id=“example_app_id”
–app_secret=“example_app_secret”
– Assign this host a certificate for TLS, otherwise it would use the one
– set in the global section (if any).
– Note that old-style SSL on port 5223 only supports one certificate, and will always
– use the global one.
…
speakerstats_component = “speakerstats.lmjit1.dev2dev.net”
conference_duration_component = “conferenceduration.lmjit1.dev2dev.net”
– we need bosh
modules_enabled = {
“bosh”;
“pubsub”;
“ping”; – Enable mod_ping
“speakerstats”;
“external_services”;
“conference_duration”;
“muc_lobby_rooms”;
}
c2s_require_encryption = false
lobby_muc = “lobby.lmjit1.dev2dev.net”
main_muc = “conference.lmjit1.dev2dev.net”
– muc_lobby_whitelist = { “recorder.lmjit1.dev2dev.net” } – Here we can whitelist jibri to enter lobby enabled rooms

Component “conference.lmjit1.dev2dev.net” “muc”
restrict_room_creation = true
storage = “memory”
modules_enabled = {
“muc_meeting_id”;
“muc_domain_mapper”;
–“token_verification”;
}
admins = { “focus@auth.lmjit1.dev2dev.net” }
muc_room_locking = false
muc_room_default_public_jids = true

– internal muc component
Component “internal.auth.lmjit1.dev2dev.net” “muc”
storage = “memory”
modules_enabled = {
“ping”;
}
admins = { “focus@auth.lmjit1.dev2dev.net”, “jvb@auth.lmjit1.dev2dev.net” }
muc_room_locking = false
muc_room_default_public_jids = true

– added by Hawke as per :
– How to configure Jitsi Meet with multiple Videobridge nodes - DEV Community
– for enabling multiple jvbs

Component “jitsi-videobridge1a.lmjit1.dev2dev.net”
– //This is the domain name of the second videobridge
…
– This can be found on first VB, under /etc/jitsi/videobridge/config

–

–
Component “jvb2.lmjit1.dev2dev.net”
– //This is the domain name of the second videobridge
…
– This can be found on second VB, under /etc/jitsi/videobridge/config

VirtualHost “auth.lmjit1.dev2dev.net”
modules_enabled = { “limits_exception”; }
ssl = {
key = “/etc/prosody/certs/auth.lmjit1.dev2dev.net.key”;
certificate = “/etc/prosody/certs/auth.lmjit1.dev2dev.net.crt”;
}
authentication = “internal_hashed”

– added by Hawke as per:
– How to configure Jitsi Meet with multiple Videobridge nodes - DEV Community
admins = {
“jitsi-videobridge1a.lmjit1.dev2dev.net”,
“jvb2.lmjit1.dev2dev.net”,
}

– Proxy to jicofo’s user JID, so that it doesn’t have to register as a component.
Component “focus.lmjit1.dev2dev.net” “client_proxy”
target_address = “focus@auth.lmjit1.dev2dev.net”

Component “speakerstats.lmjit1.dev2dev.net” “speakerstats_component”
muc_component = “conference.lmjit1.dev2dev.net”

Component “conferenceduration.lmjit1.dev2dev.net” “conference_duration_component”
muc_component = “conference.lmjit1.dev2dev.net”

Component “lobby.lmjit1.dev2dev.net” “muc”
storage = “memory”
restrict_room_creation = true
muc_room_locking = false
muc_room_default_public_jids = true

I am most suspicious of my prosody config.

#11

This should just be your domain:

JVB_HOSTNAME=lmjit1.dev2dev.net/

There are quite a number of misconfigurations. I suggest searching through the forum to find out how to loadbalance multiple JVBs.

#12

Alas that is probably why I have the configuration issues. I have been trying to figure it out from half a dozen different forum postings in addition to those other sources already listed and seem to conflict with other sometimes. Just trying to muddle through. Thanks for that pointer there, I’ll keep hammering away .

#13

So the JVB_HOSTNAME= is not the hostname of the JVB instance, it is just the overall domain and the listing of the nginx/meet/prosody/jicofo instance instead?
I see that is mistake for jitsi-videobridge.lmjit1.dev2dev.net because I had for jvb2, the jvb2.lmjit1.dev2dev.net, but you are saying that JVB_HOSTNAME should just be JVB_HOSTNAME=lmjit1.dev2dev.net not jvb1.lmjit1.dev2dev.net (config on jvb1), and jvb2.lmjit1.dev2dev.net (config on jvb2). Sorry I’m so confused by that. Thanks for clarifying (assuming that I’m understanding now how that is corrected as far as that entry now).

#14

Yes, it should be the hostname of the XMPP domain (where prosody is hosted).
Then you have to set JVB_HOST to be the IP address of the JMS server.

I think this post should help too:

1 Like
#15

That is helpful in part, yes. Going through the steps as per link you listed. Thank you.

The JVBs I setup in AWS do not actually have public IP addresses, so that is a big issue, yes? (port 10000, (I read elsewhere one port for each or was that just for NAT setup?, 10000 for jvb1, 10001 for jvb2, 10002 for jvb3, etc.)?

The only static public IP (assigned Elastic IP (EIP)) for this cluster available right now is the Nginx/Jitsi Meet instance (lmjit1), all the other components are in private AWS subnet. I could shut down some other R&D instances and juggle around the 5 EIPs for the account, but is that really the correct way to handle this?

Looks like a problem: I have to have more public IP addresses for those JVBs? But already maxed out for the 5 elastic IPs for the account in AWS for other instances, and so for the 2 additional JVBs to have public IPs they will be dynamic not static, so if I follow those directions and manually put in the public dynamic IP that will be broken next time they are stopped/started?
This is just R&D environment so is shutdown every night.

How does someone make this work in AWS environment without auto-scaling if for example they want:
Static public (elastic IP EIP) for jitsi page (nginx + meet + prosody + jicofo),
Static public IP for 10 JVBs? Other than auto-scaling, how would one make that work with the AWS limit of only 5 EIPs? (sorry if these are too many newb questions).

Is this going to need to follow something like one the various NAT instructions I’ve seen?
Does that work with multiple JVBs? Or am I just confusing things even more here?
Guess I should get some sleep and come back at this after, I’m going in circles now.

Cheers!

#16

Getting elastic IPs makes sense only if you want to be able to give your clients IPs/IP ranges to whitelist on their firewalls.

Otherwise normal dynamic IPs are totally OK - you just add some small automation scripts inside the JVB servers to rewrite the configs with the current IP (you retrieve it from the AWS APIs) and then restart JVB to make the changes active.

1 Like
#17

@rpgresearch,

The following are the automation script and related systemd service

/usr/local/sbin/jvb-config
/etc/systemd/system/jvb-config.service

2 Likes
#18

Exactly, this is one way to do it. Just a small note - I personally prefer to make all the nicknames not random, but based on something that will help me quickly distinguish the nodes (jvb, jibri, etc.) when I go through the logs. For example something like “shard_name-node_type-IP” - this tells me right away in which shard/region to look for the server, is it jvb/jibri and what is the IP. So this way if you check the Jicofo log and you see something suspicious and need to login to that node to see what’s wrong, you just select the IP and middleclick/paste it in an ssh terminal - and you’re in. :wink: Helps a lot if it’s a large platform with a lot of (changing) IPs.

2 Likes