Disappearing cameras with multiple videobridges and octo

Hi.
We have setup with one ubuntu instance with jitsi stack(without JVB), and two ubuntu instances with JVBs. The problem is we see only half of all cameras or none. There is no errors on the jitsi-meet frontend in consloe. The only problem is the one below, but I couldn’t find any solution.

JVB 2021-09-03 10:20:52.785 WARNING: [317] [confId=74e10bfc328aff62 gid=182855 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN] OctoEndpointMessageTransport.endpointMessage#142: Unable to find endpoint to send EndpointMessage to: c58ed61e

JVB 2021-09-03 09:32:57.685 WARNING: [232] [confId=6b7117e9b737156a gid=163037 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN] OctoEndpointMessageTransport.endpointMessage#142: Unable to find endpoint to send EndpointMessage to: a82a560c
JVB 2021-09-03 09:32:57.714 WARNING: [243] [confId=6b7117e9b737156a gid=163037 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN] OctoEndpointMessageTransport.endpointMessage#142: Unable to find endpoint to send EndpointMessage to: a82a560c

our config.js OUR_DOMAIN.-config.js - Pastebin.com
our jicofo configs jicofo-configs - Pastebin.com
our jvb1 configs jvb-1-configs - Pastebin.com
out jvb2 configs jvb-2-configs - Pastebin.com

Samples of our logs during tests:
jvb log

JVB 2021-09-03 09:29:30.090 INFO: [185] [confId=6b7117e9b737156a gid=163037 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN] ConfOctoTransport.addHandler#471: Adding handler for ep ID ffffffff
JVB 2021-09-03 09:29:30.090 INFO: [185] [relayId=0.0.0.0:4096] BridgeOctoTransport.addHandler#88: Adding handler for conference 163037
JVB 2021-09-03 09:29:30.090 WARNING: [185] [relayId=0.0.0.0:4096] BridgeOctoTransport.addHandler#92: Replacing an existing packet handler for gid=163037
JVB 2021-09-03 09:29:30.095 INFO: [185] [confId=6b7117e9b737156a epId=d8bd3546 local_ufrag=b0vrb1felfro7f gid=163037 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f] Agent.gatherCandidates#622: Gathering candidates for component stream-d8bd3546.RTP.
JVB 2021-09-03 09:29:30.097 INFO: [188] [confId=6b7117e9b737156a gid=163037 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN] Conference.dominantSpeakerChanged#424: ds_change ds_id=d8bd3546
JVB 2021-09-03 09:29:30.098 INFO: [185] [confId=6b7117e9b737156a epId=d8bd3546 gid=163037 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN] Endpoint.setTransportInfo#679: Ignoring empty DtlsFingerprint extension: <transport xmlns='urn:xmpp:jingle:transports:ice-udp:1'><fingerprint xmlns='urn:xmpp:jingle:apps:dtls:0' required='false'/></transport>
JVB 2021-09-03 09:29:30.498 INFO: [185] [confId=6b7117e9b737156a gid=163037 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN] OctoEndpoints.lambda$addEndpoints$6#169: Creating Octo endpoint f0fc95e1
JVB 2021-09-03 09:29:30.499 INFO: [185] [confId=6b7117e9b737156a gid=163037 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN] ConfOctoTransport.addHandler#471: Adding handler for ep ID f0fc95e1
JVB 2021-09-03 09:29:30.629 INFO: [185] [confId=6b7117e9b737156a epId=d8bd3546 gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN] DtlsTransport.setSetupAttribute#120: The remote side is acting as DTLS client, we'll act as server
JVB 2021-09-03 09:29:30.629 INFO: [185] [confId=6b7117e9b737156a epId=d8bd3546 local_ufrag=b0vrb1felfro7f gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN] IceTransport.startConnectivityEstablishment#184: Starting the Agent without remote candidates.
JVB 2021-09-03 09:29:30.629 INFO: [185] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] Agent.startConnectivityEstablishment#713: Start ICE connectivity establishment.
JVB 2021-09-03 09:29:30.629 INFO: [185] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] Agent.initCheckLists#949: Init checklist for stream stream-d8bd3546
JVB 2021-09-03 09:29:30.629 INFO: [185] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] Agent.setState#923: ICE state changed from Waiting to Running.
JVB 2021-09-03 09:29:30.629 INFO: [185] [confId=6b7117e9b737156a epId=d8bd3546 local_ufrag=b0vrb1felfro7f gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN] IceTransport.iceStateChanged#323: ICE state changed old=Waiting new=Running
JVB 2021-09-03 09:29:30.630 INFO: [185] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ConnectivityCheckClient.startChecks#142: Start connectivity checks.
JVB 2021-09-03 09:29:30.642 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] Agent.triggerCheck#1714: Add peer CandidatePair with new reflexive address to checkList: CandidatePair (State=Frozen Priority=7890166909142827007):
	LocalCandidate=candidate:1 1 tcp 2130706431 172.31.43.115 4443 typ host
	RemoteCandidate=candidate:10000 1 tcp 1837072639 89.186.9.131 56476 typ prflx
JVB 2021-09-03 09:29:30.677 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] Agent.triggerCheck#1714: Add peer CandidatePair with new reflexive address to checkList: CandidatePair (State=Frozen Priority=7889885434166116351):
	LocalCandidate=candidate:1 1 tcp 2130706431 172.31.43.115 4443 typ host
	RemoteCandidate=candidate:10001 1 tcp 1837007103 89.186.9.131 62510 typ prflx
JVB 2021-09-03 09:29:30.680 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] Agent.triggerCheck#1714: Add peer CandidatePair with new reflexive address to checkList: CandidatePair (State=Frozen Priority=7889603959189405695):
	LocalCandidate=candidate:1 1 tcp 2130706431 172.31.43.115 4443 typ host
	RemoteCandidate=candidate:10002 1 tcp 1836941567 89.186.9.131 50271 typ prflx
JVB 2021-09-03 09:29:30.695 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] Agent.triggerCheck#1714: Add peer CandidatePair with new reflexive address to checkList: CandidatePair (State=Frozen Priority=7962083765692399615):
	LocalCandidate=candidate:3 1 udp 2130706431 172.31.43.115 10000 typ host
	RemoteCandidate=candidate:10003 1 udp 1853817087 89.186.9.131 50373 typ prflx
JVB 2021-09-03 09:29:30.705 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ConnectivityCheckClient.processSuccessResponse#630: Pair succeeded: 172.31.43.115:4443/tcp/host -> 89.186.9.131:56476/tcp/prflx (stream-d8bd3546.RTP).
JVB 2021-09-03 09:29:30.705 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 componentId=1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f name=stream-d8bd3546 epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ComponentSocket.addAuthorizedAddress#99: Adding allowed address: 89.186.9.131:56476/tcp
JVB 2021-09-03 09:29:30.705 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ConnectivityCheckClient.processSuccessResponse#639: Pair validated: 3.124.45.227:4443/tcp/srflx -> 89.186.9.131:56476/tcp/prflx (stream-d8bd3546.RTP).
JVB 2021-09-03 09:29:30.705 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] DefaultNominator.strategyNominateFirstValid#142: Nominate (first valid): 3.124.45.227:4443/tcp/srflx -> 89.186.9.131:56476/tcp/prflx (stream-d8bd3546.RTP).
JVB 2021-09-03 09:29:30.705 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] Agent.nominate#1787: verify if nominated pair answer again
JVB 2021-09-03 09:29:30.705 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ConnectivityCheckClient.processSuccessResponse#708: IsControlling: true USE-CANDIDATE:false.
JVB 2021-09-03 09:29:30.726 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ConnectivityCheckClient.processSuccessResponse#630: Pair succeeded: 172.31.43.115:10000/udp/host -> 89.186.9.131:50373/udp/prflx (stream-d8bd3546.RTP).
JVB 2021-09-03 09:29:30.726 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 componentId=1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f name=stream-d8bd3546 epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ComponentSocket.addAuthorizedAddress#99: Adding allowed address: 89.186.9.131:50373/udp
JVB 2021-09-03 09:29:30.726 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ConnectivityCheckClient.processSuccessResponse#639: Pair validated: 3.124.45.227:10000/udp/srflx -> 89.186.9.131:50373/udp/prflx (stream-d8bd3546.RTP).
JVB 2021-09-03 09:29:30.726 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ConnectivityCheckClient.processSuccessResponse#708: IsControlling: true USE-CANDIDATE:false.
JVB 2021-09-03 09:29:30.729 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] Agent.triggerCheck#1714: Add peer CandidatePair with new reflexive address to checkList: CandidatePair (State=Frozen Priority=7889322484212695039):
	LocalCandidate=candidate:1 1 tcp 2130706431 172.31.43.115 4443 typ host
	RemoteCandidate=candidate:10004 1 tcp 1836876031 89.186.9.131 54958 typ prflx
JVB 2021-09-03 09:29:30.730 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ConnectivityCheckClient.processSuccessResponse#630: Pair succeeded: 172.31.43.115:4443/tcp/host -> 89.186.9.131:62510/tcp/prflx (stream-d8bd3546.RTP).
JVB 2021-09-03 09:29:30.731 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 componentId=1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f name=stream-d8bd3546 epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ComponentSocket.addAuthorizedAddress#99: Adding allowed address: 89.186.9.131:62510/tcp
JVB 2021-09-03 09:29:30.731 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ConnectivityCheckClient.processSuccessResponse#639: Pair validated: 3.124.45.227:4443/tcp/srflx -> 89.186.9.131:62510/tcp/prflx (stream-d8bd3546.RTP).
JVB 2021-09-03 09:29:30.731 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ConnectivityCheckClient.processSuccessResponse#708: IsControlling: true USE-CANDIDATE:false.
JVB 2021-09-03 09:29:30.732 WARNING: [185] [confId=6b7117e9b737156a gid=163037 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN] OctoEndpointMessageTransport.endpointMessage#142: Unable to find endpoint to send EndpointMessage to: f0fc95e1
JVB 2021-09-03 09:29:30.733 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ConnectivityCheckClient.processSuccessResponse#630: Pair succeeded: 172.31.43.115:4443/tcp/host -> 89.186.9.131:50271/tcp/prflx (stream-d8bd3546.RTP).
JVB 2021-09-03 09:29:30.733 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 componentId=1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f name=stream-d8bd3546 epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ComponentSocket.addAuthorizedAddress#99: Adding allowed address: 89.186.9.131:50271/tcp
JVB 2021-09-03 09:29:30.733 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ConnectivityCheckClient.processSuccessResponse#639: Pair validated: 3.124.45.227:4443/tcp/srflx -> 89.186.9.131:50271/tcp/prflx (stream-d8bd3546.RTP).
JVB 2021-09-03 09:29:30.733 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ConnectivityCheckClient.processSuccessResponse#708: IsControlling: true USE-CANDIDATE:false.
JVB 2021-09-03 09:29:30.762 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ConnectivityCheckClient.processSuccessResponse#630: Pair succeeded: 3.124.45.227:4443/tcp/srflx -> 89.186.9.131:56476/tcp/prflx (stream-d8bd3546.RTP).
JVB 2021-09-03 09:29:30.762 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ConnectivityCheckClient.processSuccessResponse#639: Pair validated: 3.124.45.227:4443/tcp/srflx -> 89.186.9.131:56476/tcp/prflx (stream-d8bd3546.RTP).
JVB 2021-09-03 09:29:30.762 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ConnectivityCheckClient.processSuccessResponse#708: IsControlling: true USE-CANDIDATE:true.
JVB 2021-09-03 09:29:30.762 INFO: [193] [confId=6b7117e9b737156a gid=163037 stats_id=Arely-2w1 conf_name=poc@conference.jitsi-vm.OUR_DOMAIN ufrag=b0vrb1felfro7f epId=d8bd3546 local_ufrag=b0vrb1felfro7f] ConnectivityCheckClient.processSuccessResponse#723: Nomination confirmed for pair: 3.124.45.227:4443/tcp/srflx -> 89.186.9.131:56476/tcp/prflx (stream-d8bd3546.RTP).

our jicofo.log

Jicofo 2021-09-03 10:20:12.691 INFO: [293] AutoOwnerRoleManager.electNewOwner#98: Electing new owner: ChatMember[poc@conference.jitsi-vm.OUR_DOMAIN/236b46e5, jid: null]@2007859480
Jicofo 2021-09-03 10:20:12.702 INFO: [293] [room=poc@conference.jitsi-vm.OUR_DOMAIN meeting_id=819aadaf-298f-4a8a-89db-777773755aca] JitsiMeetConferenceImpl.onMemberLeft#1055: Member left:727abcd1
Jicofo 2021-09-03 10:20:12.703 INFO: [293] [room=poc@conference.jitsi-vm.OUR_DOMAIN meeting_id=819aadaf-298f-4a8a-89db-777773755aca] JitsiMeetConferenceImpl.terminateParticipant#1092: Terminating 727abcd1, reason: gone, send session-terminate: false
Jicofo 2021-09-03 10:20:12.703 INFO: [293] AbstractOperationSetJingle.terminateSession#330: Terminate session: poc@conference.jitsi-vm.OUR_DOMAIN/727abcd1, reason: gone, send terminate: false
Jicofo 2021-09-03 10:20:12.703 INFO: [293] [room=poc@conference.jitsi-vm.OUR_DOMAIN meeting_id=819aadaf-298f-4a8a-89db-777773755aca] JitsiMeetConferenceImpl.terminateParticipant#1120: Removed participant 727abcd1 removed=true
Jicofo 2021-09-03 10:20:12.703 INFO: [293] [room=poc@conference.jitsi-vm.OUR_DOMAIN meeting_id=819aadaf-298f-4a8a-89db-777773755aca] JitsiMeetConferenceImpl$BridgeSession.terminate#2570: Expiring channels for: poc@conference.jitsi-vm.OUR_DOMAIN/727abcd1 on: Bridge[jid=jvbbrewery@internal.auth.jitsi-vm.OUR_DOMAIN/55faad5c-6c4d-49d1-9b05-1e9a752cf268, relayId=0.0.0.0:4096, region=region1, stress=0.05]
Jicofo 2021-09-03 10:20:56.800 INFO: [293] [room=poc@conference.jitsi-vm.OUR_DOMAIN meeting_id=819aadaf-298f-4a8a-89db-777773755aca] JitsiMeetConferenceImpl.onMemberLeft#1055: Member left:f18360bc
Jicofo 2021-09-03 10:20:56.800 INFO: [293] [room=poc@conference.jitsi-vm.OUR_DOMAIN meeting_id=819aadaf-298f-4a8a-89db-777773755aca] JitsiMeetConferenceImpl.terminateParticipant#1092: Terminating f18360bc, reason: gone, send session-terminate: false
Jicofo 2021-09-03 10:20:56.800 INFO: [293] AbstractOperationSetJingle.terminateSession#330: Terminate session: poc@conference.jitsi-vm.OUR_DOMAIN/f18360bc, reason: gone, send terminate: false
Jicofo 2021-09-03 10:20:56.801 INFO: [293] [room=poc@conference.jitsi-vm.OUR_DOMAIN meeting_id=819aadaf-298f-4a8a-89db-777773755aca] JitsiMeetConferenceImpl.terminateParticipant#1120: Removed participant f18360bc removed=true
Jicofo 2021-09-03 10:20:56.801 INFO: [293] [room=poc@conference.jitsi-vm.OUR_DOMAIN meeting_id=819aadaf-298f-4a8a-89db-777773755aca] JitsiMeetConferenceImpl$BridgeSession.terminate#2570: Expiring channels for: poc@conference.jitsi-vm.OUR_DOMAIN/f18360bc on: Bridge[jid=jvbbrewery@internal.auth.jitsi-vm.OUR_DOMAIN/d8b5134c-6442-425a-b5e6-7fe47b1d3d50, relayId=0.0.0.0:4096, region=region1, stress=0.03]
Jicofo 2021-09-03 10:20:56.826 INFO: [293] [room=poc@conference.jitsi-vm.OUR_DOMAIN meeting_id=819aadaf-298f-4a8a-89db-777773755aca] JitsiMeetConferenceImpl.onMemberLeft#1055: Member left:db985b03
Jicofo 2021-09-03 10:20:56.826 INFO: [293] [room=poc@conference.jitsi-vm.OUR_DOMAIN meeting_id=819aadaf-298f-4a8a-89db-777773755aca] JitsiMeetConferenceImpl.terminateParticipant#1092: Terminating db985b03, reason: gone, send session-terminate: false
Jicofo 2021-09-03 10:20:56.826 INFO: [293] AbstractOperationSetJingle.terminateSession#330: Terminate session: poc@conference.jitsi-vm.OUR_DOMAIN/db985b03, reason: gone, send terminate: false
Jicofo 2021-09-03 10:20:56.827 INFO: [293] [room=poc@conference.jitsi-vm.OUR_DOMAIN meeting_id=819aadaf-298f-4a8a-89db-777773755aca] JitsiMeetConferenceImpl.terminateParticipant#1120: Removed participant db985b03 removed=true
Jicofo 2021-09-03 10:20:56.827 INFO: [293] [room=poc@conference.jitsi-vm.OUR_DOMAIN meeting_id=819aadaf-298f-4a8a-89db-777773755aca] JitsiMeetConferenceImpl$BridgeSession.terminate#2570: Expiring channels for: poc@conference.jitsi-vm.OUR_DOMAIN/db985b03 on: Bridge[jid=jvbbrewery@internal.auth.jitsi-vm.OUR_DOMAIN/55faad5c-6c4d-49d1-9b05-1e9a752cf268, relayId=0.0.0.0:4096, region=region1, stress=0.04]
Jicofo 2021-09-03 10:20:56.828 INFO: [293] [room=poc@conference.jitsi-vm.OUR_DOMAIN meeting_id=819aadaf-298f-4a8a-89db-777773755aca] JitsiMeetConferenceImpl.onMemberLeft#1055: Member left:c58ed61e
Jicofo 2021-09-03 10:20:56.828 INFO: [293] [room=poc@conference.jitsi-vm.OUR_DOMAIN meeting_id=819aadaf-298f-4a8a-89db-777773755aca] JitsiMeetConferenceImpl.terminateParticipant#1092: Terminating c58ed61e, reason: gone, send session-terminate: false

our prosody.log

Sep 03 10:18:11 mod_bosh	info	New BOSH session, assigned it sid '49a3b133-c0b9-41a8-adcd-d35e7661152d'
Sep 03 10:18:11 bosh49a3b133-c0b9-41a8-adcd-d35e7661152d	info	Authenticated as jv1bm1g65vzcdq66@jitsi-vm.OUR_DOMAIN
Sep 03 10:18:12 mod_bosh	info	New BOSH session, assigned it sid '687f97bc-009e-4a17-be6a-54e9c2b7a5bc'
Sep 03 10:18:12 bosh687f97bc-009e-4a17-be6a-54e9c2b7a5bc	info	Authenticated as rwhzuwep3l9yutxk@jitsi-vm.OUR_DOMAIN
Sep 03 10:18:14 mod_bosh	info	New BOSH session, assigned it sid '54d48114-435f-45a3-af99-7f9ab5841154'
Sep 03 10:18:26 mod_bosh	info	New BOSH session, assigned it sid '5818d0d7-cdee-41a3-8ed5-66a5a4308ce9'
Sep 03 10:18:26 bosh5818d0d7-cdee-41a3-8ed5-66a5a4308ce9	info	Authenticated as wj9e5vc07gact3f2@jitsi-vm.OUR_DOMAIN
Sep 03 10:18:29 bosh54d48114-435f-45a3-af99-7f9ab5841154	info	Authenticated as jw3gwj8fs_z5a1fc@jitsi-vm.OUR_DOMAIN
Sep 03 10:18:31 mod_bosh	info	New BOSH session, assigned it sid '164a201a-73e5-4eb1-badf-18b5d8b33fc1'
Sep 03 10:18:31 bosh164a201a-73e5-4eb1-badf-18b5d8b33fc1	info	Authenticated as dvpawlbm_0696xr5@jitsi-vm.OUR_DOMAIN
Sep 03 10:19:30 mod_bosh	info	New BOSH session, assigned it sid '7d8ebdb3-fc1e-4b5e-91de-568df06a95bd'
Sep 03 10:19:31 bosh7d8ebdb3-fc1e-4b5e-91de-568df06a95bd	info	Authenticated as h7pdno0k3hua2gl_@jitsi-vm.OUR_DOMAIN
Sep 03 10:19:31 mod_bosh	info	New BOSH session, assigned it sid '763fcda4-1a47-4925-8048-d8a2dcbc2a64'
Sep 03 10:19:32 mod_bosh	info	New BOSH session, assigned it sid 'e378c414-4a67-4dd2-9926-6239254f5a0a'
Sep 03 10:19:32 bosh763fcda4-1a47-4925-8048-d8a2dcbc2a64	info	Authenticated as 5mqig2vl_w_cntwr@jitsi-vm.OUR_DOMAIN
Sep 03 10:19:32 boshe378c414-4a67-4dd2-9926-6239254f5a0a	info	Authenticated as 8p8wv0ba3c4aja2f@jitsi-vm.OUR_DOMAIN
Sep 03 10:19:32 mod_bosh	info	New BOSH session, assigned it sid 'e0105aa9-5b9c-4b6a-b447-0de3b3b48c33'
Sep 03 10:19:33 mod_bosh	info	New BOSH session, assigned it sid '865258ed-5035-49d0-a24f-cc2d872f6b20'
Sep 03 10:19:33 boshe0105aa9-5b9c-4b6a-b447-0de3b3b48c33	info	Authenticated as nl0edunzmvm8kbcf@jitsi-vm.OUR_DOMAIN
Sep 03 10:19:33 bosh865258ed-5035-49d0-a24f-cc2d872f6b20	info	Authenticated as nswgdyl0gtyxwx7f@jitsi-vm.OUR_DOMAIN
Sep 03 10:19:33 mod_bosh	info	New BOSH session, assigned it sid '14db4fb4-607b-4475-9604-e7f7ea7fd008'
Sep 03 10:19:34 mod_bosh	info	New BOSH session, assigned it sid 'e220492f-5aa3-4553-bef5-e7d98270f3c7'
Sep 03 10:19:34 bosh14db4fb4-607b-4475-9604-e7f7ea7fd008	info	Authenticated as trhiz-b-mh22ddc4@jitsi-vm.OUR_DOMAIN
Sep 03 10:19:35 boshe220492f-5aa3-4553-bef5-e7d98270f3c7	info	Authenticated as _k4g32-cdogevnfk@jitsi-vm.OUR_DOMAIN
Sep 03 10:19:35 mod_bosh	info	New BOSH session, assigned it sid '3ed0c71c-496a-4871-b49f-463f0384f291'
Sep 03 10:19:35 bosh3ed0c71c-496a-4871-b49f-463f0384f291	info	Authenticated as lmldbi3aheeb0zni@jitsi-vm.OUR_DOMAIN
Sep 03 10:20:56 bosh865258ed-5035-49d0-a24f-cc2d872f6b20	info	BOSH client disconnected: session close
Sep 03 10:20:56 boshe378c414-4a67-4dd2-9926-6239254f5a0a	info	BOSH client disconnected: session close
Sep 03 10:20:56 bosh14db4fb4-607b-4475-9604-e7f7ea7fd008	info	BOSH client disconnected: session close
Sep 03 10:20:56 bosh3ed0c71c-496a-4871-b49f-463f0384f291	info	BOSH client disconnected: session close
Sep 03 10:20:57 boshe220492f-5aa3-4553-bef5-e7d98270f3c7	info	BOSH client disconnected: session close
Sep 03 10:20:58 bosh5818d0d7-cdee-41a3-8ed5-66a5a4308ce9	info	BOSH client disconnected: session close
Sep 03 10:21:58 speakerstats.jitsi-vm.OUR_DOMAIN:speakerstats_component	warn	A module has been configured that triggers external events.
Sep 03 10:21:58 speakerstats.jitsi-vm.OUR_DOMAIN:speak

Do you have some ideas about what is wrong?

Thanks for your help.

Regards.
Mateusz Sokołowski

  • Are TCP/9090 and UDP/4096 open on JVB?

  • Did you add websocket block to the Nginx config?


Yes, they are open.

I think i did jitsi-vm.OUR_DOMAIN.conf - Pastebin.com

    # colibri (JVB) websockets for jvb1
    location ~ ^/colibri-ws/1/(.*) {
        proxy_pass http://FIRST_JVB:9090/colibri-ws/1/$1$is_args$args;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        tcp_nodelay on;
    }

    location ~ ^/colibri-ws/2/(.*) {
        proxy_pass http://SECOND_JVB:9090/colibri-ws/2/$1$is_args$args;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        tcp_nodelay on;
    }

The Nginx config seems OK.

For ports, it’s needed to check the accessibilities. Are you sure these ports are allowed on the firewall?

Thanks for your reply.
Looks like JVBs are reachable.

root@ip-**-117:/etc/jitsi/videobridge# systemctl stop jitsi-videobridge2
root@ip-**-117:/etc/jitsi/videobridge# nc -l 9090
123
root@ip-**-117:/etc/jitsi/videobridge# nc -ul 4096^C
c: command not found
root@ip-**-117::/etc/jitsi/videobridge# nc -ul 4096
123
^C
root@ip-**-117::/etc/jitsi/videobridge# 
root@ip-**-115:/etc/jitsi/videobridge# systemctl stop jitsi-videobridge2
root@ip-**-115:/etc/jitsi/videobridge# nc -l 9090
123
root@ip-**-115:/etc/jitsi/videobridge# nc -ul 4096
123
^C
root@ip-**-115:/etc/jitsi/videobridge# 
``

Does it work when there is only one active JVB?

  • when jvb1 is stopped and jvb2 is running…

  • when jvb1 is running and jvb2 is stopped…

Both videobridges are working, all cameras are roughly working but there is some flickering time to time (cameras disappearing for a second or two) around And there is some insufficient bandwidth problem. I’am investigating it.

JVB 2021-09-03 14:00:20.742 WARNING: [99] [confId=6001bfaf50bbeef5 epId=4e6d81d5 gid=174717 stats_id=Mavis-03e conf_name=poc@conference.jitsi-vm.kingsch.at] EndpointMessageTransport.endpointMessage#534: Unable to find endpoint to send EndpointMessage to: 1a4134ec
JVB 2021-09-03 14:00:20.742 WARNING: [99] [confId=6001bfaf50bbeef5 epId=e409b7ca gid=174717 stats_id=Brooks-H8p conf_name=poc@conference.jitsi-vm.kingsch.at] EndpointMessageTransport.endpointMessage#534: Unable to find endpoint to send EndpointMessage to: 1a4134ec
JVB 2021-09-03 14:00:20.782 WARNING: [297] [confId=6001bfaf50bbeef5 epId=49d86cac gid=174717 stats_id=Mavis-03e conf_name=poc@conference.jitsi-vm.kingsch.at] EndpointMessageTransport.endpointMessage#534: Unable to find endpoint to send EndpointMessage to: 1a4134ec
JVB 2021-09-03 14:00:21.233 INFO: [76] [confId=6001bfaf50bbeef5 epId=c59f6fa0 gid=174717 stats_id=Kaitlin-6cX conf_name=poc@conference.jitsi-vm.kingsch.at] BandwidthAllocator.allocate#359: Endpoints were suspended due to insufficient bandwidth (bwe=731788 bps): e409b7ca,46763bca,4e6d81d5
JVB 2021-09-03 14:00:21.257 WARNING: [285] [confId=6001bfaf50bbeef5 epId=0c47236c gid=174717 stats_id=Mavis-03e conf_name=poc@conference.jitsi-vm.kingsch.at] EndpointMessageTransport.endpointMessage#534: Unable to find endpoint to send EndpointMessage to: 1a4134ec
JVB 2021-09-03 14:00:21.294 WARNING: [286] [confId=6001bfaf50bbeef5 epId=28068529 gid=174717 stats_id=Brooks-H8p conf_name=poc@conference.jitsi-vm.kingsch.at] EndpointMessageTransport.endpointMessage#534: Unable to find endpoint to send EndpointMessage to: 1a4134ec
JVB 2021-09-03 14:00:21.733 WARNING: [168] [confId=6001bfaf50bbeef5 epId=011ac62e gid=174717 stats_id=Mavis-03e conf_name=poc@conference.jitsi-vm.kingsch.at] EndpointMessageTransport.endpointMessage#534: Unable to find endpoint to send EndpointMessage to: 1a4134ec
JVB 2021-09-03 14:00:23.344 WARNING: [285] [confId=6001bfaf50bbeef5 epId=50c13dce gid=174717 stats_id=Brooks-H8p conf_name=poc@conference.jitsi-vm.kingsch.at] EndpointMessageTransport.endpointMessage#534: Unable to find endpoint to send EndpointMessage to: 1a4134ec
JVB 2021-09-03 14:00:24.582 WARNING: [96] [confId=6001bfaf50bbeef5 epId=46763bca gid=174717 stats_id=Brooks-H8p conf_name=poc@conference.jitsi-vm.kingsch.at] EndpointMessageTransport.endpointMessage#534: Unable to find endpoint to send EndpointMessage to: 1a4134ec
JVB 2021-09-03 14:00:25.329 INFO: [74] [confId=6001bfaf50bbeef5 epId=4e6d81d5 gid=174717 stats_id=Mavis-03e conf_name=poc@conference.jitsi-vm.kingsch.at] BandwidthAllocator.allocate#359: Endpoints were suspended due to insufficient bandwidth (bwe=538116 bps): 46763bca
JVB 2021-09-03 14:00:26.089 INFO: [26] HealthChecker.run#171: Performed a successful health check in PT0.000012021S. Sticky failure: false
JVB 2021-09-03 14:00:26.335 INFO: [73] [confId=6001bfaf50bbeef5 epId=4e6d81d5 gid=174717 stats_id=Mavis-03e conf_name=poc@conference.jitsi-vm.kingsch.at] BandwidthAllocator.allocate#359: Endpoints were suspended due to insufficient bandwidth (bwe=582165 bps): 46763bca
JVB 2021-09-03 14:00:26.557 INFO: [96] [confId=6001bfaf50bbeef5 epId=46aa79ae gid=174717 stats_id=Brooks-H8p conf_name=poc@conference.jitsi-vm.kingsch.at] BandwidthAllocator.allocate#359: Endpoints were suspended due to insufficient bandwidth (bwe=467254 bps): 0c47236c,e409b7ca
JVB 2021-09-03 14:00:26.576 INFO: [286] [confId=6001bfaf50bbeef5 epId=9ab2db4a gid=174717 stats_id=Mavis-03e conf_name=poc@conference.jitsi-vm.kingsch.at] BandwidthAllocator.allocate#359: Endpoints were suspended due to insufficient bandwidth (bwe=223516 bps): 46aa79ae,cd091d42,49d86cac,0c47236c,e409b7ca
JVB 2021-09-03 14:00:26.793 INFO: [168] [confId=6001bfaf50bbeef5 epId=4e6d81d5 gid=174717 stats_id=Mavis-03e conf_name=poc@conference.jitsi-vm.kingsch.at] BandwidthAllocator.allocate#359: Endpoints were suspended due to insufficient bandwidth (bwe=582165 bps): 50c13dce,46763bca
JVB 2021-09-03 14:00:27.337 INFO: [77] [confId=6001bfaf50bbeef5 epId=4e6d81d5 gid=174717 stats_id=Mavis-03e conf_name=poc@conference.jitsi-vm.kingsch.at] BandwidthAllocator.allocate#359: Endpoints were suspended due to insufficient bandwidth (bwe=629738 bps): 49d86cac,46763bca
JVB 2021-09-03 14:00:28.337 INFO: [73] [confId=6001bfaf50bbeef5 epId=4e6d81d5 gid=174717 stats_id=Mavis-03e conf_name=poc@conference.jitsi-vm.kingsch.at] BandwidthAllocator.allocate#359: Endpoints were suspended due to insufficient bandwidth (bwe=681117 bps): 50c13dce,46763bca
JVB 2021-09-03 14:00:29.340 INFO: [77] [confId=6001bfaf50bbeef5 epId=4e6d81d5 gid=174717 stats_id=Mavis-03e conf_name=poc@conference.jitsi-vm.kingsch.at] BandwidthAllocator.allocate#359: Endpoints were suspended due to insufficient bandwidth (bwe=736606 bps): 46763bca
JVB 2021-09-03 14:00:30.342 INFO: [73] [confId=6001bfaf50bbeef5 epId=4e6d81d5 gid=174717 stats_id=Mavis-03e conf_name=poc@conference.jitsi-vm.kingsch.at] BandwidthAllocator.allocate#359: Endpoints were suspended due to insufficient bandwidth (bwe=796534 bps): 46763bca