Participants get out of sync with other participants in same room, have to refresh browser (rejoin room) as workaround. What is proper fix?

Greetings. We keep having a problems with, over time, users getting out of sync with seeing and/or hearing other users. This happens with even just a few users. most recent tests were just 5-6 users in one room.

A workaround is to have everyone leave the room and come back every 15 to 30 minutes or so, bu tthat is not a fix. Hoping for suggestions to provide an actual fix. Thanks kindly!

Scenario:
Users join room.
Everyone can hear and see everyone else. We have meeting, over time User 1 will say they can no longer see (or hear) user 2, while while users 2 through 5 say they can still see and hear everyone else, only user 1 can’t see/hear user 2.
If user 1 drops and comes back, sometimes that fixes it, and other times users 2-5 all have to drop and come back, or everyone has to drop out and come back to make it so everyone sees and hears each other again.
This out of room sync issues (what is the jitsi technical term for his?) can occur at any time but typically after 5-20 minutes of meeting time. Sometimes after 30 minutes.
This is with all users in the USA (most on the East coast, a few on the West coast)at AWS in Virginia.
We definitely see this issue with anyone not using Chrome 95 on Windows. But even when we forced everyone participating to use Windows with Chrome version 95, we still see this issue in every meeting.
We are not seeing any bandwidth bwe messages. If there are specific messages I should be looking for in any specific log, please let me know.

Here is the setup information
Jitsi Videobridge2 Version 2.1-570-gb802be83-1
Jitsi (meet web) Version 5415
JVB instance c5n.4xlarge
JMS c5n.9xlarge.
Servers running Ubuntu 18.04.6 LTS
All running Chrome Version: 95.0.4638.54

Additional info:
On JMS instance:
jitsi-meet-web/stable,now 1.0.5415-1 all [installed]
WebRTC JavaScript video conferences

jitsi-meet-web-config/stable,now 1.0.5415-1 all [installed]
Configuration for web serving of Jitsi Meet

jitsi-meet-tokens/stable,now 1.0.5415-1 all [installed]
Prosody token authentication plugin for Jitsi Meet

jitsi-meet-prosody/stable,now 1.0.5415-1 all [installed,automatic]
Prosody configuration for Jitsi Meet

jicofo/stable,now 1.0-813-1 all [installed]
JItsi Meet COnference FOcus

On JVB Instance:
jitsi-videobridge2/stable,now 2.1-570-gb802be83-1 all [installed]
WebRTC compatible Selective Forwarding Unit (SFU)

Appreciate any suggestions!

Some suggestions:

  • repeat the same test on ‘meet.jit.si’. If you can reproduce the same issue, most probably this is a client side issue. If not then it’s server side issue

  • try a newer distro. Ubuntu 20.04 or Debian 11 Bullseye

  • try a standalone Jitsi without an additional JVB. Don’t customize anything at first

Going to do some more experimenting, but found changing the following settings so far improved the out of sync issue, until we enabled the waiting room. Once we enabled waiting room people started to fall out of sync. I haven’t confirmed this as causal yet, could just have been a cooincidence, but we we made it 55 minutes into meeting without anyone losing sync, then the moment we enabled waiting room, all of a sudden it fell apart, an then everyone needed to leave to other meetings before we could experiment further.
I explicitly forced the AWS recognition (it didn’t do anything otherwise):

Here are settings changes that seemed to help (I think, not yet certain), on the separate JVB instance:
/etc/jitsi/videobridge/sip-communicator.properties

modified by hawke 20211028 20:39

#Default: false
#Explicitly disables the AWS mapping harvester. By default the harvester is enabled if ice4j detects that it is running in the AWS network.
org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=false

modified by hawke 20211028 20:51

#Default: false
#Force the use of the AWS mapping harvester, even if ice4j did not detect that it is running in the AWS network.
org.ice4j.ice.harvest.FORCE_AWS_HARVESTER=true

Remmed out the NAT_HARVESTER_LOCAL and public entries completely.
This seemed to make a significant difference, again until we enabled waiting room.
Will be testing net week if the waiting room enabling was a cause or just correlative coincidence.

Will report back here later next week when I have clearer data.

I haven’t ruled out the other suggestions, this is just what was already in the queue, I’ll check into the other options later if needed. Thanks to everyone for the suggestions.
Cheers!

Did you get any further with debugging this yet?

I’ve had a number of user reports describing exactly the same symptoms using a standalone server as well as another one that has additional videobridges. Both instances use the latest stable, i.e. the same versions listed above. Most concerning is the fact that audio is affected. Audio used to be rock solid and while occasional video problems are less than ideal, reliable audio is a must.

Our jitsi instances are used for small group meetings that rarely exceed 10 participants. Lobby/waiting room does not seem to matter. The problems also appears in meetings where waiting room is not used. It doesn’t seem to be a problem of bandwidth estimation either. Disabling BWE did not help. Neither did using only chromium/chrome as browser.

Sometimes the audio problem started after screen sharing. At least once it affected the person who had shared their screen and was heard perfectly by everyone else while doing so. The problem may also appear more frequently in meetings where users turn off their microphone whenever they are not speaking. None of this is reproducible however.

Any suggestions?