Video lags behind audio

Hi everyone.
This is an old issue that we’re experiencing again. The audio and video are not synced.
I read on other threads that the fix that worked was disabling AWS harvester (which we did).

I tried to research further, all the way down to the timestamp configurations on the bridge and chrome. Still I wasn’t able to find anything on how we can fix this.

Anyone has any experience or thoughts about this?

Some background:

  • Our bridge runs on 1Gbps up and down, guaranteed.
  • There’s no CPU load on the JVB (5% CPU usage).
  • Users are using Chrome. It seems to only happen through Chrome.
  • Users tested from multiple internet sources.
  • The issue doesn’t happen when we reroute traffic through our Coturn via TLS.

Thank you all.

We found that low end devices, when activating virtual background, cause a high load on the cpu that causes the issue.

Did anyone else encountered this issue?

Im thinkong of writing a script to limit simulcast to 360p when vbg is activated.

I hate lowering the resolution. I wish CPUs will catch up already.

Thanks for sharing this, is the problem whatever the virtual background ? I’d assume that a plain (just a color)l background could be easier on the Cpu than a photo. Also, it could be possible that the Cpu load could be lowered by switching back to VP8 (that’s another way of lowering the resoluton too)

I’m not sure - our users usually use images. I can’t tell them just use plain color, sometimes I do tell them to get better computers though lol.

If we limit the sender constraints when vbg is activated it should lower the cpu usage even when on VP9 I assume? I’m not sure there’s an option to stop VP9 and instantly switch to VP8 during a live session.

Would love your thoughts on this. Thanks

There is such logic already, but it is when an endpoint that does not support vp9 joins the conference everyone switches to vp8. So it can be added as an extra condition …

1 Like

Truth be told, I’m not sure that using a plain color background would be significant so forget it.

Yes, lowering the sent resolution is a good way to lessen the load.

I forgot to say that there are several features that the browser can and will do without saying much but with great impact;
The traditional advice is to disable speech indicator, but there are also new features involving lots of image handling such as face centering, expression detection, what else. If you have low power computers as client, it’s necessary to follow closely the updates to Jitsi-meet to ensure that these new CPU-hungry features are disabled.

We are also experiencing high cpu load with virtual backgrounds. So bad that we removed the toolbar button for the feature. VP9 and high resolution are more important for us. While not cutting edge, our devices are not low end.

Which endpoint would that be? Chrome, android and ios support VP9. Are you referring to other browsers?

Did you use the Tensor js library? We had the exact same issue when we tried applying VBG before Jitsi added the wasm module. When we switched to wasm everything is much much better but for low-end devices.

Totally agree on the video quality. I can’t stand low-resolution.

Are these features on right now? Where can I find them - I’ll hunt’em down and disable.

Thanks for the tip.

you can find (almost) all configurable features under the reference file:

/usr/share/jitsi-meet-web-config/config.js

this file holds the current options for the installed software version.

Your config.js under /etc/jitsi/meet is never changed when upgrading Jitsi-meet, but default options are used nonetheless and can be found in the reference config.js. Be aware the most of the time, the displayed default value for a new option is the true default, but problems can happen and it’s not unthinkable that sometimes a mistake is made, the displayed default is false, but the default in the code is true. So it’s better to set it explicitly to false when one don’t want an option.
Maybe try to set

disableAudioLevels: true
faceLandmarks → everything set to false
disableAddingBackgroundImages: true → will stop users to add heavy images

The only cure is obviously to get a more powerful CPU.

Hmm I’ll check. Haven’t touched this file in a while now. Thanks

strictly speaking, ‘touching’ (changing) this file is useless and a bad idea, but it should be read when upgrading Jitsi-meet version, searching for new options.

1 Like

Just had this video lagging again this time on a desktop with i7 10th gen and 64GB ram.
I went into the windows task manager and raised the priority of Chrome’s processes - seemed the rectify the issue.

This user did not use VBG :confused: but does have some other processes open. CPU was at 40%.

I wonder if Chrome is being throttled after a while being opened in the background.

I wonder how the user realized that the video was lagging if Chrome was in the background.

Whatever, it’s well known that the process in the foreground (focused) has a higher priority under Windows.
About the 64 G Ram, it’s good to remember that some applications can be extremely hungry, Chrome being a case especially when applications are leaking memory - Jitsi-meet has been pointed at recently in a Github issue IIRC.
Also system configuration problems (exchange file), antivirus, all the usual suspects too.

Yeah I agree. We’re trying to find a process to insert in the JS to keep Chrome as a focused process.

The person is one of my dev mates. He had several jitsi tabs open as we were testing something and then myself and another noticed the mismatching audio video.

We’ll keep monitor and update this thread.

Side note
In the config.js there’s a enableLipSync flag set to false.
Is this the right config? Why would it be off? It’s kinda counter-intuitive.

here all the documentation I have found on lip sync (outside of references to ‘our nasty hack’:slight_smile:

/**
 * Here we're doing audio + video stream merging in order to take advantage
 * of WebRTC's lip-sync feature.
 *
 * In Jitsi-meet we obtain separate audio and video streams and send their SSRC
 * description to Jicofo which then propagates that to other conference
 * participants. Because they are separate stream, the WebRTC stack wil not try
 * to synchronize audio and video streams which will often result in bad user
 * experience.
 *
 * We are not merging the streams on the client which would result in all
 * clients receiving them merged, because of few issues:
 * 1. Chrome will not play audio for such merged stream if the user has video
 *    muted: https://bugs.chromium.org/p/chromium/issues/detail?id=403710
 *    So we will only merge the stream if the owner has his video unmuted at
 *    the time when it's being advertised to other participant. This is subject
 *    to some race conditions and will not always work. That's why this hack is
 *    not enabled by default.
 * 2. We need separate streams for doing video mute and screen streaming. When
 *    video is being muted it's stream is being removed and added on unmute. And
 *    when we're doing screen streaming the video stream is replaced with the
 *    screen one. Now when the stream is merged audio and video are becoming
 *    tracks of the same stream and on the client side we need to be able to see
 *    when they change. Unfortunately "track added" and "track removed" events
 *    are only supported by Chrome. Firefox will display the video fine after
 *    track change, but there will be no event, Temasys has no events and track
 *    changes are not supported.

Note that ‘Temasys’ is something lost is the mist of times. If you enable this, you may go where dragon lives. But fortuna juvat audaces like they used to say while burying the pioneers.

We just ran a test on meet.ji.si. It appears there’s a correction Jitsi applied to re-sync the audio and video. The video seems to “jump” back into synchronization with the audio.
@damencho @gpatel-fr do you know where this correction is happening? Is it on the JVB or the endpoint device?

Thanks.

See the short recorded sample below from my Dropbox:
Side note - the people shown in the video are NOT professional actors :joy:

1 Like

I think this is the webrtc jitter buffer in the client.

Hmm how is it being activated? Is there anything in the Jitsi library that we’re missing?

Clearly something is activated on meet.jit.si and is off on ours…

Thanks