Audio/video quality troubleshooting, tips and advice

Hi everyone,

I know there are plenty of threads with the term “quality” but I am hoping to collaborate with the community to build shared knowledge on troubleshooting.

We’ve been receiving intermittent complaints about video/audio quality on reasonably sized bridges (e.g. 3-8 people) where the users are falling back to Google Meet and not having the same issues.

Inevitably some users of any conferencing platform will have poor internet connections or client side issues that will impact quality but it seems other products are able to compensate for this (or at least thats the perception).

In the last couple of weeks we’ve deployed rtcstats and Elastic APM to help us understand what’s happening. I am working through mountains of data to identify insights, mainly focused on the following at the moment:

  1. Making sense of huge amounts of RTCSTATS data, utilizing a lot of fippo’s great work.
  2. Looking at network factors, including QoS, jitter and latency how they influence rtc.
  3. Comparing configuration and packet captures between conferences on how our implementation and

Does anyone have any tips or tricks, anything you can share with myself and the community that will help on this journey?

(I plan to share back for what I learn along this long and dark journey) :wink:

Many thanks!

1 Like

so a brief update and hoping to inspire others to chime in :wink:

Put a lot of work into understanding the rtcstats data and parsing it into forms that provide some insight. It’s clear that we’re seeing a high degree of changes in resolution and frame rate on a number of connections but there’s nothing apparent in the ICE flow that would support the why its happening.

Similarly there’s no indication of excessive packet loss, latency or jitter (although there are always end user specific exceptions here) but nothing platform wide that would suggest issues.

The focus for me now is the richness in the browser logs and it’s clear there’s at least a couple of areas for further exploration:

  1. It appears that the transition from P2P to JVB (and back) is not always happening cleanly:

2020-10-06T15:52:56.389Z [modules/RTC/TraceablePeerConnection.js] <A._remoteTrackAdded>: TPC[3,p2p:false] remote track added: c7ca1b9f-b8ab-4d8f-a101-5f24a6ea462b-2 video
2020-10-06T15:52:56.390Z [modules/RTC/TraceablePeerConnection.js] <A._remoteTrackAdded>: TPC[3,p2p:false] associated ssrc b592d152 522690395
2020-10-06T15:52:56.390Z [JitsiConference.js] <re.onRemoteTrackAdded>: Trying to add remote JVB track, when in P2P - IGNORED

  1. The client appears to be triggering downgrades in resolution, either via “setReceiverVideoConstraint” or other events, for example what appear to be unfounded “mute” actions.

Any thoughts, tips or shared experience welcome!

Very good topic! @saghul, who in the team would be best to help out on this?

Definitely curious about this! I posted something similar a while back:

@Jayamini_Rodrigo or @damencho any suggestions on these errors that cause bad call qualities without explanation?

We had discovered some quality issues when there is screen sharing in the meeting, next week there will be stable update, fixes are already in unstable repo, you can update from there to check whether it changes it for you.
There are few more fixes around quality that will also go in.

@damencho thanks for the update, appreciated (especially on a weekend).

Would you be able to share a little detail that would help us validate if it is the issue we have ahead of the new release, direct message would be great if thats better.

Well we had seen sessions with screensharing that cause some packetloss report, there was a oversanding bug fixed in the current stable release, I think. And also some buffers in the bridge allocating more memory causing higher garbage collecting times that cause audio delay, which is in fact caused by the new chrome 86.

1 Like

Thanks @damencho we’ve certainly experienced the memory issue on the JVB’s and resulting lag in audio compared to video. We’ve bumped up our heap and scheduled reloads to manage it in the meantime. Looking forward to the other quality fixes during the next release!

1 Like

We’ve also been seeing high memory and audio lag. Thanks for confirming it’s not just us!

Please can you expand on what settings changes you’ve made to counter this?
I’m seeing memory usage jump to max in a couple of hours so a regular reload isn’t quite enough.

Interestingly we’re only seeing this on our “JVB only” servers and not on the “managing” server. Perhaps this is just how things are being load balanced.

Hi @dogsbody-rob , there’s a number of posts on the forums if you search for “jvb memory and chrome 86”, although I think this one is probably is best link as it points you to fix:

1 Like