8x8 good network, but p2p switches back to LD quality

Good morning,

We are busy switching from our own hosted Jitsi instances to 8x8 and the first experiences are great, only we do not get any feedback on our technical requests. Which is strange because it is a paid service, but let’s try on this forum. We are using Jitsi for an eHealth platform, always one-on-one, preferably p2p. We notice that often the quality of the video goes back to LD, while the network quality is fine:

Sometimes the quality is SD, or when working locally it is HD for a while, but it changes over time and most of the time it is between LD and SD. It is strange, because the up-/download speed is sufficient and no package-loss. Would enabling the e2eping matter? We read that this is only for indication.

We have hundreds of therapy sessions a week, always p2p (or psp-turn), so this should be solved first before we can switch to 8x8.

Thanks a lot!
Marc

Hi @marctemp1, when you are talking about SD/LD and HD what do you mean? Are you looking at the color of the indicators or?
Cause on the screenshot the upload and download seem fine and the resolution is HD (1280x72).

Hi @damencho , thanks for the swift reply. With LD/SD/HD I ment the indication in the top bar next to the session timer, I assume this corresponds with 180p/360p/720p of the incoming video.

Sorry for the confusion, I tried to ask two questions at once:

Q1: Why is the connection indicator in my first post red, while all connection parameters seem to be ok? High bitrates, no package loss and (after some testing with e2eping enabled today) a low E2E-RTT. Still the connection remains poor. How is the poor quality indication defined in the API?

Q2: In our (one-on-one always, doctor/client) calls we deal many times with bad quality, while the internet connection on both sides seems to be ok and no high CPU load. This is annoying and one of the reasons we want to switch to 8x8. (other reasons are the higher availability and zero maintenance cost)

We did some testing with two of our client’s PC’s via our own Jitsi server, 8x8.vc and meet.jit.si and they all seem to have the same behavior. The CPU usage during Jitsi meeting in Chrome of this client is about 25% and the internet upload speed is quite stable 2MB/s:

Speedtest F

It is surprising that the outgoing bitrate is very low (65Kbps). Same for the connection and video quality (LD):

Low upload

We did this test on meet.jit.si, but 8x8.vc and our own server act the same. The outgoing bitrate is going up and down and after a while we noticed that it seems to be linked to the availableOutgoingBitrate measured by Chrome:

Outgoing bitrate Jitsi

However, when we close Jitsi and open Google Meet, the availableOutgoingBitrate is higher and is much more stable:

Outgoing bitrate Google

The question is: why is the outgoing bitrate much lower then we have available (causing low quality video for about 50% of our clients) and do you have any advice how to improve this?

NB, we only use Jitsi for one-on-one calls (p2p or p2p-turn), so it doesn’t matter at all if we adjust any settings which have a negative impact on calls for more than 2 participants (like multicasting or whatever).

Thanks a lot, we are really struggling with this for 1,5 year now.

I’m not sure, we will check that. You should not see red in this situation.

For the second question, can you do the same test but disable the p2p in the call, so even for 2 participants to make sure the call goes through the bridge.

Do you see any difference?

Hi @damencho,

Call via the JVB (p2p.enable=false) doesn’t change the behavior of the quality (and Chrome’s AvailableOutgoingBitrate), only difference between the bridge and p2p is the indication in the Jitsi API. I have tested today with:

  • PC1 on a fast internet connection (up/down 1Gbit/s) and
  • PC2 on wifi a bit further from the router, but still a stable connection (all wifi bar’s full on Windows 10, up ~90 Mbit/s and down ~40Mbit/s):

Speedtest PC2

Via P2P, the local indication on PC2 is about equal to the remote indication of PC2 seen from PC1:
Jitsi via p2p

Via JVB, same behavior wrt quality, the same local indication on PC2, but the remote indication of PC2 seen from PC1 is now suddenly much better:

This maybe caused by the JVB, telling PC1 that PC2 has a good connection, while it is actually poor. This doesn’t really bother us, it might have something to do with the false indication we noted before (Question 1 in my previous post)

The real problem (for us) and our theory on this:
What bothers us is the poor quality of many users in our one-on-one calls and I believe that this is not particularly caused by any p2p or one-on-one call related issue in our Jitsi config, but because of our users notice this much more because our API is always used in full screen mode. In a group calls with (much) more than 2 participants the majority of the users are in tileview and the poor connections are not really noticed. We have always two participants looking to each other for at least an hour and always in full screen view. In that case a 180p video will be noticed straight away and I assume that’s why we get so many complaints about the quality of Jitsi. I hope this theory makes sense, because I tried to prove it with some additional tests today. Sorry for the long texts, but we hope you can help us out on this.

I did some further testing today between Jitsi/8x8/our-own-server and Google Meet and the problem seems not to be related to Jitsi, but to the Chrome WebRTC AvailableOutgoingBitrate, which absolutely doesn’t represent the actual available upload bandwidth. It is much lower than the actual availability, especially when users have a slightly lower bandwidth, like our PC2 on wifi. Btw, it doesn’t matter if we call with Jitsi or with Google Meet in Chrome, the problem seems to be in the browser/WebRTC, not in the VC-software. Here’s for example our WebRTC log using Jitsi via 8x8 on PC2:
8x8

Seems to be a bit unstable, right? It seems that when the wifi is ok (or when using a cable connection), Chrome gives a stable availableOutgoingBitrate of 4Mbit/s, but when the wifi quality slightly decreases (but still a full wifi bar in Windows 10), this unstable availableOutgoingBitrate phenomena occurs. The result is that the Jitsi video from PC2 is continiously switching between SD (360p) and LD (180p). I thought that this was caused by the wifi connection of PC2, which might be very unstable and slow, but speedtest.net showed a stable 30-40 Mbit/s upload every time we test it. Same story as we hear from our users, telling us that their internet is fine (Zoom works very well, blah, blah.) but Jitsi quality on our platform is always poor.

We did another test with Google Meet and after a couple of minutes we noticed that the availableOutgoingBitrate went down to less than 100Kbit/s, like with Jitsi before. Immediately we started speedtest.net on PC2 and surprisingly it showed a stable 20Mbit/s, which we could also see in our network monitor on the right. Strange thing is that Chrome’s availableOutgoingBitrate remained low at the exact same time:

I have also added a screenshot after the test, where you can see that the availableOutgoingBitrate is continiously very low (causing Jitsi to send a poor 180p) while in the same time the upload speed is proven to be 20Mbit/s:
WebRTC after speedtest

I did another test with Jitsi (8x8) and Google Meet in parallel and this showed that both VC-tools struggle with availableOutgoingBitrate on non-optimal wifi connections. For both Jitsi as Google Meet the quality of the outgoing video quality went exactly up and down with the availableOutgoingBitrate and not in parallel! From time to time Jitsi had a much better quality then Google Meet, and vice versa. Note that during these 5 minutes recording I ran speedtest.net several times, resulting in an upload bandwidth of 20-30Mbit/s each time. The wifi/internet upload bandwidth is perfectly capable to send 720p, so why does PC2 send 180p quality video?!

I repeated this on Chrome/Edge 94 and Chrome Beta 95 multiple times, all with comparable behavior, from time to time very low availableOutgoingBitrates while the actual upload speed is 200 times better. Question rises: how does Chrome measure the availableOutgoingBitrate???

Assumed conclusion and question:
Chrome’s WebRTC availableOutgoingBitrate measurement is far lower than the real available upload bandwidth when a wifi connection is less than perfect. Jitsi (and Google Meet) use this availableOutgoingBitrate to define the quality of the outgoing video, which results in unnecessary low quality.
Final question: When we exclusively work with one-on-one calls, would it be possible to improve this quality for Jitsi? For example by enabling layerSuspension, disabling multiCasting, changing the bitrates or the newBandwidthAllocationStrategy maybe?

Thanks for thorough analysis! We’ll need to dig deeper into this.

That is mostly what the P2P mode does. It sidesteps the bridge and goes directly (even if it ends up going through TURN) so there is no simulcast, nor custom bitrate allocation strategy, browsers negotiate between them.

Hi Saghul,

Thanks for the reply. We really appreciate the help from the both of you, as this was actually a question for 8x8.

After discussing the test results with the team this morning, we concluded the same. This is not a Jitsi/8x8 problem, but a Chromium/WebRTC problem. I hope though that we can fine-tune Jitsi specifically for our “always full-screen and one-on-one” usage, which will solve it for us. Maybe we can do some further testing with p2p disabled and some of these settings. We await your response and if we can test anything else, please let me know.

Kind regards,
Marc

I work for 8x8 myself, so this well within my scope :slight_smile:

Any chance we can get a webrtc-internals dump from you? (check in chrome://webrtc-internals) Also please check “Enable diagnostic packet and event recording”. Thank you!

Just so we can try improve things / tap some shoulders, can you please share the cases you opened?