BWE Sources were suspended due to insufficient bandwidth

Hello, we are having problems with our JVB on our servers lately.

On some clients videos in tile view and filmstrip are switched off or flicker (status=inactive) because of BWE. With a reload or a click on the tile the video reappears immediately.
Especially often this happens on clients with a weak CPU or mobile internet connection, but it seems that it then spreads to clients with a very good CPU and internet connection.

Occurs with Chrome, Edge and Safari but never with Firefox.

Noticeably often triggered by screensharing, we see an improvement since we hide screen sharing in tile view.
Also if any client repeatedly switches tile view quickly it can trigger this on other clients, it also leads to a significant image degradation of all videos on all clients.

With trust-bwe=false the problem does not occur and all images are visible, but of course only as long as there is no client with a truly bad connection.

LOG:

JVB 2022-08-15 07:27:29.974 INFO: [119] [confId=bddfc473f0be17d6 conf_name=confname epId=ab49d7c8 stats_id=Bobbie-tok] BandwidthAllocator.allocate2#452: Sources were suspended due to insufficient bandwidth (bwe=182632 bps): 0bec8935-v0
JVB 2022-08-15 07:27:31.757 INFO: [113] [confId=bddfc473f0be17d6 conf_name=confname epId=13da794b stats_id=Ava-x6P] BandwidthAllocator.allocate2#452: Sources were suspended due to insufficient bandwidth (bwe=131753 bps): 0bec8935-v0
JVB 2022-08-15 07:27:32.014 INFO: [115] [confId=bddfc473f0be17d6 conf_name=confname epId=ab49d7c8 stats_id=Bobbie-tok] BandwidthAllocator.allocate2#452: Sources were suspended due to insufficient bandwidth (bwe=215102 bps): 0bec8935-v0
JVB 2022-08-15 07:27:33.775 INFO: [127] [confId=bddfc473f0be17d6 conf_name=confname epId=13da794b stats_id=Ava-x6P] BandwidthAllocator.allocate2#452: Sources were suspended due to insufficient bandwidth (bwe=155756 bps): 0bec8935-v0
JVB 2022-08-15 07:27:35.895 INFO: [120] [confId=bddfc473f0be17d6 conf_name=confname epId=13da794b stats_id=Ava-x6P] BandwidthAllocator.allocate2#452: Sources were suspended due to insufficient bandwidth (bwe=183753 bps): 0bec8935-v0
JVB 2022-08-15 07:27:43.878 INFO: [114] [confId=bddfc473f0be17d6 conf_name=confname epId=0bec8935 stats_id=Burley-68m] BandwidthAllocator.allocate2#452: Sources were suspended due to insufficient bandwidth (bwe=118887 bps): 10e912c5-v0
JVB 2022-08-15 07:27:52.604 INFO: [112] [confId=bddfc473f0be17d6 conf_name=confname epId=ab49d7c8 stats_id=Bobbie-tok] BandwidthAllocator.allocate2#452: Sources were suspended due to insufficient bandwidth (bwe=153592 bps): 10e912c5-v0
JVB 2022-08-15 07:27:52.805 INFO: [125] [confId=bddfc473f0be17d6 conf_name=confname epId=ab49d7c8 stats_id=Bobbie-tok] BandwidthAllocator.allocate2#452: Sources were suspended due to insufficient bandwidth (bwe=118392 bps): 10e912c5-v0,0bec8935-v0

Server:

Ubuntu 22.04.1 LTS
java-18-openjdk-amd64
jitsi-videobridge2 2.2-18-gade06bf8-1
CPU: Intel(R) Core(TM) i9-10940X CPU @ 3.30GHz
RAM: 32GB
Ethernet controller: Intel Corporation Ethernet Connection (2) I219-V
public IP, no NAT
over 500Mbit up/down

jvb.conf

bwe-change-threshold=0.15
thumbnail-max-height-px=180
allow-oversend-onstage=true
max-oversend-bitrate=500 kbps
trust-bwe=true
padding-period=15ms
max-time-between-calculations = 5 seconds

Any ideas how we can improve this, or better search for the reason (e.g. monitor the bwe of a client permanently)?

As the error message suggests, there’s insufficient bandwidth. The definitive fix is to improve the network conditions of the connecting clients or lower resource demands by switching codecs and constraining resolution e.t.c…

It happens much more often than before and not with Firefox on the same machines.
It also happens with 6% Cpu and 50Mbit DSL or even on computers with 500mbit fiber lines, even on clients in the same gigabit switch as the server.

We only suspect that computers with low bandwidth somehow trigger it on these well connected ones.

The bwe values are most likely incorrectly determined and do not match the speedtests in any way. It works without packet loss if we set trust-bwe=false

speedtest is not something you can use to check for the bandwidth of a udp link.
For example, we have seen a bad access point in the network bringing down the bandwidth to very low values due to faulty scanning and basically pausing the network for a short period of time, every minute or so. A thing that will affect the bandwidth estimations and you will not notice that when doing a speedtest.

Yes, that’s right, we’ve had problems like that before (other computers in the network had Windows updates, and it didn’t show up in the speedtest).
However, this affected only individual clients or clients in the same network.
Now it often affects the majority of clients at the same time, over at least 4 different ISPs.
I suspect that somehow the server or its network connection must be at least part of the problem. However, we also had the problem on a dedicated server of a hosting provider.

You can setup some iperf testing on the server and some location and see how it performs with constant udp connections

Video streaming is very sensitive to packet loss and unlike other kind of activities, depends on throughput. Even a millisecond drop in throughput can be catastrophic for video relay. BWE remains the toughest nut to crack when it comes to WebRTC - this is not a Jitsi thing. The question that often needs to be answered is: what to do when packets are lost?

I believe we had an extensive discussion on this topic in one of the threads. If I find it, I’ll link it here so you can go through it. It will help you to better understand why you’re experiencing what you’re experiencing.

I will do that

Thank you, that would be helpful.

I also thought of some other things that might help narrow it down. We will probably test if it occurs in different conferences on the same server at the same time and if it occurs on the clients at the same time if they are in 2 conferences on different servers.