Those values has nothing to do with the bandwidth estimation done on the jvb side.
Thanks @damencho that’s good to know. Do you know if there is anything else cached in the client for bandwidth estimation? Or this could be more a browser-related bug?
I don’t see the browser affecting that … its on the bridge side and maybe monitor the bridge network when this happens … do you see cpu usage there when this happen, what is the bandwidth used at the moment of this, are there any packets dropped by the kernel … instrument those to have stats and see what those show when you see the problem …
Is it happening for all participants in the call is another good point …
Thanks @damencho for the insight, we will monitor bridge network / cpu usage and packets dropped.
We observed that this issue happens only to some participants in the conference. For example I have two computers next to each other connected to the same network. One had the issue consistently and the other one did not have the issue. When we observe the issue, the
bitrateController.bweBps is very low for the affected participant IDs (seems to get stuck) while the
Bandwidth Estimation.currentEstimate is higher (more details in my previous comment in this thread).
Thank you for your help in debugging this issue, I really appreciate it.
We are still experiencing freezing and low quality issues when we have the BWE enabled.
We monitored the network and CPU usage in our videobridges, they are below the limits and they are not struggling. We also checked the dropped packets in the videobridge network interface, they are not increasing while we experiencing this issue.
When we disable BWE, all participants get excellent video quality and not video freezing. This confirms that BWE is related to the issue that we have. Based on the debug queries that we did (in one of my previous comments) we can confirm that even if the
Bandwidth Estimation.currentEstimate is high,
bweBps stays low. For some reason it seems that if a client connection struggles briefly,
bweBps will stay stuck in a low value even if the client connection bandwidth improves.
i seem to be hitting the same issue ,
Any update on this issue? this is really causing problems…
many error logs with:
BandwidthAllocator.allocate#359: Endpoints were suspended due to insufficient bandwidth (bwe=107230 bps):
and people are unable to work and share screen with presentations and docs…
I did not hear any news. What I know for sure is that with BWE disabled in our videobridge config, we don’t have this issue. For that reason we are running with BWE disabled in production, and limiting the video feeds each client receives using the new video constraints, which helps to not overload bandwidth. Only in connections below 3Mbps we see issues. We would like to enable BWE again in the future once this issue is fixed.
we are stuck with old UI and cannot upgrade till this issue is resolved…
We also have this problem after we upgraded from jvb version 2.1-416-g2f43d1b4-1 to jvb version 2.1-538-g062e9f56-1.
Could it be that the allocation strategy changed between those versions? It might give an indication where to look for a solution.
Does simulcast still work if we trustBwe=false? We always want a video to be delivered to the viewer. Quality loss is acceptable but entirey disabling is not.
Our live system is affected be the issue. Any help would be highly appreciated.
@ip972, please stop flooding.
You are not the only one with that kind of issue but your messages are not helping (at all).
If you want this to be corrected so badly, consider hiring a developer to fix this for you.
Same issue here. We eventually need to disable BWE.
I updated 2 days ago.
Here is the output of ~$ tail /var/log/apt/history.log
Start-Date: 2021-10-04 18:14:54
Commandline: apt upgrade
Requested-By: Alfred (1000)
cloud-init:amd64 (21.2-3-g899bfaa9-0ubuntu2~18.04.1, 21.3-1-g6803368d-0ubuntu1~18.04.3),
php7.2-common:amd64 (7.2.24-0ubuntu0.18.04.8, 7.2.24-0ubuntu0.18.04.9),
php7.2-cli:amd64 (7.2.24-0ubuntu0.18.04.8, 7.2.24-0ubuntu0.18.04.9),
php7.2-fpm:amd64 (7.2.24-0ubuntu0.18.04.8, 7.2.24-0ubuntu0.18.04.9),
libsystemd0:amd64 (237-3ubuntu10.50, 237-3ubuntu10.52),
php7.2-mysql:amd64 (7.2.24-0ubuntu0.18.04.8, 7.2.24-0ubuntu0.18.04.9),
python-apt-common:amd64 (1.6.5ubuntu0.6, 1.6.5ubuntu0.7),
udev:amd64 (237-3ubuntu10.50, 237-3ubuntu10.52),
php7.2-json:amd64 (7.2.24-0ubuntu0.18.04.8, 7.2.24-0ubuntu0.18.04.9),
php7.2-opcache:amd64 (7.2.24-0ubuntu0.18.04.8, 7.2.24-0ubuntu0.18.04.9),
php7.2-curl:amd64 (7.2.24-0ubuntu0.18.04.8, 7.2.24-0ubuntu0.18.04.9),
libudev1:amd64 (237-3ubuntu10.50, 237-3ubuntu10.52),
php7.2-xml:amd64 (7.2.24-0ubuntu0.18.04.8, 7.2.24-0ubuntu0.18.04.9),
python3-distupgrade:amd64 (1:18.04.44, 1:18.04.45),
ubuntu-release-upgrader-core:amd64 (1:18.04.44, 1:18.04.45),
systemd-sysv:amd64 (237-3ubuntu10.50, 237-3ubuntu10.52),
php7.2-mbstring:amd64 (7.2.24-0ubuntu0.18.04.8, 7.2.24-0ubuntu0.18.04.9),
php7.2-readline:amd64 (7.2.24-0ubuntu0.18.04.8, 7.2.24-0ubuntu0.18.04.9),
php7.2-gd:amd64 (7.2.24-0ubuntu0.18.04.8, 7.2.24-0ubuntu0.18.04.9),
libpam-systemd:amd64 (237-3ubuntu10.50, 237-3ubuntu10.52),
golang-docker-credential-helpers:amd64 (0.5.0-2, 0.5.0-2ubuntu0.1),
systemd:amd64 (237-3ubuntu10.50, 237-3ubuntu10.52),
libnss-systemd:amd64 (237-3ubuntu10.50, 237-3ubuntu10.52),
libgnutls30:amd64 (3.5.18-1ubuntu1.4, 3.5.18-1ubuntu1.5),
python3-apt:amd64 (1.6.5ubuntu0.6, 1.6.5ubuntu0.7),
base-files:amd64 (10.1ubuntu2.10, 10.1ubuntu2.11)
End-Date: 2021-10-04 18:15:22
So, can I conclude that one or more of these has caused the problem?
What do we all have in common?
Do we all use Ubuntu?
Do we all use php7.2-fpm?
Do we all use nginx?
I hope this will help us to drill down to the package that caused this, can anyone help me with a hint to quickly revert these packages to the previous version?
Thanks in advance…
I will try this. With latest stable having same issue with only 13 participants and only one client having camera on, jvb log showed person with camera on suspended but video continue to deliver but Audio was lost.
Not sure why Audio was suspended and video worked fine.
Disabling BWE is not an ideal solution. Have you tried upgrading to the latest unstable JVB?
So I can confirm, all jitsi packages were unchanged.
Edit: I just downgraded php7.2-xxx
Let you know if the problem is solved in 6 hours.
My apt command was:
sudo apt-get install php7.2-common:amd64=7.2.24-0ubuntu0.18.04.8 php7.2-cli:amd64=7.2.24-0ubuntu0.18.04.8 php7.2-opcache:amd64=7.2.24-0ubuntu0.18.04.8 php7.2-json:amd64=7.2.24-0ubuntu0.18.04.8 php7.2-readline:amd64=7.2.24-0ubuntu0.18.04.8 php7.2-fpm=7.2.24-0ubuntu0.18.04.8 php7.2-curl=7.2.24-0ubuntu0.18.04.8 php7.2-gd=7.2.24-0ubuntu0.18.04.8 php7.2-mbstring=7.2.24-0ubuntu0.18.04.8 php7.2-mysql=7.2.24-0ubuntu0.18.04.8 php7.2-xml=7.2.24-0ubuntu0.18.04.8
Curious: what version of Chrome are you using? Do you have an origin trial for Plan B?
After also downgrading all others in the list, we had no audio problems in the meeting. Also logfile showed no “bandwidth problems”
To be compleet, I list the other commands I used:
sudo apt install libsystemd0:amd64=237-3ubuntu10.50 udev=237-3ubuntu10.50 libudev1=237-3ubuntu10.50 systemd-sysv=237-3ubuntu10.50 libpam-systemd=237-3ubuntu10.50 libnss-systemd=237-3ubuntu10.50 systemd=237-3ubuntu10.50
sudo apt install cloud-init=18.2-14-g6d48d265-0ubuntu1
sudo apt install libgnutls30=3.5.18-1ubuntu1.3
sudo apt install base-files=10.1ubuntu2.2
We still have this issue and was able to locate that the problem is the BWE. Disabled everything is ok, enabled we experience the issues described above. It was not related to the Chrome version. And we have activated Plan B. Did anybody from the developers pushed anything in the Bridge module that would fix this issue? Any help would be appreciated, since running a conference without BWE is not OK at all.