Frequent video "freezing" during conferences

[jrapdx]

Since mid-December we’ve noticed problems during meetings, biggest is frequent video “freezing”. The issue occurs unevenly across participant locations, connections to one particular site being especially troublesome for unknown reasons. Most meetings are limited to 2 parties.

My Jitsi server runs on a modest VPS, 4 cpu, 4GB ram. Typically it shows <10% cpu utilization, <25/100GB ram usage. Since I’m the only admin/user, the system is lightly loaded.

I’ve been trying to sort out what’s going on. Various tests have been inconclusive. Currently the server is monitored with netdata (GitHub - netdata/netdata: Real-time performance monitoring, done right! https://www.netdata.cloud). This is a screenshot of a netdata “chart” showing ipv4 udp packets during a recent videoconference:

The obvious instances of sudden loss in udp reception coincided exactly with “frozen” video.

To follow up, iperf was used to generate a UDP load in isochronous mode (emulates video traffic):

iperf -c thinairarts.com -p 1194 -u -e -t 600 -i 10 --isochronous --full-duplex
------------------------------------------------------------
Client connecting to thinairarts.com, UDP port 1194 with pid 3408 (1 flows)
Isochronous: 60.00 frames/sec mean=20.0 Mbit/s, stddev=0.000 bit/s, Period/IPG=16.67/0.000 ms
TOS set to 0x0 (Nagle on)
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
...
[  1] 0.00-600.01 sec  1.40 GBytes  20.0 Mbits/sec  0/0 1740 pps 36000/2/2
[  1] Sent 1043975 datagrams

Here’s the corresponding netdata trace:

Though iperf causes greater packet transfer (vs. videoconference traffic) no “gaps” were observed like those seen during the Jitsi session.

IOW something happens during videoconferences that didn’t show up in testing. Of course iperf couldn’t actually replicate the way Jitsi works, but even so iperf results don’t appear to show intrinsic network problems that account for the video performance issues.

So at this point the “freezing” problem remains unresolved. Any information or hints will be greatly appreciated.

[damencho]

Not setting these can cause similar behavior jitsi-videobridge/20-jvb-udp-buffers.conf at master · jitsi/jitsi-videobridge · GitHub, are those set?

[damencho]

The other thing is the RAM usage, if you use all components on one machine 4 is not optimal, as jicofo and jvb are configured to get up to 3GB, and this can be just garbage collection running frequently in some periods …
If you use single machine give it at least 8 GB.

[Freddie]

@damencho

If you use single machine give it at least 8 GB.

@damencho This is a bit puzzling to me. The OP reports hosting mostly 2-party meetings. Baring the possibility that they have several hundreds of those meetings running concurrently, why wouldn’t a 4CPU/4GB server suffice? I think it’s also telling that they’re reporting this behavior just started in mid-December (not sure if they upgraded their server to a newer version around the same time). But the inference is that this same server worked fine for them prior to that. If the recommendation now is to provide at least 8GB for a standalone deployment of Jitsi, this challenges what - in my mind - has been one of the best sells of Jitsi - the fact that it doesn’t require much physical infrastructure to run (bandwidth being the most important resource). Jicofo and JVB have always had that 3GB configuration, so what’s different now to require 8GB of RAM in this simple use case?

[damencho]

Hum, I don’t know … Maybe you are right if memory is not enough you will see other problems…
But in general its not good idea to have 4GB and allow just two services to take up to 6, as a minimum you should minimize the usage of those two services.

@jrapdx is it possible you upgraded java and you started using 17 instead of 11?

[jrapdx]

Yup, set exactly as recommended:

net.core.rmem_max=10485760
net.core.netdev_max_backlog=100000

[damencho]

What about the java version?

[jrapdx]

AFAIK the only java installed is openjdk-8:

$ java -version
openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-8u312-b07-0ubuntu1~20.04-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)

[jrapdx]

FWIW during the videoconference there was ~2.45GB RAM available to applications:

[damencho]

Can you switch to java 11 and see how it goes?

[jrapdx]

Worth a shot, I’ll install 11 and report back.

[jrapdx]

Java 11 installed without difficulty. Then restarted everything and tried various connections. (Though I had to wait until today to test the most troublesome connection.)

Results were mixed depending partly on which browser I used. Firefox was very glitchy, at times totally lost video or frame rate dropped to <10. Video was quite blurry.

Webkit/chromium-based browsers did better, video images clearer, but connection was “Nonoptimal” or “Poor” about 50% of the time. (Not reflected in server monitoring re: udp4/6 packet transfer rate.) Curiously bitrate (per Jitsi UI) was higher in jvb-mediated vs p2p meetings.

Today I was finally able to conduct a 35 minute test session with the remote participant. This went well, no loss of video, no freezing, bitrate stayed “Good” throughout.

At this point it appears using Java 11 has relieved the video freezing problems we were having. I suppose more time is needed before declaring total success, but so far the outcome is favorable. Big thanks to @damencho and @Freddie for your help!