[SOLVED] Looks like a memory leak

Description

I have a fresh install on a fresh server, and it’s leaking memory and crashing

Here’s a couple screenshots
image

You can see where I rebooted the server, we started a call, and it grew to full memory usage within an hour.

Current behavior

Leaking lots of memory and crashing server.

Expected Behavior

No memory leaks

Possible Solution

shrug

Steps to reproduce

Turn on server, start a meeting. Watch memory usage sky rocket.

Environment details

SkySilk VPS

root@jitsi:~# apt install jitsi-meet
Reading package lists... Done
Building dependency tree
Reading state information... Done
jitsi-meet is already the newest version (2.0.4416-1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@jitsi:~#
root@jitsi:~# cat /proc/version
Linux version 4.15.18-24-pve (build@pve) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)) #1 SMP PVE 4.15.18-52 (Thu, 05 Dec 2019 10:14:17 +0100)
root@jitsi:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.4 LTS
Release:        18.04
Codename:       bionic

I started a github bug report here, but have had no responses and wanted to see if there’s better following here. https://github.com/jitsi/jitsi-meet/issues/5930

By default jvb and jicofo needs to be able to use up to 3GB of memory -Xmx3072m, if that is not enough they can crash.
For jvb you can control that by VIDEOBRIDGE_MAX_MEMORY=3072m
You can set it as env variable or in /usr/share/jitsi-videobridge/lib/videobridge.rc

2 Likes

Oh sick! That should be part of the config guides, especially since a lot of lightweight VPS start at 2GB or lower.

That would also explain why when I paid to use 4 GB ram for 24 hours I couldn’t get it to crash. RAM just grew to 3.1GB and stopped there.

it’s not as such but there are hints

1 Like

That’s a good file. Thanks for the link!

So I’ve notice that it’s not quite working as expected. It’s using more than 1200mb of RAM.

root@jitsi:~# ps aux --sort -rss
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
jvb         2040  0.0 33.5 7394580 1373076 ?     Ssl  06:37  27:27 java -Xmx1200m -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp
jicofo      2026  0.0 12.0 11285856 493980 ?     Sl   06:37   0:17 java -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Dnet.java.sip.communic

I’ve also noticed jicofo has it’s own -Xmx parameter. Does that mean the sum of the two is really how much RAM I’d need? Also, is there a correct location for setting it for jicofo?

Edit: found it hardcoded in /usr/share/jicofo/jicofo.sh

1 Like

I tried 6 users on a dedicated server with only 2Gb yesterday. CPU hit 99.3%.

I have just set VIDEOBRIDGE_MAX_MEMORY=1800m

I’ll report back what happens.

So, so far, I set jvb to 1200m and jicofo to 500m and it’s been working steady. I’m going to do a couple more long multi-person calls this weekend see how it does

Just finished with few 8-person calls (multiple hours long) and it held steady with jvb set to 1200m and jicofo set to 500m.

I upgraded the server to one with 8Gb, 8 cores and 100Mb/s bit rate. It can manage about ten users. I am running netdata on the server and nothing looks excessive at the server end. I am beginning to believe the problems are with the client devices.

You should be able to get more than 10 users on that based on how well my 1 core CPU, 2 Gb RAM server can run 8 users with only about 50% CPU utilization.

Idk what’s not working for you, but even client devices can be pretty basic. I’ve run it on chromebooks and old cellphones no problem.

It was breaking down badly above 12 users. The bandwidth is only
100Mb/s and the graph shows a maximum of 52% of network capacity
being used. I can send you the netdata screen shots if you would
like to see them.

I have now set the resolution to 720p, frame rate to 20 - 60
frames/sec, stopped the audio level indicator, and set last-N to
5. I run another test tomorrow.

It may be that a few of us use hi-Def screens and don’t reduce
the window size.

Mike

That doesn’t make a ton of sense to me. I have mine set to 1080p, and then I upped the screenshare framerate to 30 fps from 5, sharing at 1440p. If you really only have 100Mbps rates, you could be running to issues in a worse case scenario.

I found a nice post that explains how simulcast works with JVB that might help you understand how the bandwidth is being utilized. Help on Simulcast

I don’t think the clients’ screen resolutions have anything to do with the server utilization.

Thanks, I did have a good idea of how the transmitted screen
resolution worked.

I will be running a test in about 40 minutes time with the new
settings that I described. Once I see how that works I will decide
whether to upgrade to 1Gb/s for only £5 per month. It should be a
no brainer!

Hi,

I’ve started a Jitsi instance on a fresh Ubuntu18.04 VPS with 8GB RAM and 4vCPUS. Works like a charm!

After six days, I’m noticing that the RAM memory use was just growing and growing. The graphic bellow shows the picture:

The CPU peaks are the exactly at the moment where the calls happened. The memory use, however, keeps growing and it doesn’t free after the call is ended.

I restarted jitsi-videobridge2, and the memory was freed.

Is this behavior expected? All memory settings mentioned in this thread are set as the default values.

Thanks for the awesome software. Best,

Looks like JVB and Jicofo use up to 3072MB of memory each. I don’t think Java starts freeing memory until the memory its been allocated maxes out. Hence when I lower the limits as I explain above, I can make it so it grows to 2GB total and never goes past that. If you have 8 GB of RAM and nothing else on the system, you can stick with these two defaults and I expect you’d never see more than 6 GB + system ram usage. It looks like you restarted it before it finished growing to that point.

1 Like

Hi @Ethan_Spitz,

if you have 2GB RAM, is it require to split equally?
jvb 1024 + Jicofo 1024 ?

TIA