JVB memory usage

On the jitsi call from February 10th there was some discussion around the memory usage of the JVB. It was mentioned that the JVB should be using only a few hundred MB of memory and upping the heap limit could cause longer pauses for garbage collection.

Last fall we were having JVBs run out of memory so we upped the heap limit and the container memory limit (and also the instance size as a precaution). We are running a quite old JVB (build 1109) but are working on upgrading to JVB 2. With our current production bridge (1109 build), the container memory usage is typically using about 500 MB after a deploy and rises up to 1 - 1.5 GB within the first day or two and ends up rising up to 2 - 3 GB in a stair step pattern and then hovering there for a while.

A couple questions:

  1. Is the memory footprint of the new JVB that much lower than the old JVB or is there possibly something else going on?
  2. We use the prometheus exporter (https://github.com/karrieretutor/jitsi-prom-exporter/) but it lacks the java metrics. Has there been any talk about having a prometheus metrics endpoint built into the bridge to get both application and java metrics? We might be able to add that if it would be a welcome change.
1 Like

Hi Devin,

The new bridge should be a bit more efficient, but the difference isn’t major. What you observe about the memory usage of the jvm process is normal. It doesn’t mean that all this memory is in use by the java application. Do you have a reason to think it is causing any problems?

I don’t think anyone from our team is working on a prometheus exporter. Can it work as a separate process on the same machine? This is what we do with datadog.

Boris

We have been seeing some issues with streams freezing and I wanted to eliminate potential causes so I was curious if java GC pauses could be impacting that but we currently don’t have visibility into that in production.

I was looking around to see if we could use an exporter to get the java metrics and found https://github.com/prometheus/jmx_exporter. I’m not quite sure if it will do what we need but that might work for our purposes.

I want software releases memory once it’s not used. Cache is okay for me if it’s released when new processes ask for memory. How to tune Java to this free memory behaviour?

I think this explains our troubles from a few days ago. I followed the quick-install instructions to set Jitsi Meet up on a box with 1GB RAM. Things worked great for several days and then one day, it all fell apart, with the logs indicating it was out of memory.

Not being a Java guy, I sent the full commands from ps for prosody, jicofo and jvb to our team where an astute colleague said (paraphrasing) “I see a problem with -Xmx3072m [used in both the jvb and jicofo commands]. We don’t have 3GB. It will never garbage collect.”

So… should I shrink it to -Xmx768m? Or will a daily

systemctl restart jicofo.service  
systemctl restart jitsi-videobridge2.service
systemctl restart prosody.service

be enough to keep a rather small Jitsi Meet alive? I doubt we’d get up to 10 people in one meeting.