JVB memory usage

On the jitsi call from February 10th there was some discussion around the memory usage of the JVB. It was mentioned that the JVB should be using only a few hundred MB of memory and upping the heap limit could cause longer pauses for garbage collection.

Last fall we were having JVBs run out of memory so we upped the heap limit and the container memory limit (and also the instance size as a precaution). We are running a quite old JVB (build 1109) but are working on upgrading to JVB 2. With our current production bridge (1109 build), the container memory usage is typically using about 500 MB after a deploy and rises up to 1 - 1.5 GB within the first day or two and ends up rising up to 2 - 3 GB in a stair step pattern and then hovering there for a while.

A couple questions:

  1. Is the memory footprint of the new JVB that much lower than the old JVB or is there possibly something else going on?
  2. We use the prometheus exporter (https://github.com/karrieretutor/jitsi-prom-exporter/) but it lacks the java metrics. Has there been any talk about having a prometheus metrics endpoint built into the bridge to get both application and java metrics? We might be able to add that if it would be a welcome change.

Hi Devin,

The new bridge should be a bit more efficient, but the difference isn’t major. What you observe about the memory usage of the jvm process is normal. It doesn’t mean that all this memory is in use by the java application. Do you have a reason to think it is causing any problems?

I don’t think anyone from our team is working on a prometheus exporter. Can it work as a separate process on the same machine? This is what we do with datadog.

Boris

We have been seeing some issues with streams freezing and I wanted to eliminate potential causes so I was curious if java GC pauses could be impacting that but we currently don’t have visibility into that in production.

I was looking around to see if we could use an exporter to get the java metrics and found https://github.com/prometheus/jmx_exporter. I’m not quite sure if it will do what we need but that might work for our purposes.