VideoBridge HealthCheck looping

Hi,

I’ve tried searching something similar, but could not find.
We have a jitsi server running on private EC2 instance. We are using a bit older version, because we are connecting to it from app and we can’t easily update. We are using these versions:

jitsi-meet                       2.0.5390-3                         all          WebRTC JavaScript video conferences
jitsi-meet-prosody               1.0.4628-1                         all          Prosody configuration for Jitsi Meet
jitsi-meet-web                   1.0.4628-1                         all          WebRTC JavaScript video conferences
jitsi-meet-web-config            1.0.4628-1                         all          Configuration for web serving of Jitsi Meet
jitsi-videobridge2               2.1-416-g2f43d1b4-1                all          WebRTC compatible Selective Forwarding Unit (SFU)

We had already three incidences, when EC2 instance starts using almost 100% CPU, we cant’t connect to it thru ssh, only solution is reboot it. Then after everything spins up again, everything starts to working again. These incidence are quite rare, once in a few weeks, but it is still annoying.

I found in /var/log/jitsi/jvb.log that around that time, when CPU hit 100% VideoBridge performed a lot of health checks, I’m suspecting that this caused extra load on CPU, but not sure how to debug it correctly and fix. Here is log:

2021-06-23 17:23:46.165 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:23:56.165 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:24:06.165 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000002S. Sticky failure: false
2021-06-23 17:24:16.165 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000002S. Sticky failure: false
2021-06-23 17:24:26.165 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000002S. Sticky failure: false
2021-06-23 17:24:36.165 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:24:46.066 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:24:46.165 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:24:56.165 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:25:06.165 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:25:16.166 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000002S. Sticky failure: false
2021-06-23 17:25:26.165 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:25:36.166 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000002S. Sticky failure: false
2021-06-23 17:25:46.066 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:25:46.165 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:25:56.165 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000002S. Sticky failure: false
2021-06-23 17:26:06.167 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000002S. Sticky failure: false
2021-06-23 17:26:16.165 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000002S. Sticky failure: false
2021-06-23 17:26:26.252 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000002S. Sticky failure: false
2021-06-23 17:26:39.321 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.06749S. Sticky failure: false
2021-06-23 17:28:09.637 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:34:06.295 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000002S. Sticky failure: false
2021-06-23 17:35:21.114 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:39:03.727 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.030929S. Sticky failure: false
2021-06-23 17:39:40.804 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:40:03.944 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000003S. Sticky failure: false
2021-06-23 17:40:27.633 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:40:52.941 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000002S. Sticky failure: false
2021-06-23 17:41:17.043 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:41:42.214 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000003S. Sticky failure: false
2021-06-23 17:42:06.969 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:43:40.909 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.020813S. Sticky failure: false
2021-06-23 17:44:05.377 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:44:38.531 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:45:07.056 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000003S. Sticky failure: false
2021-06-23 17:45:33.278 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:45:57.214 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000003S. Sticky failure: false
2021-06-23 17:46:16.117 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:46:33.979 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000002S. Sticky failure: false
2021-06-23 17:46:51.462 WARNING: [20] [hostname=localhost id=shard] MucClient$PingFailedListenerImpl.pingFailed#805: Ping failed, the XMPP connection needs to reconnect.
2021-06-23 17:46:57.252 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:46:57.356 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:46:57.356 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:46:57.356 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:46:57.356 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:46:57.356 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:46:57.356 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:46:57.357 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:46:57.357 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:46:57.357 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:46:57.357 INFO: [22] VideobridgeExpireThread.expire#140: Running expire()
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.357 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000005S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.358 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.359 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.360 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.360 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.360 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.360 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.360 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.360 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.360 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-06-23 17:46:57.360 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.360 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-06-23 17:46:57.360 INFO: [25] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false

This is the problem, the connection to prosody failed, you need to check why is that. There were numerous fixes around the xmpp connection in the newer versions.

Mind that if you are running your client in chrome, you will need to apply for reverse original trial for plan b, as that becomes disabled by default and in order to use unified plan, you need to update to latest … we haven’t switched it completely by default but that will happen in the next month and if we find bugs o the way new updates will come … so, In general, you should be able to update as browsers changes every 6 weeks and Chrome and FF are planning to move two 4 weeks schedule so the client sometimes need to follow browser releases …

First thanks for a heads up about future releases!

I’ve looked at /var/log/prosody/*, I couldn’t find anything useful. *.log file has no entries between last room closure and the time server restarted. *.err has only this:

Jun 23 17:56:45 portmanager     error   Error binding encrypted port for https: No certificate present in SSL/TLS configuration for https port 5281

But this is after server reboot and as I googled about it, it seemed that everyone is just ignoring it.

Maybe I have to check some other log file?

Is jvb and prosody on the same machine?

Yes, they are on a same EC2 instance.

Is the available RAM enough? This is the only thing I can think about.
You can check the syslog around that time … it is good idea to also monitor prosody process cpu usage, not only the whole system cpu.

AWS does not show RAM usage by default, and we currently do not have anything else to monitor our server over time.

We are running t2.micro with 1GB RAM and 1vCPU. All our calls are between two users, so they should be using p2p mode and not use a lot of server resources. We did some monitoring with ‘top’ command, when we have 2-3 active rooms, we see some increase in CPU usage at around 20-30%. But ram usage are always the same, around 100MB are always free (doesn’t matter if there is no active rooms or there are 3 active rooms).

All incidence were happening when there were no active rooms. I’ve looked at syslogs, it looks like that updates were running at that time, so it is possible that they used all RAM that were free and kernel turned off prosody to free some RAM?

│Jun 23 17:26:00 jitsi systemd[1]: Starting Daily apt download activities...                                                                                                                                                  │
│Jun 23 17:26:04 jitsi dbus-daemon[480]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by ':1.77' (uid=0 pid=33235 comm="/usr/bin/gdbus call --system --dest │
│Jun 23 17:26:04 jitsi PackageKit: daemon start                                                                                                                                                                               │
│Jun 23 17:26:04 jitsi systemd[1]: Starting PackageKit Daemon...                                                                                                                                                              │
│Jun 23 17:26:04 jitsi dbus-daemon[480]: [system] Successfully activated service 'org.freedesktop.PackageKit'                                                                                                                 │
│Jun 23 17:26:04 jitsi systemd[1]: Started PackageKit Daemon.                                                                                                                                                                 │
│Jun 23 17:28:16 jitsi systemd-networkd[435]: eth0: Could not set DHCPv4 address: Connection timed out                                                                                                                        │
│Jun 23 17:30:19 jitsi systemd-networkd[435]: eth0: Failed                                                                                                                                                                    │
│Jun 23 17:31:47 jitsi PackageKit: daemon quit                                                                                                                                                                                │
│Jun 23 17:33:35 jitsi packagekitd[33238]: Error releasing name org.freedesktop.PackageKit: Timeout was reached                                                                                                               │
│Jun 23 17:37:03 jitsi systemd[1]: packagekit.service: Succeeded.                                                                                                                                                             │
│Jun 23 17:42:27 jitsi systemd[1]: snapd.service: Watchdog timeout (limit 5min)!                                                                                                                                              │
│Jun 23 17:42:54 jitsi systemd[1]: snapd.service: Killing process 502 (snapd) with signal SIGABRT.                                                                                                                            │
│Jun 23 17:44:16 jitsi systemd[1]: snapd.service: State 'stop-watchdog' timed out. Terminating.                                                                                                                               │
│Jun 23 17:46:56 jitsi systemd[1]: snapd.service: State 'stop-sigterm' timed out. Killing.                                                                                                                                    │
│Jun 23 17:47:05 jitsi systemd[1]: snapd.service: Killing process 502 (snapd) with signal SIGKILL.                                                                                                                            │
│Jun 23 17:47:18 jitsi systemd[1]: snapd.service: Main process exited, code=killed, status=9/KILL                                                                                                                             │
│Jun 23 17:47:25 jitsi systemd[1]: snapd.service: Failed with result 'watchdog'.                                                                                                                                              │
│Jun 23 17:47:29 jitsi systemd[1]: snapd.service: Scheduled restart job, restart counter is at 1.                                                                                                                             │
│Jun 23 17:47:39 jitsi systemd[1]: Stopped Snap Daemon.                                                                                                                                                                       │
│Jun 23 17:49:23 jitsi systemd[1]: Starting Snap Daemon...

By default jicofo and jvb are configured to use up to 3GB of Ram and if you haven’t adjusted that and your deployment has less than 8GB of ram you will run into troubles …

2 Likes

Oh :man_facepalming: I’ve did installation based on this there was nothing about setting ram usage, so I did not knew about these settings.

I found this guide where he uses probably same EC2 instance as we, I will try later today to make changes to our configs, will see how it will perform after changes :crossed_fingers:

I’ve changed settings as in a guide I mentioned previous. At a moment it looks better, on idle server has around 240 free MB of RAM (previous was around 100MB). Performance for user seems to be the same.

We also set up this morning rule for server, so it would restart every night. First this was as a short term solution for this problem, but now we are thinking maybe it’s worth leaving this rule, as we do not have any activity during the night, so we can then make sure that each night we would have clear start-up of a server.

I hope this problem is fixed for us, because of night server restart or because of configuration update :crossed_fingers:

@damencho thanks for a quick support

How can this be adjusted in the Docker container setup?