Jitsi-meet becoming unresponsive

Hi all,

First of all wonderful tool and I’d very much like to have my colleagues also being able to use this in ‘secure domain’ mode. With the excellent help of the manuals we had it set up quiet nicely, though with a problem.

After some time the server becomes unresponsive, because (lack of prereading) we didn’t first install nginx or apache - so we are in jetty mode (this might be cause/effect for the issue? not sure).
Sometimes it helps to restart prosody using prosodyctl, other times the whole site isn’t responsing and I have to restart jitsi-videobridge.
Running on an Ubuntu 18.04 - AWS instance, installed jitsi-meet version as of yesterday. We don’t have many users, but I did change the systemd settings as indicated.

Logging does not show anything particular, it’s currently down, but jicofo doesn’t log anything after someone left a channel an hour ago. (left out users details in logs below). After getting below logs I’ve restarted jitsi-videobridge and it works for a while, including the prosody indicative /http-bind

What could be going on/wrong here? Besides the apt install and letsencrypt we took the basic steps to secure domain and finalize install

  • modify site in /etc/prosody/conf.avail/ anonymous->internal-plain
  • add XMPP+site to jicofo’s sip properties file
  • register a couple dozen users with prosodyctl
  • rebooted the machine

Not sure why yesterday prosody stopped working at all (see lowest log), but today it seems that jitsi-videobridge (i.e. jetty) becomes unresponsive. If caused due to not installing a webserver (apache/nginx) as proxy, how should we do that ‘after the fact’? What else could we be looking into? Journal’s don’t show anything particular related, load on the system is very low as well.

Thanks for any clues/indications/forgot-to-reads,

Tom

Logging information

Jicofo:

Jicofo 2020-03-19 15:01:47.286 WARNING: [4165] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Participant not found for ...@...../5340bd6e terminated already or never started ?
Jicofo 2020-03-19 15:01:47.289 INFO: [4165] org.jitsi.jicofo.FocusManager.log() Disposed conference for room: ...@..... conference count: 0
Jicofo 2020-03-19 15:09:22.350 INFO: [15] org.jitsi.jicofo.auth.AbstractAuthAuthority.log() Expiring session:AuthSessio

Jvb still actively logging

JVB 2020-03-19 15:59:39.061 INFO: [45] org.jitsi.videobridge.health.Health.log() Performed a successful health check in 11ms. Sticky failure: false
JVB 2020-03-19 15:59:39.691 FINE: [253] org.jitsi.videobridge.xmpp.ComponentImpl.processIQ() (serving component 'JitsiVideobridge') Processing IQ (packetId hERLh-32009): <iq type="get" to="jitsi-videobridge.our.jit.si" from="focus@auth.our.jit.si/focus18132966863" id="hERLh-32009"><healthcheck xmlns="http://jitsi.org/protocol/healthcheck"/></iq>
JVB 2020-03-19 15:59:39.691 FINE: [253] org.jitsi.videobridge.xmpp.ComponentImpl.processIQRequest() (serving component 'JitsiVideobridge') Processing IQ request (packetId hERLh-32009).
JVB 2020-03-19 15:59:39.692 FINE: [253] org.jitsi.videobridge.xmpp.ComponentImpl.processIQ() (serving component 'JitsiVideobridge') Responding to IQ (packetId hERLh-32009) with: <iq to="focus@auth.our.jit.si/focus18132966863" from="jitsi-videobridge.our.jit.si" id="hERLh-32009" type="result"/>
JVB 2020-03-19 15:59:47.152 FINE: [277] org.jitsi.videobridge.xmpp.ComponentImpl.processIQ() (serving component 'JitsiVideobridge') Processing IQ (packetId qcMoM-1724): <iq id="qcMoM-1724" type="result" to="jitsi-videobridge.our.jit.si" from="our.jit.si"/>

prosody

|Mar 19 14:58:09 bosh0b26086e-7df7-4aa6-842d-b747d922d2b5|info|BOSH client disconnected|
|---|---|---|
|Mar 19 15:00:48 mod_bosh|info|New BOSH session, assigned it sid '3916e7c0-0d0b-4fc1-8125-6f270fc1e556'|
|Mar 19 15:00:48 bosh3916e7c0-0d0b-4fc1-8125-6f270fc1e556|info|BOSH client disconnected|
|Mar 19 15:00:51 mod_bosh|info|New BOSH session, assigned it sid '0864ab09-7174-46d9-a00e-27190dbb2df9'|
|Mar 19 15:00:51 bosh0864ab09-7174-46d9-a00e-27190dbb2df9|info|Authenticated as ...@our.jit.si|
|Mar 19 15:01:47 bosh0864ab09-7174-46d9-a00e-27190dbb2df9|info|BOSH client disconnected|

systemctl for videobridge

   Active: active (running) since Thu 2020-03-19 14:48:04 UTC; 1h 13min ago
  Process: 16226 ExecStartPost=/bin/bash -c echo $MAINPID > /var/run/jitsi-videobridge/jitsi-videobridge.pid (code=exited, status=0/SUCCESS)
 Main PID: 16225 (java)
    Tasks: 112 (limit: 65000)
   CGroup: /system.slice/jitsi-videobridge.service

Prosody status

   Active: active (exited) since Wed 2020-03-18 12:56:48 UTC; 1 day 3h ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 0 (limit: 65000)
   CGroup: /system.slice/prosody.service

although ps ax shows

16078 ?        S      0:02 lua5.1 /usr/lib/prosody/../../bin/prosody

prosody log found this morning (when we found our first crash) - > do note that this didn’t happen today, although we did restart prosody along with videobridge a couple of time

Mar 18 15:24:18 general info    Shutting down...
Mar 18 15:24:18 general info    Shutdown status: Cleaning up
Mar 18 15:24:18 general info    Shutdown complete
Mar 19 07:51:51 general info    Hello and welcome to Prosody version 0.10.0
Mar 19 07:51:51 general info    Prosody is using the select backend for connection handling
Mar 19 07:51:51 portmanager     info    Activated service 'component' on [127.0.0.1]:5347, [::1]:5347

The easiest way is starting from scratch, uninstalling all.
The other option is to stop everything, install nginx, use the template to configure your virtual host and activate it: jitsi-meet/jitsi-meet.example at master · jitsi/jitsi-meet · GitHub, then remove the jetty settings from jvb and restart jvb and jicofo in this order.

Tnx Damian,

Would that also improve the server stability? I did the other option for now, seems to work fine, I’ll wait and see if this improves things.

Cheers, Tom

Yep, nginx behaves better than jetty.

Hello Damencho,

I am having the same issue on my install. I customized a few things and would rather not reinstall the entire thing. I’d like to do the “other option.” Would you be able to list out the steps needed to stop everything, install nginx, configure the virt host and remove the jetty settings, etc?
I’m new to this and would like to get the steps right. Is that possible?
Thanks!

If you update from latest stable it should migrate you from using jetty to using nginx.

Perfect. Thank you. I think this is working now. I see nginx running. I don’t see anything related to jetty anymore, but I can’t say I remember seeing it before. Is there a way to fully verify that it is no longer using jetty at all?

Also, side bar, is it OK to have all of the "HeapDumpOnOutOfMemoryError PIDs in the screenshot below?