How to debug jicofo memory errors? (jitsi-meet participants not seeing each other.)

I installed jitsi-meet on a Debian / Ubuntu 18 following the quick install guide.

I can access the service at meet.mydomain.com/sessionname and see my self.

But participants don’t see each other and neither do they see a thumbnail of others.

In /var/log/jitsi/jicofo.log the following error is being repeated during the session:

Jicofo 2020-03-24 19:20:27.071 SEVERE: [60] util.UtilActivator.uncaughtException().122 An uncaught exception occurred in thread=Thread[pool-11-thread-1,5,main] and message was: unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:717)
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1025)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Jicofo 2020-03-24 19:20:33.269 SEVERE: [41] org.jitsi.xmpp.component.ComponentBase.log() Ping timeout for ID: SjtsZ-94
Jicofo 2020-03-24 19:20:33.359 SEVERE: [41] org.jitsi.xmpp.component.ComponentBase.log() Ping timeout for ID: SjtsZ-95

The output of free -h reads like this:

              total        used        free      shared  buff/cache   available
Mem:           8,0G        552M        7,3G         22M        189M        7,4G
Swap:            0B          0B          0B

In /var/log/prosody/prosody.log each new participant joining the conversation produces an entry of the form

Mar 24 19:20:03 mod_bosh        info    New BOSH session, assigned it sid 'e51afbf6-21f6-4ff9-ae3c-fbd5519fdf58'
Mar 24 19:20:03 boshe51afbf6-21f6-4ff9-ae3c-fbd5519fdf58        info    Authenticated as 8b40cec7-c788-4452-916b-0b36922f6530@meet.mydomain.com
Mar 24 19:25:24 bosh65d50eac-027c-4f69-b32b-61d865cab631        info    BOSH client disconnected

I am an experienced Linux user but could not help myself. After two days of debugging I’m now seeking your help. Thanks a lot.

I’m aware of this thread but think my RAM is enough: [Jicofo] Java OutOfMemoryError

Are you using latest packages, from unstable? You better try those.

Thanks for your reply! I am now using the unstable sources. The same error appears in the jicofo log and still the conference participants don’t see video or thumbnails of each other.

Is there anything I could try?

Jicofo by default runs with option to use up to 3GB of memory. We had never seen that to be exhausted even on very busy machines … So not sure what is going on, and how do you manage to run into OutOfMemoryError …

But is there a way to debug this further? I find the interplay of all these components rather complicated … Maybe this is not the original error but something else is going wrong. Do you have an idea which log files I should watch primarily or how else to debug this situation?

Interesting fact: I ran into the very same error on two different machines. The first was a Debian / Ubuntu 18 virtual Server with Plesk, Apache2 and 2gb of memory. So I ordered a new server, with a completely new system (same software configuration) but 8GB RAM. And I keep running into the exact same error.

Maybe it is something about the Debian packages?

Maybe it is not a memory issue but the primary error is the line

Jicofo 2020-03-24 19:20:33.269 SEVERE: [41] org.jitsi.xmpp.component.ComponentBase.log() Ping timeout for ID: SjtsZ-94

What does “ping timout” mean?

We have seen that when jicofo is not able to send messages to prosody.

Hi,
I had the same same issue on a virtual Debian 9, but it was my fault…
Just to be sure: Did you also notice the part regarding the values to be maintained in /etc/systemd/system.conf in the installation guide?

I was trying anything next to everything before I re-read the guide carefully and changed the system.conf -> made my day!

1 Like

Thanks a lot @Toberl, that solved my problem!! I falsely thought the systemd settings in the Installation Guide applied only when you are behind NAT! So I got that wrong.

Where do i found this information? Which values?
Forgive me my blind eyes but in the jitsi-docu i saw nothing about system.conf … my debian server comes out of the box from strato … maybe i m taking the wrong words to ask aunt google?
values /etc/systemd/system.conf debian 9
values /etc/systemd/system.conf debian 9 virtual

ok - autsch - ich habs gefunden - ups - i found it. hope that it helps … ill be back

okay - as often:
be attentive and look closely - this problem is solved. jitsi works … thank you!

Just for the sake of completeness: What I did was configuring

DefaultLimitNOFILE=65000
DefaultLimitNPROC=65000
DefaultTasksMax=65000

in /etc/systemd/system.conf.

yep - that works fine