java.lang.OutOfMemoryError: unable to create new native thread

I have a virtual server with Debian 9 and Jitsi-meet installed (nothing else).

/etc/systemd/system.conf includes the following lines:

[Manager]
DefaultLimitNOFILE=65000
DefaultLimitNPROC=65000
DefaultTasksMax=65000

/etc/systemd/logind.conf:
UserTasksMax=80%

For a single conference with 8 participants I get the following exception in jicofo.log after some time:

java.lang.OutOfMemoryError: unable to create new native thread

The problem occurred after my last update of jitsi-meet. No problems before.

My server has 8 gb ram, mostly free.

I would be very grateful for hints.

What is the output of systemctl status jicofo

jicofo.service - LSB: Jitsi conference Focus
   Loaded: loaded (/etc/init.d/jicofo; generated; vendor preset: enabled)
   Active: active (running) since Wed 2020-11-18 06:42:27 CET; 10h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 251 ExecStart=/etc/init.d/jicofo start (code=exited, status=0/SUCCESS)
    Tasks: 282 (limit: 65000)
   CGroup: /system.slice/jicofo.service
           └─279 java -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djdk.tls.ephemeralDHKeySize=2048 -Dconfig.file=/etc/jitsi/jicofo/jicofo.conf -Dnet.java.sip.communicator.SC_HOME_DIR

Last line is cut

And you reproduce this every time?

Hi,
I have exact the same problem. A fresh install (quick install self-hosting) and with 1 user.
I have a Let’s Encrypt certificate but did not yet setup a secure domain

free -m
              total        used        free      shared  buff/cache   available
Mem:          16384         558       15253           1         571       15824
Swap:             0           0           0

cat /proc/sys/kernel/threads-max 
2058707
ps axms | wc -l
308

I’m trying to install jitsi-meet for a few days now and today I started over with a complete fresh canvas (apt purge everything :wink: )

The passed days I had errors due to configuration problems but this one is new. Did something change in the stable branch the last few days?

systemctl status jicofo
● jicofo.service - LSB: Jitsi conference Focus
   Loaded: loaded (/etc/init.d/jicofo; generated)
   Active: active (running) since Fri 2020-11-20 12:09:37 CET; 5min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 209 ExecStart=/etc/init.d/jicofo start (code=exited, status=0/SUCCESS)
    Tasks: 146 (limit: 165)
   CGroup: /system.slice/jicofo.service
           └─231 java -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djdk.tls.ephemeralDHKeySize=2048 -Dconfig.file=/etc/jitsi/jicofo/jicofo.conf -Dnet.java.sip.communicato
systemd[1]: jicofo.service: Failed with result 'exit-code'.
systemd[1]: Stopped LSB: Jitsi conference Focus.
systemd[1]: jicofo.service: Found left-over process 2841 (java) in control group while starting unit. Ignoring.
systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
systemd[1]: Starting LSB: Jitsi conference Focus...
systemd[1]: jicofo.service: Control process exited, code=exited status=2
systemd[1]: jicofo.service: Failed with result 'exit-code'.
systemd[1]: Failed to start LSB: Jitsi conference Focus.
Starting LSB: Jitsi conference Focus...
Started LSB: Jitsi conference Focus.

Result is that I can’t unmute my video and mic.
Thanks, R

You see your jicofo is currently running with 146 threads and its limit is 165, a little load and will break it.

Didn’t see that, where do I configure a higher value? In /etc/systemd/system.conf ?
I don’t have DefaultTasksMax set so it should default to 512. So where does the limit 165 come from?

EDIT:
The answer is yes.
After increasing the max=4096
Still the same problem, although it looks like it is running (active). The web gives the same problem.

systemctl status jicofo
 jicofo.service - LSB: Jitsi conference Focus
   Loaded: loaded (/etc/init.d/jicofo; generated)
   Active: active (running) since Fri 2020-11-20 14:27:29 CET; 1min 26s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1126 ExecStop=/etc/init.d/jicofo stop (code=exited, status=0/SUCCESS)
  Process: 1131 ExecStart=/etc/init.d/jicofo start (code=exited, status=0/SUCCESS)
    Tasks: 91 (limit: 4096)
   CGroup: /system.slice/jicofo.service
           └─1140 java -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djdk.tls.ephemeralDHKeySize=2048 -Dconfig.file=/etc/jitsi/jicofo/jicofo.conf -Dnet.java.sip.communicat
    systemd[1]: Failed to start LSB: Jitsi conference Focus.
    systemd[1]: jicofo.service: Found left-over process 234 (java) in control group while starting unit. Ignoring.
    systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
    systemd[1]: Starting LSB: Jitsi conference Focus...
    systemd[1]: Started LSB: Jitsi conference Focus.

what says
cat /proc/sys/kernel/pid_max

32768
EDIT:
Sorry I don’t know. Got the pid_max after I increased to

DefaultLimitNOFILE=65000
DefaultLimitNPROC=65000
DefaultTasksMax=65000

I don’t think that kernel parameters are impacted by systemd changes. This value is strange - what’s your OS ?

Ubuntu 18.0.4
uname -a
Linux my.server.x 4.15.0 #1 SMP Tue Jun 9 12:58:54 MSK 2020 x86_64 x86_64 x86_64 GNU/Linux

Hello guys,
After I edited the /etc/systemd/system.conf file and restarted the server (shurtdown -r now).
It seems that I don’t have any more problems with running jicofo and the videobridge.
systemctl status of both indicated they are running as they should be.
I still have the problem that I can’t unmute my video and microphone but I don’t think it is related to this topic and I don’t want to hijack this thread.
So thank for your help here and I’ll open a new thread to figure-out what’s wrong.
R.

Yes. I even reinstalled the server (this time with Ubuntu 18.04) -> same problem.

In the meantime I have set up a second server at another hoster. The new one does not seem to have the problem. So I suspect that my old provider (Strato) is somehow causing the problem.