Fresh install ==> OutOfMemory Execption

Install & Config
#1

Dear all,
just installed a fresh Linux Server and went through Quick-Install-Doc. Unfortunately i am quite new to jitsi and got stuck analysing the error.
Jitsi starts well, but as soon as i enter a new room i get several processes creating MemoryDumps and Java OOM Execptions in the logs. Others are unable to join the room.

NAME=“Ubuntu”
VERSION=“18.04.4 LTS (Bionic Beaver)”
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME=“Ubuntu 18.04.4 LTS”
VERSION_ID=“18.04”

Jicofo 2020-04-08 08:41:45.374 SCHWERWIEGEND: [54] util.UtilActivator.uncaughtException().122 An uncaught exception occurred in thread=Thread[pool-12-thread-1,5,main] and message was: unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1025)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Jicofo 2020-04-08 08:41:46.863 SCHWERWIEGEND: [41] org.jitsi.xmpp.component.ComponentBase.log() Ping timeout for ID: yc4EY-149
Jicofo 2020-04-08 08:41:46.906 SCHWERWIEGEND: [41] org.jitsi.xmpp.component.ComponentBase.log() Ping timeout for ID: yc4EY-150
Jicofo 2020-04-08 08:41:47.240 INFORMATION: [28] org.jitsi.jicofo.xmpp.BaseBrewery.removeInstance().378 Removed brewery instance: jvbbrewery@internal.auth.<sub.mydomain.tld>/55a2574d-ed86-4f43-8a1a-f28cf68cb7e4
Jicofo 2020-04-08 08:41:47.240 INFORMATION: [28] org.jitsi.jicofo.xmpp.BaseBrewery.notifyInstanceOffline().184 A bridge left the MUC: jvbbrewery@internal.auth.<sub.mydomain.tld>/55a2574d-ed86-4f43-8a1a-f28cf68cb7e4
Jicofo 2020-04-08 08:41:47.240 INFORMATION: [28] org.jitsi.jicofo.bridge.BridgeSelector.log() Removing JVB: jvbbrewery@internal.auth.<sub.mydomain.tld>/55a2574d-ed86-4f43-8a1a-f28cf68cb7e4
Jicofo 2020-04-08 08:41:47.241 INFORMATION: [43] org.jitsi.jicofo.bridge.JvbDoctor.log() Stopping health-check task for: jvbbrewery@internal.auth.<sub.mydomain.tld>/55a2574d-ed86-4f43-8a1a-f28cf68cb7e4
Jicofo 2020-04-08 08:41:56.861 SCHWERWIEGEND: [41] org.jitsi.xmpp.component.ComponentBase.log() Ping timeout for ID: yc4EY-164
Jicofo 2020-04-08 08:41:56.906 SCHWERWIEGEND: [41] org.jitsi.xmpp.component.ComponentBase.log() Ping timeout for ID: yc4EY-165

how can i find out why defaults do not work for me ?

root@hostname:~# cat /proc/cat /var/run/jitsi-videobridge/jitsi-videobridge.pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65000 65000 processes
Max open files 65000 65000 files
Max locked memory 16777216 16777216 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 1545098 1545098 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

top - 09:23:00 up 43 min, 1 user, load average: 0,00, 0,01, 0,05
Tasks: 25 gesamt, 1 laufend, 24 schlafend, 0 gestoppt, 0 Zombie
%CPU(s): 1,2 be, 0,1 sy, 0,0 ni, 98,6 un, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
KiB Spch : 8388608 gesamt, 7901608 frei, 225320 belegt, 261680 Puff/Cache
KiB Swap: 0 gesamt, 0 frei, 0 belegt. 8161756 verfü Spch

java -Xmx3072m -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/videobridge/logging.properties -cp /usr/share/jitsi-videobridge/jitsi-videobridge.jar:/usr/share/jitsi-videobridge/lib/* org.jitsi.videobridge.Main --host=localhost --domain=<sub.mydomain.tld> --port=5347 --secret=secret --apis=,

any ideas ?

No media streams (neither video nor audio) works after standard installation
#2

this is strange, how are external processes such as nginx supposed to access the videobridge if it does not expose an api ? While I am certainly no jitsi expert, I’d take a look at cat /etc/jitsi/videobridge/config and try to add something like
JVB_OPTS=–apis=rest,

you should have a file that looks like this

JVB_HOSTNAME=your-url

# sets the hostname of the XMPP server (default: domain if set, localhost otherwise)
JVB_HOST=

# sets the port of the XMPP server (default: 5275)
JVB_PORT=5347

# sets the shared secret used to authenticate to the XMPP server
JVB_SECRET=your-secret

# extra options to pass to the JVB daemon
JVB_OPTS=--apis=rest,


# adds java system props that are passed to jvb (default are for home and logging config file)
JAVA_SYS_PROPS="-Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/videobridge/logging.properties"
#3

Recommended is 8GB of RAM for jitsi-meet on single machine.

#4

All right, but what’s your point ?

#5

I was saying that because of this exception.

#6

Hi,
I have the same error.

by checking the status of jicofo with
systemctl status jicofo.service
the server said, that the service failed to start. Perhaps because of the fact “Tasks: 58 (limit: 60)”.

Is there a possibility to chance the limit?

Jitsi broken after Debian upgrade - Could not reserve enough space for 3145728KB object heap
#7

This is how this is done in the bridge:


Yeah this limit of 60 is very low compared to the default ones …

#8

In the original message, the top output quoted was saying the system had indeed 8 Gb Ram.

#9

Yep, I got confused by the first part which is OutOfMemoryError , but the second one: unable to create new native thread says it, it maybe a memory not enough to create native threads, something we had seen, but also the limit.

#10

In my case the problem was the low task limit and some DNS problems which could be solved. Now the server runs:)

Great Tool.

Thanks for your help.

#11

@niox where can i set those TASK limits ? In my file (/lib/systemd/system/jitsi-videobridge2.service) the limit is set to 65000. As I mentioned it is a clean, fresh install.

my exact error looks like:

Jicofo 2020-04-15 16:07:38.479 SCHWERWIEGEND: [452] util.UtilActivator.uncaughtException().122 An uncaught exception occurred in thread=Thread[pool-405-thread-1,5,main] and message was: unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)

How can I find out what limit is being reached? Files? Tasks? mem?

#12

The problem is jicofo (at least the log you provided). You need to set it maybe for jicofo user somehow as it doesn’t use systems unitfile as jvb. Check ulimit.

#13

Yes indeed I had noticed it but did not think about what followed.
systemctl cat jicofo
# /run/systemd/generator.late/jicofo.service
# Automatically generated by systemd-sysv-generator
and so:
systemctl cat jitsi-videobridge2
# /lib/systemd/system/jitsi-videobridge2.service
(…)
[Service]
(…)
# more threads for this process
TasksMax=65000
# allow more open files for this process
LimitNPROC=65000
LimitNOFILE=65000

lack of time to implement and test it in debian packaging I guess.

#14

Sry for the late answer.

I did this, too:
Open the file /etc/systemd/system.conf and add the following lines at the end.

DefaultLimitNOFILE=65000
DefaultLimitNPROC=65000
DefaultTasksMax=65000
#15

i have the same error :
org.jitsi.xmpp.component.ComponentBase.log() Ping timeout for ID: 9SBm3-121 after installing ldap