Crash with a conference with approx. 15 users

Install & Config
#1

If a conference reaches about 15 users, Jitsi crashes. Only one user had video enabled, all other’s video were disabled and only audio were unmuted.

The error log shows

/var/log/jitsi/jicofo.log

java.net.SocketException: Broken pipe (Write failed)
        at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
        at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)
        at java.base/sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:312)
        at java.base/sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:316)
        at java.base/sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:153)
        at java.base/java.io.OutputStreamWriter.flush(OutputStreamWriter.java:254)
        at java.base/java.io.BufferedWriter.flush(BufferedWriter.java:257)
        at org.dom4j.io.XMLWriter.flush(XMLWriter.java:272)
        at org.jivesoftware.whack.ExternalComponent.send(ExternalComponent.java:371)
        at org.jivesoftware.whack.ExternalComponentManager.sendPacket(ExternalComponentManager.java:269)
        at org.xmpp.component.AbstractComponent.send(AbstractComponent.java:925)
        at org.jitsi.xmpp.component.ComponentBase.access$400(ComponentBase.java:36)
        at org.jitsi.xmpp.component.ComponentBase$PingTask.run(ComponentBase.java:577)
        at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
        at java.base/java.util.TimerThread.run(Timer.java:506)

/var/log/jitsi/jvb.log

Exception in thread "org.ice4j.ice.harvest.AbstractUdpListener thread for 85.215.93.43:10000/udp" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.base/java.lang.Thread.start0(Native Method)
        at java.base/java.lang.Thread.start(Thread.java:803)
        at org.ice4j.socket.MergingDatagramSocket$SocketContainer.init(MergingDatagramSocket.java:733)
        at org.ice4j.socket.MergingDatagramSocket$SocketContainer.<init>(MergingDatagramSocket.java:709)
        at org.ice4j.socket.MergingDatagramSocket.doAdd(MergingDatagramSocket.java:279)
        at org.ice4j.socket.MergingDatagramSocket.add(MergingDatagramSocket.java:244)
        at org.ice4j.ice.harvest.SinglePortUdpHarvester$MyCandidate.addSocket(SinglePortUdpHarvester.java:378)
        at org.ice4j.ice.harvest.SinglePortUdpHarvester$MyCandidate.access$000(SinglePortUdpHarvester.java:211)
        at org.ice4j.ice.harvest.SinglePortUdpHarvester.maybeAcceptNewSession(SinglePortUdpHarvester.java:152)
        at org.ice4j.ice.harvest.AbstractUdpListener.runInHarvesterThread(AbstractUdpListener.java:326)
        at org.ice4j.ice.harvest.AbstractUdpListener.access$000(AbstractUdpListener.java:46)
        at org.ice4j.ice.harvest.AbstractUdpListener$1.run(AbstractUdpListener.java:246)
[194377.900s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.

Obviously, JVB were unable to create a new thread and crashed.

Question: How do I find out which limit was reached and caused the crash (physical memory, open file limit, number of total threads)?

My setup:

  • 8GB RAM, no swap
  • DefaultTasksMax=65000
  • DefaultLimitNOFILE=65000
  • DefaultLimitNPROC=65000
#2

you don’t have a Virtuozzo container as server (I think that a german hoster called Strato or something like that does this thing) ?

1 Like
#3

Yes, I do. You are totally right, my server is a Vrituozzo container and is hosted by Strato

#4

this is a very well known problem. You will find lot of threads about Jicofo creating a big number of threads and Virtuozzo config crashing Jitsi. The only workaround that has been found to allow Jitsi to host a modest number of guests is to stop Jicofo health monitoring - that is creating a large number a threads if I understand correctly. I am speaking from memory here if this is not helping you could try to find the thread in the forum with the search function, it was less than a month ago I think.

1 Like
#5

Do you have some pointers for me? I search for the keywords “Virtuozzo” but this only returned my own post, searching for “crash health monitoring” or “crash health check” or “crash too many threads” did no unveil anything neither. Obviously, I am too stupid to find the right post.

#6

here it is:

FTR I searched for ‘strato’ + my handle.