Fork: retry: Resource temporarily unavailable

Hello,

I was successfully hosting jitsi-meet on our server for the last year until last week. I ran apt upgrade and restarted the server and since than I have a lot of trouble. I also tried to run apt purge and reinstall it - without success.

When connection to a meeting I get the message unfortunately something went wrong, we are trying to fix this. Additionally, some other services on the server are unreachable (like ssh).

Below I added some outputs and parts of the log file, if you would need to see the entire log file I can upload it as well - just need to remove personal data first.

Ubuntu 18.04.5 LTS

Jitsi Versions

root@server:~# dpkg -l | grep jitsi
ii  jitsi-meet                         2.0.5142-1                                        all          WebRTC JavaScript video conferences
ii  jitsi-meet-prosody                 1.0.4466-1                                        all          Prosody configuration for Jitsi Meet
ii  jitsi-meet-web                     1.0.4466-1                                        all          WebRTC JavaScript video conferences
ii  jitsi-meet-web-config              1.0.4466-1                                        all          Configuration for web serving of Jitsi Meet
ii  jitsi-videobridge2                 2.1-376-g9f12bfe2-1                               all          WebRTC compatible Selective Forwarding Unit (SFU)

free output

root@server:~# free -m
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
              total        used        free      shared  buff/cache   available
Mem:           2048         905         943          30         198        1110
Swap:             0           0           0

top output

top - 03:04:28 up 1 day,  7:52,  2 users,  load average: 0,00, 0,02, 0,05
Tasks:  71 total,   2 running,  69 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0,3 us,  0,5 sy,  0,0 ni, 99,2 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
KiB Mem :  2097152 total,   983488 free,   931328 used,   182336 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  1132376 avail Mem
# ...
 1698 jicofo    20   0 5939968 301048   8732 S   0,0 14,4   1:44.39 java
 1789 jvb       20   0 5679668 173216   6428 S   0,0  8,3   3:19.74 java

Threats of jicofo

root@server:~# ps -o nlwp 1698
NLWP
 271

Threats of jvb

root@server:~# ps -o nlwp 1789
NLWP
  41

JVB LOG

2020-11-06 01:05:28.344 EXCEPTION: [15] RetryStrategy$TaskRunner.run#198: org.jivesoftware.smack.sasl.SASLErrorException: SASLError using SCRAM-SHA-1: not-authorized
org.jivesoftware.smack.sasl.SASLErrorException: SASLError using SCRAM-SHA-1: not-authorized
	at org.jivesoftware.smack.SASLAuthentication.authenticationFailed(SASLAuthentication.java:292)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1100)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
	at java.lang.Thread.run(Thread.java:748)
    
2020-11-06 01:05:42.355 WARNUNG: [19] org.jivesoftware.smack.AbstractXMPPConnection.callConnectionClosedOnErrorListener: Connection XMPPTCPConnection[not-authenticated] (0) closed with error
org.jivesoftware.smack.XMPPException$StreamErrorException: system-shutdown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
<stream:error><system-shutdown xmlns='urn:ietf:params:xml:ns:xmpp-streams'/><text>Received SIGTERM</text></stream:error>
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1064)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
	at java.lang.Thread.run(Thread.java:748)

jconfo

# 20s intervall
Jicofo 2020-11-06 01:37:35.697 SCHWERWIEGEND: [185] org.jitsi.jicofo.Main.log() An uncaught exception occurred in thread=Thread[Jicofo Scheduledpool-3-thread-97,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:717)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1025)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
    
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:717)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
	at org.jivesoftware.smack.AbstractXMPPConnection.asyncGo(AbstractXMPPConnection.java:1725)
	at org.jivesoftware.smack.AbstractXMPPConnection.invokeStanzaCollectorsAndNotifyRecvListeners(AbstractXMPPConnection.java:1229)
	at org.jivesoftware.smack.AbstractXMPPConnection.processStanza(AbstractXMPPConnection.java:1112)
	at org.jivesoftware.smack.AbstractXMPPConnection.parseAndProcessStanza(AbstractXMPPConnection.java:1096)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection.access$500(XMPPTCPConnection.java:151)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1044)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
    
Jicofo 2020-11-06 17:55:33.038 INFORMATION: [72] org.jitsi.jicofo.FocusManager.log() Exception while trying to start the conference
net.java.sip.communicator.service.protocol.OperationFailedException: Failed to join the room
	at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.joinAs(ChatRoomImpl.java:298)
	at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.join(ChatRoomImpl.java:209)
	at org.jitsi.jicofo.JitsiMeetConferenceImpl.joinTheRoom(JitsiMeetConferenceImpl.java:581)
	at org.jitsi.jicofo.JitsiMeetConferenceImpl.start(JitsiMeetConferenceImpl.java:404)
	at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:465)
	at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:419)
	at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:394)
	at org.jitsi.jicofo.xmpp.FocusComponent.handleConferenceIq(FocusComponent.java:337)
	at org.jitsi.jicofo.xmpp.FocusComponent.handleIQSetImpl(FocusComponent.java:228)
	at org.jitsi.xmpp.component.ComponentBase.handleIQSet(ComponentBase.java:362)
	at org.xmpp.component.AbstractComponent.processIQRequest(AbstractComponent.java:515)
	at org.xmpp.component.AbstractComponent.processIQ(AbstractComponent.java:289)
	at org.xmpp.component.AbstractComponent.processQueuedPacket(AbstractComponent.java:239)
	at org.xmpp.component.AbstractComponent.access$100(AbstractComponent.java:81)
	at org.xmpp.component.AbstractComponent$PacketProcessor.run(AbstractComponent.java:1051)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.jivesoftware.smack.SmackException$NoResponseException: No response received within reply timeout. Timeout was 15000ms (~15s). Waited for response using: AndFilter: (StanzaTypeFilter: Presence, OrFilter: (AndFilter: (FromMatchesFilter (ignoreResourcepart): test123@conference.meet.ballaeron.de, MUCUserStatusCodeFilter: status=110), AndFilter: (FromMatchesFilter (full): test123@conference.meet.ballaeron.de/focus, StanzaIdFilter: id=CiIDe-24383, PresenceTypeFilter: type=error))).
	at org.jivesoftware.smack.SmackException$NoResponseException.newWith(SmackException.java:111)
	at org.jivesoftware.smack.SmackException$NoResponseException.newWith(SmackException.java:98)
	at org.jivesoftware.smack.StanzaCollector.nextResultOrThrow(StanzaCollector.java:260)
	at org.jivesoftware.smackx.muc.MultiUserChat.enter(MultiUserChat.java:355)
	at org.jivesoftware.smackx.muc.MultiUserChat.createOrJoin(MultiUserChat.java:498)
	at org.jivesoftware.smackx.muc.MultiUserChat.createOrJoin(MultiUserChat.java:444)
	at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.joinAs(ChatRoomImpl.java:240)
	... 17 more
    
Jicofo 2020-11-06 18:24:42.913 SCHWERWIEGEND: [16] org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.log() Failed to connect/login: The following addresses failed: 'localhost:5222' failed because: localhost/127.0.0.1 exception: java.net.ConnectException: Connection refused (Connection refused), localhost/0:0:0:0:0:0:0:1 exception: java.net.ConnectException: Connection refused (Connection refused)
org.jivesoftware.smack.SmackException$ConnectionException: The following addresses failed: 'localhost:5222' failed because: localhost/127.0.0.1 exception: java.net.ConnectException: Connection refused (Connection refused), localhost/0:0:0:0:0:0:0:1 exception: java.net.ConnectException: Connection refused (Connection refused)
	at org.jivesoftware.smack.SmackException$ConnectionException.from(SmackException.java:278)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection.connectUsingConfiguration(XMPPTCPConnection.java:619)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection.connectInternal(XMPPTCPConnection.java:902)
	at org.jivesoftware.smack.AbstractXMPPConnection.connect(AbstractXMPPConnection.java:383)
	at org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.doConnect(XmppProtocolProvider.java:269)
	at org.jitsi.retry.RetryStrategy$TaskRunner.run(RetryStrategy.java:193)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
    
Jicofo 2020-11-06 18:24:43.750 SCHWERWIEGEND: [29] org.jitsi.meet.ComponentMain.log() java.net.ConnectException: Connection refused (Connection refused), host:localhost, port:5347
org.xmpp.component.ComponentException: java.net.ConnectException: Connection refused (Connection refused)
	at org.jivesoftware.whack.ExternalComponent.connect(ExternalComponent.java:296)
	at org.jivesoftware.whack.ExternalComponentManager.addComponent(ExternalComponentManager.java:242)
	at org.jivesoftware.whack.ExternalComponentManager.addComponent(ExternalComponentManager.java:222)
	at org.jitsi.meet.ComponentMain.lambda$getConnectCallable$0(ComponentMain.java:285)
	at org.jitsi.retry.RetryStrategy$TaskRunner.run(RetryStrategy.java:193)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:607)
	at org.jivesoftware.whack.ExternalComponent.connect(ExternalComponent.java:174)
	... 11 more
    
Jicofo 2020-11-06 20:24:07.263 SCHWERWIEGEND: [277] org.jitsi.jicofo.bridge.JvbDoctor.log() Error when doing health-check on: jvbbrewery@internal.auth.meet.ballaeron.de/6325e46c-b4fe-49ae-b883-e8fc94b11326
net.java.sip.communicator.service.protocol.OperationFailedException: No response or failed otherwise: <iq to='jvbbrewery@internal.auth.meet.ballaeron.de/6325e46c-b4fe-49ae-b883-e8fc94b11326' id='AsnOp-2455' type='get'><healthcheck xmlns='http://jitsi.org/protocol/healthcheck'/></iq>
	at org.jitsi.impl.protocol.xmpp.XmppProtocolProvider$XmppConnectionAdapter.sendPacketAndGetReply(XmppProtocolProvider.java:667)
	at org.jitsi.jicofo.bridge.JvbDoctor$HealthCheckTask.doHealthCheck(JvbDoctor.java:251)
	at org.jitsi.jicofo.bridge.JvbDoctor$HealthCheckTask.run(JvbDoctor.java:198)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
	at org.jivesoftware.smack.StanzaCollector.nextResult(StanzaCollector.java:190)
	at org.jivesoftware.smack.StanzaCollector.nextResult(StanzaCollector.java:168)
	at org.jitsi.impl.protocol.xmpp.XmppProtocolProvider$XmppConnectionAdapter.sendPacketAndGetReply(XmppProtocolProvider.java:655)
	... 9 more

Hello MarvinX,
ran into the same Problem with the same errorlogs. Jitsi was running smooth on an older version (beginning of 2020). System is a virtual server ubuntu 18.04 (4 Core, 8GB) hostet by strato.de.
Did you find any solutions for this problem?
Kind regards
Jan

I have the same problem with my strato vserver.

Hello Jan,
unfortunately not. I did some research and it might be due to a threat leak but I wasn’t able to confirm it.

Not likely. The jvb message says it can’t authenticate to prosody. And the jicofo message says out of memory. Maybe it’s unrelated. Or jicofo has failed so many times because of the jvb problem it had eaten all memory. All these problems have possibly a root cause in the authentication problem of jvb (SASL error). What’s necessary at this point is to enable full debug log on prosody to see if it’s indeed a bad password.

Thanks for your reply and sorry that it took me so long.

I fixed the Error regarding the SASLError. After restarting, I got only Info and some Warning log messages. After a few hours though I got the OutOfMemoryError again (repeating).

free -h

root@lvps176-28-18-58:~# free -h
          total        used        free      shared  buff/cache   available
Mem:           2,0G        1,0G        695M         30M        293M        954M
Swap:            0B          0B          0B

JVB ps -o nlwp

root@server:~# ps -o nlwp 29867
NLWP
  45

Jconfo ps -o nlwp

root@server:~# ps -o nlwp 29951
NLWP
 268

I’m not sure if this actually has any meaning but the Task IDs shown in htop go up to 31000.
As described by other users above, I am using a V-sercer as well.

Do you have any ideas on this?
Thanks in advance!

so if I understand you correctly, it works normally for a few hours then you get the exact same error message in the jicofo log ? and the limits as displayed by systemctl status are higher than the number you get ?

BTW I see you have only 2 Gb Ram, you know that 4Gb are recommended because Jicofo and Jvb are configured by default to limits appropriate for this amount of memory right ?

exactly. I was able to join a conference with 3 devices and experienced no problems at all. The next day, meetings where interrupted. (probably due to the OutOfMemoryError) I connected via ssh and at this that time, I already often got an Resource temporarily unavailable error when trying to run some bash commands. Also this stayed even though no one was trying to create / join a meeting. I was only able to resolve this (temporary) by restarting the server.

The outputs I posten where taken every time before I restarted.

Interesting. As the Self-Hosting Guide mentions that the default values are enough for up to 20 participants and we use less than 20 I didn’t take a look into this at first. Now I noticed, that the actual value is very low.

root@server:~# systemctl show --property DefaultTasksMax
DefaultTasksMax=450

/etc/systemd/system.conf

[Manager]
# ...
DefaultTasksMax=90%

All other values are commented-out.
Do you know if there could be any good reasons for this and if it’s save to change the value?

yes there are probably good reasons such as avoiding fork bombs, and the problem with jicofo is that in the Debian packaging there is no systemd configuration so the system defaults are used (for videobridge there is an explicit systemd configuration setting the task max parameter to a high value).

90% imply that the absolute maximum of tasks for a process in your system would be 500. That seems absurdly low. On an inactive system jicofo takes about 300 tasks. Raise it to 2000.

How can I do so?
I know I can edit /etc/systemd/system.conf but 500 is not enough either. Or is this a limitation of V-server provider?

# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1540589
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 62987
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
# cat /proc/sys/kernel/pid_max
32768
# cat /proc/sys/kernel/threads-max
3081178

well, I said raise it to 2000, not 500 :slight_smile:

Maybe I understood something wrong but if the current value of 90% indicates a MaxTask of 500 (100%), would it have any effect to set it to 2000?

of course not, unless you reboot.

Ok so I updated it and changed it to 2000 and rebooted.

# systemctl show --property DefaultTasksMax
DefaultTasksMax=2000

However the problem remains the same :frowning:.
Any additional ideas?

I also noticed that after I stop all services with:

service jicofo stop
service jitsi-videobridge2 stop
service prosody stop

Some tasks of jicofo user are still running. (Showing up in top, showing many tasks in htop)

I uploaded the entire log files below. I just replaced my domain with domain.com and my IP with xxx.xxx.xxx.xxx

jicofo.log (241.5 KB)
jvb.log (195.8 KB)

Screenshot after I stopped all services. (There are many more entries of jiconfo)

It is recommended at least 8 GB for a typical Jitsi installation. If you don’t have enough RAM you should decrease the reserved RAM for jicofo and jvb

Add the following line to /etc/jitsi/jicofo/config

JICOFO_MAX_MEMORY=1024m

and the following to /etc/jitsi/videobridge/config

VIDEOBRIDGE_MAX_MEMORY=1024m

then restart the system.

what says
systemctl show jicofo --property TasksCurrent,TasksMax

After jitsi-meet is running for a few hours (when the error came up again):

# systemctl show jicofo --property TasksCurrent,TasksMax
TasksCurrent=273
TasksMax=2000

There was no meeting running between the reboot and the first Error.

Done so. No effect. (Error occurred after I joined a meeting with two devices)

Should it be VIDEOBRIDGE_MAX_MEMORY OR JVB_MAX_MEMORY?

is this vps a vm or a container, is there a
/proc/user_beancounters entry and if yes, what’s the value for numproc ?

First of all thanks a lot for the help and sorry for my long response time.

As I understand it’s a VM but I’m not 100% sure on that.

# cat /proc/user_beancounters | grep numproc
            numproc                       186                  186                  500                  500                    6

From what I can read on the Internet, it’s an OpenVZ container and it’s indeed limiting the handles count. It’s managed by your hoster, so changing the config at your level will not change anything.
The only possibility while keeping the same VPS would be to recompile Jicofo code to ask for less threads since it’s not a configurable parameter in Jitsi-meet.