Fork: retry: Resource temporarily unavailable

Hello,

I was successfully hosting jitsi-meet on our server for the last year until last week. I ran apt upgrade and restarted the server and since than I have a lot of trouble. I also tried to run apt purge and reinstall it - without success.

When connection to a meeting I get the message unfortunately something went wrong, we are trying to fix this. Additionally, some other services on the server are unreachable (like ssh).

Below I added some outputs and parts of the log file, if you would need to see the entire log file I can upload it as well - just need to remove personal data first.

Ubuntu 18.04.5 LTS

Jitsi Versions

root@server:~# dpkg -l | grep jitsi
ii  jitsi-meet                         2.0.5142-1                                        all          WebRTC JavaScript video conferences
ii  jitsi-meet-prosody                 1.0.4466-1                                        all          Prosody configuration for Jitsi Meet
ii  jitsi-meet-web                     1.0.4466-1                                        all          WebRTC JavaScript video conferences
ii  jitsi-meet-web-config              1.0.4466-1                                        all          Configuration for web serving of Jitsi Meet
ii  jitsi-videobridge2                 2.1-376-g9f12bfe2-1                               all          WebRTC compatible Selective Forwarding Unit (SFU)

free output

root@server:~# free -m
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
              total        used        free      shared  buff/cache   available
Mem:           2048         905         943          30         198        1110
Swap:             0           0           0

top output

top - 03:04:28 up 1 day,  7:52,  2 users,  load average: 0,00, 0,02, 0,05
Tasks:  71 total,   2 running,  69 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0,3 us,  0,5 sy,  0,0 ni, 99,2 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
KiB Mem :  2097152 total,   983488 free,   931328 used,   182336 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  1132376 avail Mem
# ...
 1698 jicofo    20   0 5939968 301048   8732 S   0,0 14,4   1:44.39 java
 1789 jvb       20   0 5679668 173216   6428 S   0,0  8,3   3:19.74 java

Threats of jicofo

root@server:~# ps -o nlwp 1698
NLWP
 271

Threats of jvb

root@server:~# ps -o nlwp 1789
NLWP
  41

JVB LOG

2020-11-06 01:05:28.344 EXCEPTION: [15] RetryStrategy$TaskRunner.run#198: org.jivesoftware.smack.sasl.SASLErrorException: SASLError using SCRAM-SHA-1: not-authorized
org.jivesoftware.smack.sasl.SASLErrorException: SASLError using SCRAM-SHA-1: not-authorized
	at org.jivesoftware.smack.SASLAuthentication.authenticationFailed(SASLAuthentication.java:292)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1100)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
	at java.lang.Thread.run(Thread.java:748)
    
2020-11-06 01:05:42.355 WARNUNG: [19] org.jivesoftware.smack.AbstractXMPPConnection.callConnectionClosedOnErrorListener: Connection XMPPTCPConnection[not-authenticated] (0) closed with error
org.jivesoftware.smack.XMPPException$StreamErrorException: system-shutdown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
<stream:error><system-shutdown xmlns='urn:ietf:params:xml:ns:xmpp-streams'/><text>Received SIGTERM</text></stream:error>
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1064)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
	at java.lang.Thread.run(Thread.java:748)

jconfo

# 20s intervall
Jicofo 2020-11-06 01:37:35.697 SCHWERWIEGEND: [185] org.jitsi.jicofo.Main.log() An uncaught exception occurred in thread=Thread[Jicofo Scheduledpool-3-thread-97,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:717)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1025)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
    
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:717)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
	at org.jivesoftware.smack.AbstractXMPPConnection.asyncGo(AbstractXMPPConnection.java:1725)
	at org.jivesoftware.smack.AbstractXMPPConnection.invokeStanzaCollectorsAndNotifyRecvListeners(AbstractXMPPConnection.java:1229)
	at org.jivesoftware.smack.AbstractXMPPConnection.processStanza(AbstractXMPPConnection.java:1112)
	at org.jivesoftware.smack.AbstractXMPPConnection.parseAndProcessStanza(AbstractXMPPConnection.java:1096)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection.access$500(XMPPTCPConnection.java:151)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1044)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
    
Jicofo 2020-11-06 17:55:33.038 INFORMATION: [72] org.jitsi.jicofo.FocusManager.log() Exception while trying to start the conference
net.java.sip.communicator.service.protocol.OperationFailedException: Failed to join the room
	at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.joinAs(ChatRoomImpl.java:298)
	at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.join(ChatRoomImpl.java:209)
	at org.jitsi.jicofo.JitsiMeetConferenceImpl.joinTheRoom(JitsiMeetConferenceImpl.java:581)
	at org.jitsi.jicofo.JitsiMeetConferenceImpl.start(JitsiMeetConferenceImpl.java:404)
	at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:465)
	at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:419)
	at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:394)
	at org.jitsi.jicofo.xmpp.FocusComponent.handleConferenceIq(FocusComponent.java:337)
	at org.jitsi.jicofo.xmpp.FocusComponent.handleIQSetImpl(FocusComponent.java:228)
	at org.jitsi.xmpp.component.ComponentBase.handleIQSet(ComponentBase.java:362)
	at org.xmpp.component.AbstractComponent.processIQRequest(AbstractComponent.java:515)
	at org.xmpp.component.AbstractComponent.processIQ(AbstractComponent.java:289)
	at org.xmpp.component.AbstractComponent.processQueuedPacket(AbstractComponent.java:239)
	at org.xmpp.component.AbstractComponent.access$100(AbstractComponent.java:81)
	at org.xmpp.component.AbstractComponent$PacketProcessor.run(AbstractComponent.java:1051)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.jivesoftware.smack.SmackException$NoResponseException: No response received within reply timeout. Timeout was 15000ms (~15s). Waited for response using: AndFilter: (StanzaTypeFilter: Presence, OrFilter: (AndFilter: (FromMatchesFilter (ignoreResourcepart): test123@conference.meet.ballaeron.de, MUCUserStatusCodeFilter: status=110), AndFilter: (FromMatchesFilter (full): test123@conference.meet.ballaeron.de/focus, StanzaIdFilter: id=CiIDe-24383, PresenceTypeFilter: type=error))).
	at org.jivesoftware.smack.SmackException$NoResponseException.newWith(SmackException.java:111)
	at org.jivesoftware.smack.SmackException$NoResponseException.newWith(SmackException.java:98)
	at org.jivesoftware.smack.StanzaCollector.nextResultOrThrow(StanzaCollector.java:260)
	at org.jivesoftware.smackx.muc.MultiUserChat.enter(MultiUserChat.java:355)
	at org.jivesoftware.smackx.muc.MultiUserChat.createOrJoin(MultiUserChat.java:498)
	at org.jivesoftware.smackx.muc.MultiUserChat.createOrJoin(MultiUserChat.java:444)
	at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.joinAs(ChatRoomImpl.java:240)
	... 17 more
    
Jicofo 2020-11-06 18:24:42.913 SCHWERWIEGEND: [16] org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.log() Failed to connect/login: The following addresses failed: 'localhost:5222' failed because: localhost/127.0.0.1 exception: java.net.ConnectException: Connection refused (Connection refused), localhost/0:0:0:0:0:0:0:1 exception: java.net.ConnectException: Connection refused (Connection refused)
org.jivesoftware.smack.SmackException$ConnectionException: The following addresses failed: 'localhost:5222' failed because: localhost/127.0.0.1 exception: java.net.ConnectException: Connection refused (Connection refused), localhost/0:0:0:0:0:0:0:1 exception: java.net.ConnectException: Connection refused (Connection refused)
	at org.jivesoftware.smack.SmackException$ConnectionException.from(SmackException.java:278)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection.connectUsingConfiguration(XMPPTCPConnection.java:619)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection.connectInternal(XMPPTCPConnection.java:902)
	at org.jivesoftware.smack.AbstractXMPPConnection.connect(AbstractXMPPConnection.java:383)
	at org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.doConnect(XmppProtocolProvider.java:269)
	at org.jitsi.retry.RetryStrategy$TaskRunner.run(RetryStrategy.java:193)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
    
Jicofo 2020-11-06 18:24:43.750 SCHWERWIEGEND: [29] org.jitsi.meet.ComponentMain.log() java.net.ConnectException: Connection refused (Connection refused), host:localhost, port:5347
org.xmpp.component.ComponentException: java.net.ConnectException: Connection refused (Connection refused)
	at org.jivesoftware.whack.ExternalComponent.connect(ExternalComponent.java:296)
	at org.jivesoftware.whack.ExternalComponentManager.addComponent(ExternalComponentManager.java:242)
	at org.jivesoftware.whack.ExternalComponentManager.addComponent(ExternalComponentManager.java:222)
	at org.jitsi.meet.ComponentMain.lambda$getConnectCallable$0(ComponentMain.java:285)
	at org.jitsi.retry.RetryStrategy$TaskRunner.run(RetryStrategy.java:193)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:607)
	at org.jivesoftware.whack.ExternalComponent.connect(ExternalComponent.java:174)
	... 11 more
    
Jicofo 2020-11-06 20:24:07.263 SCHWERWIEGEND: [277] org.jitsi.jicofo.bridge.JvbDoctor.log() Error when doing health-check on: jvbbrewery@internal.auth.meet.ballaeron.de/6325e46c-b4fe-49ae-b883-e8fc94b11326
net.java.sip.communicator.service.protocol.OperationFailedException: No response or failed otherwise: <iq to='jvbbrewery@internal.auth.meet.ballaeron.de/6325e46c-b4fe-49ae-b883-e8fc94b11326' id='AsnOp-2455' type='get'><healthcheck xmlns='http://jitsi.org/protocol/healthcheck'/></iq>
	at org.jitsi.impl.protocol.xmpp.XmppProtocolProvider$XmppConnectionAdapter.sendPacketAndGetReply(XmppProtocolProvider.java:667)
	at org.jitsi.jicofo.bridge.JvbDoctor$HealthCheckTask.doHealthCheck(JvbDoctor.java:251)
	at org.jitsi.jicofo.bridge.JvbDoctor$HealthCheckTask.run(JvbDoctor.java:198)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
	at org.jivesoftware.smack.StanzaCollector.nextResult(StanzaCollector.java:190)
	at org.jivesoftware.smack.StanzaCollector.nextResult(StanzaCollector.java:168)
	at org.jitsi.impl.protocol.xmpp.XmppProtocolProvider$XmppConnectionAdapter.sendPacketAndGetReply(XmppProtocolProvider.java:655)
	... 9 more

Hello MarvinX,
ran into the same Problem with the same errorlogs. Jitsi was running smooth on an older version (beginning of 2020). System is a virtual server ubuntu 18.04 (4 Core, 8GB) hostet by strato.de.
Did you find any solutions for this problem?
Kind regards
Jan

I have the same problem with my strato vserver.

Hello Jan,
unfortunately not. I did some research and it might be due to a threat leak but I wasn’t able to confirm it.

Not likely. The jvb message says it can’t authenticate to prosody. And the jicofo message says out of memory. Maybe it’s unrelated. Or jicofo has failed so many times because of the jvb problem it had eaten all memory. All these problems have possibly a root cause in the authentication problem of jvb (SASL error). What’s necessary at this point is to enable full debug log on prosody to see if it’s indeed a bad password.