Got SIGTERM in prosody. Please help - this is 2nd time in two days

So We have our own setup of jitsi-meet where we are running latest stable version of jitsi. So we had lot of meeting scheduled at one particular time and around 500 to 600 participants jumped to different meeting at same time. at that time jitsi meet kind of crashed. on digging up prosody logs this is what I got at same timestamp —

Aug 12 17:34:58 bosh2a4ad35c-9b53-4138-aff0-5dadc2f28ea1	info	Authenticated as 7dff43e5-eb3a-4c39-b059-002ceaa532bd@newton-jitsi-meet.theboloapp.com
Aug 12 17:34:58 mod_posix	warn	Received SIGTERM
Aug 12 17:34:58 startup	info	Shutting down: Received SIGTERM
Aug 12 17:34:58 c2s561bc28ea360	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc2719080	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc2b02c90	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc2c32600	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc2580760	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc26179b0	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc259c5f0	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc22d9ee0	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc2c94630	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc2d55d60	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc28ca3b0	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc2781a50	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc29e4280	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc1f38050	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc289fba0	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc2b94d20	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc2bffa50	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc24a6740	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc2bde000	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc2cd78d0	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc2467dd0	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc2bef3b0	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc2aac440	info	Client disconnected: connection closed
Aug 12 17:34:58 c2s561bc2ddb110	info	Client disconnected: connection closed

This is logs from Jicofo during same timestamp.

Jicofo 2020-08-12 17:34:58.398 WARNING: [55] org.jivesoftware.smack.AbstractXMPPConnection.callConnectionClosedOnErrorListener() Connection XMPPTCPConnection[focus@auth.newton-jitsi-meet.theboloapp.com/focus35193334613483] (0) closed with error
org.jivesoftware.smack.XMPPException$StreamErrorException: system-shutdown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
<stream:error><system-shutdown xmlns='urn:ietf:params:xml:ns:xmpp-streams'/><text>Received SIGTERM</text></stream:error>
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1064)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
	at java.lang.Thread.run(Thread.java:748)
Jicofo 2020-08-12 17:34:58.398 SEVERE: [55] org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.log() XMPP connection closed on error: system-shutdown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
<stream:error><system-shutdown xmlns='urn:ietf:params:xml:ns:xmpp-streams'/><text>Received SIGTERM</text></stream:error>
Jicofo 2020-08-12 17:34:58.398 INFO: [55] org.jitsi.jicofo.ProtocolProviderHandler.log() XmppProtocolProvider(focus@auth.newton-jitsi-meet.theboloapp.com/focus35193334613483 (Jabber)): RegistrationStateChangeEvent[ oldState=Registered; newState=RegistrationState=Unregistered; reasonCode=-1; reason=null]
Jicofo 2020-08-12 17:34:58.398 INFO: [55] org.jitsi.jicofo.xmpp.BaseBrewery.log() Left brewery room: JibriBrewery@internal.auth.newton-jitsi-meet.theboloapp.com
Jicofo 2020-08-12 17:34:58.399 INFO: [55] org.jitsi.jicofo.xmpp.BaseBrewery.log() Removed brewery instance: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-188835780
Jicofo 2020-08-12 17:34:58.399 INFO: [55] org.jitsi.jicofo.recording.jibri.JibriDetector.log() Jibri: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-188835780 went offline
Jicofo 2020-08-12 17:34:58.399 INFO: [55] org.jitsi.jicofo.xmpp.BaseBrewery.log() Removed brewery instance: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-622894515
Jicofo 2020-08-12 17:34:58.399 INFO: [55] org.jitsi.jicofo.recording.jibri.JibriDetector.log() Jibri: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-622894515 went offline
Jicofo 2020-08-12 17:34:58.399 SEVERE: [300] org.jitsi.jicofo.recording.jibri.JibriSession.log() Jibri went offline: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-188835780 for room: meeting-puglz0sl67@conference.newton-jitsi-meet.theboloapp.com
Jicofo 2020-08-12 17:34:58.399 INFO: [55] org.jitsi.jicofo.xmpp.BaseBrewery.log() Removed brewery instance: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-441409261
Jicofo 2020-08-12 17:34:58.399 INFO: [55] org.jitsi.jicofo.recording.jibri.JibriDetector.log() Jibri: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-441409261 went offline
Jicofo 2020-08-12 17:34:58.399 INFO: [300] org.jitsi.jicofo.recording.jibri.JibriSession.log() Got Jibri status update: Jibri jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-188835780 has status off and failure reason error, current Jibri jid is jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-188835780
Jicofo 2020-08-12 17:34:58.399 INFO: [55] org.jitsi.jicofo.xmpp.BaseBrewery.log() Removed brewery instance: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-446632251
Jicofo 2020-08-12 17:34:58.399 INFO: [55] org.jitsi.jicofo.recording.jibri.JibriDetector.log() Jibri: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-446632251 went offline
Jicofo 2020-08-12 17:34:58.399 INFO: [300] org.jitsi.jicofo.recording.jibri.JibriSession.log() Jibri failed, trying to fall back to another Jibri
Jicofo 2020-08-12 17:34:58.399 INFO: [55] org.jitsi.jicofo.xmpp.BaseBrewery.log() Removed brewery instance: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-432950185
Jicofo 2020-08-12 17:34:58.399 INFO: [55] org.jitsi.jicofo.recording.jibri.JibriDetector.log() Jibri: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-432950185 went offline
Jicofo 2020-08-12 17:34:58.400 INFO: [300] org.jitsi.jicofo.recording.jibri.JibriSession.log() Starting session with Jibri jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-467227243
Jicofo 2020-08-12 17:34:58.400 INFO: [300] org.jitsi.jicofo.recording.jibri.JibriSession.log() Starting Jibri jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-467227243 for stream ID: null in room: meeting-puglz0sl67@conference.newton-jitsi-meet.theboloapp.com
Jicofo 2020-08-12 17:34:58.400 INFO: [55] org.jitsi.jicofo.xmpp.BaseBrewery.log() Removed brewery instance: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-434468987
Jicofo 2020-08-12 17:34:58.400 INFO: [55] org.jitsi.jicofo.recording.jibri.JibriDetector.log() Jibri: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-434468987 went offline
Jicofo 2020-08-12 17:34:58.400 INFO: [55] org.jitsi.jicofo.xmpp.BaseBrewery.log() Removed brewery instance: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-448671679
Jicofo 2020-08-12 17:34:58.400 INFO: [55] org.jitsi.jicofo.recording.jibri.JibriDetector.log() Jibri: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-448671679 went offline
Jicofo 2020-08-12 17:34:58.400 INFO: [55] org.jitsi.jicofo.xmpp.BaseBrewery.log() Removed brewery instance: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-454781760
Jicofo 2020-08-12 17:34:58.400 INFO: [55] org.jitsi.jicofo.recording.jibri.JibriDetector.log() Jibri: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-454781760 went offline
Jicofo 2020-08-12 17:34:58.400 SEVERE: [300] org.jitsi.jicofo.recording.jibri.JibriSession.log() Failed to send start Jibri IQ: net.java.sip.communicator.service.protocol.OperationFailedException: No connection - unable to send packet: <iq to='jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-467227243' id='7OBYv-21420' type='set'><jibri xmlns='http://jitsi.org/protocol/jibri' action='start' recording_mode='file' room='meeting-puglz0sl67@conference.newton-jitsi-meet.theboloapp.com' session_id='rakejqjvjylllvpp' app_data='{"file_recording_metadata":{}}'/></iq>
net.java.sip.communicator.service.protocol.OperationFailedException: No connection - unable to send packet: <iq to='jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-467227243' id='7OBYv-21420' type='set'><jibri xmlns='http://jitsi.org/protocol/jibri' action='start' recording_mode='file' room='meeting-puglz0sl67@conference.newton-jitsi-meet.theboloapp.com' session_id='rakejqjvjylllvpp' app_data='{"file_recording_metadata":{}}'/></iq>
	at org.jitsi.impl.protocol.xmpp.XmppProtocolProvider$XmppConnectionAdapter.sendPacketAndGetReply(XmppProtocolProvider.java:663)
	at org.jitsi.jicofo.recording.jibri.JibriSession.sendJibriStartIq(JibriSession.java:574)
	at org.jitsi.jicofo.recording.jibri.JibriSession.startInternal(JibriSession.java:370)
	at org.jitsi.jicofo.recording.jibri.JibriSession.start(JibriSession.java:335)
	at org.jitsi.jicofo.recording.jibri.JibriSession.retryRequestWithAnotherJibri(JibriSession.java:642)
	at org.jitsi.jicofo.recording.jibri.JibriSession.handleJibriStatusUpdate(JibriSession.java:712)
	at org.jitsi.jicofo.recording.jibri.JibriSession.access$600(JibriSession.java:49)
	at org.jitsi.jicofo.recording.jibri.JibriSession$JibriEventHandler.handleEvent(JibriSession.java:820)
	at org.jitsi.eventadmin.EventAdminImpl.callEventHandler(EventAdminImpl.java:209)
	at org.jitsi.eventadmin.EventAdminImpl.access$000(EventAdminImpl.java:35)
	at org.jitsi.eventadmin.EventAdminImpl$1.run(EventAdminImpl.java:189)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.jivesoftware.smack.SmackException$NotConnectedException: The connection XMPPTCPConnection[focus@auth.newton-jitsi-meet.theboloapp.com/focus35193334613483] (0) is no longer connected. done=true smResumptionPossible=false
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketWriter.throwNotConnectedExceptionIfDoneAndResumptionNotPossible(XMPPTCPConnection.java:1327)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection.throwNotConnectedExceptionIfAppropriate(XMPPTCPConnection.java:358)
	at org.jivesoftware.smack.AbstractXMPPConnection.sendStanza(AbstractXMPPConnection.java:670)
	at org.jivesoftware.smack.AbstractXMPPConnection.createStanzaCollectorAndSend(AbstractXMPPConnection.java:776)
	at org.jivesoftware.smack.AbstractXMPPConnection.createStanzaCollectorAndSend(AbstractXMPPConnection.java:765)
	at org.jitsi.impl.protocol.xmpp.XmppProtocolProvider$XmppConnectionAdapter.sendPacketAndGetReply(XmppProtocolProvider.java:640)
	... 15 more
Jicofo 2020-08-12 17:34:58.400 INFO: [55] org.jitsi.jicofo.xmpp.BaseBrewery.log() Removed brewery instance: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-587922889
Jicofo 2020-08-12 17:34:58.400 INFO: [55] org.jitsi.jicofo.recording.jibri.JibriDetector.log() Jibri: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-587922889 went offline
Jicofo 2020-08-12 17:34:58.400 INFO: [300] org.jitsi.jicofo.recording.jibri.JibriSession.log() Failed to fall back to another Jibri, this session has now failed: org.jitsi.jicofo.recording.jibri.JibriSession$StartException: Internal server error
org.jitsi.jicofo.recording.jibri.JibriSession$StartException: Internal server error
	at org.jitsi.jicofo.recording.jibri.JibriSession.startInternal(JibriSession.java:376)
	at org.jitsi.jicofo.recording.jibri.JibriSession.start(JibriSession.java:335)
	at org.jitsi.jicofo.recording.jibri.JibriSession.retryRequestWithAnotherJibri(JibriSession.java:642)
	at org.jitsi.jicofo.recording.jibri.JibriSession.handleJibriStatusUpdate(JibriSession.java:712)
	at org.jitsi.jicofo.recording.jibri.JibriSession.access$600(JibriSession.java:49)
	at org.jitsi.jicofo.recording.jibri.JibriSession$JibriEventHandler.handleEvent(JibriSession.java:820)
	at org.jitsi.eventadmin.EventAdminImpl.callEventHandler(EventAdminImpl.java:209)
	at org.jitsi.eventadmin.EventAdminImpl.access$000(EventAdminImpl.java:35)
	at org.jitsi.eventadmin.EventAdminImpl$1.run(EventAdminImpl.java:189)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Jicofo 2020-08-12 17:34:58.400 INFO: [55] org.jitsi.jicofo.xmpp.BaseBrewery.log() Removed brewery instance: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-447785272
Jicofo 2020-08-12 17:34:58.401 INFO: [55] org.jitsi.jicofo.recording.jibri.JibriDetector.log() Jibri: jibribrewery@internal.auth.newton-jitsi-meet.theboloapp.com/jibri-instanse-447785272 went offline
Jicofo 2020-08-12 17:34:58.401 INFO: [300] org.jitsi.jicofo.recording.jibri.JibriRecorder.log() Got jibri status off and failure error

Can someone please help?? I am running latest stable version of jitsi-meet in ubuntu 18.04 installed using apt-get.

What is the available RAM on the machine?

@damencho

for server where jicofo and prosody runs it is 8GB and 4 core
for each video bridge it is 4GB and 2 core - around 10 were running when this happened
for each jibri each is 4GB and 2 core - around 50 were running when this happened

thanks

I upgraded to new stable version from old unstable version two days ago only. I never faced such issue in old unstable version.

I was suspecting not enough memory for prosody and the systemd restarts it, but that should be enough for both … do you have monitoring of the memory, how does it look like when that happen.

since it happened yesterday, i was suspecting it to happen again so i was actively monitoring it and RAM being used was around 1.5GB when this happened. other thing i noted was once i restart prosody, everything works fine but jibri instances are not available for recording.

So you did the restart? I kind of got lost here, so what was the problem that made you restart it?
The error ...StreamErrorException: system-shutdown... is normally seen when prosody is being restarted …
So the first restart from the logs above at 17:34:58 is not done by you, and conferences are not working and you need to restart prosody again?

so system went down, platform was not responding. nobody was able to join. those who have joined saw a connecting screen. then i restarted prosody and everything was fine except recordings were not getting started which maybe the case that jibri was not registering itself after prosody restart.

@damencho if you need more logs around same time, I can provide you with that too. thanks

If this happen check js console why clients so the reload, check nginx config and prosody and jicofo configs for anything unusual around that time.
The logs when you did the restart are not very useful. Jibri should reconnect after the restart, check jibri logs if that do not happen.

sure will do that, today also there are bunch of meetings at same time. if jitsi crashed will have all the logs ready. thanks

hey @damencho

So today I did not face issue where everything crashed.

only thing that i changed was I was not auto starting recording as soon as moderator of meeting joined meeting. I used to start recording using startRecording command of iframeApi.

i started a session before peak traffic so I can get logs if things crashed. I joined session using three different devices and started recording. So during peak traffic, one of my instance was disconnected but joined again after it showed reconnecting screen, unlikely i closed tab without copying logs. also when peak arroived, recording stoped. So I tried to start recording again but recording was not getting started though I was having nearly 50 jibri instances running. this is only error log from jibri that i could find –

2020-08-13 17:35:58.089 SEVERE: [3039] org.jitsi.jibri.selenium.JibriSelenium.run() An error occurred while joining the call: org.openqa.selenium.WebDriverException: <unknown>: Failed to read the 'localStorage' property from 'Window': Access is denied for this document.
  (Session info: chrome=78.0.3904.97)
  (Driver info: chromedriver=78.0.3904.105 (60e2d8774a8151efa6a00b1f358371b1e0e07ee2-refs/branch-heads/3904@{#877}),platform=Linux 4.15.0-109-generic x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 0 milliseconds
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: 'a068b55ad9c6', ip: '172.18.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '4.15.0-109-generic', java.version: '1.8.0_252'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Capabilities {acceptInsecureCerts: false, acceptSslCerts: false, applicationCacheEnabled: false, browserConnectionEnabled: false, browserName: chrome, chrome: {chromedriverVersion: 78.0.3904.105 (60e2d8774a81..., userDataDir: /tmp/.com.google.Chrome.c1viuO}, cssSelectorsEnabled: true, databaseEnabled: false, goog:chromeOptions: {debuggerAddress: localhost:46217}, handlesAlerts: true, hasTouchScreen: false, javascriptEnabled: true, locationContextEnabled: true, mobileEmulationEnabled: false, nativeEvents: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, proxy: Proxy(), rotatable: false, setWindowRect: true, strictFileInteractability: false, takesHeapSnapshot: true, takesScreenshot: true, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unexpectedAlertBehaviour: ignore, unhandledPromptBehavior: ignore, version: 78.0.3904.97, webStorageEnabled: true}
Session ID: fe84ccc58f2e6afeff90b5a8fb6f99fb with stack: 
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:214)
org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:166)
org.openqa.selenium.remote.http.JsonHttpResponseCodec.reconstructValue(JsonHttpResponseCodec.java:40)
org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:80)
org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:44)
org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:83)
org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:543)
org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:480)
org.jitsi.jibri.selenium.JibriSelenium.setLocalStorageValues(JibriSelenium.kt:175)
org.jitsi.jibri.selenium.JibriSelenium.access$setLocalStorageValues(JibriSelenium.kt:112)
org.jitsi.jibri.selenium.JibriSelenium$joinCall$1.run(JibriSelenium.kt:255)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

these are the only error log that I got from browser console that I did not close

2020-08-13T17:10:37.970Z [modules/RTC/TraceablePeerConnection.js] <R.close>:  Closing TPC[3,p2p:true]...
Logger.js:154 2020-08-13T17:10:38.001Z [modules/xmpp/SdpConsistency.js] <o.makeVideoPrimarySsrcsConsistent>:  TPC[1,p2p:false] no SSRC found for the recvonly videostream!
Logger.js:154 2020-08-13T17:10:38.008Z [JitsiConference.js] Resumed media transfer over the JVB connection!
Logger.js:154 2020-08-13T17:10:38.131Z [JitsiMeetJS.js] <Object.getGlobalOnErrorHandler>:  UnhandledError: null Script: null Line: null Column: null StackTrace:  Error: Strophe: TypeError: Cannot read property 'addListener' of null
    at s._onMediaSessionStarted (QualityController.js:37)
    at a.<anonymous> (QualityController.js:22)
    at a.emit (events.js:152)
    at JitsiConference.js:2669
    at I.Handler.handler (strophe.umd.js:2712)
    at I.Handler.run (strophe.umd.js:1875)
    at strophe.umd.js:3157
    at Object.forEachChild (strophe.umd.js:830)
    at I.Connection._dataRecv (strophe.umd.js:3146)
    at P.Bosh._onRequestStateChange (strophe.umd.js:5012)
    at Object.r.Strophe.log (strophe.util.js:89)
    at Object.fatal (strophe.umd.js:1402)
    at Object._handleError (strophe.umd.js:1308)
    at I.Handler.run (strophe.umd.js:1877)
    at strophe.umd.js:3157
    at Object.forEachChild (strophe.umd.js:830)
    at I.Connection._dataRecv (strophe.umd.js:3146)
    at P.Bosh._onRequestStateChange (strophe.umd.js:5012)
o @ Logger.js:154
getGlobalOnErrorHandler @ JitsiMeetJS.js:609
window.onerror @ middleware.js:107
callErrorHandler @ GlobalOnErrorHandler.js:61
r.Strophe.log @ strophe.util.js:89
fatal @ strophe.umd.js:1402
_handleError @ strophe.umd.js:1308
run @ strophe.umd.js:1877
(anonymous) @ strophe.umd.js:3157
forEachChild @ strophe.umd.js:830
_dataRecv @ strophe.umd.js:3146
_onRequestStateChange @ strophe.umd.js:5012
XMLHttpRequest.send (async)
d @ strophe.umd.js:5123
_processRequest @ strophe.umd.js:5137
_throttledRequestHandler @ strophe.umd.js:5290
_onIdle @ strophe.umd.js:4901
_onIdle @ strophe.umd.js:3881
(anonymous) @ strophe.umd.js:5257
setTimeout (async)
_send @ strophe.umd.js:5256
send @ strophe.umd.js:2583
sendIQ @ strophe.umd.js:2740
sendIQ @ XmppConnection.js:420
terminate @ JingleSessionPC.js:1469
ie._stopP2PSession @ JitsiConference.js:3176
ie._maybeStartOrStopP2P @ JitsiConference.js:3120
ie.onMemberJoined @ JitsiConference.js:1499
a.emit @ events.js:152
onPresence @ ChatRoom.js:592
onPresence @ strophe.emuc.js:103
run @ strophe.umd.js:1875
(anonymous) @ strophe.umd.js:3157
forEachChild @ strophe.umd.js:830
_dataRecv @ strophe.umd.js:3146
_onRequestStateChange @ strophe.umd.js:5012
XMLHttpRequest.send (async)
d @ strophe.umd.js:5123
_processRequest @ strophe.umd.js:5137
_throttledRequestHandler @ strophe.umd.js:5290
_onIdle @ strophe.umd.js:4901
_onIdle @ strophe.umd.js:3881
(anonymous) @ strophe.umd.js:5257
setTimeout (async)
_send @ strophe.umd.js:5256
send @ strophe.umd.js:2583
send @ XmppConnection.js:400
onJingle @ strophe.jingle.js:246
run @ strophe.umd.js:1875
(anonymous) @ strophe.umd.js:3157
forEachChild @ strophe.umd.js:830
_dataRecv @ strophe.umd.js:3146
_onRequestStateChange @ strophe.umd.js:5012
XMLHttpRequest.send (async)
d @ strophe.umd.js:5123
_processRequest @ strophe.umd.js:5137
_throttledRequestHandler @ strophe.umd.js:5290
_onIdle @ strophe.umd.js:4901
_onIdle @ strophe.umd.js:3881
(anonymous) @ strophe.umd.js:5257
setTimeout (async)
_send @ strophe.umd.js:5256
send @ strophe.umd.js:2583
sendIQ @ strophe.umd.js:2740
sendIQ @ XmppConnection.js:420
sendIceCandidates @ JingleSessionPC.js:729
(anonymous) @ JingleSessionPC.js:638
setTimeout (async)
sendIceCandidate @ JingleSessionPC.js:634
peerconnection.onicecandidate @ JingleSessionPC.js:424
R.peerconnection.onicecandidate @ TraceablePeerConnection.js:269
Logger.js:154 2020-08-13T17:10:38.131Z [modules/xmpp/strophe.util.js] <Object.r.Strophe.log>:  Strophe: TypeError: Cannot read property 'addListener' of null
    at s._onMediaSessionStarted (https://newton-jitsi-meet.theboloapp.com/libs/lib-jitsi-meet.min.js?v=4289:17:76417)
    at a.<anonymous> (https://newton-jitsi-meet.theboloapp.com/libs/lib-jitsi-meet.min.js?v=4289:17:76270)
    at a.emit (https://newton-jitsi-meet.theboloapp.com/libs/lib-jitsi-meet.min.js?v=4289:1:129023)
    at https://newton-jitsi-meet.theboloapp.com/libs/lib-jitsi-meet.min.js?v=4289:10:185750
    at I.Handler.handler (https://newton-jitsi-meet.theboloapp.com/libs/lib-jitsi-meet.min.js?v=4289:1:26339)
    at I.Handler.run (https://newton-jitsi-meet.theboloapp.com/libs/lib-jitsi-meet.min.js?v=4289:1:21746)
    at https://newton-jitsi-meet.theboloapp.com/libs/lib-jitsi-meet.min.js?v=4289:1:30184
    at Object.forEachChild (https://newton-jitsi-meet.theboloapp.com/libs/lib-jitsi-meet.min.js?v=4289:1:13408)
    at I.Connection._dataRecv (https://newton-jitsi-meet.theboloapp.com/libs/lib-jitsi-meet.min.js?v=4289:1:30033)
    at P.Bosh._onRequestStateChange (https://newton-jitsi-meet.theboloapp.com/libs/lib-jitsi-meet.min.js?v=4289:1:50018)
o @ Logger.js:154
r.Strophe.log @ strophe.util.js:90
fatal @ strophe.umd.js:1402
_handleError @ strophe.umd.js:1308
run @ strophe.umd.js:1877
(anonymous) @ strophe.umd.js:3157
forEachChild @ strophe.umd.js:830
_dataRecv @ strophe.umd.js:3146
_onRequestStateChange @ strophe.umd.js:5012
XMLHttpRequest.send (async)
d @ strophe.umd.js:5123
_processRequest @ strophe.umd.js:5137
_throttledRequestHandler @ strophe.umd.js:5290
_onIdle @ strophe.umd.js:4901
_onIdle @ strophe.umd.js:3881
(anonymous) @ strophe.umd.js:5257
setTimeout (async)
_send @ strophe.umd.js:5256
send @ strophe.umd.js:2583
sendIQ @ strophe.umd.js:2740
sendIQ @ XmppConnection.js:420
terminate @ JingleSessionPC.js:1469
ie._stopP2PSession @ JitsiConference.js:3176
ie._maybeStartOrStopP2P @ JitsiConference.js:3120
ie.onMemberJoined @ JitsiConference.js:1499
a.emit @ events.js:152
onPresence @ ChatRoom.js:592
onPresence @ strophe.emuc.js:103
run @ strophe.umd.js:1875
(anonymous) @ strophe.umd.js:3157
forEachChild @ strophe.umd.js:830
_dataRecv @ strophe.umd.js:3146
_onRequestStateChange @ strophe.umd.js:5012
XMLHttpRequest.send (async)
d @ strophe.umd.js:5123
_processRequest @ strophe.umd.js:5137
_throttledRequestHandler @ strophe.umd.js:5290
_onIdle @ strophe.umd.js:4901
_onIdle @ strophe.umd.js:3881
(anonymous) @ strophe.umd.js:5257
setTimeout (async)
_send @ strophe.umd.js:5256
send @ strophe.umd.js:2583
send @ XmppConnection.js:400
onJingle @ strophe.jingle.js:246
run @ strophe.umd.js:1875
(anonymous) @ strophe.umd.js:3157
forEachChild @ strophe.umd.js:830
_dataRecv @ strophe.umd.js:3146
_onRequestStateChange @ strophe.umd.js:5012
XMLHttpRequest.send (async)
d @ strophe.umd.js:5123
_processRequest @ strophe.umd.js:5137
_throttledRequestHandler @ strophe.umd.js:5290
_onIdle @ strophe.umd.js:4901
_onIdle @ strophe.umd.js:3881
(anonymous) @ strophe.umd.js:5257
setTimeout (async)
_send @ strophe.umd.js:5256
send @ strophe.umd.js:2583
sendIQ @ strophe.umd.js:2740
sendIQ @ XmppConnection.js:420
sendIceCandidates @ JingleSessionPC.js:729
(anonymous) @ JingleSessionPC.js:638
setTimeout (async)
sendIceCandidate @ JingleSessionPC.js:634
peerconnection.onicecandidate @ JingleSessionPC.js:424
R.peerconnection.onicecandidate @ TraceablePeerConnection.js:269
Logger.js:154 2020-08-13T17:10:38.132Z [JitsiMeetJS.js] <Object.getGlobalOnErrorHandler>:  UnhandledError: null Script: null Line: null Column: null StackTrace:  Error: Strophe: error: Cannot read property 'addListener' of null
    at Object.r.Strophe.log (strophe.util.js:89)
    at Object.fatal (strophe.umd.js:1402)
    at Object._handleError (strophe.umd.js:1316)
    at I.Handler.run (strophe.umd.js:1877)
    at strophe.umd.js:3157
    at Object.forEachChild (strophe.umd.js:830)
    at I.Connection._dataRecv (strophe.umd.js:3146)
    at P.Bosh._onRequestStateChange (strophe.umd.js:5012)
o @ Logger.js:154
getGlobalOnErrorHandler @ JitsiMeetJS.js:609
window.onerror @ middleware.js:107
callErrorHandler @ GlobalOnErrorHandler.js:61
r.Strophe.log @ strophe.util.js:89
fatal @ strophe.umd.js:1402
_handleError @ strophe.umd.js:1316
run @ strophe.umd.js:1877
(anonymous) @ strophe.umd.js:3157
forEachChild @ strophe.umd.js:830
_dataRecv @ strophe.umd.js:3146
_onRequestStateChange @ strophe.umd.js:5012
XMLHttpRequest.send (async)
d @ strophe.umd.js:5123
_processRequest @ strophe.umd.js:5137
_throttledRequestHandler @ strophe.umd.js:5290
_onIdle @ strophe.umd.js:4901
_onIdle @ strophe.umd.js:3881
(anonymous) @ strophe.umd.js:5257
setTimeout (async)
_send @ strophe.umd.js:5256
send @ strophe.umd.js:2583
sendIQ @ strophe.umd.js:2740
sendIQ @ XmppConnection.js:420
terminate @ JingleSessionPC.js:1469
ie._stopP2PSession @ JitsiConference.js:3176
ie._maybeStartOrStopP2P @ JitsiConference.js:3120
ie.onMemberJoined @ JitsiConference.js:1499
a.emit @ events.js:152
onPresence @ ChatRoom.js:592
onPresence @ strophe.emuc.js:103
run @ strophe.umd.js:1875
(anonymous) @ strophe.umd.js:3157
forEachChild @ strophe.umd.js:830
_dataRecv @ strophe.umd.js:3146
_onRequestStateChange @ strophe.umd.js:5012
XMLHttpRequest.send (async)
d @ strophe.umd.js:5123
_processRequest @ strophe.umd.js:5137
_throttledRequestHandler @ strophe.umd.js:5290
_onIdle @ strophe.umd.js:4901
_onIdle @ strophe.umd.js:3881
(anonymous) @ strophe.umd.js:5257
setTimeout (async)
_send @ strophe.umd.js:5256
send @ strophe.umd.js:2583
send @ XmppConnection.js:400
onJingle @ strophe.jingle.js:246
run @ strophe.umd.js:1875
(anonymous) @ strophe.umd.js:3157
forEachChild @ strophe.umd.js:830
_dataRecv @ strophe.umd.js:3146
_onRequestStateChange @ strophe.umd.js:5012
XMLHttpRequest.send (async)
d @ strophe.umd.js:5123
_processRequest @ strophe.umd.js:5137
_throttledRequestHandler @ strophe.umd.js:5290
_onIdle @ strophe.umd.js:4901
_onIdle @ strophe.umd.js:3881
(anonymous) @ strophe.umd.js:5257
setTimeout (async)
_send @ strophe.umd.js:5256
send @ strophe.umd.js:2583
sendIQ @ strophe.umd.js:2740
sendIQ @ XmppConnection.js:420
sendIceCandidates @ JingleSessionPC.js:729
(anonymous) @ JingleSessionPC.js:638
setTimeout (async)
sendIceCandidate @ JingleSessionPC.js:634
peerconnection.onicecandidate @ JingleSessionPC.js:424
R.peerconnection.onicecandidate @ TraceablePeerConnection.js:269
Logger.js:154 2020-08-13T17:10:38.132Z [modules/xmpp/strophe.util.js] <Object.r.Strophe.log>:  Strophe: error: Cannot read property 'addListener' of null
o @ Logger.js:154
r.Strophe.log @ strophe.util.js:90
fatal @ strophe.umd.js:1402
_handleError @ strophe.umd.js:1316
run @ strophe.umd.js:1877
(anonymous) @ strophe.umd.js:3157
forEachChild @ strophe.umd.js:830
_dataRecv @ strophe.umd.js:3146
_onRequestStateChange @ strophe.umd.js:5012
XMLHttpRequest.send (async)
d @ strophe.umd.js:5123
_processRequest @ strophe.umd.js:5137
_throttledRequestHandler @ strophe.umd.js:5290
_onIdle @ strophe.umd.js:4901
_onIdle @ strophe.umd.js:3881
(anonymous) @ strophe.umd.js:5257
setTimeout (async)
_send @ strophe.umd.js:5256
send @ strophe.umd.js:2583
sendIQ @ strophe.umd.js:2740
sendIQ @ XmppConnection.js:420
terminate @ JingleSessionPC.js:1469
ie._stopP2PSession @ JitsiConference.js:3176
ie._maybeStartOrStopP2P @ JitsiConference.js:3120
ie.onMemberJoined @ JitsiConference.js:1499
a.emit @ events.js:152
onPresence @ ChatRoom.js:592
onPresence @ strophe.emuc.js:103
run @ strophe.umd.js:1875
(anonymous) @ strophe.umd.js:3157
forEachChild @ strophe.umd.js:830
_dataRecv @ strophe.umd.js:3146
_onRequestStateChange @ strophe.umd.js:5012
XMLHttpRequest.send (async)
d @ strophe.umd.js:5123
_processRequest @ strophe.umd.js:5137
_throttledRequestHandler @ strophe.umd.js:5290
_onIdle @ strophe.umd.js:4901
_onIdle @ strophe.umd.js:3881
(anonymous) @ strophe.umd.js:5257
setTimeout (async)
_send @ strophe.umd.js:5256
send @ strophe.umd.js:2583
send @ XmppConnection.js:400
onJingle @ strophe.jingle.js:246
run @ strophe.umd.js:1875
(anonymous) @ strophe.umd.js:3157
forEachChild @ strophe.umd.js:830
_dataRecv @ strophe.umd.js:3146
_onRequestStateChange @ strophe.umd.js:5012
XMLHttpRequest.send (async)
d @ strophe.umd.js:5123
_processRequest @ strophe.umd.js:5137
_throttledRequestHandler @ strophe.umd.js:5290
_onIdle @ strophe.umd.js:4901
_onIdle @ strophe.umd.js:3881
(anonymous) @ strophe.umd.js:5257
setTimeout (async)
_send @ strophe.umd.js:5256
send @ strophe.umd.js:2583
sendIQ @ strophe.umd.js:2740
sendIQ @ XmppConnection.js:420
sendIceCandidates @ JingleSessionPC.js:729
(anonymous) @ JingleSessionPC.js:638
setTimeout (async)
sendIceCandidate @ JingleSessionPC.js:634
peerconnection.onicecandidate @ JingleSessionPC.js:424
R.peerconnection.onicecandidate @ TraceablePeerConnection.js:269
Logger.js:154 2020-08-13T17:10:38.132Z [modules/xmpp/strophe.util.js] <Object.r.Strophe.log>:  Strophe: Removing Strophe handlers due to uncaught exception: Cannot read property 'addListener' of null
o @ Logger.js:154
r.Strophe.log @ strophe.util.js:77
warn @ strophe.umd.js:1382
(anonymous) @ strophe.umd.js:3165
forEachChild @ strophe.umd.js:830
_dataRecv @ strophe.umd.js:3146
_onRequestStateChange @ strophe.umd.js:5012
XMLHttpRequest.send (async)
d @ strophe.umd.js:5123
_processRequest @ strophe.umd.js:5137
_throttledRequestHandler @ strophe.umd.js:5290
_onIdle @ strophe.umd.js:4901
_onIdle @ strophe.umd.js:3881
(anonymous) @ strophe.umd.js:5257
setTimeout (async)
_send @ strophe.umd.js:5256
send @ strophe.umd.js:2583
sendIQ @ strophe.umd.js:2740
sendIQ @ XmppConnection.js:420
terminate @ JingleSessionPC.js:1469
ie._stopP2PSession @ JitsiConference.js:3176
ie._maybeStartOrStopP2P @ JitsiConference.js:3120
ie.onMemberJoined @ JitsiConference.js:1499
a.emit @ events.js:152
onPresence @ ChatRoom.js:592
onPresence @ strophe.emuc.js:103
run @ strophe.umd.js:1875
(anonymous) @ strophe.umd.js:3157
forEachChild @ strophe.umd.js:830
_dataRecv @ strophe.umd.js:3146
_onRequestStateChange @ strophe.umd.js:5012
XMLHttpRequest.send (async)
d @ strophe.umd.js:5123
_processRequest @ strophe.umd.js:5137
_throttledRequestHandler @ strophe.umd.js:5290
_onIdle @ strophe.umd.js:4901
_onIdle @ strophe.umd.js:3881
(anonymous) @ strophe.umd.js:5257
setTimeout (async)
_send @ strophe.umd.js:5256
send @ strophe.umd.js:2583
send @ XmppConnection.js:400
onJingle @ strophe.jingle.js:246
run @ strophe.umd.js:1875
(anonymous) @ strophe.umd.js:3157
forEachChild @ strophe.umd.js:830
_dataRecv @ strophe.umd.js:3146
_onRequestStateChange @ strophe.umd.js:5012
XMLHttpRequest.send (async)
d @ strophe.umd.js:5123
_processRequest @ strophe.umd.js:5137
_throttledRequestHandler @ strophe.umd.js:5290
_onIdle @ strophe.umd.js:4901
_onIdle @ strophe.umd.js:3881
(anonymous) @ strophe.umd.js:5257
setTimeout (async)
_send @ strophe.umd.js:5256
send @ strophe.umd.js:2583
sendIQ @ strophe.umd.js:2740
sendIQ @ XmppConnection.js:420
sendIceCandidates @ JingleSessionPC.js:729
(anonymous) @ JingleSessionPC.js:638
setTimeout (async)
sendIceCandidate @ JingleSessionPC.js:634
peerconnection.onicecandidate @ JingleSessionPC.js:424
R.peerconnection.onicecandidate @ TraceablePeerConnection.js:269
Logger.js:154 2020-08-13T17:10:38.134Z [modules/xmpp/strophe.jingle.js] <g.onJingle>:  on jingle session-terminate from meeting-ww3xnvv8py@conference.newton-jitsi-meet.theboloapp.com/b86a15ca 

thanks for your help

Are you having more than a thousand concurrent connections?
Maybe be switching to epoll network backend will improve https://prosody.im/doc/network_backend
I’m still not sure where is the problem, just guessing.
And make sure nginx and prosody file handlers limit are high enough for the number of connections.

not thousand but can say max upto 700 and with video bridges and jibri about 60 connections are added more.

sure will increase those limits and switch to epoll network.

thanks

hey @damencho,

thanks for your help, so i did the asked changes and there was improvement but still lot of were facing disconnection issue. Seem like horizontal scaling of prosody is needed. Can you recommend any documentation that can help me to achieve it?

Regards,
Gaurav Kapatia

Checkout the video tutorials about scaling, this involves adding haproxies infront of the shards so you can stick sessions by url parameter room.

Also search the forum there are multiple posts about this.

thanks! just saw the video about using HAproxy, it was quite interesting. will try to build it :slight_smile: