Sigterm after update to jitsi-meet_8194

himbeere · January 12, 2023, 5:34pm

Hello one and all.

After upgrading to:

JVB crashes with sigterm. No config change on my side.

JVB 2023-01-12 18:23:11.478 WARNING: [54] org.jivesoftware.smack.AbstractXMPPConnection.callConnectionClosedOnErrorListener: Connection XMPPTCPConnection[jvb@auth.meetings.mycloud.de/AO_ltUIM] (0) closed with error
org.jivesoftware.smack.XMPPException$StreamErrorException: system-shutdown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
<stream:error><system-shutdown xmlns='urn:ietf:params:xml:ns:xmpp-streams'/><text>Received SIGTERM</text></stream:error>
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:984)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$700(XMPPTCPConnection.java:916)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:939)
        at java.base/java.lang.Thread.run(Thread.java:829)
JVB 2023-01-12 18:23:11.482 WARNING: [54] [hostname=localhost id=shard] MucClient$2.connectionClosedOnError#341: Closed on error:
org.jivesoftware.smack.XMPPException$StreamErrorException: system-shutdown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
<stream:error><system-shutdown xmlns='urn:ietf:params:xml:ns:xmpp-streams'/><text>Received SIGTERM</text></stream:error>
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:984)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$700(XMPPTCPConnection.java:916)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:939)
        at java.base/java.lang.Thread.run(Thread.java:829)
JVB 2023-01-12 18:23:14.437 WARNING: [27] [hostname=localhost id=shard] MucClient.setPresenceExtensions#467: Cannot set presence extension: not connected.
JVB 2023-01-12 18:23:15.130 INFO: [19] [hostname=localhost id=shard] MucClient$2.connected#321: Connected.
JVB 2023-01-12 18:23:15.130 INFO: [19] [hostname=localhost id=shard] MucClient.lambda$getConnectAndLoginCallable$9#646: Logging in.
JVB 2023-01-12 18:23:15.135 INFO: [19] [hostname=localhost id=shard] MucClient$2.authenticated#327: Authenticated, b=false
JVB 2023-01-12 18:23:15.135 INFO: [19] [hostname=localhost id=shard] MucClient$MucWrapper.join#751: Leaving a MUC we already occupy.
JVB 2023-01-12 18:23:15.142 WARNING: [19] [hostname=localhost id=shard] MucClient$MucWrapper.leave#726: Error while trying to leave a MUC: 
org.jivesoftware.smack.XMPPException$XMPPErrorException: XMPP error reply received from jvbbrewery@internal.auth.meetings.mycloud.de/e9c83d22-763c-45a3-b448-4dfb2ee2d08a: XMPPError: item-not-found - cancel
        at org.jivesoftware.smack.XMPPException$XMPPErrorException.ifHasErrorThenThrow(XMPPException.java:171)
        at org.jivesoftware.smack.XMPPException$XMPPErrorException.ifHasErrorThenThrow(XMPPException.java:165)
        at org.jivesoftware.smack.StanzaCollector.nextResultOrThrow(StanzaCollector.java:284)
        at org.jivesoftware.smack.StanzaCollector.nextResultOrThrow(StanzaCollector.java:228)
        at org.jivesoftware.smackx.muc.MultiUserChat.leave(MultiUserChat.java:798)
        at org.jitsi.xmpp.mucclient.MucClient$MucWrapper.leave(MucClient.java:722)
        at org.jitsi.xmpp.mucclient.MucClient$MucWrapper.join(MucClient.java:752)
        at org.jitsi.xmpp.mucclient.MucClient.joinMucs(MucClient.java:370)
        at org.jitsi.xmpp.mucclient.MucClient.lambda$getConnectAndLoginCallable$9(MucClient.java:667)
        at org.jitsi.retry.RetryStrategy$TaskRunner.run(RetryStrategy.java:167)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
JVB 2023-01-12 18:23:15.197 INFO: [19] [hostname=localhost id=shard] MucClient$MucWrapper.join#771: Joined MUC: jvbbrewery@internal.auth.meetings.mycloud.de
JVB 2023-01-12 18:28:05.681 WARNING: [57] org.jivesoftware.smack.AbstractXMPPConnection.callConnectionClosedOnErrorListener: Connection XMPPTCPConnection[jvb@auth.meetings.mycloud.de/E4LVS3L3] (0) closed with error
org.jivesoftware.smack.XMPPException$StreamErrorException: system-shutdown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
<stream:error><system-shutdown xmlns='urn:ietf:params:xml:ns:xmpp-streams'/><text>Received SIGTERM</text></stream:error>
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:984)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$700(XMPPTCPConnection.java:916)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:939)
        at java.base/java.lang.Thread.run(Thread.java:829)
JVB 2023-01-12 18:28:05.684 WARNING: [57] [hostname=localhost id=shard] MucClient$2.connectionClosedOnError#341: Closed on error:
org.jivesoftware.smack.XMPPException$StreamErrorException: system-shutdown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
<stream:error><system-shutdown xmlns='urn:ietf:params:xml:ns:xmpp-streams'/><text>Received SIGTERM</text></stream:error>
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:984)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$700(XMPPTCPConnection.java:916)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:939)
        at java.base/java.lang.Thread.run(Thread.java:829)
JVB 2023-01-12 18:28:09.437 WARNING: [27] [hostname=localhost id=shard] MucClient.setPresenceExtensions#467: Cannot set presence extension: not connected.
JVB 2023-01-12 18:28:10.354 INFO: [19] [hostname=localhost id=shard] MucClient$2.connected#321: Connected.
JVB 2023-01-12 18:28:10.354 INFO: [19] [hostname=localhost id=shard] MucClient.lambda$getConnectAndLoginCallable$9#646: Logging in.
JVB 2023-01-12 18:28:10.365 INFO: [19] [hostname=localhost id=shard] MucClient$2.authenticated#327: Authenticated, b=false
JVB 2023-01-12 18:28:10.366 INFO: [19] [hostname=localhost id=shard] MucClient$MucWrapper.join#751: Leaving a MUC we already occupy.
JVB 2023-01-12 18:28:10.377 WARNING: [19] [hostname=localhost id=shard] MucClient$MucWrapper.leave#726: Error while trying to leave a MUC: 
org.jivesoftware.smack.XMPPException$XMPPErrorException: XMPP error reply received from jvbbrewery@internal.auth.meetings.mycloud.de/e9c83d22-763c-45a3-b448-4dfb2ee2d08a: XMPPError: item-not-found - cancel
        at org.jivesoftware.smack.XMPPException$XMPPErrorException.ifHasErrorThenThrow(XMPPException.java:171)
        at org.jivesoftware.smack.XMPPException$XMPPErrorException.ifHasErrorThenThrow(XMPPException.java:165)
        at org.jivesoftware.smack.StanzaCollector.nextResultOrThrow(StanzaCollector.java:284)
        at org.jivesoftware.smack.StanzaCollector.nextResultOrThrow(StanzaCollector.java:228)
        at org.jivesoftware.smackx.muc.MultiUserChat.leave(MultiUserChat.java:798)
        at org.jitsi.xmpp.mucclient.MucClient$MucWrapper.leave(MucClient.java:722)
        at org.jitsi.xmpp.mucclient.MucClient$MucWrapper.join(MucClient.java:752)
        at org.jitsi.xmpp.mucclient.MucClient.joinMucs(MucClient.java:370)
        at org.jitsi.xmpp.mucclient.MucClient.lambda$getConnectAndLoginCallable$9(MucClient.java:667)
        at org.jitsi.retry.RetryStrategy$TaskRunner.run(RetryStrategy.java:167)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

Someone has an idea what’s going on here?

cheers, t.

damencho · January 12, 2023, 5:36pm

What happens if you restart jvb?

Boris_Grozev · January 12, 2023, 5:49pm

This is normal behavior. On upgrade prosody restarts because it receives a TERM. The bridge loses the XMPP connection, then re-connects. There’s a stack trace logged with “Error while trying to leave a MUC”, which in this case is benign.

What is the expected behavior? Is something not working?

himbeere · January 12, 2023, 7:53pm

Authentication with internal_hashed does not work anymore. So you say the sigterm is not related?

himbeere · January 12, 2023, 8:21pm

Alright. So there is a problem with authentication somehow then.

himbeere · January 12, 2023, 8:55pm

After restarting jvb the error in jvb.log is gone, yes. But the problem persists. A look to the other log files revealed from jicofo.log

root@meetings:~# grep -r "Lost a bridge" /var/log/jitsi/*
/var/log/jitsi/jicofo.log:Jicofo 2023-01-12 18:18:06.468 WARNING: [54] BridgeSelector.removeJvbAddress#108: Lost a bridge: jvbbrewery@internal.auth.meetings.mycloud.de/e9c83d22-763c-45a3-b448-4dfb2ee2d08a
/var/log/jitsi/jicofo.log:Jicofo 2023-01-12 18:19:44.622 WARNING: [31] BridgeSelector.removeJvbAddress#108: Lost a bridge: jvbbrewery@internal.auth.meetings.mycloud.de/e9c83d22-763c-45a3-b448-4dfb2ee2d08a
/var/log/jitsi/jicofo.log:Jicofo 2023-01-12 18:23:11.488 WARNING: [39] BridgeSelector.removeJvbAddress#108: Lost a bridge: jvbbrewery@internal.auth.meetings.mycloud.de/e9c83d22-763c-45a3-b448-4dfb2ee2d08a
/var/log/jitsi/jicofo.log:Jicofo 2023-01-12 18:28:05.693 WARNING: [41] BridgeSelector.removeJvbAddress#108: Lost a bridge: jvbbrewery@internal.auth.meetings.mycloud.de/e9c83d22-763c-45a3-b448-4dfb2ee2d08a
/var/log/jitsi/jicofo.log:Jicofo 2023-01-12 21:49:51.647 WARNING: [20] BridgeSelector.removeJvbAddress#108: Lost a bridge: jvbbrewery@internal.auth.meetings.mycloud.de/e9c83d22-763c-45a3-b448-4dfb2ee2d08a
root@meetings:~#

That wasn’t there before so often. Maybe once a month or so.

Boris_Grozev · January 12, 2023, 8:59pm

Yeah, that’s just prosody being restarted

Both the bridge and jicofo authenticate using internal_hashed, and your logs show they are both able to login.

saghul · January 13, 2023, 10:04am

There is a known problem with authentication on that release, we will be releasing a new stable release shortly.

himbeere · January 13, 2023, 11:55am

Thanks!

saghul · January 13, 2023, 2:02pm

The new stable release is out.

himbeere · January 13, 2023, 3:04pm

Yes! Works as expected!

Emirhan_Sener · January 19, 2023, 10:43pm

having same issue with docker:latest installation. What is the solution for it?

saghul · January 20, 2023, 7:15am

latest has been deprecated for a long time. Please use an actual release.

Emirhan_Sener · January 20, 2023, 9:49am

I pulled images written below:

-jitsi/prosody:unstable-2023-01-20
-jitsi/jicofo:unstable-2023-01-20
-jitsi/jvb:unstable-2023-01-20
-jitsi/jibri:unstable-2023-01-20
-emir6195/cbot-jitsi-applem1:latest (for web, my build from actual project on github.)

below i shared my .env
ENABLE_AUTH=1

Enable guest access

ENABLE_GUESTS=1

Select authentication type: internal, jwt, ldap or matrix

AUTH_TYPE=internal

When docker is up, i execute these commands below:

docker-compose exec prosody /bin/bash
prosodyctl --config /config/prosody.cfg.lua register emir6195 meet.jitsi Msal.550550

When i try to login meeting, console gives this error.

app.bundle.min.js?v=139:178 2023-01-20T09:36:03.965Z [features/authentication] authenticateAndUpgradeRole failed

{authenticationError: ‘not-acceptable’, message: “Missing mandatory attribute ‘machine-uid’”}

authenticationError: “not-acceptable”

message: “Missing mandatory attribute ‘machine-uid’”

[[Prototype]]: Object

lib-jitsi-meet.min.js?v=139:2 2023-01-20T09:36:03.966Z [JitsiMeetJS.ts] <Object.getGlobalOnErrorHandler>: UnhandledError: [object Object] Script: null Line: null Column: null StackTrace: n/a

r @ lib-jitsi-meet.min.js?v=139:2

getGlobalOnErrorHandler @ lib-jitsi-meet.min.js?v=139:2

window.onunhandledrejection @ app.bundle.min.js?v=139:178

merhaba:1 Uncaught (in promise)

{authenticationError: ‘not-acceptable’, message: “Missing mandatory attribute ‘machine-uid’”}

authenticationError: “not-acceptable”

message: “Missing mandatory attribute ‘machine-uid’”

[[Prototype]]: Object

jicofo logs is like that :

cbot-meet-release-jicofo-1 | Jicofo 2023-01-20 12:45:45.005 INFO: [23] ConferenceIqHandler.handleConferenceIq#76: Focus request for room: merhaba@muc.meet.jitsi
cbot-meet-release-jicofo-1 | Jicofo 2023-01-20 12:45:51.021 INFO: [23] ConferenceIqHandler.handleConferenceIq#76: Focus request for room: merhaba@muc.meet.jitsi
cbot-meet-release-jicofo-1 | Jicofo 2023-01-20 12:45:57.016 INFO: [23] ConferenceIqHandler.handleConferenceIq#76: Focus request for room: merhaba@muc.meet.jitsi

Is there something I am missing?
Appreciate for your help <3

saghul · January 20, 2023, 10:11am

This is fixed in the latest stable release.

	r	@	lib-jitsi-meet.min.js?v=139:2
	getGlobalOnErrorHandler	@	lib-jitsi-meet.min.js?v=139:2
	window.onunhandledrejection	@	app.bundle.min.js?v=139:178