When prosody restarts, do all other services also need to restart?

I’m working on getting Jitsi setup in Kubernetes and it looks like I have everything working (testing with three separate computers all in a conference). I’m now trying to test the resiliency of the system and I noticed that if I restart the prosody server, nothing works anymore until I restart the jvb, web and jicofo services as well. By nothing works, I mean all new conferences either crash when the second user joins or just will fail to transmit audio/video.

Is this expected? Is there any way to “kick” the other services without a full restart? Any suggestions on how to handle this scenario?

Thanks

Do you see any jicofo exceptions? I was just fixing similar problem. Part of the fix is in unstable repo.

Yep. Once prosody comes back up, jicofo will have the following in its logs:

Jicofo 2019-03-26 20:04:07.095 INFO: [120] org.jitsi.jicofo.FocusManager.log() Created new focus for test@muc.meet.jitsi@auth.meet.jitsi. Conference count 1,options:
Jicofo 2019-03-26 20:04:07.099 INFO: [120] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Lip-sync enabled in test@muc.meet.jitsi
Jicofo 2019-03-26 20:04:07.103 INFO: [120] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Joining the room: test@muc.meet.jitsi
Jicofo 2019-03-26 20:04:07.213 INFO: [37] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Chat room event ChatRoomMemberPresenceChangeEvent[type=MemberJoined sourceRoom=org.jitsi.impl.protocol.xmpp.ChatRoomImpl@18782f75 member=ChatMember[test@muc.meet.jitsi/0l3t3xw_, jid: null]@760103145]
Jicofo 2019-03-26 20:04:07.221 INFO: [37] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Granted owner to test@muc.meet.jitsi/0l3t3xw_
Jicofo 2019-03-26 20:04:07.223 INFO: [37] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Member test@muc.meet.jitsi/0l3t3xw_ joined.
Jicofo 2019-03-26 20:04:13.107 INFO: [123] org.jitsi.jicofo.xmpp.FocusComponent.handleConferenceIq().402 Focus request for room: test@muc.meet.jitsi
Jicofo 2019-03-26 20:04:13.440 INFO: [37] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Chat room event ChatRoomMemberPresenceChangeEvent[type=MemberJoined sourceRoom=org.jitsi.impl.protocol.xmpp.ChatRoomImpl@18782f75 member=ChatMember[test@muc.meet.jitsi/3id1fyoe, jid: null]@2032246572]
Jicofo 2019-03-26 20:04:13.442 INFO: [37] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Member test@muc.meet.jitsi/3id1fyoe joined.
Jicofo 2019-03-26 20:04:13.469 SEVERE: [37] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Can not invite participant -- no bridge available.
Jicofo 2019-03-26 20:04:13.480 SEVERE: [37] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Can not invite participant -- no bridge available.
Jicofo 2019-03-26 20:04:13.954 INFO: [37] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Chat room event ChatRoomMemberPresenceChangeEvent[type=MemberLeft sourceRoom=org.jitsi.impl.protocol.xmpp.ChatRoomImpl@18782f75 member=ChatMember[test@muc.meet.jitsi/0l3t3xw_, jid: 0l3t3xw_m8aupsny@meet.jitsi/BIO8C3sZ]@760103145]
Jicofo 2019-03-26 20:04:13.954 INFO: [37] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Owner has left the room !
Jicofo 2019-03-26 20:04:13.969 INFO: [37] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Granted owner to test@muc.meet.jitsi/3id1fyoe
Jicofo 2019-03-26 20:04:13.969 INFO: [37] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Member test@muc.meet.jitsi/0l3t3xw_ is leaving
Jicofo 2019-03-26 20:04:13.971 INFO: [37] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Removed participant: true, test@muc.meet.jitsi/0l3t3xw_
Jicofo 2019-03-26 20:04:14.144 INFO: [37] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Chat room event ChatRoomMemberPresenceChangeEvent[type=MemberLeft sourceRoom=org.jitsi.impl.protocol.xmpp.ChatRoomImpl@18782f75 member=ChatMember[test@muc.meet.jitsi/3id1fyoe, jid: 3id1fyoe4dzntnr9@meet.jitsi/qXlslWDH]@2032246572]
Jicofo 2019-03-26 20:04:14.146 INFO: [37] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Owner has left the room !
Jicofo 2019-03-26 20:04:14.149 INFO: [37] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Member test@muc.meet.jitsi/3id1fyoe is leaving
Jicofo 2019-03-26 20:04:14.149 INFO: [37] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Removed participant: true, test@muc.meet.jitsi/3id1fyoe

Are you using the bridge with component or with mucs?
When using with component jicofo was checking every minute …

Honestly, I’m not sure. I based the configuration on what is in the docker-jitsi-meet repo. Is there an easy way to tell?

I am also one having the same issue. In my Kubernetes deployment where jitsi-meet, jvb & prosody are configured as LoadBalancer type services and they are running on separate pods and have ClusterIPs for each of them.

I have configured the XMPP_DOMAIN, AUTH_XMPP_DOMAIN and MUX_XMPP_DOMAIN as follows :
XMPP_DOMAIN=172.20.3.122
AUTH_XMPP_DOMAIN=auth.172.20.3.122
MUC_XMPP_DOMAIN=muc.172.20.3.122

The external IP for all the services are set to 172.20.3.22

Able to join the bridge and the conference crashes when a new user joins

Is my networking config is valid? Or else am I missing something?
Any help would be appreciated…
Thanks in advance.

Sivaven

I’m not familiar with Kubernetes but how are you making sure a given conference always hits the same prosody? A conference “foo” must be on a single prosody server or you’ll have a situation similar to a split brain.

Also, have you set your DOCKER_HOST_ADDRESS as the public IP of the jvb?

Hi Saghul,

I have deployed the Jitsi server components on Kube as shown below:

cbuser@opencartmaster:~$ kubectl get all -n default
NAME                                READY   STATUS    RESTARTS   AGE
pod/jicofo-5695c66957-cjp65         1/1     Running   0          12h
pod/jvb-8d6466896-w6ms7             1/1     Running   0          12h
pod/prosody-5f84ff9dc9-cdl9s        1/1     Running   0          12h
pod/web-8647f6f6d7-khrjl            1/1     Running   0          12h


NAME                           TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                                        AGE
service/jvb-udp                LoadBalancer   10.103.156.223   172.20.3.122   10000:30383/UDP                                12h
service/kubernetes             ClusterIP      10.96.0.1        <none>         443/TCP                                        2d12h
service/prosody                LoadBalancer   10.108.132.112   172.20.3.122   5222:31887/TCP,5280:32407/TCP,5347:32245/TCP   12h
service/web                    LoadBalancer   10.104.234.45    172.20.3.122   80:31980/TCP,443:31943/TCP                     12h


NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/jicofo         1/1     1            1           12h
deployment.apps/jvb            1/1     1            1           12h
deployment.apps/prosody        1/1     1            1           12h
deployment.apps/web            1/1     1            1           12h

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/jicofo-5695c66957         1         1         1       12h
replicaset.apps/jvb-8d6466896             1         1         1       12h
replicaset.apps/prosody-5f84ff9dc9        1         1         1       12h
replicaset.apps/web-8647f6f6d7            1         1         1       12h

The pods are kind of a Kube object which executes single/multiple containers. The jitsi components are running on separate pods and each pod only have one replica of container (replicaset.apps section in the output).

So there is only one prosody(XMPP) server and all the requests are routed to it whenever an event occurs.

The parameter set as

DOCKER_HOST_ADDRESS=172.20.3.122

where 172.20.3.122 is the public IP of JVB pod/container.

Thanks for lending your hands on resolving this issue. Kindly let me know any other configurations are required to figure out the issue.

You need the public IP of the service, not the public LAN IP. Unless you want to run it LAN-only, that is.

Hi Saghul, it was my mistake that i had mentioned the ip as Pod/Container public IP. Actually it is the public IP of JVB service.

And finally we figured it out where the env variable JVB_BREWERY_MUC was missed in the Jicofo configuration caused the users to automatically drop off from the bridge.

Jicofo 2019-08-29 00:30:08.590 INFO: [106] org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.reconnectingIn().684 XMPP reconnecting in: 10
Jicofo 2019-08-29 00:30:09.591 INFO: [106] org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.reconnectingIn().684 XMPP reconnecting in: 9
Jicofo 2019-08-29 00:30:10.591 INFO: [106] org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.reconnectingIn().684 XMPP reconnecting in: 8
Jicofo 2019-08-29 00:30:11.592 INFO: [106] org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.reconnectingIn().684 XMPP reconnecting in: 7
Jicofo 2019-08-29 00:30:12.592 INFO: [106] org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.reconnectingIn().684 XMPP reconnecting in: 6
Jicofo 2019-08-29 00:30:13.593 INFO: [106] org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.reconnectingIn().684 XMPP reconnecting in: 5
Jicofo 2019-08-29 00:30:14.594 INFO: [106] org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.reconnectingIn().684 XMPP reconnecting in: 4
Jicofo 2019-08-29 00:30:15.595 INFO: [106] org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.reconnectingIn().684 XMPP reconnecting in: 3
Jicofo 2019-08-29 00:30:16.242 SEVERE: [109] org.jitsi.xmpp.component.ComponentBase.log() Ping timeout for ID: rVQr3-1463
Jicofo 2019-08-29 00:30:16.595 INFO: [106] org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.reconnectingIn().684 XMPP reconnecting in: 2
Jicofo 2019-08-29 00:30:17.596 INFO: [106] org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.reconnectingIn().684 XMPP reconnecting in: 1
Jicofo 2019-08-29 00:30:18.596 INFO: [106] org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.reconnectingIn().684 XMPP reconnecting in: 0
Jicofo 2019-08-29 00:30:18.596 INFO: [106] org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.reconnectingIn().684 XMPP reconnecting in: 0
Jicofo 2019-08-29 00:30:18.713 SEVERE: [106] org.jitsi.jicofo.xmpp.BaseBrewery.start().196 Failed to create room: <no value>@internal.muc.172.20.3.122
net.java.sip.communicator.service.protocol.OperationFailedException: Invalid room name
        at org.jitsi.impl.protocol.xmpp.OperationSetMultiUserChatImpl.createChatRoom(OperationSetMultiUserChatImpl.java:106)
        at org.jitsi.jicofo.xmpp.BaseBrewery.start(BaseBrewery.java:187)
        at org.jitsi.jicofo.xmpp.BaseBrewery.maybeStart(BaseBrewery.java:141)
        at org.jitsi.jicofo.xmpp.BaseBrewery.registrationStateChanged(BaseBrewery.java:166)
        at org.jitsi.jicofo.ProtocolProviderHandler.registrationStateChanged(ProtocolProviderHandler.java:141)
        at net.java.sip.communicator.service.protocol.AbstractProtocolProviderService.fireRegistrationStateChanged(AbstractProtocolProviderService.java:187)
        at net.java.sip.communicator.service.protocol.AbstractProtocolProviderService.fireRegistrationStateChanged(AbstractProtocolProviderService.java:141)
        at org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.notifyConnected(XmppProtocolProvider.java:335)
        at org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.access$000(XmppProtocolProvider.java:62)
        at org.jitsi.impl.protocol.xmpp.XmppProtocolProvider$XmppConnectionListener.authenticated(XmppProtocolProvider.java:623)
        at org.jivesoftware.smack.AbstractXMPPConnection.callConnectionAuthenticatedListener(AbstractXMPPConnection.java:1297)
        at org.jivesoftware.smack.AbstractXMPPConnection.afterSuccessfulLogin(AbstractXMPPConnection.java:572)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection.afterSuccessfulLogin(XMPPTCPConnection.java:379)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection.loginInternal(XMPPTCPConnection.java:444)
        at org.jivesoftware.smack.AbstractXMPPConnection.login(AbstractXMPPConnection.java:491)
        at org.jivesoftware.smack.AbstractXMPPConnection.login(AbstractXMPPConnection.java:448)
        at org.jivesoftware.smack.ReconnectionManager$2.run(ReconnectionManager.java:294)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.jxmpp.stringprep.XmppStringprepException: XmppStringprepException caused by '<no value>@internal.muc.172.20.3.122': org.jxmpp.stringprep.XmppStringprepException: Localpart must not contain '<'
        at org.jxmpp.jid.impl.JidCreate.entityBareFrom(JidCreate.java:591)
        at org.jitsi.impl.protocol.xmpp.OperationSetMultiUserChatImpl.createChatRoom(OperationSetMultiUserChatImpl.java:102)
        ... 17 more
Caused by: org.jxmpp.stringprep.XmppStringprepException: Localpart must not contain '<'
        at org.jxmpp.stringprep.simple.SimpleXmppStringprep.localprep(SimpleXmppStringprep.java:74)
        at org.jxmpp.stringprep.XmppStringPrepUtil.localprep(XmppStringPrepUtil.java:62)
        at org.jxmpp.jid.parts.Localpart.from(Localpart.java:137)
        at org.jxmpp.jid.impl.LocalAndDomainpartJid.<init>(LocalAndDomainpartJid.java:46)
        at org.jxmpp.jid.impl.JidCreate.entityBareFrom(JidCreate.java:589)
        ... 18 more
Jicofo 2019-08-29 00:30:18.718 INFO: [106] org.jitsi.jicofo.ComponentsDiscovery.log() New component discovered: auth.172.20.3.122, IQ Stanza (query jabber:iq:version) [to=focus@auth.172.20.3.122/focus148968938263665,from=auth.172.20.3.122,id=rVQr3-1474,type=result,]
Jicofo 2019-08-29 00:30:18.719 INFO: [106] org.jitsi.jicofo.JitsiMeetServices.log() Detected XMPP server version: IQ Stanza (query jabber:iq:version) [to=focus@auth.172.20.3.122/focus148968938263665,from=auth.172.20.3.122,id=rVQr3-1474,type=result,]
Jicofo 2019-08-29 00:30:18.720 INFO: [106] org.jitsi.jicofo.ComponentsDiscovery.log() New component discovered: muc.172.20.3.122, null
Jicofo 2019-08-29 00:30:18.721 INFO: [106] org.jitsi.jicofo.JitsiMeetServices.log() MUC component discovered: muc.172.20.3.122
Jicofo 2019-08-29 00:30:18.723 INFO: [106] org.jitsi.jicofo.ComponentsDiscovery.log() New component discovered: focus.172.20.3.122, null
Jicofo 2019-08-29 00:30:18.724 INFO: [106] org.jitsi.jicofo.ComponentsDiscovery.log() Services re-discovery interval: 30000
Jicofo 2019-08-29 00:30:18.724 INFO: [106] org.jitsi.jicofo.FocusManager.log() XMPP provider reg state: RegistrationState=Registered
Jicofo 2019-08-29 00:33:11.769 INFO: [126] org.jitsi.jicofo.xmpp.FocusComponent.handleConferenceIq().401 Focus request for room: aaa12345@muc.172.20.3.122
Jicofo 2019-08-29 00:33:11.771 INFO: [126] org.jitsi.jicofo.FocusManager.log() Created new focus for aaa12345@muc.172.20.3.122@auth.172.20.3.122. Conference count 1,options:
Jicofo 2019-08-29 00:33:11.775 INFO: [126] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Lip-sync enabled in aaa12345@muc.172.20.3.122
Jicofo 2019-08-29 00:33:11.776 INFO: [126] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Joining the room: aaa12345@muc.172.20.3.122
Jicofo 2019-08-29 00:33:12.379 INFO: [114] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Chat room event ChatRoomMemberPresenceChangeEvent[type=MemberJoined sourceRoom=org.jitsi.impl.protocol.xmpp.ChatRoomImpl@68fa6ce5 member=ChatMember[aaa12345@muc.172.20.3.122/fbf47e17, jid: null]@1715310474]
Jicofo 2019-08-29 00:33:12.387 INFO: [114] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Granted owner to aaa12345@muc.172.20.3.122/fbf47e17
Jicofo 2019-08-29 00:33:12.387 INFO: [114] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Member aaa12345@muc.172.20.3.122/fbf47e17 joined.
Jicofo 2019-08-29 00:34:39.686 INFO: [129] org.jitsi.jicofo.xmpp.FocusComponent.handleConferenceIq().401 Focus request for room: aaa12345@muc.172.20.3.122
Jicofo 2019-08-29 00:34:40.277 INFO: [114] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Chat room event ChatRoomMemberPresenceChangeEvent[type=MemberJoined sourceRoom=org.jitsi.impl.protocol.xmpp.ChatRoomImpl@68fa6ce5 member=ChatMember[aaa12345@muc.172.20.3.122/aabcef82, jid: null]@1644300181]
Jicofo 2019-08-29 00:34:40.277 INFO: [114] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Member aaa12345@muc.172.20.3.122/aabcef82 joined.
Jicofo 2019-08-29 00:34:40.278 SEVERE: [114] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Can not invite participant -- no bridge available.
Jicofo 2019-08-29 00:34:40.278 SEVERE: [114] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Can not invite participant -- no bridge available.
Jicofo 2019-08-29 00:34:41.861 INFO: [114] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Chat room event ChatRoomMemberPresenceChangeEvent[type=MemberLeft sourceRoom=org.jitsi.impl.protocol.xmpp.ChatRoomImpl@68fa6ce5 member=ChatMember[aaa12345@muc.172.20.3.122/fbf47e17, jid: yd5lejgdycdaawgc@172.20.3.122/Q-Jtv8cR]@1715310474]
Jicofo 2019-08-29 00:34:41.862 INFO: [114] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Owner has left the room !
Jicofo 2019-08-29 00:34:41.869 INFO: [114] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Granted owner to aaa12345@muc.172.20.3.122/aabcef82
Jicofo 2019-08-29 00:34:41.871 INFO: [114] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Member aaa12345@muc.172.20.3.122/fbf47e17 is leaving
Jicofo 2019-08-29 00:34:41.871 INFO: [114] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Removed participant: true, aaa12345@muc.172.20.3.122/fbf47e17
Jicofo 2019-08-29 00:34:42.728 INFO: [114] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Chat room event ChatRoomMemberPresenceChangeEvent[type=MemberLeft sourceRoom=org.jitsi.impl.protocol.xmpp.ChatRoomImpl@68fa6ce5 member=ChatMember[aaa12345@muc.172.20.3.122/aabcef82, jid: 0-s2l4bwt52_zkem@172.20.3.122/uJIC_n69]@1644300181]
Jicofo 2019-08-29 00:34:42.728 INFO: [114] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Owner has left the room !
Jicofo 2019-08-29 00:34:42.728 INFO: [114] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Member aaa12345@muc.172.20.3.122/aabcef82 is leaving
Jicofo 2019-08-29 00:34:42.729 INFO: [114] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Removed participant: true, aaa12345@muc.172.20.3.122/aabcef82
Jicofo 2019-08-29 00:34:42.746 INFO: [114] org.jitsi.jicofo.FocusManager.log() Disposed conference for room: aaa12345@muc.172.20.3.122 conference count: 0

Primarily this issue will be caused by any missing configuration on Jicofo server to discover JVB component.

@Saghal, Thank you very much for volunteering to sort out this issue.

Not that I helped much, but I’m glad you got it working!

@sivaven Hi sivaven, do you have a documentation of how to deploy jitsi onto kubernetes? I’m facing with the same situation and need your advice, Thanks!