Jicofo don’t listen/connect to prosody client_proxy - act 2 (latest 2 stable builds)

Hello,

I’m running into the same issue as I had in Jicofo don't listen/connect to prosody client_proxy (stable 15 apr) with the 2 latest builds. Last working builds I tested was :

  • jitsi-meet-web-config-1.0.5913-2
  • jibri-8.0_114_g20e233e-2
  • jicofo-1.0_862-2
  • jitsi-videobridge2-2.1_634_gff8609ad-2
  • jitsi-meet-tokens-1.0.5913-2
  • jitsi-meet-web-1.0.5913-2

Current not working build :

jicofo-1.0_954-2
jibri-8.0_114_g20e233e-2
jitsi-meet-web-config-1.0.6776-2
jitsi-meet-web-1.0.6776-2
jitsi-videobridge2-2.2_9_g8cded16e-2
jitsi-meet-tokens-1.0.6776-2

Actually I’m not sure this is the same issue but the consequences are the same, prosody does nothing of the requests sent to focus, but this time the subscribe command has been done properly.

I updated using the references files and the builds content :

  • My prosody configuration with all the fixes including the muc_room_locking and muc_room_default_public_jids things I saw on a topic
  • My prosody modules with yours (from the build)
  • My old and new videobridge configuration (old sip and new .conf) using references
  • My jicofo configuration including proper client-proxy and all the client configuration

The only clue I have is an error in jicofo that seems more related to jvb - jicofo communication :

XMPP error reply received from jvbbrewery@internal.auth.xxx: XMPPError: forbidden - auth [Only owners can configure rooms]

Sadly, ressources I can find about this doesn’t help me :

Debian package scripts fail to update Prosody configuration files · Issue #10282 · jitsi/jitsi-meet · GitHub => My prosody configuration is a copy/paste from yours
Cant Create JibriBrewery Room in Jicofo · Issue #349 · jitsi/jicofo · GitHub => My internal auth storage is set to memory

Thank you in advance for any help !

Is jicofo jid set to be admin ?

Another option is to stop everything start first prospdy, then jicofo and as last jvb.

Hello !

Is jicofo jid set to be admin ?

By jicofo jid you mean focus ?

My admins are :

  • focus in conference component
  • focus in breakout component
  • focus and jvb in internal auth component

Another option is to stop everything start first prospdy, then jicofo and as last jvb.

Yes I already start services in this order with sleep in my start script

Self-Hosting Guide - Debian/Ubuntu server | Jitsi Meet

Yes I know this link, I’m on it pretty much every day :wink:

Thanks in advance !

I think I posted that in wrong thread.

Add focus to the internal muc component, jitsi-meet/prosody.cfg.lua-jvb.example at f42772ec5bcc87ff6de17423d36df9bcad6e770d · jitsi/jitsi-meet · GitHub

Yeah that’s already the case, our files are nearly identical :

My admins are :
[…]

  • focus and jvb in internal auth component

@damencho Do you have any idea or need some access to test ? We are starting the production phase for our customers really soon and want to integrate new Jitsi version

And what is the problem? When you restart the server jicofo does not connect to the brewery room? I don’t see why you will need to restart a server in production.

Not sure why jicofo is not considered admin for your deployment. Try setting it as admin in the global prosody config.

Actually I’m not sure the brewery issue is the real problem because Jicofo still says that it is connected to videobridge (actually I don’t understand at all what is brewery, brewery user and their purpose).

What I know is that jicofo and videobridge starts, prosody receive and accept connexion from jibri, videobridge and jicofo (which seems to have an anonymous user) and then when I try to join a room client side the browser send the request to prosody and prosody says service unavailable like if he doesn’t know what to do with the request sent to focus.xxx.

The behavior is the same than in Jicofo don't listen/connect to prosody client_proxy (stable 15 apr)

If you want I can provide screenshots but except the error I showed you it is the same behavior (but I think this is not the same issue in the back)

Well are you sure jicofo connects and authenticates with the jid that is used for the prosodyctl mod_roster_command subscribe … Check it in prosody logs, what do you see there ?

You are getting that error cause the proxy module cannot find the jicofo user to send it to, the target address target_address.

No you are right it might be the issue.

We can see here jibri and jicofo auth with their users but jicofo with an anonymous user.

But I’m pretty sure that this was already the case before… Do you have any idea about that ?

EDIT : Btw these are my configurations

/etc/jitsi/jicofo/config

image

/etc/jitsi/jicofo/jicofo.conf

image

prosody

Clean jicofo logs and restart it and upload them here.

Yes sir =>

Jicofo 2022-12-02 15:59:31.901 INFOS: [1] Main.main#47: Starting Jicofo.
Jicofo 2022-12-02 15:59:32.108 INFOS: [1] JitsiConfig.<clinit>#47: Initialized newConfig: merge of /etc/jitsi/jicofo/jicofo.conf: 1,system properties,reference.conf @ jar:file:/usr/share/jicofo/lib/jicofo-selector-1.0-954.jar!/reference.conf: 1
Jicofo 2022-12-02 15:59:32.111 INFOS: [1] ReadOnlyConfigurationService.reloadConfiguration#51: loading config file at path /etc/jitsi/jicofo/sip-communicator.properties
Jicofo 2022-12-02 15:59:32.114 INFOS: [1] JitsiConfig.<clinit>#68: Initialized legacyConfig: sip communicator props (no description provided)
Jicofo 2022-12-02 15:59:32.122 INFOS: [1] JitsiConfig$Companion.reloadNewConfig#94: Reloading the Typesafe config source (previously reloaded 0 times).
Jicofo 2022-12-02 15:59:32.741 INFOS: [1] JicofoServices.createAuthenticationAuthority#199: Authentication service disabled.
Jicofo 2022-12-02 15:59:32.808 AVERTISSEMENT: [1] [xmpp_connection=client] XmppProviderImpl.createXmppConnection#172: Disabling TLS certificate verification!
Jicofo 2022-12-02 15:59:32.910 INFOS: [1] XmppServices.<init>#48: No dedicated Service XMPP connection configured, re-using the client XMPP connection.
Jicofo 2022-12-02 15:59:32.913 INFOS: [1] XmppServices.<init>#63: No Jigasi detector configured.
Jicofo 2022-12-02 15:59:32.963 INFOS: [1] BridgeSelector.<init>#57: Using org.jitsi.jicofo.bridge.SingleBridgeSelectionStrategy
Jicofo 2022-12-02 15:59:32.963 AVERTISSEMENT: [1] JicofoServices.<init>#100: JVB health-checks disabled
Jicofo 2022-12-02 15:59:32.969 INFOS: [1] [type=bridge brewery=jvbbrewery] BaseBrewery.<init>#101: Initialized with JID=jvbbrewery@internal.auth.gofast-preprod-comm.ceo-vision.com
Jicofo 2022-12-02 15:59:33.154 INFOS: [1] [type=jibri brewery=jibribrewery] BaseBrewery.<init>#101: Initialized with JID=jibribrewery@internal.auth.gofast-preprod-comm.ceo-vision.com
Jicofo 2022-12-02 15:59:33.155 INFOS: [1] JicofoServices.<init>#129: No SIP Jibri detector configured.
Jicofo 2022-12-02 15:59:33.170 INFOS: [16] [xmpp_connection=client] XmppProviderImpl.doConnect#209: Connected, JID=null
Jicofo 2022-12-02 15:59:33.181 INFOS: [1] JicofoServices.<init>#143: Starting HTTP server with config: host=null, port=8888, tlsPort=8843, isTls=false, keyStorePath=null, sendServerVersion=true.
Jicofo 2022-12-02 15:59:33.240 INFOS: [16] [xmpp_connection=client] XmppProviderImpl.fireRegistrationStateChanged#314: Set replyTimeout=PT15S
Jicofo 2022-12-02 15:59:33.280 INFOS: [21] AvModerationHandler.registrationChanged#118: Discovered av_moderation component at avmoderation.gofast-preprod-comm.ceo-vision.com.
Jicofo 2022-12-02 15:59:33.283 INFOS: [22] ConferenceIqHandler.registrationChanged#192: Using breakout room component address: breakout.gofast-preprod-comm.ceo-vision.com
Jicofo 2022-12-02 15:59:33.356 INFOS: [27] [type=jibri brewery=jibribrewery] BaseBrewery.addInstance#342: Added brewery instance: jibribrewery@internal.auth.gofast-preprod-comm.ceo-vision.com/jibri
Jicofo 2022-12-02 15:59:33.357 INFOS: [32] [type=bridge brewery=jvbbrewery] BaseBrewery.addInstance#342: Added brewery instance: jvbbrewery@internal.auth.gofast-preprod-comm.ceo-vision.com/videobridge
Jicofo 2022-12-02 15:59:33.368 INFOS: [27] JibriDetector.onInstanceStatusChanged#98: Creating a new instance for jibribrewery@internal.auth.gofast-preprod-comm.ceo-vision.com/jibri, available = true
Jicofo 2022-12-02 15:59:33.381 INFOS: [1] org.eclipse.jetty.server.Server.doStart: jetty-11.0.10; built: 2022-06-16T20:42:17.891Z; git: d21dded5817960ec3c753a7ba02ef86f7c9ed89e; jvm 11.0.17+8-LTS
Jicofo 2022-12-02 15:59:33.385 INFOS: [32] BridgeSelector.addJvbAddress#92: Added new videobridge: Bridge[jid=jvbbrewery@internal.auth.gofast-preprod-comm.ceo-vision.com/videobridge, version=2.2.9-g8cded16e, relayId=null, region=null, stress=0,00]
Jicofo 2022-12-02 15:59:33.395 GRAVE: [25] [type=jibri brewery=jibribrewery] BaseBrewery.start#176: Failed to create room.
org.jivesoftware.smack.XMPPException$XMPPErrorException: XMPP error reply received from jibribrewery@internal.auth.gofast-preprod-comm.ceo-vision.com: XMPPError: forbidden - auth [Only owners can configure rooms]
        at org.jivesoftware.smack.XMPPException$XMPPErrorException.ifHasErrorThenThrow(XMPPException.java:171)
        at org.jivesoftware.smack.XMPPException$XMPPErrorException.ifHasErrorThenThrow(XMPPException.java:165)
        at org.jivesoftware.smack.StanzaCollector.nextResultOrThrow(StanzaCollector.java:284)
        at org.jivesoftware.smack.StanzaCollector.nextResultOrThrow(StanzaCollector.java:228)
        at org.jivesoftware.smackx.muc.MultiUserChat.getConfigurationForm(MultiUserChat.java:844)
        at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.joinAs(ChatRoomImpl.java:295)
        at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.join(ChatRoomImpl.java:224)
        at org.jitsi.jicofo.xmpp.BaseBrewery.start(BaseBrewery.java:170)
        at org.jitsi.jicofo.xmpp.BaseBrewery.maybeStart(BaseBrewery.java:131)
        at org.jitsi.jicofo.xmpp.BaseBrewery.registrationChanged(BaseBrewery.java:153)
        at org.jitsi.impl.protocol.xmpp.AbstractXmppProvider.lambda$fireRegistrationStateChanged$0(AbstractXmppProvider.java:91)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Jicofo 2022-12-02 15:59:33.396 GRAVE: [24] [type=bridge brewery=jvbbrewery] BaseBrewery.start#176: Failed to create room.
org.jivesoftware.smack.XMPPException$XMPPErrorException: XMPP error reply received from jvbbrewery@internal.auth.gofast-preprod-comm.ceo-vision.com: XMPPError: forbidden - auth [Only owners can configure rooms]
        at org.jivesoftware.smack.XMPPException$XMPPErrorException.ifHasErrorThenThrow(XMPPException.java:171)
        at org.jivesoftware.smack.XMPPException$XMPPErrorException.ifHasErrorThenThrow(XMPPException.java:165)
        at org.jivesoftware.smack.StanzaCollector.nextResultOrThrow(StanzaCollector.java:284)
        at org.jivesoftware.smack.StanzaCollector.nextResultOrThrow(StanzaCollector.java:228)
        at org.jivesoftware.smackx.muc.MultiUserChat.getConfigurationForm(MultiUserChat.java:844)
        at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.joinAs(ChatRoomImpl.java:295)
        at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.join(ChatRoomImpl.java:224)
        at org.jitsi.jicofo.xmpp.BaseBrewery.start(BaseBrewery.java:170)
        at org.jitsi.jicofo.xmpp.BaseBrewery.maybeStart(BaseBrewery.java:131)
        at org.jitsi.jicofo.xmpp.BaseBrewery.registrationChanged(BaseBrewery.java:153)
        at org.jitsi.impl.protocol.xmpp.AbstractXmppProvider.lambda$fireRegistrationStateChanged$0(AbstractXmppProvider.java:91)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Jicofo 2022-12-02 15:59:33.705 AVERTISSEMENT: [1] org.glassfish.jersey.server.wadl.WadlFeature.configure: JAXBContext implementation could not be found. WADL feature is disabled.
Jicofo 2022-12-02 15:59:33.813 AVERTISSEMENT: [1] org.glassfish.jersey.internal.inject.Providers.checkProviderRuntime: A provider org.jitsi.rest.Version registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.jitsi.rest.Version will be ignored. 
Jicofo 2022-12-02 15:59:33.814 AVERTISSEMENT: [1] org.glassfish.jersey.internal.inject.Providers.checkProviderRuntime: A provider org.jitsi.rest.prometheus.Prometheus registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.jitsi.rest.prometheus.Prometheus will be ignored. 
Jicofo 2022-12-02 15:59:34.064 INFOS: [1] org.eclipse.jetty.server.handler.ContextHandler.doStart: Started o.e.j.s.ServletContextHandler@58833798{/,null,AVAILABLE}
Jicofo 2022-12-02 15:59:34.075 INFOS: [1] org.eclipse.jetty.server.AbstractConnector.doStart: Started ServerConnector@10895b16{HTTP/1.1, (http/1.1)}{0.0.0.0:8888}
Jicofo 2022-12-02 15:59:34.088 INFOS: [1] org.eclipse.jetty.server.Server.doStart: Started Server@64524dd{STARTING}[11.0.10,sto=0] @2377ms

When you restart jicofo you should see:

Dec 02 14:59:49 c2s55b6c855b150	info	Authenticated as focus@auth.domain.com

And in jicofo logs:

Jicofo 2022-12-02 14:59:49.717 INFO: [17] [xmpp_connection=client] XmppProviderImpl.doConnect#205: Connected, JID=null

You don’t have this in prosody?

I have

Jicofo 2022-12-02 15:59:33.170 INFOS: [16] [xmpp_connection=client] XmppProviderImpl.doConnect#209: Connected, JID=null

But

Dec 02 15:59:33 c2s3b997f0      info    Authenticated as sgk-d1hbcb8hbrhw@gofast-preprod-comm.ceo-vision.com

You miss this in your jicofo config: client-proxy: "focus.domain.com",

Do you have sip-communicator.properties?

That’s your problem.

Maybe upload all jicofo configs here masking private info …

I have the proxy conf :

Give me a minute I will upload the sip and 2 config files