Upgraded to latest version, now "Something went wrong"

You log doesn’t look good. Here is what I see when starting jicofo:

Jicofo 2021-06-02 15:14:14.028 INFO: [1] Main.main#49: Starting Jicofo.
Jicofo 2021-06-02 15:14:14.243 INFO: [1] JitsiConfig.<clinit>#47: Initialized newConfig: merge of /etc/jitsi/jicofo/jicofo.conf: 1,system properties,reference.conf @ jar:file:/usr/share/jicofo/jicofo.jar!/reference.conf: 1
Jicofo 2021-06-02 15:14:14.245 INFO: [1] ReadOnlyConfigurationService.reloadConfiguration#51: loading config file at path /etc/jitsi/jicofo/sip-communicator.properties
Jicofo 2021-06-02 15:14:14.247 INFO: [1] JitsiConfig.<clinit>#68: Initialized legacyConfig: sip communicator props (no description provided)
Jicofo 2021-06-02 15:14:14.247 INFO: [1] JitsiConfig$Companion.reloadNewConfig#94: Reloading the Typesafe config source (previously reloaded 0 times).
Jicofo 2021-06-02 15:14:14.835 INFO: [1] XmppServices.<init>#43: No dedicated Service XMPP connection configured, re-using the client XMPP connection.
Jicofo 2021-06-02 15:14:14.876 INFO: [11] [xmpp_connection=client] XmppProviderImpl.doConnect#203: Connected, JID= null
Jicofo 2021-06-02 15:14:15.177 INFO: [1] [type=jigasi brewery=jigasibrewery] BaseBrewery.<init>#101: Initialized with JID=jigasibrewery@internal.auth.mydomain
Jicofo 2021-06-02 15:14:15.177 INFO: [11] [xmpp_connection=client] XmppProviderImpl.fireRegistrationStateChanged#328: Set replyTimeout=PT15S
Jicofo 2021-06-02 15:14:15.217 INFO: [1] [type=jigasi brewery=jigasibrewery] BaseBrewery.start#172: Joined the room.
Jicofo 2021-06-02 15:14:15.247 INFO: [1] BridgeSelector.<init>#79: Using org.jitsi.jicofo.bridge.SingleBridgeSelectionStrategy
Jicofo 2021-06-02 15:14:15.252 INFO: [1] [type=bridge brewery=jvbbrewery] BaseBrewery.<init>#101: Initialized with JID=jvbbrewery@internal.auth.mydomain
Jicofo 2021-06-02 15:14:15.265 INFO: [15] [type=bridge brewery=jvbbrewery] BaseBrewery.addInstance#338: Added brewery instance: jvbbrewery@internal.auth.mydomain/opaaaaa
Jicofo 2021-06-02 15:14:15.278 INFO: [1] [type=bridge brewery=jvbbrewery] BaseBrewery.start#172: Joined the room.
Jicofo 2021-06-02 15:14:15.286 INFO: [15] BridgeSelector.addJvbAddress#118: Added new videobridge: Bridge[jid=jvbbrewery@internal.auth.mydomain/opaaaaa, relayId=null, region=us-west-2, stress=0.00]
Jicofo 2021-06-02 15:14:15.292 INFO: [15] JvbDoctor.addBridge#140: Scheduled health-check task for: jvbbrewery@internal.auth.mydomain/opaaaaa
Jicofo 2021-06-02 15:14:15.303 INFO: [1] [type=jibri brewery=jibribrewery] BaseBrewery.<init>#101: Initialized with JID=jibribrewery@internal.auth.mydomain
Jicofo 2021-06-02 15:14:15.313 INFO: [1] [type=jibri brewery=jibribrewery] BaseBrewery.start#172: Joined the room.
Jicofo 2021-06-02 15:14:15.314 INFO: [1] JicofoServices.<init>#110: No SIP Jibri detector configured.
Jicofo 2021-06-02 15:14:15.341 INFO: [1] FocusManager.start#142: Initialized octoId=99
Jicofo 2021-06-02 15:14:15.350 INFO: [1] JicofoServices.createAuthenticationAuthority#211: Authentication service disabled.
Jicofo 2021-06-02 15:14:15.351 INFO: [1] IqHandler.init#60: Registering IQ handlers with XmppConnection.
Jicofo 2021-06-02 15:14:15.357 INFO: [1] JicofoServices.<init>#155: Starting HTTP server with config: host=null, port=8888, tlsPort=8843, isTls=false, keyStorePath=null, sendServerVersion=true.
Jicofo 2021-06-02 15:14:15.420 INFO: [1] org.eclipse.jetty.util.log.Log.initialized: Logging initialized @1656ms to org.eclipse.jetty.util.log.Slf4jLog
Jicofo 2021-06-02 15:14:15.568 INFO: [1] org.eclipse.jetty.server.Server.doStart: jetty-9.4.40.v20210413; built: 2021-04-13T20:42:42.668Z; git: b881a572662e1943a14ae12e7e1207989f218b74; jvm 1.8.0_292-8u292-b10-0ubuntu1~18.04-b10
Jicofo 2021-06-02 15:14:16.313 WARNING: [1] org.glassfish.jersey.internal.inject.Providers.checkProviderRuntime: A provider org.jitsi.rest.Version registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.jitsi.rest.Version will be ignored.
Jicofo 2021-06-02 15:14:16.570 INFO: [1] org.eclipse.jetty.server.handler.ContextHandler.doStart: Started o.e.j.s.ServletContextHandler@21e20ad5{/,null,AVAILABLE}
Jicofo 2021-06-02 15:14:16.588 INFO: [1] org.eclipse.jetty.server.AbstractConnector.doStart: Started ServerConnector@29a60c27{HTTP/1.1, (http/1.1)}{0.0.0.0:8888}
Jicofo 2021-06-02 15:14:16.589 INFO: [1] org.eclipse.jetty.server.Server.doStart: Started @2825ms

Yeah, I gathered there ought to be more, but I’m afraid this is what I’ve got :slight_smile: From your logs I get the impression you’ve got a jicofo.conf instead of all the older stuff? Can you show me an example so I can build from there?

Maybe, installing fresh on another machine and then comparing the config files would help…

:+1:

Mine is almost empty:

i# cat /etc/jitsi/jicofo/jicofo.conf
# Jicofo HOCON configuration. See /usr/share/jicofo/jicofo.jar/reference.conf for
#available options, syntax, and default values.
jicofo {
  sctp {
    enabled = false
  }
  xmpp: {
    client: {
      client-proxy: focus.mydomain
    }
  }
  conference {
    enable-auto-owner = true
  }
}

Here you can see all available settings: jicofo/reference.conf at master · jitsi/jicofo · GitHub

Ok, so the logging.properties had stuff set to warning instead of all, here is a snippet of jicofo starting up with the package default logging.properties. But still, no log entries when a conference is opened…

Jicofo 2021-06-03 08:31:22.365 INFO: [1] Main.main#49: Starting Jicofo.
Jicofo 2021-06-03 08:31:22.639 INFO: [1] JitsiConfig.<clinit>#47: Initialized newConfig: merge of system properties,reference.conf @ jar:file:/usr/share/jicofo/jicofo.jar!/reference.conf: 1
Jicofo 2021-06-03 08:31:22.642 INFO: [1] ReadOnlyConfigurationService.reloadConfiguration#51: loading config file at path /etc/jitsi/jicofo/sip-communicator.properties
Jicofo 2021-06-03 08:31:22.644 INFO: [1] JitsiConfig.<clinit>#68: Initialized legacyConfig: sip communicator props (no description provided)
Jicofo 2021-06-03 08:31:22.645 INFO: [1] JitsiConfig$Companion.reloadNewConfig#94: Reloading the Typesafe config source (previously reloaded 0 times).
Jicofo 2021-06-03 08:31:23.329 INFO: [1] XmppServices.<init>#40: No dedicated Service XMPP connection configured, re-using the client XMPP connection.
Jicofo 2021-06-03 08:31:23.382 INFO: [1] BridgeSelector.<init>#79: Using org.jitsi.jicofo.bridge.SingleBridgeSelectionStrategy
Jicofo 2021-06-03 08:31:23.387 INFO: [1] [type=bridge brewery=jvbbrewery] BaseBrewery.<init>#101: Initialized with JID=jvbbrewery@internal.auth.<meet-host>
Jicofo 2021-06-03 08:31:23.416 INFO: [1] JicofoServices.<init>#103: No Jibri detector configured.
Jicofo 2021-06-03 08:31:23.417 INFO: [1] JicofoServices.<init>#109: No SIP Jibri detector configured.
Jicofo 2021-06-03 08:31:23.419 INFO: [1] JicofoServices.<init>#115: No Jigasi detector configured.
Jicofo 2021-06-03 08:31:23.450 WARNING: [1] FocusManager.start#134: Jicofo ID is not set correctly set (value=0). Configure a valid value [1-65535] by setting org.jitsi.jicofo.SHORT_ID in sip-communicator.properties or jicofo.octo.id in jicofo.conf. Future versions will require this for Octo.
Jicofo 2021-06-03 08:31:23.464 INFO: [1] JicofoServices.createAuthenticationAuthority#217: Authentication service disabled.
Jicofo 2021-06-03 08:31:23.472 INFO: [1] IqHandler.init#95: Registering IQ handlers with XmppConnection.
Jicofo 2021-06-03 08:31:23.479 INFO: [1] JicofoServices.<init>#161: Starting HTTP server with config: host=null, port=8888, tlsPort=8843, isTls=false, keyStorePath=null.
Jicofo 2021-06-03 08:31:23.627 INFO: [1] org.eclipse.jetty.util.log.Log.initialized: Logging initialized @1585ms to org.eclipse.jetty.util.log.Slf4jLog
Jicofo 2021-06-03 08:31:23.741 INFO: [1] org.eclipse.jetty.server.Server.doStart: jetty-9.4.35.v20201120; built: 2020-11-20T21:17:03.964Z; git: bdc54f03a5e0a7e280fab27f55c3c75ee8da89fb; jvm 11.0.11+9-post-Debian-1deb10u1
Jicofo 2021-06-03 08:31:24.041 INFO: [14] [xmpp_connection=client] XmppProviderImpl.doConnect#200: Connected, JID= null
Jicofo 2021-06-03 08:31:24.260 INFO: [14] [type=bridge brewery=jvbbrewery] BaseBrewery.start#172: Joined the room.
Jicofo 2021-06-03 08:31:24.262 INFO: [27] [type=bridge brewery=jvbbrewery] BaseBrewery.addInstance#338: Added brewery instance: jvbbrewery@internal.auth.<meet-host>/7b9e69c1-163f-5f1a-98c0-071fbdca585c
Jicofo 2021-06-03 08:31:24.268 INFO: [14] [xmpp_connection=client] XmppProviderImpl.fireRegistrationStateChanged#325: Set replyTimeout=PT15S
Jicofo 2021-06-03 08:31:24.286 INFO: [27] BridgeSelector.addJvbAddress#118: Added new videobridge: Bridge[jid=jvbbrewery@internal.auth.<meet-host>/7b9e69c1-163f-5f1a-98c0-071fbdca585c, relayId=null, region=null, stress=0.00]
Jicofo 2021-06-03 08:31:24.289 INFO: [27] JvbDoctor.addBridge#140: Scheduled health-check task for: jvbbrewery@internal.auth.<meet-host>/7b9e69c1-163f-5f1a-98c0-071fbdca585c
Jicofo 2021-06-03 08:31:24.291 INFO: [27] [type=bridge brewery=jvbbrewery] BaseBrewery.addInstance#338: Added brewery instance: jvbbrewery@internal.auth.<meet-host>/03ac679c-2338-5a44-b5e0-9bbdf2f43612
Jicofo 2021-06-03 08:31:24.293 INFO: [27] BridgeSelector.addJvbAddress#118: Added new videobridge: Bridge[jid=jvbbrewery@internal.auth.<meet-host>/03ac679c-2338-5a44-b5e0-9bbdf2f43612, relayId=null, region=null, stress=0.00]
Jicofo 2021-06-03 08:31:24.294 INFO: [27] JvbDoctor.addBridge#140: Scheduled health-check task for: jvbbrewery@internal.auth.<meet-host>/03ac679c-2338-5a44-b5e0-9bbdf2f43612
Jicofo 2021-06-03 08:31:24.296 INFO: [27] [type=bridge brewery=jvbbrewery] BaseBrewery.addInstance#338: Added brewery instance: jvbbrewery@internal.auth.<meet-host>/c9fb9b7d-c6b3-5687-85a1-8cec5c625ede
Jicofo 2021-06-03 08:31:24.298 INFO: [27] BridgeSelector.addJvbAddress#118: Added new videobridge: Bridge[jid=jvbbrewery@internal.auth.<meet-host>/c9fb9b7d-c6b3-5687-85a1-8cec5c625ede, relayId=null, region=null, stress=0.00]
Jicofo 2021-06-03 08:31:24.303 INFO: [27] JvbDoctor.addBridge#140: Scheduled health-check task for: jvbbrewery@internal.auth.<meet-host>/c9fb9b7d-c6b3-5687-85a1-8cec5c625ede
Jicofo 2021-06-03 08:31:24.397 WARNING: [1] org.glassfish.jersey.server.wadl.WadlFeature.configure: JAXBContext implementation could not be found. WADL feature is disabled.
Jicofo 2021-06-03 08:31:24.520 WARNING: [1] org.glassfish.jersey.internal.inject.Providers.checkProviderRuntime: A provider org.jitsi.rest.Version registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.jitsi.rest.Version will be ignored. 
Jicofo 2021-06-03 08:31:24.834 INFO: [1] org.eclipse.jetty.server.handler.ContextHandler.doStart: Started o.e.j.s.ServletContextHandler@48f4713c{/,null,AVAILABLE}
Jicofo 2021-06-03 08:31:24.851 INFO: [1] org.eclipse.jetty.server.AbstractConnector.doStart: Started ServerConnector@73386d72{HTTP/1.1, (http/1.1)}{0.0.0.0:8888}
Jicofo 2021-06-03 08:31:24.852 INFO: [1] org.eclipse.jetty.server.Server.doStart: Started @2812ms

Now it looks different than before, it finds 3 bridges.
What is the output in jicofo log if you open 2 tabs in the same room? What is the console log error from the clients?

Hi Damian,

to my ‘untrained eye’ it looks just like the snippets we’ve seen earlier in this thread:

2021-06-03T13:27:40.070Z [JitsiMeetJS.js] <getGlobalOnErrorHandler>:  UnhandledError: Focus error, retry after 1000 Script: null Line: null Column: null StackTrace:  l.prototype._allocateConferenceFocusError@https://<meet-host>/libs/lib-jitsi-meet.min.js?v=4985:10:165609
l.prototype.allocateConferenceFocus/</<@https://<meet-host>/libs/lib-jitsi-meet.min.js?v=4985:10:164660
sendIQ/a<@https://<meet-host>/libs/lib-jitsi-meet.min.js?v=4985:1:31531
run@https://<meet-host>/libs/lib-jitsi-meet.min.js?v=4985:1:26830
_dataRecv/<@https://<meet-host>/libs/lib-jitsi-meet.min.js?v=4985:1:35268
forEachChild@https://<meet-host>/libs/lib-jitsi-meet.min.js?v=4985:1:18492
_dataRecv@https://<meet-host>/libs/lib-jitsi-meet.min.js?v=4985:1:35117
_onRequestStateChange@https://<meet-host>/libs/lib-jitsi-meet.min.js?v=4985:1:55102
Logger.js:154:22
2021-06-03T13:27:40.072Z [modules/xmpp/moderator.js] <l.prototype._allocateConferenceFocusError>:  Focus error, retry after 1000 
<iq id="f8166c10-7a9d-4520-99ab-a58a47726821:sendIQ" xmlns="jabber:client" from="focus.<meet-host>" type="error" to="pek8rpucsvbeyo-c@<meet-host>/IEHgc44-">
Logger.js:154:22
2021-06-03T13:27:40.129Z [features/overlay] <componentDidMount>:  The conference will be reloaded after 13 seconds. Logger.js:154:22
2021-06-03T13:27:40.146Z [features/base/lastn] <Ht>:  There is no active conference, not updating last N Logger.js:154:22
2021-06-03T13:27:40.163Z [conference.js] <_onConferenceFailed>:  CONFERENCE FAILED: conference.focusDisconnected focus.<meet-host> 1 Logger.js:154:22
2021-06-03T13:27:40.177Z [modules/RTC/RTCUtils.js] <obtainAudioAndVideoPermissions/s<>:  Got media constraints:  {"video":false,"audio":{"autoGainControl":true,"deviceId":"LyWHAr+MDZDgmf2xrFR4nH5He4ni12kd5QB5LounKyE=","echoCancellation":true,"noiseSuppression":true}} Logger.js:154:22
2021-06-03T13:27:40.433Z [modules/RTC/RTCUtils.js] <_getUserMedia/</<>:  onUserMediaSuccess Logger.js:154:22
2021-06-03T13:27:40.433Z [modules/RTC/JitsiLocalTrack.js] <_setStream>:  Setting new MSID: {b01e8ab5-a4f5-4ab6-8dc0-298578ee4a0c} {9353f1b4-db94-4e5e-8885-6c45d9d7170b} on LocalTrack[undefined,audio] Logger.js:154:22
2021-06-03T13:27:40.435Z [modules/detection/VADAudioAnalyser.js] <_trackAdded/this._vadInitTracker<>:  Created VAD emitter for track:  PCM2902 Audio Codec Analog Stereo Logger.js:154:22
2021-06-03T13:27:40.526Z [modules/UI/videolayout/LargeVideoManager.js] <scheduleLargeVideoUpdate/<>:  hover in local Logger.js:154:22
2021-06-03T13:27:41.171Z [modules/xmpp/moderator.js] <l.prototype.setFocusUserJid>:  Focus jid set to:  undefined Logger.js:154:22
2021-06-03T13:27:41.172Z [modules/xmpp/moderator.js] <l.prototype.createConferenceIq>:  Session ID: null machine UID: fb3abdcbbea52b1f5c4eed520d068d64 Logger.js:154:22

Also, again, there is nothing in the jicofo logs when I open conferences. It’s completely quiet. Prosody logs aren’t informative either, they just show:

Jun 03 13:31:40 mod_bosh info New BOSH session, assigned it sid ‘1071ce57-77d8-48bb-8125-520529c9a201’
Jun 03 13:31:40 bosh1071ce57-77d8-48bb-8125-520529c9a201 info Authenticated as 7dxs9vbf3mbcjkst@
Jun 03 13:31:57 bosh1071ce57-77d8-48bb-8125-520529c9a201 info BOSH client disconnected: session close

Just for kicks, can you confirm that the only real change to the prosody config is adding the “client_proxy” (with quotes) at the end of the Component line for focus?

Thanks,
Florian

Yes and the command jitsi-meet/jitsi-meet-prosody.postinst at 1b200abaa7aa4ca1ac5ac78c67877b97a87fc2b5 · jitsi/jitsi-meet · GitHub

I would guess that the error:

Maybe due to not executed the command… if you can get the full error from the console or from the Network tab, can help …

I had actually executed that mod_roster_command earlier, but did so once more to make sure. Unfortunately to no avail. Doesn’t give any console output either btw.

I have some trouble getting the data you are looking for from the console, but it looks inside that it shows:

<error type="cancel">
  <service-unavailable xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"></service-unavailable>
</error>

Hope this helps…

You are hitting this I think … jitsi-meet/mod_client_proxy.lua at b6f7f8fba7ba81c5a0ab5477a1d103b7279bf38f · jitsi/jitsi-meet · GitHub
It is like jitsi-meet/prosody.cfg.lua-jvb.example at 1b200abaa7aa4ca1ac5ac78c67877b97a87fc2b5 · jitsi/jitsi-meet · GitHub
is not connected: “focus@auth.yourdomain”

Is your config correct having target_address = "focusUser@auth.jitmeet.example.com"

Hi Damian,

Yes! That pointer was the solution, and it’s also why I was looking for the exact change/rewrite in one of my earlier posts.

I did have the added “client_proxy” but the next line was still set with the component_secret, instead of the target_address. After adding that and restarting prosody everything connected and the conference opens as expected.

So, post-mortem: The newer jitsi packages were not dealing well with config templates pushed from the ansible repository where it concerned prosody/jicofo. Now at least we figured out where the relevant changes in the ansible stuff needs to go.

Whooof, that was a major ghost-hunt. Thank you so much for sticking with me.

Have a great evening!
Florian

That’s why just doing a clean install somewhere and compare configs can give you some quick answers.