Octo: Not working using muc approach

Hi @damencho and @Boris_Grozev.
I have still a problem to get a configuration for cascading bridges. After configuring jitsi to use xmpp muc to handle and manage the JVBs in 2 regions, i got the message “Unfortunately something went wrong…” when the second participant is joining the room.
I tried using stable and unstable jitsi. Apparently. the bridge is not known by jicofo
My configuration has been done following the thread: how-to-add-the-secondary-jvb-to-main-jitsi-server/
Excerpt from jicofo.log:
Jicofo 2019-05-07 12:10:06.936 SEVERE: [55] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Can not invite participant – no bridge available.

Jicofo 2019-05-07 12:10:06.938 SEVERE: [55] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Can not invite participant – no bridge available.

Excerpt of the jvb.log
JVB 2019-05-07 12:01:33.705 WARNING: [25] org.jitsi.videobridge.EndpointMessageTransport.log() SCTP connection with 9e1b25f99ef8152b not ready yet.

JVB 2019-05-07 12:01:33.705 WARNING: [25] org.jitsi.videobridge.EndpointMessageTransport.log() No available transport channel, can’t send a message

JVB 2019-05-07 12:01:33.741 INFO: [28] org.jitsi.videobridge.health.Health.log() Performed a successful health check in 106ms. Sticky failure: false

JVB 2019-05-07 12:01:34.936 INFO: [24] org.jitsi.xmpp.mucclient.MucClientManager.log() Setting a presence extension: org.jitsi.xmpp.extensions.colibri.ColibriStatsExtension@2a012e71

JVB 2019-05-07 12:01:39.937 INFO: [24] org.jitsi.xmpp.mucclient.MucClientManager.log() Setting a presence extension: org.jitsi.xmpp.extensions.colibri.ColibriStatsExtension@23b51807

JVB 2019-05-07 12:01:43.741 INFO: [28] org.jitsi.videobridge.Videobridge.log() CAT=stat create_conf,conf_id=b20ba87c83115141 conf_name=null,logging=false,conf_count=1,ch_count=0,v_streams=0

Have you add this: How to add the secondary jvb to main jitsi-server ?

@damencho:

Yes I did add this to each bridge.
Maybe a review of my config could help pin point something I have missed? Also as mentioned to @Boris_Grozev, I would like to trace the expected traffic between each bridges and the shard(s). these stats

Hi @damencho and @Boris_Grozev. I am uploading the configuration files from my shard us-west.
Let me know if it helps understanding my problem.
jicofo_ sip-communicator.properties.txt (952 Bytes)
jicofo_config.txt (1.1 KB)
jvb_config.txt (883 Bytes)
jvb_sip-communicator.properties.txt (1.1 KB)
prosody.cfg.lua.txt (155 Bytes)
sfo.example.com.cfg.lua.txt (2.0 KB)

@damencho
Found the problem…in my sip-communicator.properties for videobridge, all the muc configurations are on the format:
org.jitsi.videobridge.shard.HOSTNAME=localhost
instead of
org.jitsi.videobridge. xmpp.user .shard.HOSTNAME=localhost

@damencho. I have a problem to add other JVBs installed on different servers:
unable to find valid certification path to requested target
and in /etc/jitsi/jicofo/sip-communicator.properties I already added:
org.jitsi.jicofo.ALWAYS_TRUST_MODE_ENABLED=true
Note: I am installing on additional servers (VMs) dedicated to videobridge by only deploying jitis-videobridge. I do not install the whole jitsi meet on the jvb servers.
prosody.log
ssl handshake error: sslv3 alert certificate unknown
jvb.log
JVB 2019-05-15 04:20:42.271 SEVERE: [40] org.jitsi.xmpp.mucclient.MucClientManager.log() Failed to initialize and start a MucClient:
org.jivesoftware.smack.SmackException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1072)
at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:994)
at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1010)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
at org.jivesoftware.smack.tcp.XMPPTCPConnection.proceedTLSReceived(XMPPTCPConnection.java:810)
at org.jivesoftware.smack.tcp.XMPPTCPConnection.access$1200(XMPPTCPConnection.java:151)
at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1067)
… 3 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
at sun.security.validator.Validator.validate(Validator.java:262)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621)
… 13 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)

I do not understand which certificates are required on the JVB servers
Thanks,
Christian

Jvb creates a xmpp connection to prosody and tries to verify the certificate of that connection, as we use self-sgned certs for those xmpp connections the connection fails with the above message. You can disable that check with:
org.jitsi.videobridge.xmpp.user.shard.DISABLE_CERTIFICATE_VERIFICATION=true.

Thanks @damencho. I included it in my muc configuration since the beginning. I followed the config example from @Boris_Grozev:
org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_TRANSPORT=muc
org.jitsi.videobridge.STATISTICS_INTERVAL=5000

org.jitsi.videobridge.xmpp.user.shard.HOSTNAME=localhost
org.jitsi.videobridge.xmpp.user.shard.DOMAIN=auth.boris.jitsi.net
org.jitsi.videobridge.xmpp.user.shard.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard.PASSWORD=XXX
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS=JvbBrewery@internal.auth.boris.jitsi.net
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME=jvb1
org.jitsi.videobridge.xmpp.user.shard.DISABLE_CERTIFICATE_VERIFICATION=true

So the JVB hosted on the same server than jicofo and prosody connect without problem to prosody and muc room is created and the conf works well.
the certificate error happens when i start the second JVB which is installed as stand alone media server on a different VM.
Should I do something on this stand alone vm with jitsi videobridge only?
on the JVB quick capture shows the tls exchange:
with 1XX.2XX.X3X.XX as JVB stand alone IP
and XX8.1XX.2XX.XX as Prosody / Jicofo server IP

8 0.013678958 1XX.2XX.X3X.XX → XX8.1XX.2XX.XX XMPP/XML 129 STARTTLS 
9 0.015753071 XX8.1XX.2XX.XX → 1XX.2XX.X3X.XX XMPP/XML 118 PROCEED 

10 0.019783588 1XX.2XX.X3X.XX → XX8.1XX.2XX.XX TLSv1.2 272 Client Hello
11 0.024552983 XX8.1XX.2XX.XX → 1XX.2XX.X3X.XX TLSv1.2 1746 Server Hello, Certificate, Server Key Exchange, Certificate Request, Server Hello Done
12 0.024586683 1XX.2XX.X3X.XX → XX8.1XX.2XX.XX TCP 68 52930 → 5222 [ACK] Seq=439 Ack=2120 Win=33792 Len=0 TSval=3891777267 TSecr=3185608098
13 0.026870665 1XX.2XX.X3X.XX → XX8.1XX.2XX.XX TLSv1.2 75 Alert (Level: Fatal, Description: Certificate Unknown)

Thanks!

Is this the second bridge config, the one that is not where prosody is?

Yes. It is the configuration example. Hostname and other values replaced with public IPs and correct username etc… I can post the configuration of the standalone config if it is better to understand the context

You get the cert error even with DISABLE_CERTIFICATE_VERIFICATION?

Maybe post the whole bridge log from the beginning with the error?

Thanks @damencho. It works well now.
Next step is interconnecting 2 shards:
Shard1 with 2 JVBs (jvb11 & jvb12)
Shard2 with 2 JVBs (jvb21 & jvb 22)
jvb11 & jvb12 connected to Shard1 & Shard2 mucs
jvb21 & jvb22 connected to Shard1 & Shard2 mucs
I assume that besides the description of each shard ( org.jitsi.videobridge. xmpp.user .shardx.*) in sip-communicator.properties files I need to have the following line:
org.jitsi.videobridge.AUTHORIZED_SOURCE_REGEXP=focus@xxxxxxx.net for each xmpp shard domain, right? Or is it more relevant to have the same xmpp domain for all shards?
Thanks

It is a regexp show you can come up with something that covers your domains.

Thanks! It makes sense. As best practice or in your experience at Jitsi, each shard share the same xmpp domain like jitsi.example.com or each shard has its own XMPP domain or sub domain?

For meet.jit.si we are using same domain meet.jit.si everywhere.

Thanks @damencho.
So i am confused about the usage of certificates for the same XMPP domain: for each shard which is using the same XMPP domain (i.e. jitsi.example.com) , I should copy /etc/prosody/certs/jitsi.example.* and /etc/prosody/certs/auth.jitsi.example.com.* on every shard? or import it via prosodyctl cert import?
Thanks

These certs are internal and there is no need to copy/import or move them around, leave everything by default.
The only thing is that jvbs may be connecting to the auth.* domain, so if you are running jvb octo communication through secured network as recommended, than you just need to disable cert checking in jvb. Otherwise you need to copy the public auth… certificate and make it trusted on the jvb machine.