PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed

Hello, I had a working installation of Jitsi and I updated it a few days ago with latest release. The installed system did not work, so I tried again from scratch, following the “self hosting guide”. The installation still have problems.

These are the steps I’ve done:

# systemctl stop prosody jitsi-meet jitsi-videobridge2.service coturn jicofo
# dpkg --purge prosody jitsi-meet-prosody jitsi-videobridge2 jitsi-meet-web-config jitsi-meet-web jitsi-meet-turnserver jitsi-meet-tokens jitsi-meet jicofo coturn lua5.2 luarocks lua-any
# rm -fr /usr/share/jitsi-videobridge /etc/jitsi/videobridge /etc/jitsi /var/lib/prosody /etc/prosody/certs 
# apt install jitsi-meet

This is the error I see:

# tail -47 /var/log/jitsi/jicofo.log 
Jicofo 2022-11-19 12:52:20.151 AVVERTENZA: [96] org.jivesoftware.smack.AbstractXMPPConnection.callConnectionClosedOnErrorListener: Connection XMPPTCPConnection[not-authenticated] (0) closed with error
javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:353)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:296)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:291)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
	at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1416)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection.proceedTLSReceived(XMPPTCPConnection.java:733)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection.access$1400(XMPPTCPConnection.java:131)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:990)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$700(XMPPTCPConnection.java:916)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:939)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
	at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369)
	at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:275)
	at java.base/sun.security.validator.Validator.validate(Validator.java:264)
	at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1341)
	... 17 more
Caused by: java.security.cert.CertPathValidatorException: signature check failed
	at java.base/sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)
	at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:224)
	at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:144)
	at java.base/sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83)
	at java.base/java.security.cert.CertPathValidator.validate(CertPathValidator.java:309)
	at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:364)
	... 23 more
Caused by: java.security.SignatureException: Signature does not match.
	at java.base/sun.security.x509.X509CertImpl.verify(X509CertImpl.java:422)
	at java.base/sun.security.provider.certpath.BasicChecker.verifySignature(BasicChecker.java:166)
	at java.base/sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:147)
	at java.base/sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
	... 28 more

Do you have any hints on what to check? I saw similar errors and they were due to not following the guide, but I did follow the guide, so the error may be different.

Thank you,
Giuseppe

Welcome to the forum.

This is not the correct way to do a reinstall; you purged and then tried a reinstall. The error you’ve noted suggests you have problems with certs. And that makes sense because you removed prosody certs in your purge.

You are right: the old certificates should have been removed. But the installation should have created new ones, isn’t it? In fact I reinstalled prosody, as well.

Thank you,
Giuseppe

Purge everything as you did before with the folder deletion.
Execute this line

And then start over with the installation.
Did that help?

No, that did not fix the problem.
Should I remove everything from /var/lib/prosody/ and /etc/prosody after the purge completes?

Thank you,
Giuseppe

You may check the following but don’t apply it directly if there is a desktop on this server:

Hum, strange… The problem are the certs old ones got such in the local trust store on the computer and when you generate new ones they are ignored when the command is executed.

Basically I think the following should fix a deployment broken in that way:

rm /usr/local/share/ca-certificates/$JICOFO_AUTH_DOMAIN.crt
update-ca-certificates -f
ln -sf /var/lib/prosody/$JICOFO_AUTH_DOMAIN.crt /usr/local/share/ca-certificates/$JICOFO_AUTH_DOMAIN.crt
update-ca-certificates -f
service jicofo restart

You just need to fill in JICOFO_AUTH_DOMAIN.

I /usr/local/share/ca-certificates there was a link to the certificate file in /var/lib/prosody/ . The update-ca-certificates did not fix the problem.

So, I am removing and purging all packages again, then I am removing the configuration directories, including the ones with prosody certificates and its link. Here it is:

# systemctl stop prosody jitsi-meet jitsi-videobridge2.service coturn jicofo
Failed to stop jitsi-meet.service: Unit jitsi-meet.service not loaded.

# dpkg --purge prosody jitsi-meet-prosody jitsi-videobridge2 jitsi-meet-web-config jitsi-meet-web jitsi-meet-turnserver jitsi-meet-tokens jitsi-meet jicofo coturn lua5.2 luarocks lua-any
(Lettura del database... 128064 file e directory attualmente installati.)
[...]
groupdel: group 'prosody' does not exist
dpkg: attenzione: nel rimuovere prosody, la directory "/var/lib/prosody" è risultata non vuota e non viene rimossa
dpkg: attenzione: nel rimuovere prosody, la directory "/etc/prosody/certs" è risultata non vuota e non viene rimossa
Elaborazione dei trigger per systemd (241-7~deb10u8)...
Elaborazione dei trigger per man-db (2.8.5-2)...
Elaborazione dei trigger per libc-bin (2.28-10+deb10u2)...

# rm -fr /usr/share/jitsi-videobridge /etc/jitsi /var/lib/prosody /etc/prosody/

# ls -l /usr/local/share/ca-certificates/
totale 0

Then, I reinstall from scratch. This will not find nor use any old certificate.

# LANG=C apt install jitsi-meet
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  coturn jicofo jitsi-meet-prosody jitsi-meet-turnserver jitsi-meet-web jitsi-meet-web-config jitsi-videobridge2 lua-any luarocks prosody
Suggested packages:
  sip-router lua-dbi-mysql lua-dbi-postgresql lua-dbi-sqlite3 lua-event lua-ldap
Recommended packages:
  lua-readline lua-unbound
The following NEW packages will be installed:
  coturn jicofo jitsi-meet jitsi-meet-prosody jitsi-meet-turnserver jitsi-meet-web jitsi-meet-web-config jitsi-videobridge2 lua-any luarocks prosody
0 upgraded, 11 newly installed, 0 to remove and 1 not upgraded.
Need to get 512 kB/75.4 MB of archives.
After this operation, 131 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
[...]

At the first question, I enter my hostname, the one to be used in all URLs. At the second question I select the Let’s Encrypt certificate provider. At the third I input my email address to be used in the certificate request. At the fourth I discard the telephony support.

The installation goes to end without any errors:

[...]
Setting up luarocks (3.8.0+dfsg1-1prosody~buster1) ...
Setting up jitsi-meet-turnserver (1.0.6776-1) ...
Configuring turnserver
Setting up jitsi-meet (2.0.8044-1) ...
Processing triggers for systemd (241-7~deb10u8) ...
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for libc-bin (2.28-10+deb10u2) ...
Scanning processes...                                                                                                                                                                                              
Scanning processor microcode...                                                                                                                                                                                    
Scanning linux images...                                                                                                                                                                                           

Running kernel seems to be up-to-date.

The processor microcode seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

Then, I check the link for the certificate and the log directory:

# ls -l /usr/local/share/ca-certificates
totale 0
lrwxrwxrwx 1 root staff 41 nov 20 15:35 auth.smaug.domain.tld.crt -> /var/lib/prosody/auth.smaug.domain.tld.crt

# ls -l /var/log/jitsi/
totale 532
-rw-r--r-- 1 jicofo jitsi 484468 nov 20 15:39 jicofo.log
-rw-r--r-- 1 jvb    jitsi  55377 nov 20 15:35 jvb.log

And here you may see the error:

# tail -30 /var/log/jitsi/jicofo.log 
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection.proceedTLSReceived(XMPPTCPConnection.java:733)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection.access$1400(XMPPTCPConnection.java:131)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:990)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$700(XMPPTCPConnection.java:916)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:939)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
	at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369)
	at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:275)
	at java.base/sun.security.validator.Validator.validate(Validator.java:264)
	at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1341)
	... 17 more
Caused by: java.security.cert.CertPathValidatorException: signature check failed
	at java.base/sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)
	at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:224)
	at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:144)
	at java.base/sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83)
	at java.base/java.security.cert.CertPathValidator.validate(CertPathValidator.java:309)
	at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:364)
	... 23 more
Caused by: java.security.SignatureException: Signature does not match.
	at java.base/sun.security.x509.X509CertImpl.verify(X509CertImpl.java:422)
	at java.base/sun.security.provider.certpath.BasicChecker.verifySignature(BasicChecker.java:166)
	at java.base/sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:147)
	at java.base/sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
	... 28 more

All of this happens on a completely updated Debian 10 machine:

# lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 10 (buster)
Release:	10
Codename:	buster

using the prosody suggested by Jitsi, and latest jicofo:

# apt policy prosody
prosody:
  Installato: 0.12.1-1~buster1
  Candidato:  0.12.1-1~buster1
  Tabella versione:
 *** 0.12.1-1~buster1 500
        500 http://packages.prosody.im/debian buster-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     0.11.2-1+deb10u4 500
        500 http://deb.debian.org/debian buster/main amd64 Packages
        500 http://security.debian.org/debian-security buster/updates/main amd64 Packages

# apt policy jicofo | head
jicofo:
  Installato: 1.0-954-1
  Candidato:  1.0-954-1
  Tabella versione:
 *** 1.0-954-1 500
        500 https://download.jitsi.org stable/ Packages
        100 /var/lib/dpkg/status
     1.0-940-1 500
        500 https://download.jitsi.org stable/ Packages
     1.0-934-1 500

Have you run the update certificates with -f option to force rebuild the trust store?

Yes, I copied the command from here. I’ve just checked the bash command history, and I may confirm this.

Or, do you mean I should have run the command after the purge?

Thank you,
Giuseppe

Yes, after cleaning everything.

I just did it. Nothing changed.

Yeah no idea, its something around the truststore having the wrong certs.

You can disable the certificate verification in jicofo.conf
This is the setting jicofo/reference.conf at 4db09bda3eeb30ec6c86e9a828d987006de0f0de · jitsi/jicofo · GitHub

I have no idea either, but I just solved it. After removing all Jitsi related packages and after removing some configuration files (as already shown), I also autoremoved (with purge) all remaining packages that were not required anymore but were installed as dependencies.
This fixed the problem, since the following installation worked like a charm.
Thank you for your time.