Android App Rejecting Let's Encrypt Certificates for TURNS

I’ve run into a problem (open issue on GitHub here) where the Android app appears to be rejecting the Let’s Encrypt certificate chain generated via the “install-letsencrypt-cert.sh” script when attempting to talk TURNS with coturn.

Packet captures indicate that the proper certificate chain is sent to the Android app by coturn and that the app is rejecting the certificate due to the CA being unknown. This appears to only affect Let’s Encrypt certificates as the app can successfully talk TURNS with coturn after switching to an InCommon signed certificate and chain.

Has anyone else happen to run into this issue?

The letsencrypt certificate is supposed to be cross-signed by another authority:
if you browse to letsencrypt.org and you click on the lock icon you’ll see that it is signed by (issued by) Let’s Encrypt Authority X3, the cross-signed certificate (as such it should be trusted by old devices). In your trace do you see a similar certificate or is it different ?

I do see the cross-signed certificate in my packet captures. In the TLS Server Hello packet containing the certificates the following are sent to the client:

  • The server certificate signed by “Let’s Encrypt Authority X3”
  • The “Let’s Encrypt Authority X3” certificate signed by “DST Root CA X3”

The app installation I’m testing with is running on Android 10.

I think now that the problem is the missing cross-signed certificate
Here is the issue where this is tracked. You’ll notice that it is said that there is a fix to solve this problem until 2021. Can you try to add the missing cross-signed certificate manually ? You could get away by simply concatenating the certificate here to the crt file pointed to by ssl_certificate in the nginx config.
It this works, it could be that there is a big problem in the jitsi app, it should use the Android store that should include the Letsencrypt root now (you said in the github issue that the certificate is good in Android browser so the Android cert store should be all right I guess).
Maybe a jitsi person chould know if jitsi Android app use its own certificates ? Sorry I’m too lazy to check this now :slight_smile:

Both coturn and nginx are serving a chain with the cross-signed CA certificate after running the “install-letsencrypt-cert.sh” script (fullchain.pem). Just to make sure that it’s not a problem with the fullchain.pem file generated by certbot I manually concatenated the current server certificate and the cross-signed CA certificate in a separate file and configured nginx and coturn to use it with the same results.

The app appears perfectly happy with the Let’s Encrypt chain (the server certificate plus the cross-signed CA certificate) for communicating with everything but the TURN server: I can start and join conferences on the app with a Let’s Encrypt chain running on nginx and have no media issues when using a certificate chain signed by another CA on the TURN server. It makes me wonder if the TURN and/or ICE component of the app is looking at a different certificate store than the rest of the app (I’m not a mobile developer, so I’m not sure where to start in looking through the code).

Yes I missed this.
I know a bit about certs but not a lot about TURN unfortunately. Maybe I’ll take a look tomorrow how it’s really working in Jitsi.

Right, I have looked a bit more at this and I’m still mystified by the system. Problem is that I don’t need it all, I have full control on my firewall and I just let in UDP port 10000 and all is well.
So how is it supposed to work is still murky. I have setup a test instance with the turn thingy installed and it seems the turn server is installed.
dpkg --get-selections | grep coturn
coturn install

What’s strange is that I don’t see how installing a turn server on the same host with jitsi is installed could get things any better; since firewall is supposed to block UDP, how a software on the same host can solve the problem is just strange. Using an external turn server as pointed here needs the 10000-20000 range open. However if you can open the 10000-20000 range you can also open the single UDP port 10000 and simply not use coturn. Baffling.
Anyway, the coturn server use a specific certificate as pointed here and if you really use coturn it’s in /etc/turnserver.conf that you’ll find the reference to the used certificate. If turn server is on a different host as it’s supposed to be, obviously it must have a different name on its certificate (it could be the same certificate if you use a DNS validated certificate, but you need to transfert it somehow on the turn server using sftp or something else)
Hope my uninformed ramblings could help you some :slight_smile:

It seems as if there is indeed a problem with the Android app not accepting Let’sEncrypt certificates for turns:// connections. I’ve left a comment with more details on a Github issue that already existed:

if I understand correctly, the problem is that Certbot configures correctly the Nginx server with the cross-signed certificate but that the setup of the certificate in the turn server includes only the modern, not cross-signed certificate. And that Jitsi should correct the Android app build before March 2021 because at this point the cross-signed certificate will not be valid anymore…

Its not about what is configured on the server side, nginx and the turn server should both serve the fullchain including intermediate CAs and the leave certificate.

The problem – as I understand it – is on the client side:

  • If the Android app connects to the webserver (nginx) via https it verifies the server certificate using the list of trusted root CAs that is built into the Android OS. Let’s Encrypt certificates are verified successfully.
  • However, if the app cannot establish a connection to the videobridge directly and needs to fallback to the turnserver a “different part” of the Jitsi Meet Android app is used to establish the turns:// connection: This connection is apparently handled by a webrtc library (which seems to be included in the Jitsi App as a native library via react-webrtc). And the webrtc library does not use the list of trusted CAs which is provided by the Android OS. Instead, another list of trusted CAs is compiled into this library and only the CAs contained in this hardcoded list are used to verify the certificate that the app receives from the turn server. Unfortunately, this hardcoded list of CAs does not include the trust anchor that’s needed to trust a certificate issued by Let’s Encrypt.

In my opinion, this is an unfortunate design of the webrtc library in multiple ways.

  • First, the hardcoded list of root CAs is badly chosen because the list seems to be published by google for a different purpose: It appears to be a list of CAs that are issuing certificates for any of the various google services and thus may be required to be trusted if one wants to use these google services.
  • Second, a library should not come with a hardcoded list of CAs, especially if it does not provide an API to override the list. If Google and Mozilla have a hard time to maintain a list of trustworthy CAs, how (ad why) should a library maintain its own list.
  • Third, apps that include this library are required to upgrade to the latest release of the library on a regular basis just to keep the list of CAs up to date.
  • Fourth, the existence of this CA list and the fact that it is used by the webrtc library for TURNS connections seems to be badly documented,

The above is how I understand the issue and I may of course be wrong on some points. Hopefully, @saghul can shed some light on this, please?