ERR_EMPTY_RESPONSE Jitsi connections fail for clients that don't pass ALPN during SSL/TLS handshake (ClientHello)

Hi,

I’ve observed that some of my corporate clients were unable to connect to my Jitsi deployments, after I updated deploy configurations to use latest packages that updated Nginx multiplexing Regexps for h2 and http/1.1 in /etc/nginx/modules-enabled/60-jitsi-meet.conf.

My clients complained of an ERR_EMPTY_RESPONSE on their Chrome browser when they tried to access my Jitsi URL from their official laptops. At the same time, I saw this in my /var/log/nginx/error.log:

2020/08/11 06:24:08 [error] 10929#10929: *27725 recv() failed (104: Connection reset by peer) while proxying and reading from upstream, client: <client-ip>, server: 0.0.0.0:443, upstream: "127.0.0.1:5349", bytes from/to client:330/3253, bytes from/to upstream:3253/609

I added an Nginx log_format to print ssl_preread_alpn_protocols in my access_log and RCA revealed that these client machines weren’t sending ALPN values somehow, even with latest versions of Chrome/Firefox browsers. I was able to reproduce an empty response behavior via cURL on my Mac OS machine that has an installation of cURL that is known to not send ALPN.

As per online forums and this StackOverflow answer, it seems like client machines have an antivirus/firewall installation that downgrades HTTP2 connections and strips ALPN values. The ngx_stream_ssl_preread_module on my server thus receives a blank value for ssl_preread_alpn_protocols. As a result, my Jitsi installation is unable to distinguish between an HTTP request and a TURN request and defaults everything to my TURN server (running on port 5349) which returns an empty response of course.

Temporary solution that works for now

The whole point for me to rely on TURN was because corporate clients were unable to use Jitsi installs from their official machines with or without VPN. I switched back to my previous setup with a dedicated Coturn server running on port 443 and a dedicated Jitsi server serving at port 443. I also removed /etc/jitsi/modules-enabled/60-jitsi-meet.conf and updated my Nginx configuration to use port 443 rather than 4444. This has allowed for smooth connectivity as now regardless of ALPN values, my Nginx server serves Jitsi and my Coturn server serves p2p requests.

However, there are some clients who are able to access my Jitsi web interface but their p2p fails and they keep getting thrown out of the call. They are not able to see other participants either. IMO, we need a better solution to this as neither a separate TURN server completely covers all scenarios, nor multiplexing on the basis of ALPN values works for all clients.

@damencho @saghul What do you think? Is there a better way to do this?

AFAIK this solution has no future, it has been stated somewhere on the Github (sorry lazy this morning) that the Jitsi project is planning to use a separate turn domain in the future and remove ALPN.

about p2p if it fails you can always downgrade it by disabling it in the config.js.

My coturn chronicles

1 Like

Thanks for sharing, @emrah and @gpatel-fr . Using a separate turndomain indeed seems logical. I anyway generate a separate certificate for my turnserver, so that should work just fine.

About downgrading p2p, I can’t do UDP because it’s blocked on the client-side. What other options do I have?

If you disable p2p in the config;js, one-on-one communications wll get through jvb, so what could be the problem with turn ?

But doesn’t JVB communicate over UDP port 10000 only? The reason I need TURN is because clients can’t communicate with JVB directly over UDP. So downgrading p2p to JVB is not an option for me. Is there any other option?

beats me - I have thought that TURN was doing an interface between TCP 443 and UDP 10000.

workstation – tcp --> port 443 turn server – udp --> port 10000 jvb

if it does not work like that how meet.jit.si can be connected through the famous corporate firewall when users are more than 2 ?

That is exactly how it works.

But here we are talking about TURN configuration not working out in cases where ALPN is not sent. In that case, assuming that we use turndomain for multiplexing instead of using ALPN, I don’t see any other option to downgrade upon (falling back on JVB is not an option as we have just established). In another thread, @damencho mentioned that it could be due to the fact that I’m only relying on Let’s Encrypt certificate and not using fullchain. I’m going to try that out to see if it works.

@gpatel-fr I had a feeling we have discussed my use case before. [Here] (UnhandledError: Uncaught NotSupportedError: Failed to construct 'RTCPeerConnection': Failed to initialize native PeerConnection)

Indeed what you are meaning by p2p is not at all what I understand. For me p2p is point to point, that is
workstation --> (intercession of jicofo) --> workstation
I think that the p2p referenced in the config.js is the meaning I understand, that is, browser s talking to browser without any role for JVB.

However if a distinct domain is used for turn, what is exactly the problem for clients connecting to the turn server without ALPN ? the dispatching should be done by nginx (or whatever proxy is used) based on the domain name as for any old TLS connection. To be candid I never tested it myself but I intend to :slight_smile:

p2p configuration in config.js along with useStunTurn set to true would force using STUN servers and a TURN relay for ICE negotiation, as advertised in turncredentials configuration set in my prosody config. It propagates the following to each XMPP client who connects, for ICE negotiation in p2p mode:

  1. A STUN server host and port
  2. A TURN server host and port
  3. A turns or TURN over SSL host and port

In my use case:

  1. Direct connection to JVB via UDP fails.
  2. STUN connectivity fails because even with public address discovery, my clients are not allowed to establish p2p connections with each other behind NAT.
  3. ICE then falls on a TURN relay provided by Coturn and translates client IP addresses into TURN’s public IP Address and a unique port map that it uses to relay packets between clients via TURN’s public IP address.
  4. If even the TURN relay was not reachable or a client was unable to connect to it, ICE fails and the conference fails for that client.

This is as per my understanding of how WebRTC works behind NAT. Others may comment for corrections.

Ref: Why Does Your WebRTC Product Need a TURN Server