Ramblings about reverse proxying coturn behind NAT, ssl - tls external termination

Hello
as the title imply, this is not asking for help nor a how-to, it’s just a few disorganized and boring thoughts about my experience with Coturn and proxying behind a NAT.

Basic networking knowledge is necessary to understand this coturn stuff.

Coturn is the way to get through firewall blocking the normal communication with videobridge, port 10000/UDP.
To get around this, all communications should be able to work with only port 443 (https)
The current solution for the packaged, ready-to-use ‘Debian’ distribution is to use a feature called
ALPN: Application Layer Protocol Negotiation
Before doing a TLS handshake, the nginx server scans the network frame and is able to do a dispatching based on the trafic kind.
Coturn is TCP, NOT HTTP.
So the Jitsi configuration for nginx switches all HTTP trafic to the normal Web Jitsi stack, and the rest goes to Coturn.

This is working. Note however, that about 2 posts a day on this forum complain that unfortunately something went wrong when 3 users try to connect.
Why is it so is unclear since if port 10000 is blocked Coturn should work out of the box and allow a connection through port 443.

Another consideration is that Jitsi devs have announced that they want to use another system, where coturn is listening on a different IP, so coturn is actually a dedicated computer (or VM, or container)
This is a good idea because Coturn maintainer is not at all well disposed about any kind of proxying; see this github post:


This is important because it means that a proxying solution will not scale. For ‘business’ configuration this may not be a problem because well they have money and another IP address is not a significant expense when you pay for 50 Gigabits/s of connectivity.

So I tried to use default Coturn setup.
I succeeeded without much problem thanks to the good folks on this forum posting tips and how-to.

Then I tried to do what I intended all along, proxying Coturn without using an additional IP address.
Because paying for another IP address is not jusfified by the meetings usage I have.

So as my front end I replaced nginx by haproxy.
As far as I know, nginx is not able to proxy completely coturn in its open-source version.
Here is an extract of the setup I tested this week-end.
Here 1.2.3.4 by any public IP, and example.com any domain.

frontend anyhttp.example.com
    bind 1.2.3.4:80
    option httplog
    acl letsencrypt url_beg /.well-known/acme-challenge
    acl http  ssl_fc,not
    http-request redirect scheme https if http !letsencrypt
    use_backend weblocal if letsencrypt

frontend anytls.example.com
    bind 1.2.3.4:443 ssl crt /etc/haproxy/certs/cert.pem
    mode tcp
    option tcplog

    acl host_turn ssl_fc_sni turn.example.com
    acl host_jitsi ssl_fc_sni meeting.example.com

    use_backend jitsi if host_jitsi
    use_backend coturn if host_turn

    default_backend weblocal

backend weblocal
    option httpclose
    option forwardfor
    server localhost localhost:80

backend jitsi
    option httpclose
    option forwardfor
    server jitsi container_jitsi:80

backend coturn
    mode tcp
    server jitsi jitsi:3478

The let’s encrypt configuration is not essential, it’s just to show that it’s possible.
The Let’sEncrypt setup actually takes the fullchain and privkey live results, concatenate them and put the result in /etc/haproxy/certs/cert.pem. Be aware that haproxy is extremely touchy about
certificates. Concatenation order matters, rights on the certs directory matter. Error messages are about as instructive as nothing.

The backend weblocal is actually my previous nginx web setup that I changed to listen to only localhost:80
It is serving let’s encrypt requests on HTTP as well as (on HTTPS) other services that I have left out.
The let’sencrypt certificate has been created with the 2 DNS entries meeting.example.com and turn.example.com
They are pointing to the public IP address of the server.

‘container_jitsi’ is here the LXD container where I have a slightly customized Debian installation.
The forwardfor is there to allow the Jitsi nginx in the container to log real ip address (using the eponym module)

Now the Jitsi container configuration modifications.

a) nginx
The Jitsi nginx has been modified to remove everything related to https. It is only listening to port 80.
This is pretty straightforward, removing all between the server_name for port 80 and the root /usr/share/jitsi-meet; in the https section.
So the 2 servers sections are merged into one, serving only http.

b) prosody

turncredentials = {
--  { type = "stun", host = "meeting.example.com", port = "4446" },
--  { type = "turn", host = "meeting.example.com", port = "4446", transport = "udp" },
  { type = "turns", host = "turn.example.com", port = "443", transport = "tcp" }
};

c) coturn
I removed the certificates stuff - unneeded since I handle certificates only at the front end level.
There is a small problem here, if the certificates are removed coturn will stop to listen to tcp.
That’s because coturn is a crafty software, it sees in the config file the instruction ‘no-tcp’, sees also certificates, hence it concludes that the user really wants to use tcp and ignore the no-tcp instruction. If the certificates are removed, the no-tcp line takes effect.
So to use coturn without certificates, the no-tcp line has to be commented out.
After that ss -tapnu | grep turn should show that coturn is listening on 3478/tcp (that’s the value that my haproxy setup is using)

In my Jitsi-generated config file I had also a strange line:

external-ip=;; connection timed out; no servers could be reached

this is because I am installing Jitsi in a container that has by default no access to Internet.
The install process tries to access a STUN server to get the external public IP address, fails and this is the result.
So this has to be fixed by hand:

external-ip=1.2.3.4/10.0.0.2

where 1.2.3.4 is the public IP address of the Jitsi server and 10.0.0.1 of the Jitsi container.
This is a problem that could happen with any Jitsi coturn setup in NAT context.
It may be a reason why coturn does not work with some out of the box installations.

Now that the front end can connect to Coturn, coturn itself has also to actually connect to the jvb.
This is actually done in 2 parts:

b-1) coturn has to know where to connect. The client (Chrome, Firefox…) sends a Dns name (the one setup in the prosody config file)
This could be resolved by the jitsi container (server) to a public IP address. This will be NOT good.
So the /etc/hosts or the hostname of the container (server) has to be set so that the public DNS name resolve to the private IP address of the JVB.
Public IP address -> haproxy -> (private ip address) coturn jitsi -> (private ip address) jvb

b-2) coturn has to actually accept to connect to this address. By default it is setup by Jitsi configuration to deny just that.
All private IP addresses ranges are blacklisted.
In the turn log there will be lines like:

Sep 7 08:09:33 jitsitest turnserver: 705: A peer IP 10.0.0.2 denied in the range: 10.0.0.0-10.255.255.255

when the coturn server is asked to serve a client.
To allow it, the denying line has to be commented out like that:

#denied-peer-ip=10.0.0.0-10.255.255.255

This may be another reason why coturn does not work out of the box with some installations.

Miscellaneous:

  • you can turn udp off (no-udp) it’s useless to have coturn listen to udp it serves only to have huge output to ss -tapnu | grep turn.
    When you are asking coturn to listen to port 3478 it listens automatically to port 3479 because it cans. No idea if this can be turned off.

  • Logging: by default Jitsi configuration will spam syslog. If this is not wanted (I don’t) it’s possible to do that:
    #syslog
    log-file=/var/log/turnserver/coturn.log
    for some mysterious reason, this will block initial configuration dumping to the log.
    The ‘verbose’ instruction allows to set a very annoying log spamming but can be useful for debugging.

  • It’s possible to enable coturn supervision
    ** telnet: warning do not do that on open internet server !
    uncomment no-cli and add a password with cli-password=verysecretpassword
    Then telnet localhost 5766 (or nc 127.0.0.1 5766).
    2 commands of interest:
    pc : dump configuration
    ps: dump coturn current connexions state, allow to confirm that coturn is actually used and look at bandwith used.
    Users having UDP/10000 access may also open coturn sessions even if they are not used. If there are connexions issues, they may be switched to use coturn (not sure about that but I think to have seen it). Generally speaking connections issues are bad with networking.
    It’s also possible to enable web supervision:
    ** web-admin-port=8081 (I use 8081 because 8080 can often used elsewhere and coturn silently ignores this instruction if the port is already used)
    Then;
    turnadmin -A -u admin -p verysecretpassword
    Web supervision has same capabilities.

Limitations

  1. performance: coturn proxying will be bad for performance (see above comment of Coturn maintainer).
    Coturn should be avoided if possible anyway because, well, TCP.

  2. logging: coturn client IP address logging will be incorrect because of proxying.
    There is no workaround since Coturn maintainer does not support proxying. He has included proxying in recent versions but only for the Stun protocol, not for turn.

Both problems can be addressed by, well, using a second IP address. This may be costly for a limited usage, and not possible for home users, though.

1 Like