Fallback to port 443 does not work as expected

Install & Config
#1

Hi folks,

after semi-hijacking another thread (sorry for that) I decided to create an own one, as I do not want to disturb the other topic. After going back and forth too often while troubleshooting I decided to start with a fresh server, mybe that brings me to the finish line :smiley:

Whats the goal?
Running jitsi instance which itself sits behind a NAT. Connections should fall back to TCP 443 only, if the client sits behind a corporate firewall. If possible, everything running on a single server VM (only up to 20 participants at all)

With what did I start?
fresh server Ubuntu 20.04. in private network (192.168.x.x) behind firewall with NAT with the following ports forwarded to the VM:
TCP Ports 443, 80, 22, 5349
UDP Ports 10000, 3478

DNS entries for both meet.blahblah.com and turn.blahblah.com on public DNS servers and internal DNS servers (internal the hostnames get resolved with the DMZ IP adress)

What did I do?
Followed the selfhosting guide

  • apt install apt-transport-https
  • apt install openjdk-8-jdk
  • hostnamectl set-hostname meet.blahblah.com
  • apt install jitsi-meet (after adding the repo)
  • /usr/share/jitsi-meet/scripts/install-letsencrypt-cert.sh
  • Added the following to /etc/jitsi/videobridge/sip-communicator.properties

org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=192.168.x.x
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=195.x.x.x
COMMENTED org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES

  • Added the following to /etc/systemd/system.conf

DefaultLimitNOFILE=65000
DefaultLimitNPROC=65000
DefaultTasksMax=65000

At this point jitsi is working fine, when there are no corporate firewalls involved, both 2 and more participants. I then followed the part ā€œUse TURN SERVER on port 443ā€ in the guide (https://jitsi.github.io/handbook/docs/devops-guide/turn). Afterwards, everything was still working as before.

What did I do different than the description in the guide?

  • added local LAN address to hosts instead of public IP as the VM is behind a NAT
  • added local LAN adress as turn_backend in the nginx module

What works:

  • 2 or 3 way calls are still working after change turn port to 443

What does not work:

  • Fallback to 443 still does not work.
  • Turnserver is still bound to the meet.blahblah.com certificate. Should I edit the turnserver.conf accordingly? Is there anything else to do?

Additional questions:

  • Is 4443 needed or not? The selfhost guide says so and so
  • Is certbot already configured to reissue the cert?

Help is highly appreciated, thanks so much in advance

#2

This is strange this step should handle that:

systemctl stop nginx
DOMAIN="turn-jitsi-meet.example.com"
EMAIL="myemail@example.com"
TURN_HOOK=/etc/letsencrypt/renewal-hooks/deploy/0000-coturn-certbot-deploy.sh
cp /usr/share/jitsi-meet-turnserver/coturn-certbot-deploy.sh $TURN_HOOK
chmod u+x $TURN_HOOK
sed -i "s/jitsi-meet.example.com/$DOMAIN/g" $TURN_HOOK

/usr/bin/certbot certonly --noninteractive \
    --standalone \
    -d $DOMAIN \
    --agree-tos --email $EMAIL \
    --deploy-hook $TURN_HOOK

Did that one finish successfully?

#3

Yes, everything finished successfully, here is the output

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Running deploy-hook command: /etc/letsencrypt/renewal-hooks/deploy/0000-coturn-certbot-deploy.sh
    Output from deploy-hook command 0000-coturn-certbot-deploy.sh:
    Configuring turnserver
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/turn.blahblah.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/turn.blahblah.com/privkey.pem
   Your cert will expire on 2021-03-03. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Edit: Files are all there

/usr/share/doc/coturn/examples/etc/turn_server_cert.pem
/usr/share/doc/coturn/examples/etc/turn_server_pkey.pem
/usr/share/doc/coturn/examples/etc/turn_client_cert.pem
/usr/share/doc/coturn/examples/etc/turn_client_pkey.pem
/etc/coturn/certs/turn.blahblah.com.privkey.pem
/etc/coturn/certs/turn.blahblah.com.fullchain.pem
/etc/coturn/certs/meet.blahblah.com.fullchain.pem
/etc/coturn/certs/meet.blahblah.com.privkey.pem
/etc/letsencrypt/archive/turn.blahblah.com/fullchain1.pem
/etc/letsencrypt/archive/turn.blahblah.com/cert1.pem
/etc/letsencrypt/archive/turn.blahblah.com/privkey1.pem
/etc/letsencrypt/archive/turn.blahblah.com/chain1.pem
/etc/letsencrypt/live/turn.blahblah.com/cert.pem
/etc/letsencrypt/live/turn.blahblah.com/fullchain.pem
/etc/letsencrypt/live/turn.blahblah.com/chain.pem
/etc/letsencrypt/live/turn.blahblah.com/privkey.pem

#4

OK, Iā€™ve executed

sed -i ā€œs/meet.blahblah.com/$DOMAIN/gā€ $TURN_HOOK

instead of

sed -i ā€œs/jitsi-meet.example.com/$DOMAIN/gā€ $TURN_HOOK

The guide wasnā€™t 100% clear here what needs to be replaced and what doesnā€™t.
I have to doublecheck everything again tomorrow. Thank you so far. I will let you know the result tomorrow.

#5

OK, Iā€™ve doublechecked it by disabling all ports except 80/443. Unfortunatley it is not working. I can join the meeting but I am not able to see any participants. Iā€™ve checked jvb.log, jicofo.log and prosody.log and did not see ay errorā€¦ :frowning:

#6

Nginx Log states the following:

2020/12/04 07:02:37 [error] 1425#1425: *1224 recv() failed (104: Connection reset by peer) while proxying and reading from upstream, client: 107.178.239.205, server: 0.0.0.0:443, upstream: ā€œ127.0.0.1:5349ā€, bytes from/to client:0/7, bytes from/to upstream:7/420

I have a question regarding the turnserver.conf

jitsi-meet coturn config. Do not modify this line

use-auth-secret
keep-address-family
static-auth-secret=71y4fjBPCeFks36N
realm=meet.blahblah.com
cert=/etc/coturn/certs/turn.blahblah.com.fullchain.pem
pkey=/etc/coturn/certs/turn.blahblah.com.privkey.pem
no-multicast-peers
no-cli
no-loopback-peers
no-tcp-relay
no-tcp
listening-port=3478
tls-listening-port=5349
no-tlsv1
no-tlsv1_1
# https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=intermediate&openssl=1.1.0g&guideline=5.4
cipher-list=ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
# jitsi-meet coturn relay disable config. Do not modify this line
denied-peer-ip=0.0.0.0-0.255.255.255
denied-peer-ip=10.0.0.0-10.255.255.255
denied-peer-ip=100.64.0.0-100.127.255.255
denied-peer-ip=127.0.0.0-127.255.255.255
denied-peer-ip=169.254.0.0-169.254.255.255
denied-peer-ip=127.0.0.0-127.255.255.255
denied-peer-ip=172.16.0.0-172.31.255.255
denied-peer-ip=192.0.0.0-192.0.0.255
denied-peer-ip=192.0.2.0-192.0.2.255
denied-peer-ip=192.88.99.0-192.88.99.255
denied-peer-ip=192.168.0.0-192.168.255.255
denied-peer-ip=198.18.0.0-198.19.255.255
denied-peer-ip=198.51.100.0-198.51.100.255
denied-peer-ip=203.0.113.0-203.0.113.255
denied-peer-ip=240.0.0.0-255.255.255.255
syslog

How does it come, that no-tcp is set in the turnserver config. As far as I understand TCP is needed for 443?

#7

take a look here. BTW there is a search function in this forum.

1 Like
#8

Thanks, so I can just comment that, as it does nothing in my case. I am two steps further now.
After adding listening-ip and allowe-peer-ip to my turnserver.conf, it is working with chrome. Firefox and the mobile apps still fail. This helped a lot: Coturn chronicles

Do I get it right that cohosting coturn and jitsi on the same server is semi optimal? I think about adding a second VM with a dedicated IP for coturn if take makes stuff and future upgrades easierā€¦

#9

if it comes to speed, nothing can beat ā€˜networkā€™ communications through localhost, so from this point of view itā€™s optimal. If you want to do some kind of load balancing for coturn it could be different.

#10

Iā€™ve read some comments from the coturn developers, that proxying is semi-optimalā€¦ But of course, you are right performance-wise. As we have 10g network, I do not think that network troughput will be an issue.

At the moment, this whole setup drives me nuts. As far as I see, a dedicated coturn server is a widely used setup and I expect not to face that many ā€œissuesā€ like at the moment.

#11

proxying coturn is not recommended. However, load balancing is not necessarily proxying, it can be done through DNS round robin.

#12

DNS round robin for loadbalancing is a not-so-good idea imho. Thereā€™s no recognition if a host fails.

Anyway, still the same isse, I am not able to get this setup up and running complety. Iā€™ve set the coturn log to verbose and get the following output when connecting

17: session 003000000000000007: closed (2nd stage), user <> realm <meet.blahblah.com> origin <>, local 192.168.115.25:5349, remote 192.168.115.25:32802, reason: TLS/TCP socket buffer operation error (callback)

#13

I found different threads with similar problems but didnā€™t find a solution for this. Iā€™ve doublechecked that the correct certificates are bound (helped somebody else with a similar problem), but still no luck. @damencho, do you have another hint? Thanks

#14

OTOH using a straight proxy with coturn is bad for performance. Not that using TCP for Jitsi is a great idea anyway. If you care about failing hosts, you have to do some monitoring no ?

Itā€™s not totally clear what is this operation, but I think that 32802 port would be your proxy originating a connect to coturn on the TLS port. Does your proxy handles that the destination port is using TLS ? For haproxy, it would mean mode tcp. Using http mode would not work.

#15

Iā€™d say, nginx ist configured the correct way. Here is my modules.conf. 192.168.115.25 is the IP of the local machine.

stream {
map $ssl_preread_server_name $name {
meet.blahblah.com web_backend;
turn.blahblah.com turn_backend;
}

upstream web_backend {
    server 127.0.0.1:4444;
}

upstream turn_backend {
    server 192.168.115.25:5349;
}

server {
    listen 443;
    listen [::]:443;

    # since 1.11.5
    ssl_preread on;

    proxy_pass $name;

    # Increase buffer to serve video
    proxy_buffer_size 10m;
}

}

#16

Right, I have to admit that I donā€™t see the reason why your setup does not work. Time to debug in this case. Do it using a 3G connection or whatever allows you to test from the point of view of an external client.

Use openssl s_client -connect turn.blahblah.com:443 -showcerts | openssl x509 -text

and

openssl s_client -connect turn.blahblah.com:5349 -showcerts | openssl x509 -text

you should get the same certificate (the good one).

#17

Hi,

when trying your command I receive the following message on port 443, 5349 works fine

c:\OpenSSL-Win64\bin>openssl.exe s_client -connect turn.blahblah.com:443 -showcerts |openssl x509 -text
write:errno=0
unable to load certificate
4004:error:0906D06C:PEM routines:PEM_read_bio:no start line:crypto\pem\pem_lib.c:691:Expecting: TRUSTED CERTIFICATE

nginx error log says

2020/12/09 09:12:47 [error] 932#932: *3 no host in upstream ā€œā€, client: 192.168.100.57, server: 0.0.0.0:443, bytes from/to client:0/0, bytes from/to upstream:0/0

Edit: When I run a scan on ssllabs.com the correct certificate is shownā€¦

#18

it works fine for me; I think I advised you to try it from the outside of your network, and it works when itā€™s done like this (as your test with sslabs confirms). So itā€™s failing when you are trying to access it from inside your network. Maybe at some point you are directly using an internal IP address instead of a ā€˜publicā€™ name resolving locally to an internal IP address ?

#19

OK, Iā€™ve now tried it via my home network. Same output. Connecting to 5349 works fine, connecting to 443 fails with the error message.
Something seems to go wrong with the nginx upstream config. As soon as I default everything to :5349 in the upstream config I get to correct certificate via port 443.

#20

well, I have in my mailbox your message with an URL in .de - I will not post it here - and if I try it I get a certificate:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:8d:ec:2c:a8:80:88:f8:da:e5:48:.......
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = R3
        Validity
            Not Before: Dec  3 16:42:23 2020 GMT
            Not After : Mar  3 16:42:23 2021 GMT
        Subject: CN =

so I donā€™t quite understand what is happening here for you.