Problems with reverse proxy and SSL

Hello,
I’m trying to implement a particular environment of jitsi and I’m facing some problems. The goal is:
A jitsi server that communicates through internal(10.0.0.0/8 network) and external(public address) networks, and the SSL certificate is hosted on a reverse proxy equipment (it cannot be installed elsewhere) with a single wildcard (".example.com", it not supports "jitsi..example.com", for instance).

I thought to do the following:

  • Configure in my firewall two virtual IP, one internal and one external
  • In the same firewall, configure port mappings to port 80 and 443 send the traffic for the two ip cited above to the reverse proxy equipment, and the other ports for jitsi(UDP/10000 and TCP/4443) send directly to the internal address of the jitsi server
  • Configure internal(external) DNS to send my jitsi domain to the virtual internal(external) address of firewall, configured above
  • Configure the reverse proxy to correctly send the https traffic to jitsi server

But here is the problems:

  • As I said before, the SSL certificate is single wildcard, so I needed to configure a CNAME of jitsi-meet.example.com pointing to jitsi.meet.example.com. The access to the portal and creation of room works fine, and the certificate is recognized, but it seems that it not connects to the server as the jicofo log is not registering anything.
  • I was suspecting of the domain name, because on my installation I put jitsi.meet.example.com in domain, and not the jitsi-meet.example.com, as I was accessing. So I removed the redirect rule of reverse proxy and access directly the jitsi.meet.example.com, and even with certificate error, the room was created correctly and jicofo started to log something, but when I started to send some video with one of the participants, the room stos with “Something went wrong”.

I don’t know if I need to change the domain name of the jitsi installation and/or change something on NAT configuration…

I know that is a lot of information and my english is not that good, but I hope that someone comprehends and help me on these issue.

Thanks in advance.

I think that you follow the classical split brain dns design. Unfortunately, I don’t think that it will work. The videobridge needs to know the public address clients are using to use it. Either it discovers it, or you have to set it by hand in config files. Either way, videobridge can run in NAT mode, or it can run in public mode. If you have only private IP addresses, the second case can apply.
But to have an internal server (behind a NAT) having internal (private IP behind a NAT) and public Internet clients, you need an Internet router that has ‘reflexive’ or ‘hair-pinning’ capability. That is, if your internal network is 10.0.0.0/255 and your public network address is 1.2.3.4, your internal server behind redirected from 1.2.3.4 to let’s say 10.0.0.55, to address your server internal clients will have to use 1.2.3.4 as target address. The router will redirect queries to the internal server for which there exists a redirection rule.
Not all routers have this kind of capability. You can test for it easily, either it works, or not.

Even if the router has the capability, it must have the necessary power (if you have hundreds of potential internal users connected through an internal network at 2.5 Gbits/s but your router is appropriate for a 100Mbits/s internet access, this will not go very well)

If all of this is solved, the certificate problem should not be the worse. If you have a certificate dedicated to the server, it don’t matter that it is example.com or jitsi.example.com. If you want to have a single domain for the organization (on the Internet) AND the video server, it is not going to be easy. This said, I have no idea if and why this could be a problem. You can get a free certificate for a server behind a NAT, all that it is necessary is that it can be accessed through the NAT from the public internet.

Thank you for the prompt response. The following environment is working fine:

Configured the NAT in firewall to send all the traffic from a specific public address of my public range directly to the internal ip of the jitsi server.

I can communicate between internal and external clients, but without certificate, as the traffic is not passing through the reverse proxy that has the ssl certificate.

if your certificate is delivered for example.com, that is defined in the public DNS as let’s say 1.2.3.4, and your router redirect external and internal clients to your internal proxy at 10.0.0.55, said proxy having the certificate, and terminating the TLS traffic for the Jitsi-meet server, then I don’t see why it should not work. The videobridge traffic (UDP/10000) should go direct from the router to the bridge, not via the proxy, since the public certificate is not used by the bridge.

Note that I have a test server behind a proxy too (Haproxy) terminating the TLS traffic too, and I have just deleted from the Nginx Jitsi-meet configuration everything concerning TLS, that is, the link between Haproxy and Nginx is not crypted (it’s actually a link inside a computer because this Jitsi-meet setup runs in a container).

So this is possible; however I believe that Haproxy is able to proxy to a TLS server (in TCP mode), even if I have not done it yet.

Nice, good to know that it was already tested. Maybe I’m letting something pass… The configuration is exactly like you described: 80 and 443 sended to proxy to then be sended to the jitsi server, and UDP/10000 and TCP/4443 sended directly to the jitsi server.

I will search for some other NAT configurations here in firewall.

About the certificate, the question is: I need to access the jitsi in a different url that I made the instalation. In fact I need to change a “.” to a “-” in this url.

When I do this, the jicofo seems to not working properly. The URL accessed is mandatory to be the same of the jitsi instalation?

Thanks again!

I don’t understand your exact setup so it’s difficult to be absolutely sure. Your computer hostname could be jitsi.example.com and the URL could be jitsi-example.com, yes. My container ‘hostname’ is jitsitest, for example.

However, I have not changed the DNS name from the name I set while installing Jitsi-meet, the nginx server has a virtual host with the original DNS name. Changing it is doable but not trivial, there are several places to change, and if you have not done much customisations you will be better off purging everything and installing again, else you risk that in later updates your install will be broken because the original name in Debian config is applied again.

Yes… I saw that about the reinstalling is less difficult than change all the places where there are the server domain. I will teste something more and update the topic here.

Thanks for the attention.

Here is some info about jvb.log, when I try to enter with video:

JVB 2022-01-21 20:11:24.034 INFO: [105] [confId=e48406d1673c7cc3 epId=c128ebcb local_ufrag=6d6qs1fpveh3pn gid=59735 stats_id=Hanna-Zz8 conf_name=testroom@conference.jitsi.example.com] IceTransport.iceStateChanged#327: ICE state changed old=Waiting new=Running
JVB 2022-01-21 20:11:24.034 INFO: [105] [confId=e48406d1673c7cc3 gid=59735 stats_id=Hanna-Zz8 conf_name=testroom@conference.jitsi.example.com ufrag=6d6qs1fpveh3pn epId=c128ebcb local_ufrag=6d6qs1fpveh3pn] Agent.startConnectivityEstablishment#735: Trigger checks for pairs that were received before running state
JVB 2022-01-21 20:11:24.034 INFO: [105] [confId=e48406d1673c7cc3 gid=59735 stats_id=Hanna-Zz8 conf_name=testroom@conference.jitsi.example.com ufrag=6d6qs1fpveh3pn epId=c128ebcb local_ufrag=6d6qs1fpveh3pn] Agent.triggerCheck#1714: Add peer CandidatePair with new reflexive address to checkList: CandidatePair (State=Frozen Priority=7961835276064522239):
LocalCandidate=candidate:1 1 udp 2130706431 10.0.0.5 10000 typ host
RemoteCandidate=candidate:10000 1 udp 1853759231 187.68.1.124 25164 typ prflx
JVB 2022-01-21 20:11:24.034 INFO: [105] [confId=e48406d1673c7cc3 gid=59735 stats_id=Hanna-Zz8 conf_name=testroom@conference.jitsi.example.com ufrag=6d6qs1fpveh3pn epId=c128ebcb local_ufrag=6d6qs1fpveh3pn] ConnectivityCheckClient.startChecks#149: Start connectivity checks.
JVB 2022-01-21 20:11:24.076 INFO: [95] [confId=e48406d1673c7cc3 gid=59735 stats_id=Hanna-Zz8 conf_name=testroom@conference.jitsi.example.com ufrag=6d6qs1fpveh3pn epId=c128ebcb local_ufrag=6d6qs1fpveh3pn] ConnectivityCheckClient$PaceMaker.run#936: Pair failed: 10.0.0.5:10000/udp/host → 172.16.213.219:49373/udp/host (stream-c128ebcb.RTP)
JVB 2022-01-21 20:11:24.201 INFO: [91] [confId=e48406d1673c7cc3 gid=59735 stats_id=Hanna-Zz8 conf_name=testroom@conference.jitsi.example.com ufrag=6d6qs1fpveh3pn epId=c128ebcb local_ufrag=6d6qs1fpveh3pn] ConnectivityCheckClient.processSuccessResponse#644: Pair succeeded: 10.0.0.5:10000/udp/host → 187.68.1.124:25164/udp/prflx (stream-c128ebcb.RTP).

Where 10.0.0.5 would be the internal address of jitsi and 187… is an public address of the device is trying to enter the conference from the internet.

I tried to edit something on /etc/jitsi/videobridge/sip-communicator.properties, adding the public and private address… but I don’t know if it is the right way.

EDIT:
The attempt before was put the two address and coment the HARVEST_ADDRESSES, when I uncommented the HARVEST_ADDRESSES, it worked!

Thanks for all the support.

EDIT2:
The problem now is about the nature of the certificate. As I explained above, my certificate does not accept a “subsubdomain”, and I cannot use a free certificate. So the routing and pointing to reverse proxy is working now, but the certificate applied is still invalid because my domain installation was jitsi.meet.example.com, and I need to use a cname(already created) from jitsi-meet.example.com pointing to jitsi.meet.example.com. But When I access the url with “-”, it does not work, not even the jicofo works like expected, although the certificate appears valid.

I was thinking if there was a way to change this or the best way is indeed reinstall jitsi with the “-” URL.

Thanks!

Well, if you absolutely want to do that, here is a post that explains how to do it. Unfortunately I am not sure it’s all there is to it. There are reference to the domain name in jicofo config, in debian config (/var/cache/debconf). I have done it one time and not taken notes on the process, and remember than when I did an upgrade of jitsi installation, it re-instated the old domain in parts of the config - I was already aware of debconf when I did it, but it seems that I missed something. Not worth it IMO.

Nice. Thank you for the link, I will consider the reinstalling too.

EDIT:
I tried this link you send and it worked. I will create a note to the team here about the update…
Again: thank you so much for the responses, I think now the problem is solved and topic can be closed.

For the record, yes, there’s a lot more involved if you want to change both the hosting domain and xmpp domain. A default jitsi setup assumes both to be the same which prevents confusion down the road, but technically they are distinct and can be different. This used to not be possible because jibri was hardcoded to use the XMPP domain to access the conference, but there is now a config option to override that so all is good.

The linked post shows how to change the hosting domain used to access service but under the hood, the XMPP domain remains unchanged. This can be beneficial in some use cases (e.g. one can pre-build images that are installed using a pseudo domain, and the same images can be use across various deployments that are accessed using different host names), and as long as this was a deliberate choice when designing the system and done with the correct mental model in mind, it should be ok to maintain.

Also, never tried doing an upgrade on a jitsi setup with a dual domain model since we always install on fresh VM. There may be complications or confusion on upgrade. Maybe.

Certainly not something I would recommend for someone new to jitsi. It does make getting help and following instructions on the forum a tad harder since one has to mentally map examples to use the correct domain.

Hey guys, for some reason I do not understand, the solution is not valid anymore…

Even with the local and public address configured in jvb config, and commented the STUN_HARVESTER_ADRESSES commented, as stated here, I am still seeing the error of connection through the 10000 port. I’m thinking if the problem is the /etc/hosts file… I need to put there only the public address (remembering that the public address here is on a virtual ip on firewall)? Because in the default installation it put there the local ip address of the server…

this is way too unclear to make any hypothesis. Try to use tcpdump to see if packets are going to the port 10000 of your server. Remember that by default this happens only with 3 users or more.

My test server has only its private IP address and it is working fine.

I have 4 IP address here, for example:

1.2.3.4 - my public ip to internet, configured as virtual ip on firewall
10.0.0.10 - virtual ip on firewall to access from the intranet
10.0.0.1 - internal address of the jitsi server
10.0.0.5 - internal address of the reverse proxy

So we have two types of acces, from intranet and from internet:

intranet:

  • for 80 and 443: receive on 10.0.0.10 and send to 10.0.0.5
  • for 10000/udp and 4443: receive on 10.0.0.10 and send to 10.0.0.1

internet:

  • for 80 and 443: receive on 1.2.3.4 and send to 10.0.0.5
  • for 10000/udp and 4443: receive on 1.2.3.4 and send to 10.0.0.1

and the error it is showing in jvb.log is:

JVB 2022-01-24 17:02:49.444 INFO: [65] [confId=6886f81729cf372a gid=54818 stats_id=Wanda-J2T conf_name=testemoura@conference.jitsi-meet.example.com ufrag=2rn751fq6qsk3f epId=d2547647 local_ufrag=2rn751fq6qsk3f] ConnectivityCheckClient.processTimeout#874: timeout for pair: 10.0.0.1:10000/udp/host → 10.10.10.10:64534/udp/prflx (stream-d2547647.RTP), failing.

Where 10.10.10.10 is a internal computer. The same occurs when connect through internet, appearing in the logs it public ip.

Did you try as the following?

org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=10.0.0.10
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=1.2.3.4

See my first reply, I still think this will not work. For what it’s worth, I have no idea of what is a ‘virtual IP on the firewall’. I searched on the Internet and it seems a concept from Pfsense covering IP alias and other things that I did not bother to read; however I’m failing to see what could be its use in this case and your explanations are not clear to me.

I tried now with and without mapping_harvester, and it is still showing the same error.

Ok, sorry for the poor explanations. The virtual IP here is a nat with port forwading. So I can create 4 nat rules for each(internet and intranet), each one for one port (80,443,10000/udp and 4443), specifying the “public” address and the mapped one.

Here is how I test if a network can support Jitsi-meet.
First create a service running on the computer that needs to be accessed. I use python3 -m http.server.

This will create a web server running on port 8000 and displaying the current directory.

Then I test that it can be accessed on the local computer (browser http://10.0.0.10;8000). This will test the server firewall.

Then I create a NAT rule redirecting say, the port 8000 of the public address to the port 8000 of the server. I test then from the Internet that I can browse the directory all right → browser access 1.2.3.4:8000.

Last step is to do the same thing from a computer inside the private network. If browsing 1.2.3.4:8000 works from the internal, private network, the firewall/router has reflexive capability and you will be able to access the server from the inside and the outside. If not, you are out of luck.

org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=meet-jit-si-turnrelay.jitsi.net:443
org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=10.0.0.1
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=10.0.0.10

Yes, it is the same way of thinking, but I was trying to use two differents address to access the same server. One to be accessed from the internal network (10.0.0.10) and the public IP (1.2.3.4) would be accessed from the internet. Each one of them will translate to the internal address accordingly the service the client is trying to access.

Maybe it is the problem, using 2 differents address for the same server, I don’t know… maybe I need to force the internal clients to access via the public IP too (changing the DNS entry), I will try that.

EDIT:
Tried it here, but it shows the same behaviour…

JVB 2022-01-24 18:16:20.824 INFO: [67] [confId=b1068ee4c7ce74f7 gid=54818 stats_id=Derek-6of conf_name=testroom@conference.jitsi-meet.example.com ufrag=6s4sr1fq6v47sl epId=f6bd3432 local_ufrag=6s4sr1fq6v47sl] ConnectivityCheckClient.processTimeout#874: timeout for pair: 10.0.0.10:10000/udp/srflx → 192.168.0.10:61171/udp/host (stream-f6bd3432.RTP), failing.
JVB 2022-01-24 18:16:21.245 INFO: [63] [confId=b1068ee4c7ce74f7 gid=54818 stats_id=Wanda-J2T conf_name=testroom@conference.jitsi-meet.example.com ufrag=18dfr1fq6v4dou epId=b759a736 local_ufrag=18dfr1fq6v4dou] Agent.setState#908: ICE state changed from Completed to Terminated.
JVB 2022-01-24 18:16:21.245 INFO: [63] [confId=b1068ee4c7ce74f7 epId=b759a736 local_ufrag=18dfr1fq6v4dou gid=54818 stats_id=Wanda-J2T conf_name=testroom@conference.jitsi-meet.example.com] IceTransport.iceStateChanged#334: ICE state changed old=Completed new=Terminated

Same thing… the room starts, I can see the participants entering, but when someone try to transmit some video stream, the room disconnects.

As I said before, when I configure the NAT rules to send all the traffic from the 1.2.3.4 and 10.0.0.10 to 10.0.0.1, without passing the http(s) traffic through the reverse proxy, it works even without configuring the LOCAL and PUBLIC_ADDRESS on sip-communicator… So it is not appearing to be some problem in the NAT rules, but some inconsitency between the address responsible for the HTTP(S) traffic and the JVB traffic.