Problems with customized Quick Install setup

I’m having trouble with my installation of Jitsi.

TL;DR: Quick Install + custom Nginx configuration broke recently. Trying to figure out system.

Environment

I run a number of other servers on my Nginx installation with my personal set of standard configuration files & standards. I also have certbot with automatic certificate updates working the way I like it with the webroot option and some catch all configuration in Nginx.

So, installing Jitsi, I’m expecting to need to do some amount of Nginx configuration. This is what I get for running multiple services on the same hardware! :smiley:

First Try

My first try installing with the Quick Install (a month ago now, so going from memory), the configuration failed pretty ungracefully because the folder /etc/nginx/sites-available doesn’t exist on my system. It took some trouble to wipe ("purge") the original install and get it to run through the full install again. This time I made sure to create the standard /etc/nginx/*-available folders before just to see what Jitsi created so that I could recreate the configuration in my Nginx configuration folders.

Second Try

In the configuration for this attempt, I selected the option that allowed me to use my own certificates and passed in the /etc/letsencrypt/live/jitsi.example.com/* paths in the configuration.

It also created the “normal” sites-available and modules-available files. Incorporating those into my Nginx configuration system, I ended up with this configuration for the jitsi setup:

server {
  server_name jitsi.example.com video.example.com v.example.com;
  listen 80;
  listen [::]:80;

  include snips/letsencrypt.conf;
  include snips/force-main-https.conf;
}

server {
  server_name video.example.com v.example.com;

  listen 443 ssl;
  listen [::]:443 ssl;
  ssl_certificate     /etc/letsencrypt/live/jitsi.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/jitsi.example.com/privkey.pem;

  return 301 https://jitsi.example.com$request_uri;
}

server {
  listen 443 ssl;
  listen [::]:443;
  ssl_certificate     /etc/letsencrypt/live/jitsi.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/jitsi.example.com/privkey.pem;

  server_name jitsi.example.com;

  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_prefer_server_ciphers on;
  ssl_ciphers "EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA256:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EDH+aRSA+AESGCM:EDH+aRSA+SHA256:EDH+aRSA:EEC
DH:!aNULL:!eNULL:!MEDIUM:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4:!SEED";

  add_header Strict-Transport-Security "max-age=31536000";

  # Prevent any bad URLs
  location ~ ^/([a-zA-Z0-9=\?]+)$ {
    rewrite ^/(.*)$ / break;
  }

  root /usr/share/jitsi-meet;

  error_page 404 /static/404.html;

  index index.html index.htm;

  gzip on;
  gzip_types text/plain text/css application/javascript application/json;
  gzip_vary on;

  location = /config.js {
    alias /etc/jitsi/meet/jitsi.example.com-config.js;
  }

  location = /external_api.js {
    alias /usr/share/jitsi-meet/libs/external_api.min.js;
  }

  ssi on;
  ssi_types application/x-javascript application/javascript;

  #ensure all static content can always be found first
  location ~ ^/(libs|css|static|images|fonts|lang|sounds|connection_optimization|.well-known)/(.*)$ {
    add_header 'Access-Control-Allow-Origin' '*';
    alias /usr/share/jitsi-meet/$1/$2;
  }

  # BOSH
  location = /http-bind {
    proxy_pass       http://prosody/http-bind;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header Host $http_host;
  }

  # xmpp websockets
  location = /xmpp-websocket {
    proxy_pass         http://prosody/xmpp-websocket;
    proxy_http_version 1.1;
    proxy_set_header   Upgrade $http_upgrade;
    proxy_set_header   Connection "upgrade";
    proxy_set_header   Host $host;
    tcp_nodelay on;
  }

  location ~ ^/([^/?&:'"]+)$ {
    try_files $uri @root_path;
  }

  location @root_path {
    rewrite ^/(.*)$ / break;
  }

  location ~ ^/([^/?&:'"]+)/config.js$ {
    set $subdomain "$1.";
    set $subdir "$1/";

    alias /etc/jitsi/meet/jitsi.example.com-config.js;
  }

  # Anything that didn't match above, and isn't a real file, assume it's a room name and redirect to /
  location ~ ^/([^/?&:'"]+)/(.*)$ {
    set $subdomain "$1.";
    set $subdir "$1/";
    rewrite ^/([^/?&:'"]+)/(.*)$ /$2;
  }

  # BOSH for subdomains
  location ~ ^/([^/?&:'"]+)/http-bind {
    set $subdomain "$1.";
    set $subdir "$1/";
    set $prefix "$1";

    rewrite ^/(.*)$ /http-bind;
  }

  # websockets for subdomains
  location ~ ^/([^/?&:'"]+)/xmpp-websocket {
    set $subdomain "$1.";
    set $subdir "$1/";
    set $prefix "$1";

    rewrite ^/(.*)$ /xmpp-websocket;
  }
}

upstream prosody {
  server localhost:5280;
}

I also added a single exception to my Digital Ocean droplet’s firewall to allow incoming UDP 10000 connections. Note, I did not forward 4443, apparently without consequence.

This all seemed to be working. I ran a few video calls with 2 people. I ran some with ~5. All seemed to work great.

Problems

The first problem I ran into was that the videobridge crashed sporadically and didn’t have Systemd’s automatic restart set. That fix was simple enough.

Now, after a couple weeks of not using it, I can’t even join with a single user. (I have automatic updates enabled so I’m presuming a recent Jitsi update/change broke things).

The only errors I get anywhere are in the client browser:

VM10550 react_devtools_backend.js:6 2020-05-09T23:38:06.837Z [JitsiMeetJS.js] <Object.getGlobalOnErrorHandler>:  UnhandledError: null Script: null Line: null Column: null StackTrace:  Error: Focus error, retry after 8000
    at l._allocateConferenceFocusError (moderator.js:446)
    at moderator.js:373
    at O.Handler.handler (strophe.umd.js:2716)
    at O.Handler.run (strophe.umd.js:1875)
    at strophe.umd.js:3157
    at Object.forEachChild (strophe.umd.js:830)
    at O.Connection._dataRecv (strophe.umd.js:3146)
    at I.Bosh._onRequestStateChange (strophe.umd.js:5012)
r @ VM10550 react_devtools_backend.js:6
o @ md5.js:1
getGlobalOnErrorHandler @ async.js:1
window.onerror @ app.bundle.min.js?v=3992:1
callErrorHandler @ lib-jitsi-meet.min.js?v=3992:1
l._allocateConferenceFocusError @ md5.js:1
(anonymous) @ md5.js:1
(anonymous) @ lib-jitsi-meet.min.js?v=3992:1
run @ lib-jitsi-meet.min.js?v=3992:1
(anonymous) @ lib-jitsi-meet.min.js?v=3992:1
forEachChild @ lib-jitsi-meet.min.js?v=3992:1
_dataRecv @ lib-jitsi-meet.min.js?v=3992:1
_onRequestStateChange @ lib-jitsi-meet.min.js?v=3992:1
VM10550 react_devtools_backend.js:6 2020-05-09T23:38:06.838Z [modules/xmpp/moderator.js] <l._allocateConferenceFocusError>:  Focus error, retry after 8000 <iq xmlns=​"jabber:​client" type=​"error" to=​"d1366183-42a7-43d4-a160-0953350acd6d@jitsi.example.com/​f6c185a6-81bd-4323-b8c7-d1a3440524a5" from=​"focus.jitsi.example.com" id=​"5c188419-2ae2-4ec7-a92f-a65e48bdaafc:​sendIQ">​…​</iq>​
r @ VM10550 react_devtools_backend.js:6
o @ md5.js:1
l._allocateConferenceFocusError @ md5.js:1
(anonymous) @ md5.js:1
(anonymous) @ lib-jitsi-meet.min.js?v=3992:1
run @ lib-jitsi-meet.min.js?v=3992:1
(anonymous) @ lib-jitsi-meet.min.js?v=3992:1
forEachChild @ lib-jitsi-meet.min.js?v=3992:1
_dataRecv @ lib-jitsi-meet.min.js?v=3992:1
_onRequestStateChange @ lib-jitsi-meet.min.js?v=3992:1
VM10550 react_devtools_backend.js:6 2020-05-09T23:38:06.842Z [conference.js] <X._onConferenceFailed>:  CONFERENCE FAILED: conference.focusDisconnected focus.jitsi.example.com 8

I see that the problem seems to be connecting to focus.jitsi.example.com.

I’ve seen the other topics here that cover this error. The solution is basically: “fix the focus config”. So, that’s what I’m trying to do.

I am currently poking at options on my own and trying various combination. I’m certain eventually I’ll figure it out. However, it would be nice to understand what’s going on/needed and that’s what’s brought me here.

Looking at https://github.com/jitsi/jitsi-meet/blob/master/doc/manual-install.md, I see that port 4443 should be open as well. Curiously, looking at the ports currently listening on my server, I do not see 4443 anywhere. Some that I do see:

  • 4445, 4446 - turnserver - What is this? Part of the videobridge?
  • 10000 - java - Videobridge, right?
  • 5269, 5280, localhost:5347, 5222 - lua5.1 - Prosody

Also curious is that turnserver listens on all interfaces separately instead of *, including ones that don’t show on ifconfig.

Why am I here, what I’ve tried/trying

So, I’m wondering what I’ve missed that has changed.

Some of the ports I’m observing don’t match what the standard for some reason.

I previous didn’t have *.jitsi.example.com DNS records established. Now they are. I’m using CloudFlare for DNS. Previously, I believe Jitsi worked while using their “proxy” features. However I have turned those off for now just in case. That change alone has not fixed this problem.

I’m in the middle of trying combinations of these various “fixes”:

  • Open 4445-4446 TCP ports
  • Add *.jitsi.example.com to the SSL certificate
  • Add domain for internal.auth.jitsi.example.com (Noticed this sub-sub domain in a config)
  • Check for other changes in Nginx config

Either way, I’ll post my findings in the hopes that this helps someone else in the future.

Next

Thanks for reading. If anyone has any pointers or thoughts, I’d love to hear. Maybe I’m missing something simple.

If there were a document that elaborates on how each of the parts of this complicated system interact I’d be very interested in seeing that. The “Network description” map on the manual-install doc is a great start but missing a lot of, imho, ciritial details.

Finally looking at this more. I see some errors from the prosidy service:

cameron@firefly:~$ journalctl -u prosody.service -n 2
-- Logs begin at Mon 2019-08-12 07:13:05 PDT, end at Fri 2020-06-26 14:12:07 PDT. --
Jun 26 14:12:02 firefly prosody[2537]: c2s564085318610: Traceback[c2s]: /usr/lib/prosody/util/cache.lua:66: table index is nil
                                       stack traceback:
                                               /usr/lib/prosody/util/cache.lua:66: in function 'set'
                                               /usr/lib/prosody/modules/muc/mod_muc.lua:176: in function 'track_room'
                                               /usr/lib/prosody/modules/muc/mod_muc.lua:204: in function </usr/lib/prosody/modules/muc/mod_muc.lua:191>
                                               (...tail calls...)
                                               /usr/lib/prosody/modules/muc/mod_muc.lua:433: in function '?'
                                               /usr/lib/prosody/util/events.lua:79: in function </usr/lib/prosody/util/events.lua:75>
                                               (...tail calls...)
                                               /usr/lib/prosody/core/stanza_router.lua:180: in function 'core_post_stanza'
                                               /usr/lib/prosody/core/stanza_router.lua:127: in function 'core_process_stanza'
                                               /usr/lib/prosody/modules/mod_c2s.lua:275: in function 'func'
                                               /usr/lib/prosody/util/async.lua:127: in function </usr/lib/prosody/util/async.lua:125>

I found some other issues that referenced changing storage = "none" to storage = "null". Did the conf file API change? That this isn’t caught by the software in a more obvious way seems like a worthwhile improvement to make. It’s also not obvious which file needs to be changed (I found /etc/prosody/conf.d/jitsi.example.com.cfg.lua) or which of the two (if not both) storage options needs to be changed.

After restarting with that option however, I got another error:

cameron@firefly:~$ journalctl -u prosody.service -n 3
-- Logs begin at Mon 2019-08-12 07:13:05 PDT. --
Jun 26 14:17:49 firefly prosody[88727]: portmanager: Error binding encrypted port for https: No certificate present in SSL/TLS configuration for https port 5281
Jun 26 14:17:49 firefly prosody[88727]: portmanager: Error binding encrypted port for https: No certificate present in SSL/TLS configuration for https port 5281
Jun 26 14:17:59 firefly prosody[88727]: internal.auth.jitsi.example.com:muc: Error restoring room jvbbrewery@internal.auth.jitsi.example.com from storage: no data storage active

I found another post that suggested setting storage = "memory". After changing that and a restart, jitsi seems to work again!

The portmanager: Error binding encrypted port for https: No certificate present in SSL/TLS configuration for https port 5281 errors are still occurring, but I haven’t found where that causes problems, yet.

I can confirm I have the same error. Jitsi meet client says “Secure Connection Failed”. I have hosted Jitsi using quick install on Digital Ocean Debian 10 instance.

Jul 25 10:39:37 yotta systemd[1]: Started Prosody XMPP Server.
Jul 25 10:39:37 yotta prosody[14983]: portmanager: Error binding encrypted port for https: No certificate present in SSL/TLS c
Jul 25 10:39:37 yotta prosody[14983]: portmanager: Error binding encrypted port for https: No certificate present in SSL/TLS c