Javascript console full of WebSocket errors with HTTP code 404 (not found)

nagmat84 · February 6, 2021, 6:55pm

I run a default, self-hosted Jitsi installation which I have setup according to Self-Hosting Guide - Debian/Ubuntu server.

Seemingly, everything works fine, but if I open the Javascript console, I see a lot of errors (and follow-ups) like this one:

BridgeChannel.js:86 WebSocket connection to 'wss://jitsi.my-domain.tld/colibri-ws/default-id/c065e3061b8d48c7/df02eb53?pwd=25jptsga3hludossi3n145f62f' failed: Error during WebSocket handshake: Unexpected response code: 404

That is comprehensible, because the Apache virtual host which has been setup by the Jitsi installation does not contain any proxy directive which would forward such websocket request to some listener.

How do I fix that problem?

damencho · February 6, 2021, 7:02pm

github.com

jitsi/jitsi-meet/blob/c856c2051331ee1234b26d3931a2a870c352eff9/doc/debian/jitsi-meet/jitsi-meet.example-apache#L47


    </Location>
    Alias "/external_api.js" "/usr/share/jitsi-meet/libs/external_api.min.js"
    <Location /external_api.js>
        Require all granted
    </Location>
    ProxyPreserveHost on
    ProxyPass /http-bind http://localhost:5280/http-bind/
    ProxyPassReverse /http-bind http://localhost:5280/http-bind/
    ProxyPass /xmpp-websocket http://localhost:5280/xmpp-websocket
    ProxyPassReverse /xmpp-websocket http://localhost:5280/xmpp-websocket
    ProxyPassMatch ^/colibri-ws/default-id http://localhost:9090
    RewriteEngine on
    RewriteRule ^/([a-zA-Z0-9]+)$ /index.html
</VirtualHost>
# Mozilla Guideline v5.4, Apache 2.4.41, OpenSSL 1.1.1d, intermediate configuration, no OCSP
SSLProtocol         all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite      ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384

nagmat84 · February 6, 2021, 7:05pm

I added the lines. Seems as Ubuntu ships a broken default config. However, the problem has only been solved partially. Now the response code turned from 404 into 405 “method not allowed”.

damencho · February 6, 2021, 7:07pm

Apache config are behind nginx as it is not default and lack contributors … Those rules were recently added are still not in stable.

nagmat84 · February 6, 2021, 8:01pm

After some debugging and looking at the corresponding nginx config. I figured out the correct Apache config.

Wrong:

ProxyPassMatch ^/colibri-ws/default-id http://localhost:9090

Correct:

 ProxyPass          /colibri-ws/default-id ws://localhost:9090/colibri-ws/default-id
 ProxyPassReverse   /colibri-ws/default-id ws://localhost:9090/colibri-ws/default-id

You also need the Apache module mod_proxy_wstunnel.

Explanation:

The regex ^/colibri-ws/... is not necessary, because ProxyPass matches prefixes and the given regex does nothing else. No need for it.
The second parameter (the destination) was missing the /colibri-ws/default-id part, because this prefix must not be removed
Using wss as the protocol id in combination with the mod_proxy_wstunnel automatically handles all the proxy headers and upgrades the connection to ws.

damencho · February 6, 2021, 8:25pm

You can check with this guy and see what is the correct one change websocket url from http:// to ws:// by TobiasKneidl · Pull Request #8568 · jitsi/jitsi-meet · GitHub
I don’t have experience with Apache.

Maybe there should also be some code that enables that module then in postinst script when apache is configured.

nagmat84 · February 20, 2021, 12:53pm

@damencho I finally got Apache working without error. This also lead me to create my first PR. However, despite some errors in the example config for Apache, more things have to be configured in the Apache main config (outside of the virtual host for jitsi). Especially, in order to get HTTP/2 working.
Currently, the directive Protocols h2 http/1.1 has no effect, but Apache still uses HTTP 1.1. To make it work the Apache’s threading model must be switched from “preforked” to “event driven”. The latter is the modern approach and the default for a vanilla Apache installation, but unfortunately a default Debian/Ubuntu installation still uses the ancient “prefork” model, because the “event driven” approach conflicts with the way how Debian/Ubuntu include PHP support into Apache.

To cut a long story short: I would be willing to write a tutorial which explains which Apache modules need to be loaded, unloaded or replaced and what other global config options must be set (based on a default Debian/Ubuntu installation). I believe the right place for such a tutorial would be somewhere here, but I do not know how to start.

What would be the right approach to start? Just start writing an markdown document and then send it to somebody? Is the Jitsi Meet Handbook also under version control?

damencho · February 20, 2021, 3:13pm

Can this be automated? We already try to load some modules here: jitsi-meet/jitsi-meet-web-config.postinst at e525c2b2ec1abcab6b9233720ba4e1ead3d96a14 · jitsi/jitsi-meet · GitHub
And the dependency is controlled here: jitsi-meet/control at e525c2b2ec1abcab6b9233720ba4e1ead3d96a14 · jitsi/jitsi-meet · GitHub

But yeah the correct place is the handbook, maybe a new section for apache can be added next to turnserver and speakerstats in the configuration section.

nagmat84 · February 20, 2021, 3:27pm

Yes and no. Of course, you could simply overwrite the global Apache config, i.e. httpd.conf, remove mod_mpm_prefork, add mod_mpm_event and kick out mod_php. Besides mod_php, there might be more modules which needs some tweaking, to work with mod_mpm_event.

However, this automatism will very likely break a user’s setup and lead to some frustration. Moreover, I personally believe that it feels wrong to automate such a deep and invasive modification. I think administrator who use Apache should do that consciously.

Don’t get me wrong. After my latest PR, the Apache config works at least. However, it is not resource efficient. Basically, you should blame Debian for that, because they still use the ancient pre-fork model as their Apache default config. (Personally, I believe this is one of the reasons why so many people blame Apache to be so inefficient compared to Nginx, but that are jm2c.)

damencho · February 20, 2021, 3:31pm

I agree, you can create a PR with description in the handbook, thank you for your contribution. The Apache support needs some love, as we do not use it in our environment so is behind the nginx one

nagmat84 · February 20, 2021, 3:33pm

Where do I find the repository and the files for the documentation such that I can clone it and create a PR?

damencho · February 20, 2021, 3:34pm

nagmat84 · February 20, 2021, 3:35pm

Great and thank you! I will try to get that started next weekend.