Colibri-ws Websocket not working

I’ve been working through installing Jitsi on Opensuse Leap 15.1 with Apache (I know, I’m crazy, but it’s the Linux server I have available, and it’s serving other functions, so it’s what I have to work with).

I’ve learned much, and have everything working well. It’s all running with the same configuration as would be from a Quick Install I believe.

The one remaining issue that I’ve not been able to overcome is the connection to the Colibri-ws. Due to the failure of this connection, as soon as I hit 3 users the video quality goes very low. The console in Chrome shows it continually failing to connect to wss://my.server/colibri-ws/default-id/(.*). My suspicion is that I haven’t gotten the Proxy statements in Apache configured correctly. Does anyone have knowledge to pass along as to the best way to configure the Proxy piece in Apache for the websocket connections? It appears I have the http-bind working (if I break it I can’t even start the conference), and the /xmpp-websocket (not sure how to prove it, but not seeing any errors about it).

Any thoughts are appreciated.

You need the equvalent of the following location in the Apache config too

    # xmpp websockets
    location = /xmpp-websocket {
        proxy_pass http://127.0.0.1:5280/xmpp-websocket?prefix=$prefix&$args;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $http_host;
        tcp_nodelay on;
    }

This may also help: [How To] How to enable websockets (xmpp-websocket) and smacks for Prosody – However colibri (for the video bridge) looks a little different. So also see : https://github.com/jitsi/jitsi-videobridge/blob/master/doc/web-sockets.md#proxy-configuration

Thanks guys. I’ve been on both those tracks.

I decided to use tcpdump to capture the traffic on port 9090 to see exactly what’s happening. It appears I do have the Apache proxy working as the tcpdump shows that communication is getting through, but ultimately it’s throwing an authentication failure error.

In the jvb.log I’m also seeing the following:
2020-12-14 09:58:11.508 WARNING: [27] ColibriWebSocketServlet.createWebSocket#129: Received request for an nonexistent conference: colibri-ws
2020-12-14 09:58:11.628 WARNING: [32] ColibriWebSocketServlet.createWebSocket#129: Received request for an nonexistent conference: colibri-ws

I just noticed another thing, that I suspect might be the trigger, but don’t know where it’s coming from. Here are the relevant lines from my console log:
BridgeChannel.js:88 WebSocket connection to ‘wss://jherold.duckdns.org/colibri-ws/default-id/6e93e6f16131917e//a6a42650?pwd=6iiknuitk9epkkh65n8q2ak1m3’ failed: Error during WebSocket handshake: Unexpected response code: 403
_initWebSocket @ BridgeChannel.js:88
t @ BridgeChannel.js:107
Logger.js:154 2020-12-14T22:42:33.379Z [modules/RTC/BridgeChannel.js] <WebSocket.e.onclose>: Channel closed by server
Logger.js:154 2020-12-14T22:42:33.380Z [modules/RTC/BridgeChannel.js] <WebSocket.e.onclose>: Channel closed: 1006

I noticed there’s an extra slash in front of one of the ID’s, which I suspect is causing issues for the Websocket authentication. If I look at the connection information in the browser I see that’s my Participant ID, and I can see the same ID’s throughout the jvb.log. When I look at the websockets documentation as linked by @corby, under the troubleshooting section, it definitely doesn’t have the extra slash in front of the participant ID. What could cause the extra slash on the Participant ID? I suspect if I can pin that down it will start working.

Thanks again for any help that can be provided!

I finally have it functioning correctly.

Somehow I’d gotten a slash on the end of the muc URL in my config.js, and that was causing the extra slash in front of the Participant ID. Once I had that sorted it made it easier to troubleshoot the Apache proxy settings for the Colibri WebSocket.

Here are my Apache proxy settings that worked:
ProxyPassMatch ^/colibri-ws/default-id ws://localhost:9090
ProxyPassReverse ^/colibri-ws/default-id ws://localhost:9090

Much simpler than I expected. Hope it helps someone else somewhere along the line.

Can you create a PR to have this in the default apache template, the same way we have it for nginx.

@dlorehmij Could you please elaborate a bit one your solution? I tried different configurations and nothing seem to work. The server always replies with either 404 or 405 error code.

Tcpdump showed me that an error message comes from the WS: HTTP method GET is not supported by this URL.

This looks to me like Apache does not really care about the ws:// in the ProxyPass line…
How could you solve this?

Cheers

First, @damencho, I will try. I’d never heard of a PR before, so had to research what that was. I see I’ll have to create an account on github to do so. When I find some time I will.

Just a small disclaimer; I don’t consider myself any sort of expert. I’ve worked in IT for more than 3 decades, and everyday is a learning experience. This is what worked for me, and experts could probably describe a better way.

@bitone, I did realize that there probably isn’t enough info there for many. I’m thinking at some point I’ll do a write up on my whole process of how I got things working, but part of me thinks most will think what I did was crazy from the get go. We’ll see. Meanwhile, here’s the extra detail on how I got the Apache piece working.

First, you’ll need to load 3 modules for Apache to be able to complete the proxying of Websockets; proxy_module, proxy_http, and proxy_wstunnel (this is what they’re named on OpenSuSE, I don’t know other distributions or what they might be named there). My understanding is, without all 3 of these you will not be able to proxy the Websocket traffic through Apache. a2enmod is the tool to use to get them loaded, and you may have to get them installed on your distribution first using whatever tools your distribution uses.

I tried many variations on ways to proxy the traffic including using Rewrite rules instead of ProxyPass statements. Here’s how I’m currently configured, and everything seems to work:

ProxyPreserveHost on
ProxyPass /http-bind http://localhost:5280/http-bind
ProxyPassReverse /http-bind http://localhost:5280/http-bind
ProxyPass /xmpp-websocket http://localhost:5280/xmpp-websocket
ProxyPassReverse /xmpp-websocket http://localhost:5280/xmpp-websocket
ProxyPassMatch ^/colibri-ws/default-id ws://localhost:9090
ProxyPassReverse ^/colibri-ws/default-id ws://localhost:9090

I hope that helps.

@dlorehmij Thanks a lot for your instructions!

But on Debain 10 with Apache 2.4.38 this doesn’t work at all. Apparently SuSE is distributing some patched version.
According to the Apache manual, ProxyPassReverse does not take regular expressions. IMHO this line doesn’t do anything at and isn’t needed…

BTW what’s the version of your Apache?

I did some more tcpdumping on port 9090 on the lo interface when I was using those lines:

  <Location /colibri-ws/default-id>
    ProxyPass         ws://localhost:9090/colibri-ws/default-id/
    ProxyPassReverse  ws://localhost:9090/colibri-ws/default-id/
  </Location>

Apparently mod_proxy_wstunnel drops the “Updated WebSocket” header, which leads to this “HTTP method GET is not supported” error.

What worked for me in the end was some variation of the examples in the Apache documentation:

  <Location /colibri-ws/default-id>
    RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
    RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
    RewriteRule /colibri-ws/default-id/(.*) ws://localhost:9090/colibri-ws/default-id/$1 [P]
  </Location>

Cheers

I’d wondered my self about the Reverse. I’d have to admit that I don’t fully understand it, but when I was researching the Apache proxying, pretty much everything I read followed that pattern, so I did too. Of course, when I started I still had my muc URL wrong, so I was thrashing around trying to fix the wrong problem at the time. I still have all my attempts in the config, but commented out, and I have one attempt that looks nearly exactly like yours.

I’m OpenSuSE LEAP 15.1, which is running Apache 2.4.33. I was looking for a way to see the version of the proxy modules, but have strick out so far.

This is the line that worked for me:
ProxyPassMatch ^/colibri-ws/default-id ws://localhost:9090

I’ll remove the reverse and see what happens.

I removed the reverse proxies, and can confirm everything is working great. That makes my current config look like this:

    ProxyPreserveHost on
    ProxyPass /http-bind http://localhost:5280/http-bind
    ProxyPass /xmpp-websocket http://localhost:5280/xmpp-websocket
    ProxyPassMatch ^/colibri-ws/default-id ws://localhost:9090

I’ve been using this statement as well for troubleshooting:
LogLevel alert proxy:trace5

It does debug logging of all the proxy communication into the Apache error_log. You can substitute “rewrite” for “proxy” to see all the rewrite activity. Helps a lot to see exactly what’s happening. Here’s some of the log activity showing the proxy working:
[Sat Dec 19 08:30:28.853762 2020] [proxy:trace2] [pid 3190] mod_proxy.c(680): [client 192.168.1.1:45448] AH03461: attempting to match URI path ‘/colibri-ws/default-id/7a84c9d4f3c9c152/f92f9439’ against prefix ‘/http-bind’ for proxying
[Sat Dec 19 08:30:28.853827 2020] [proxy:trace2] [pid 3190] mod_proxy.c(680): [client 192.168.1.1:45448] AH03461: attempting to match URI path ‘/colibri-ws/default-id/7a84c9d4f3c9c152/f92f9439’ against prefix ‘/xmpp-websocket’ for proxying
[Sat Dec 19 08:30:28.853837 2020] [proxy:trace2] [pid 3190] mod_proxy.c(680): [client 192.168.1.1:45448] AH03461: attempting to match URI path ‘/colibri-ws/default-id/7a84c9d4f3c9c152/f92f9439’ against pattern ‘^/colibri-ws/default-id’ for proxying
[Sat Dec 19 08:30:28.853849 2020] [proxy:trace1] [pid 3190] mod_proxy.c(765): [client 192.168.1.1:45448] AH03464: URI path ‘/colibri-ws/default-id/7a84c9d4f3c9c152/f92f9439’ matches proxy handler ‘proxy:ws://localhost:9090/colibri-ws/default-id/7a84c9d4f3c9c152/f92f9439’
[Sat Dec 19 08:30:28.853925 2020] [proxy:trace2] [pid 3190] proxy_util.c(1969): [client 192.168.1.1:45448] ws: found worker ws://localhost:9090 for ws://localhost:9090/colibri-ws/default-id/7a84c9d4f3c9c152/f92f9439?pwd=5s4mh8k493sj1tlvbmmd924pse
[Sat Dec 19 08:30:28.853936 2020] [proxy:debug] [pid 3190] mod_proxy.c(1245): [client 192.168.1.1:45448] AH01143: Running scheme ws handler (attempt 0)
[Sat Dec 19 08:30:28.853945 2020] [proxy:debug] [pid 3190] proxy_util.c(2163): AH00942: WS: has acquired connection for (localhost)
[Sat Dec 19 08:30:28.853954 2020] [proxy:debug] [pid 3190] proxy_util.c(2216): [client 192.168.1.1:45448] AH00944: connecting ws://localhost:9090/colibri-ws/default-id/7a84c9d4f3c9c152/f92f9439?pwd=5s4mh8k493sj1tlvbmmd924pse to localhost:9090
[Sat Dec 19 08:30:28.854058 2020] [proxy:debug] [pid 3190] proxy_util.c(2425): [client 192.168.1.1:45448] AH00947: connected /colibri-ws/default-id/7a84c9d4f3c9c152/f92f9439?pwd=5s4mh8k493sj1tlvbmmd924pse to localhost:9090
[Sat Dec 19 08:30:28.854085 2020] [proxy:trace2] [pid 3190] proxy_util.c(2860): WS: fam 2 socket created to connect to localhost
[Sat Dec 19 08:30:28.854156 2020] [proxy:debug] [pid 3190] proxy_util.c(2894): AH02824: WS: connection established with 127.0.0.1:9090 (localhost)
[Sat Dec 19 08:30:28.854175 2020] [proxy:debug] [pid 3190] proxy_util.c(3062): AH00962: WS: connection complete to 127.0.0.1:9090 (localhost)
[Sat Dec 19 08:30:28.857081 2020] [proxy:trace2] [pid 3190] proxy_util.c(3866): [client 192.168.1.1:45448] ap_proxy_transfer_between_connections complete
[Sat Dec 19 08:30:28.857279 2020] [proxy:trace2] [pid 3190] proxy_util.c(3866): [client 192.168.1.1:45448] ap_proxy_transfer_between_connections complete
[Sat Dec 19 08:30:28.857371 2020] [proxy:trace2] [pid 3190] proxy_util.c(3866): [client 192.168.1.1:45448] ap_proxy_transfer_between_connections complete