Upgraded to Jitsi Stable 7830 (Kubernetes Setup broken)

mehtapaxshal · September 29, 2022, 8:11pm

Hi there,

My working setup is broken after upgrading from stable-7648-4 to stable-7830. I am not seeing any error in jvb, jicofo, prosody services except I see an error on my web service that’s complaining about not able to reach prosody service.

This is my configuration inside web-deployment.yaml file and is unable to reach shard-0-prosody somehow after upgrade.

env:
            - name: XMPP_SERVER
              value: shard-0-prosody
            - name: XMPP_BOSH_URL_BASE
              value: http://shard-0-prosody:5280

Any idea if there were some breaking changes that could cause this?

Also, I noticed a new config JVB_ADVERTISE_IPS on JVB. Is this something required? If so, What should it be set?

Thanks!

saghul · September 30, 2022, 8:23am

No idea sorry. Are you able to ping / telnet that container from the other container?

It’s a replacement for DOCKER_HOST_ADDRESS.

jakobd · October 11, 2022, 9:32am

I currently have the same problem, could you fix it?

mehtapaxshal · October 11, 2022, 6:32pm

Ok I tried again with both 7830 and latest 7882. Working fine now… Thanks Saghul!

hamlet · October 12, 2022, 5:31am

I also have this behavior switching from stable-7648-4 (working) to stable-7830 or 7882 (not working).

Comparing the prosody debug logs I noted a General Error “No muc_component specified. No muc to operate on!” when trying with 7830 / 7882.

@mehtapaxshal Please be so kind to share what you changed to get your setup working again on 7830 / 7882

mehtapaxshal · October 12, 2022, 9:37pm

I didn’t do anything specific…It failed for me once and then I retried again and started working for me.

Make sure - you have the required env configs set for the latest version!

hamlet · October 13, 2022, 8:51am

Could you share your env config so I can compare if you are setting something differently?

mehtapaxshal · October 13, 2022, 4:54pm

Prosody config:

containers:
        - name: prosody
          image: jitsi/prosody:stable-7882
          volumeMounts:
            - mountPath: /config
              name: prosody-config-volume
            - name: prosody
              mountPath: /prosody-plugins-custom/mod_token_moderation.lua
              subPath: mod_token_moderation.lua
          imagePullPolicy: Always
          env:
            - name: XMPP_DOMAIN
              value: <domain>
            - name: XMPP_AUTH_DOMAIN
              value: auth.<domain>
            - name: PUBLIC_URL
              value: "<domain>"
            - name: XMPP_MUC_DOMAIN
              value: muc.<domain>
            - name: XMPP_INTERNAL_MUC_DOMAIN
              value: internal-muc.<domain>
            - name: JICOFO_COMPONENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: jitsi-config
                  key: JICOFO_COMPONENT_SECRET
            - name: JVB_AUTH_USER
              value: jvb
            - name: JVB_AUTH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: jitsi-config
                  key: JVB_AUTH_PASSWORD
            - name: JICOFO_AUTH_USER
              value: focus
            - name: JICOFO_AUTH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: jitsi-config
                  key: JICOFO_AUTH_PASSWORD
            - name: JVB_TCP_HARVESTER_DISABLED
              value: "true"
            - name: ENABLE_AUTH
              value: "true"
            - name: AUTH_TYPE
              value: "jwt"
            - name: JWT_AUTH_TYPE
              value: token
            - name: JWT_TOKEN_AUTH_MODULE
              value: token_verification
            - name: JWT_ACCEPTED_AUDIENCES
              value: <aud>
            - name: JWT_ACCEPTED_ISSUERS
              value: <issuer>
            - name: JWT_APP_ID
              value: <app_id>
            - name: JWT_APP_SECRET
              valueFrom:
                secretKeyRef:
                  name: jitsi-config
                  key: JWT_APP_SECRET
            - name: ENABLE_BREAKOUT_ROOMS
              value: "false"
            - name: ENABLE_JAAS_COMPONENTS
              value: "false"
            - name: XMPP_MUC_MODULES
              value: token_moderation
            - name: JIGASI_XMPP_USER
              value: "<jigasi_user>"
            - name: JIGASI_XMPP_PASSWORD
              value: "<jigasi_pwd>"
            - name: TZ
              value: America/Los_Angeles

Jicofo Config:

containers:
        - name: jicofo
          image: jitsi/jicofo:stable-7882
          volumeMounts:
            - mountPath: /config
              name: jicofo-config-volume
          imagePullPolicy: Always
          env:
            - name: XMPP_DOMAIN
              value: <domain>
            - name: XMPP_AUTH_DOMAIN
              value: auth.<domain>
            - name: PUBLIC_URL
              value: "<domain>"
            - name: XMPP_INTERNAL_MUC_DOMAIN
              value: internal-muc.<domain>
            - name: JICOFO_COMPONENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: jitsi-config
                  key: JICOFO_COMPONENT_SECRET
            - name: JICOFO_AUTH_USER
              value: focus
            - name: JICOFO_AUTH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: jitsi-config
                  key: JICOFO_AUTH_PASSWORD
            - name: JVB_BREWERY_MUC
              value: jvbbrewery
            - name: JICOFO_ENABLE_BRIDGE_HEALTH_CHECKS
              value: "true"
            - name: ENABLE_RECORDING
              value: "true"
            - name: XMPP_RECORDER_DOMAIN
              value: recorder.<domain>
            - name: JIGASI_BREWERY_MUC
              value: "JigasiBrewery"
            - name: TZ
              value: America/Los_Angeles

hamlet · October 15, 2022, 11:08am

:@mehtapaxshal thx a lot for sharing
I tried to stick to your config as close as possible, but to no avail.

I still get (from log in web component)

my server ip - - [15/Oct/2022:12:57:45 +0200] “GET /pwa-worker.js HTTP/1.1” 200 1511 “https://jitsi.my domain/pwa-worker.js” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36 Edg/106.0.1370.42”
2022/10/15 12:58:14 [error] 261#261: *2 jitsi-meet-prosody could not be resolved (110: Operation timed out), client: 10.42.0.1, server: _, request: “GET /xmpp-websocket?room=foldingoppositesequiptoo HTTP/1.1”, host: “jitsi.my domain”

where jitsi-meet-prosody is the name of my prosody pod which can be resolved from the web pod

Nothing worthwile in the other logs.

I found that my config works as long as the web pod uses the container jitsi/web:stable-7648-4. The others (prosody, jigasi and jvb) are on stable-7882. Once I switch to jitsi/web:stable-7830 or jitsi/web:stable-7882 the setup ceases to work with “You have been disconnected” and the above error is logged.

@saghul Do you have any ideas where to look and what to check?

mehtapaxshal · October 17, 2022, 6:53pm

As Saghul mentioned - are you able to ping/telnet them? Though ping shouldn’t matter if it works for stable-7648-4 and not for stable-7882.

Could you try doing the setup from the scratch? That’s what I did it.

hamlet · October 19, 2022, 2:55pm

I am not able to ping or telnet (cmds are missing inside the container) the prosody pod, but for both web container versions, I am able to wget http://jitsi-meet-prosody:5280 and get back the index.html from prosody. So, name resolution works in general.

I have started to do tcpdumps for both web container versions and found out using wireshark that with web:stable-7882 the error is occuring inside “GET /xmpp-websocket?room=…”
The name jitsi-meet-prosody cannot be resolved and DNS requests are sent to the outside which fails as this name is a cluster-internal one.
This name is used in the env variable XMPP_BOSH_URL_BASE which is set to “http://jitsi-meet-prosody:5280” in my setup (and my setup does not work without it, checked that).

So drilling further I noted this change in meet.conf of the web container: web: fix setting prefix for subdomains. In there the proxy_pass command of nginx has been altered (the ?prefix … has been added)

proxy_pass {{ $XMPP_BOSH_URL_BASE }}/xmpp-websocket?prefix=$prefix&$args;

By changing the meet.conf in the running web container removing the “?prefix …” and restarting nginx I was able to verify that this causes the above behavior. Without “?prefix …” it works.

Unluckily I found no way to log what the proxy_pass command generates as a resulting URL. The name in the DNS request going to the outside seems to be ok (jitsi-meet-prosody)

So, hope this gives you @saghul an idea and enough infos for if and how this can be fixed. Thx in advance for your help

LeBaton · October 20, 2022, 11:06am

I’ve had the same issue, try setting in your web env variables

            - name: NGINX_RESOLVER
              value: "kube-dns.kube-system.svc.cluster.local"

hamlet · October 21, 2022, 6:27am

Thx for the tip.

That did change the error in the web container from " (110: Operation timed out)" to “(3: Host not found)”, but did not resolve the issue in general.

hamlet · October 22, 2022, 10:34am

Finally I have a working configuration again.

Resolution is to set the NGINX_RESOLVER variable like LeBaton (@LeBaton Thx again!) hinted above

AND

to use the FDQN for prosody inside XMPP_BOSH_URL_BASE.

So, in my case jitsi-meet-prosody (the shortened name) does not work, with the FDQN jitsi-meet-prosody.default.svc.cluster.local it works with stable-7882.

Cudos to this issue

github.com/jitsi-contrib/jitsi-helm

Possible Breakage in Jitsi Meet stable-7830

opened 06:35PM - 05 Oct 22 UTC

spijet

The new Jitsi Meet Docker images (tagged as `stable-7830`) introduce a breaking …change in JVB config and it's template: now the JVB image expects a comma-separated list of IP addresses to advertise in the new `` environment variable. Before that we could specify a single IP address in `DOCKER_HOST_ADDRESS` variable. I'm going to update the JVB templates and add a way to specify this new variable, test it out locally and make a PR. I'll also leave the old variable intact for the time being, in case someone still uses older versions.

which gave me the final hint.

@saghul Perhaps you may have a look especially at the comment of devium in this issue thread which explains how the behavior of proxy_pass function changes once a variable is added to the URL. Possibly, this also affects Jitsi for Docker somehow. Above my paygrade of understanding if this is the case, so just to make sure

saghul · October 24, 2022, 8:50am

Thanks for the heads up, I dropped a comment there.