Upgraded to Jitsi Stable 7830 (Kubernetes Setup broken)

Hi there,

My working setup is broken after upgrading from stable-7648-4 to stable-7830. I am not seeing any error in jvb, jicofo, prosody services except I see an error on my web service that’s complaining about not able to reach prosody service.

This is my configuration inside web-deployment.yaml file and is unable to reach shard-0-prosody somehow after upgrade.

env:
            - name: XMPP_SERVER
              value: shard-0-prosody
            - name: XMPP_BOSH_URL_BASE
              value: http://shard-0-prosody:5280

Any idea if there were some breaking changes that could cause this?

Also, I noticed a new config JVB_ADVERTISE_IPS on JVB. Is this something required? If so, What should it be set?

Thanks!

No idea sorry. Are you able to ping / telnet that container from the other container?

It’s a replacement for DOCKER_HOST_ADDRESS.

I currently have the same problem, could you fix it?

Ok I tried again with both 7830 and latest 7882. Working fine now… Thanks Saghul!

I also have this behavior switching from stable-7648-4 (working) to stable-7830 or 7882 (not working).

Comparing the prosody debug logs I noted a General Error “No muc_component specified. No muc to operate on!” when trying with 7830 / 7882.

@mehtapaxshal Please be so kind to share what you changed to get your setup working again on 7830 / 7882

I didn’t do anything specific…It failed for me once and then I retried again and started working for me.

Make sure - you have the required env configs set for the latest version!

Could you share your env config so I can compare if you are setting something differently?

Prosody config:

containers:
        - name: prosody
          image: jitsi/prosody:stable-7882
          volumeMounts:
            - mountPath: /config
              name: prosody-config-volume
            - name: prosody
              mountPath: /prosody-plugins-custom/mod_token_moderation.lua
              subPath: mod_token_moderation.lua
          imagePullPolicy: Always
          env:
            - name: XMPP_DOMAIN
              value: <domain>
            - name: XMPP_AUTH_DOMAIN
              value: auth.<domain>
            - name: PUBLIC_URL
              value: "<domain>"
            - name: XMPP_MUC_DOMAIN
              value: muc.<domain>
            - name: XMPP_INTERNAL_MUC_DOMAIN
              value: internal-muc.<domain>
            - name: JICOFO_COMPONENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: jitsi-config
                  key: JICOFO_COMPONENT_SECRET
            - name: JVB_AUTH_USER
              value: jvb
            - name: JVB_AUTH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: jitsi-config
                  key: JVB_AUTH_PASSWORD
            - name: JICOFO_AUTH_USER
              value: focus
            - name: JICOFO_AUTH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: jitsi-config
                  key: JICOFO_AUTH_PASSWORD
            - name: JVB_TCP_HARVESTER_DISABLED
              value: "true"
            - name: ENABLE_AUTH
              value: "true"
            - name: AUTH_TYPE
              value: "jwt"
            - name: JWT_AUTH_TYPE
              value: token
            - name: JWT_TOKEN_AUTH_MODULE
              value: token_verification
            - name: JWT_ACCEPTED_AUDIENCES
              value: <aud>
            - name: JWT_ACCEPTED_ISSUERS
              value: <issuer>
            - name: JWT_APP_ID
              value: <app_id>
            - name: JWT_APP_SECRET
              valueFrom:
                secretKeyRef:
                  name: jitsi-config
                  key: JWT_APP_SECRET
            - name: ENABLE_BREAKOUT_ROOMS
              value: "false"
            - name: ENABLE_JAAS_COMPONENTS
              value: "false"
            - name: XMPP_MUC_MODULES
              value: token_moderation
            - name: JIGASI_XMPP_USER
              value: "<jigasi_user>"
            - name: JIGASI_XMPP_PASSWORD
              value: "<jigasi_pwd>"
            - name: TZ
              value: America/Los_Angeles

Jicofo Config:

containers:
        - name: jicofo
          image: jitsi/jicofo:stable-7882
          volumeMounts:
            - mountPath: /config
              name: jicofo-config-volume
          imagePullPolicy: Always
          env:
            - name: XMPP_DOMAIN
              value: <domain>
            - name: XMPP_AUTH_DOMAIN
              value: auth.<domain>
            - name: PUBLIC_URL
              value: "<domain>"
            - name: XMPP_INTERNAL_MUC_DOMAIN
              value: internal-muc.<domain>
            - name: JICOFO_COMPONENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: jitsi-config
                  key: JICOFO_COMPONENT_SECRET
            - name: JICOFO_AUTH_USER
              value: focus
            - name: JICOFO_AUTH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: jitsi-config
                  key: JICOFO_AUTH_PASSWORD
            - name: JVB_BREWERY_MUC
              value: jvbbrewery
            - name: JICOFO_ENABLE_BRIDGE_HEALTH_CHECKS
              value: "true"
            - name: ENABLE_RECORDING
              value: "true"
            - name: XMPP_RECORDER_DOMAIN
              value: recorder.<domain>
            - name: JIGASI_BREWERY_MUC
              value: "JigasiBrewery"
            - name: TZ
              value: America/Los_Angeles

:@mehtapaxshal thx a lot for sharing
I tried to stick to your config as close as possible, but to no avail.

I still get (from log in web component)

my server ip - - [15/Oct/2022:12:57:45 +0200] “GET /pwa-worker.js HTTP/1.1” 200 1511 “https://jitsi.my domain/pwa-worker.js” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36 Edg/106.0.1370.42”
2022/10/15 12:58:14 [error] 261#261: *2 jitsi-meet-prosody could not be resolved (110: Operation timed out), client: 10.42.0.1, server: _, request: “GET /xmpp-websocket?room=foldingoppositesequiptoo HTTP/1.1”, host: “jitsi.my domain”

where jitsi-meet-prosody is the name of my prosody pod which can be resolved from the web pod

Nothing worthwile in the other logs.

I found that my config works as long as the web pod uses the container jitsi/web:stable-7648-4. The others (prosody, jigasi and jvb) are on stable-7882. Once I switch to jitsi/web:stable-7830 or jitsi/web:stable-7882 the setup ceases to work with “You have been disconnected” and the above error is logged.

@saghul Do you have any ideas where to look and what to check?

As Saghul mentioned - are you able to ping/telnet them? Though ping shouldn’t matter if it works for stable-7648-4 and not for stable-7882.

Could you try doing the setup from the scratch? That’s what I did it.

I am not able to ping or telnet (cmds are missing inside the container) the prosody pod, but for both web container versions, I am able to wget http://jitsi-meet-prosody:5280 and get back the index.html from prosody. So, name resolution works in general.

I have started to do tcpdumps for both web container versions and found out using wireshark that with web:stable-7882 the error is occuring inside “GET /xmpp-websocket?room=…”
The name jitsi-meet-prosody cannot be resolved and DNS requests are sent to the outside which fails as this name is a cluster-internal one.
This name is used in the env variable XMPP_BOSH_URL_BASE which is set to “http://jitsi-meet-prosody:5280” in my setup (and my setup does not work without it, checked that).

So drilling further I noted this change in meet.conf of the web container: web: fix setting prefix for subdomains. In there the proxy_pass command of nginx has been altered (the ?prefix … has been added)

proxy_pass {{ $XMPP_BOSH_URL_BASE }}/xmpp-websocket?prefix=$prefix&$args;

By changing the meet.conf in the running web container removing the “?prefix …” and restarting nginx I was able to verify that this causes the above behavior. Without “?prefix …” it works.

Unluckily I found no way to log what the proxy_pass command generates as a resulting URL. The name in the DNS request going to the outside seems to be ok (jitsi-meet-prosody)

So, hope this gives you @saghul an idea and enough infos for if and how this can be fixed. Thx in advance for your help

I’ve had the same issue, try setting in your web env variables

            - name: NGINX_RESOLVER
              value: "kube-dns.kube-system.svc.cluster.local"

Thx for the tip.

That did change the error in the web container from " (110: Operation timed out)" to “(3: Host not found)”, but did not resolve the issue in general.

Finally I have a working configuration again.

Resolution is to set the NGINX_RESOLVER variable like LeBaton (@LeBaton Thx again!) hinted above

AND

to use the FDQN for prosody inside XMPP_BOSH_URL_BASE.

So, in my case jitsi-meet-prosody (the shortened name) does not work, with the FDQN jitsi-meet-prosody.default.svc.cluster.local it works with stable-7882.

Cudos to this issue

which gave me the final hint.

@saghul Perhaps you may have a look especially at the comment of devium in this issue thread which explains how the behavior of proxy_pass function changes once a variable is added to the URL. Possibly, this also affects Jitsi for Docker somehow. Above my paygrade of understanding if this is the case, so just to make sure :slight_smile:

Thanks for the heads up, I dropped a comment there.