Kubernetes deployment - error Can not invite participant -- no bridge available

i just deployed jitsi in my k8 environment and the app seems to be running fine. the webui crashes once a meeting is set and a pair of users try to join the meeting. the error message on jicofo pod is Can not invite participant – no bridge available. not too sure on how to debug a bit further

Check jvb logs, why it cannot connect to the xmpp server.

— i get this error in a loop —

    at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
    at java.lang.Thread.run(Thread.java:748)

JVB 2019-09-26 14:21:05.705 INFO: [5982] org.jitsi.xmpp.mucclient.MucClient.log() [MucClient id=shard hostname=10.96.189.239] closed on error:
org.jivesoftware.smack.XMPPException$StreamErrorException: connection-timeout You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
stream:error</stream:error>
at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1064)
at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
at java.lang.Thread.run(Thread.java:748)
JVB 2019-09-26 14:21:06.567 INFO: [14] org.jitsi.xmpp.mucclient.MucClientManager.log() Setting a presence extension: org.jitsi.xmpp.extensions.colibri.ColibriStatsExtension@326d1c4d
JVB 2019-09-26 14:21:07.873 INFO: [18] org.jitsi.videobridge.Videobridge.log() CAT=stat create_conf,conf_id=c7190231c3fe29ac conf_name=null,logging=false,conf_count=1,ch_count=0,v_streams=0
JVB 2019-09-26 14:21:07.873 SEVERE: [18] org.jitsi.videobridge.health.Health.log() Health check failed in 0ms:
java.lang.NullPointerException
JVB 2019-09-26 14:21:11.567 INFO: [14] org.jitsi.xmpp.mucclient.MucClientManager.log() Setting a presence extension: org.jitsi.xmpp.extensions.colibri.ColibriStatsExtension@87ae431
JVB 2019-09-26 14:21:11.758 INFO: [5983] org.jitsi.xmpp.mucclient.MucClient.log() [MucClient id=shard hostname=10.96.189.239] connected
JVB 2019-09-26 14:21:11.758 INFO: [5983] org.jitsi.xmpp.mucclient.MucClient.log() [MucClient id=shard hostname=10.96.189.239] reconnection failed
JVB 2019-09-26 14:21:16.567 INFO: [14] org.jitsi.xmpp.mucclient.MucClientManager.log() Setting a presence extension: org.jitsi.xmpp.extensions.colibri.ColibriStatsExtension@3ff4da78
JVB 2019-09-26 14:21:17.873 INFO: [18] org.jitsi.videobridge.Videobridge.log() CAT=stat create_conf,conf_id=4642b7459d54c002 conf_name=null,logging=false,conf_count=1,ch_count=0,v_streams=0
JVB 2019-09-26 14:21:17.873 SEVERE: [18] org.jitsi.videobridge.health.Health.log() Health check failed in 0ms:
java.lang.NullPointerException
JVB 2019-09-26 14:21:21.255 INFO: [17] org.jitsi.videobridge.VideobridgeExpireThread.log() Running expire()
JVB 2019-09-26 14:21:21.567 INFO: [14] org.jitsi.xmpp.mucclient.MucClientManager.log() Setting a presence extension: org.jitsi.xmpp.extensions.colibri.ColibriStatsExtension@44ee5951
JVB 2019-09-26 14:21:26.567 INFO: [14] org.jitsi.xmpp.mucclient.MucClientManager.log() Setting a presence extension: org.jitsi.xmpp.extensions.colibri.ColibriStatsExtension@1ebbbaf5
JVB 2019-09-26 14:21:27.873 INFO: [18] org.jitsi.videobridge.Videobridge.log() CAT=stat create_conf,conf_id=94b1cd37db7d4767 conf_name=null,logging=false,conf_count=1,ch_count=0,v_streams=0
JVB 2019-09-26 14:21:27.873 SEVERE: [18] org.jitsi.videobridge.health.Health.log() Health check failed in 0ms:
java.lang.NullPointerException
JVB 2019-09-26 14:21:31.569 INFO: [14] org.jitsi.xmpp.mucclient.MucClientManager.log() Setting a presence extension: org.jitsi.xmpp.extensions.colibri.ColibriStatsExtension@413f8e7f
JVB 2019-09-26 14:21:36.570 INFO: [14] org.jitsi.xmpp.mucclient.MucClientManager.log() Setting a presence extension: org.jitsi.xmpp.extensions.colibri.ColibriStatsExtension@5881ab3a
JVB 2019-09-26 14:21:37.873 INFO: [18] org.jitsi.videobridge.Videobridge.log() CAT=stat create_conf,conf_id=19ee145e18b16ed8 conf_name=null,logging=false,conf_count=1,ch_count=0,v_streams=0
JVB 2019-09-26 14:21:37.873 SEVERE: [18] org.jitsi.videobridge.health.Health.log() Health check failed in 0ms:
java.lang.NullPointerException

jvb cannot connect to xmpp server

any suggestion on to get this through the line please?

— below are the svc details –

[bolaji@dhcp-10-105-13-116 approved_certs] kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE jvb-tcp LoadBalancer 10.103.136.83 <pending> 4443:32353/TCP 9d jvb-udp LoadBalancer 10.111.198.73 <pending> 10000:32537/UDP 9d prosody ClusterIP 10.96.189.239 <none> 5222/TCP,5280/TCP,5347/TCP 9d web ClusterIP 10.109.209.109 <none> 80/TCP,443/TCP 9d [bolaji@dhcp-10-105-13-116 approved_certs]

No idea what is that, but jvb uses port 5222 to connect to the xmpp server control muc room.

all i did was query kubernetes for the service details but it did not render in a tabular form on the webpage

it seems that prosody is the app running on that port

prosody ClusterIP 10.96.189.239 5222/TCP,5280/TCP,5347/TCP 9d

hi bjbojo,

In all the jitsi components (web, jicofo, prosody and jvb) use the service name pointing to prosody as XMPP_SERVER env config. It is essential here that you understand how service discovery with DNS and stuff works in k8s. Because depending on how the components are split up into k8s namespaces changes the way you are able to communicate with the xmpp service slightly… easiest approach would be to have all in one jitsi namespace.

The individual components will use XMPP_SERVER to look up the address of the xmpp server (prosody) via dns. The other domains (eg. XMPP_MUC_DOMAIN) are then used within xmpp communication.

k8s config that works for me:
prosody:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    io.kompose.service: prosody
  name: prosody
  namespace: jitsi
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      creationTimestamp: null
      labels:
        io.kompose.service: prosody
    spec:
      containers:
      - image: <prosody image>
       # imagePullPolicy: Always
        name: prosody
        env:
        - name: XMPP_DOMAIN
          value: "xmpp"
        - name: JICOFO_AUTH_USER
          value: "focus"
        - name: JICOFO_AUTH_PASSWORD
          value: "passw0rd"
        - name: JICOFO_COMPONENT_SECRET
          value: "**********"
        - name: XMPP_AUTH_DOMAIN
          value: "auth-xmpp"
        - name: XMPP_GUEST_DOMAIN
          value: "guest-xmpp"
        - name: XMPP_MUC_DOMAIN
          value: "muc-xmpp"
        - name: XMPP_INTERNAL_MUC_DOMAIN
          value: "internal-muc-xmpp"
        - name: XMPP_MODULES
          value: ""
        - name: XMPP_MUC_MODULES
          value: ""
        - name: XMPP_INTERNAL_MUC_MODULES
          value: ""
        - name: JVB_AUTH_USER
          value: "jvb"
        - name: JVB_AUTH_PASSWORD
          value: "****"
        - name: TZ
          value: "Europe/Amsterdam"
        - name: ENABLE_AUTH
          value: "0"
        - name: ENABLE_GUESTS
          value: "1"
        - name: S6_READ_ONLY_ROOT #needed if you fiddle with s6 config via configMaps
          value: "1"
        resources: {}
        ports:
        - containerPort: 5222
          protocol: TCP
          name: xmpp-client
        - containerPort: 5269
          protocol: TCP
          name: xmpp-s2s
        - containerPort: 5280
          protocol: TCP
          name: bosh-http
        - containerPort: 5347
          protocol: TCP
          name: components #i.e. external components https://prosody.im/doc/components
        volumeMounts:
        - mountPath: /etc/cont-init.d # yeah  I am doing some customization here, you wont need it
          name: prosody-s6-config
        - mountPath: /defaults/conf.d/
          name: prosody-jitsi-meet-cfg
        # - mountPath: /config/certs
        #   name: prosody-certs
      restartPolicy: Always
      volumes:
      # - name: prosody-certs
      #   persistentVolumeClaim:
      #     claimName: prosody-claim0
      - name: prosody-cfg
        configMap:
          name: prosody-cfg
          items:
          - key: prosody.cfg.lua
            path: prosody.cfg.lua
      - name: prosody-s6-config
        configMap:
          name: prosody-s6-config
      - name: prosody-jitsi-meet-cfg
        configMap:
          name: prosody-jitsi-meet-cfg
status: {}

xmpp service:

---
kind: Service
apiVersion: v1
metadata:
  name: xmpp #this becomes your XMPP_SERVER
  namespace: jitsi
  labels:
    app: prosody
spec:
  selector:
    io.kompose.service: prosody
  ports:
  - name: xmpp-client
    protocol: TCP
    port: 5222
    targetPort: xmpp-client
  - name: xmpp-s2s
    protocol: TCP
    port: 5269
    targetPort: xmpp-s2s
  - name: bosh-http
    protocol: TCP
    port: 5280
    targetPort: bosh-http
  - name: components
    protocol: TCP
    port: 5347
    targetPort: components
...

now important to set up the domains (muc domains and stuff) which you use in the config of the other components:
this is from my config map, you wont need it but it illustrates how the config from env and the xmpp config in prosody are interconnected. Basically set all the XMPP domains the same everywhere and you should be fine. Use the service name of the prosody service as XMPP_SERVER

apiVersion: v1
kind: ConfigMap
metadata:
  name: prosody-jitsi-meet-cfg
  namespace: jitsi
data:
  jitsi-meet.cfg.lua: |
    admins = { "{{ .Env.JICOFO_AUTH_USER }}@{{ .Env.XMPP_AUTH_DOMAIN }}" }
    plugin_paths = { "/prosody-plugins-custom" }
    http_default_host = "{{ .Env.XMPP_DOMAIN }}"

    VirtualHost "{{ .Env.XMPP_DOMAIN }}"
        {{ if .Env.ENABLE_AUTH | default "0" | toBool }}
        authentication = "internal_plain"
        {{ else }}
        authentication = "anonymous"
        {{ end }}
        ssl = {
            key = "/config/certs/{{ .Env.XMPP_DOMAIN }}.key";
            certificate = "/config/certs/{{ .Env.XMPP_DOMAIN }}.crt";
        }
        modules_enabled = {
            "bosh";
            "pubsub";
            "ping";
            {{ if .Env.XMPP_MODULES }}
            "{{ join "\";\n\"" (splitList "," .Env.XMPP_MODULES) }}";
            {{ end }}
        }

        c2s_require_encryption = false

    {{ if and (.Env.ENABLE_AUTH | default "0" | toBool) (.Env.ENABLE_GUESTS | default "0" | toBool) }}
    VirtualHost "{{ .Env.XMPP_GUEST_DOMAIN }}"
        authentication = "anonymous"
        c2s_require_encryption = false
    {{ end }}

    VirtualHost "{{ .Env.XMPP_AUTH_DOMAIN }}"
        ssl = {
            key = "/config/certs/{{ .Env.XMPP_AUTH_DOMAIN }}.key";
            certificate = "/config/certs/{{ .Env.XMPP_AUTH_DOMAIN }}.crt";
        }
        authentication = "internal_plain"

    Component "{{ .Env.XMPP_INTERNAL_MUC_DOMAIN }}" "muc"
        modules_enabled = {
            "ping";
            {{ if .Env.XMPP_INTERNAL_MUC_MODULES }}
            "{{ join "\";\n\"" (splitList "," .Env.XMPP_INTERNAL_MUC_MODULES) }}";
            {{ end }}
        }
        storage = "memory"
        muc_room_cache_size = 1000
        ssl = {
            key = "/config/certs/{{ .Env.XMPP_INTERNAL_MUC_DOMAIN }}.key";
            certificate = "/config/certs/{{ .Env.XMPP_INTERNAL_MUC_DOMAIN }}.crt";
        }

    Component "{{ .Env.XMPP_MUC_DOMAIN }}" "muc"
        storage = "memory"
        modules_enabled = {
            {{ if .Env.XMPP_MUC_MODULES }}
            "{{ join "\";\n\"" (splitList "," .Env.XMPP_MUC_MODULES) }}";
            {{ end }}
        }
        ssl = {
            key = "/config/certs/{{ .Env.XMPP_MUC_DOMAIN }}.key";
            certificate = "/config/certs/{{ .Env.XMPP_MUC_DOMAIN }}.crt";
        }
    {{ if .Env.XMPP_FOCUS_DOMAIN }}
    Component "{{ .Env.XMPP_FOCUS_DOMAIN }}"
        component_secret = "{{ .Env.JICOFO_COMPONENT_SECRET }}"
        ssl = {
            key = "/config/certs/{{ .Env.XMPP_FOCUS_DOMAIN }}.key";
            certificate = "/config/certs/{{ .Env.XMPP_FOCUS_DOMAIN }}.crt";
        }
    {{ else }}
    Component "focus.{{ .Env.XMPP_DOMAIN }}"
        component_secret = "{{ .Env.JICOFO_COMPONENT_SECRET }}"
    {{ end }}

many thanks for your tips… I will go over my configs again with your suggestions

Hey guys, can you please share your Kubernetes configuration for all the components (web, prosody, jicofo, jvb)?