Videobridge Pool not connecting

[niclasheu]

Hey everyone!

I tried to have a go at creating a videobridge pool which connects to multiple front ends using ansible. However, as soon as I add the second front end to the bridge, it fails to connect to both. I have attached my ansible template and jvb.log. Did anyone run into similar issues?

org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=true
org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=meet-jit-si-turnrelay.jitsi.net:443
org.jitsi.videobridge.ENABLE_STATISTICS=true

{% for host in groups['central'] %}
org.jitsi.videobridge.xmpp.user.shard-{{ loop.index }}.HOSTNAME={{ hostvars[host].inventory_hostname }}
org.jitsi.videobridge.xmpp.user.shard-{{ loop.index }}.DOMAIN=auth.{{ hostvars[host].inventory_hostname }}
org.jitsi.videobridge.xmpp.user.shard-{{ loop.index }}.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard-{{ loop.index }}.PASSWORD={{ videobridge_pass }}
org.jitsi.videobridge.xmpp.user.shard-{{ loop.index }}.MUC_JIDS=JvbBrewery@internal.auth.{{ hostvars[host].inventory_hostname }}
org.jitsi.videobridge.xmpp.user.shard-{{ loop.index }}.MUC_NICKNAME={{ inventory_hostname }}
org.jitsi.videobridge.xmpp.user.shard-{{ loop.index }}.DISABLE_CERTIFICATE_VERIFICATION=true

{% endfor %}

org.jitsi.videobridge.STATISTICS_TRANSPORT=muc,colibri
org.jitsi.videobridge.ENABLE_REST_SHUTDOWN=true
org.jitsi.videobridge.shutdown.ALLOWED_SOURCE_REGEXP=127.0.0.1
org.jitsi.videobridge.xmpp.user.shard.DISABLE_CERTIFICATE_VERIFICATION=true

And this is the log:

2021-02-01 20:31:49.254 INFO: [15] [hostname=main.meet.domain.io id=shard-1] MucClient$1.connected#259: Connected.
2021-02-01 20:31:49.254 INFO: [15] [hostname=main.meet.domain.io id=shard-1] MucClient.lambda$getConnectAndLoginCallable$7#594: Logging in.
2021-02-01 20:31:49.274 WARNING: [15] [hostname=main.meet.domain.io id=shard-1] MucClient.lambda$getConnectAndLoginCallable$7#608: Failed to login. Disconnecting to trigger a re-connect.
2021-02-01 20:31:49.276 INFO: [15] [hostname=main.meet.domain.io id=shard-1] MucClient$1.connectionClosed#271: Closed.
2021-02-01 20:31:49.766 WARNING: [27] [hostname=main.meet.domain.io id=shard-1] MucClient.setPresenceExtensions#420: Cannot set presence extension: not connected.
2021-02-01 20:31:49.766 WARNING: [27] [hostname=second.meet.domain.io id=shard-2] MucClient.setPresenceExtensions#420: Cannot set presence extension: not connected.
2021-02-01 20:31:50.139 WARNING: [16] [hostname=second.meet.domain.io id=shard-2] MucClient.lambda$getConnectAndLoginCallable$7#588: [MucClient id=shard-2 hostname=second.meet.domain.io] error connecting
org.jivesoftware.smack.XMPPException$StreamErrorException: host-unknown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
<stream:error><host-unknown xmlns='urn:ietf:params:xml:ns:xmpp-streams'/><text>This server does not serve auth.second.meet.domain.io</text></stream:error>
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1059)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
        at java.lang.Thread.run(Thread.java:748)
2021-02-01 20:31:50.145 WARNING: [205] org.jivesoftware.smack.AbstractXMPPConnection.callConnectionClosedOnErrorListener: Connection XMPPTCPConnection[not-authenticated] (0) closed with error
org.jivesoftware.smack.XMPPException$StreamErrorException: host-unknown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
<stream:error><host-unknown xmlns='urn:ietf:params:xml:ns:xmpp-streams'/><text>This server does not serve auth.second.meet.domain.io</text></stream:error>
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1064)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
        at java.lang.Thread.run(Thread.java:748)
2021-02-01 20:31:50.146 WARNING: [205] [hostname=second.meet.domain.io id=shard-2] MucClient$1.connectionClosedOnError#277: Closed on error:
org.jivesoftware.smack.XMPPException$StreamErrorException: host-unknown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
<stream:error><host-unknown xmlns='urn:ietf:params:xml:ns:xmpp-streams'/><text>This server does not serve auth.second.meet.domain.io</text></stream:error>
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1064)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
        at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
        at java.lang.Thread.run(Thread.java:748)

One Idea which I have had is an improper configuration of the jvb.conf. Mine looks like this:

videobridge {
  http-servers {
    public {
      port = 9090
    }
  }
  websockets {
    enabled = true
    domain = "main.meet.domain.io:443"
    tls = true
    server-id = test-jvb1
  }
}

This only mentions the first instance. If that is the actual problem, some guidance would be nice as I am not experienced enough to know what to do here. I took inspiration from this project:

Working Multi Jitsi-meet / multi Videobridge setup

As mentioned over on the users discussion, we have shared our Ansible playbooks that build or https://switch.ch/meet service. Here is the repository: https://github.com/switch-ch/jitsi-deploy This is NOT Plug and Play - but it should be easy enough to extract a working configuration: Features: Multiple Front-end Jitsi-Meet instances (in our case, one per university) Pool of shared video-bridges (all front end instances share the same pool of videobridges) using the new MUC feature Nginx / L…

[jcfischer]

@niclasheu

host-unknown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions

Is see a “host-unknown” error - is your DNS setup correctly?

/jc

[niclasheu]

@jcfischer

“host-unknown” error

I was able to fix that error last night, thanks. It was something wrong with the prosody config on one server.
However, the bridge still cannot connect to main.meet. The secondary works fine and it can log in, but main just won’t work for some reason. The logs are not very informative.

2021-02-02 10:54:35.784 INFO: [15] [hostname=main.meet.domain.io id=shard-1] MucClient$1.connected#259: Connected.
2021-02-02 10:54:35.785 INFO: [15] [hostname=main.meet.domain.io id=shard-1] MucClient.lambda$getConnectAndLoginCallable$7#594: Logging in.
2021-02-02 10:54:35.805 WARNING: [15] [hostname=main.meet.domain.io id=shard-1] MucClient.lambda$getConnectAndLoginCallable$7#608: Failed to login. Disconnecting to trigger a re-connect.
2021-02-02 10:54:35.806 INFO: [15] [hostname=main.meet.domain.io id=shard-1] MucClient$1.connectionClosed#271: Closed.
2021-02-02 10:54:38.559 WARNING: [27] [hostname=main.meet.domain.io id=shard-1] MucClient.setPresenceExtensions#420: Cannot set presence extension: not connected. 

The jicofo log on the main server has no mention of the videobridge. Not sure where I am going wrong here. I tried giving the bridge a new nickname for every instance it connects to, but that results in the same error.

[niclasheu]

I think I found it! in /var/lib/prosody the password for the bridge was set to 'ok' instead of the actual password. I am not sure why that is happening on one server and not the other, but changing it to the actual password worked. I will rewrite the file using a template for the future, just to make sure it won’t happen again.

Thanks, @jcfischer for taking a look!