WS enable on multiple shards with Octo and JWT

Install & Config Record & stream - Jibri
#1

Hi Jitsi Team,
We enable Websocket on our shards and seen below issue:
Config.js
websocket: ‘wss://meet.ex.com/xmpp-websocket’,
deploymentInfo: {
shard: “shard1”,
region: “us-west-2”,
userRegion: “us-west-2”
},
Prosady:
modules_enabled = {
“turncredentials”;
“bosh”;
“websocket”;
“smacks”;
smacks_max_unacked_stanzas = 5;
smacks_hibernation_time = 60;
smacks_max_hibernated_sessions = 1;
smacks_max_old_sessions = 1;
Nginx:

xmpp websockets

location = /xmpp-websocket {
    proxy_pass http://127.0.0.1:5280/xmpp-websocket?prefix=$prefix&$args;
    add_header 'x-jitsi-shard' 'shard2';
    add_header 'x-jitsi-region' 'eu-west-3';
    add_header 'Access-Control-Expose-Headers' 'X-Jitsi-Shard, X-Jitsi-Region';
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
    tcp_nodelay on;
    proxy_read_timeout 900s;
}

JVB:
videobridge {
http-servers {
public {
port = 9090
}
}
websockets {
enabled = true
domain = “meet.xe.com:443”
tls = true
server-id = 10.2.2.133
}
}
Issue:
2022-01-21T20:02:50.968Z [modules/xmpp/XmppConnection.js] Detected that shard changed from shard1 to shard2
2022-01-21T20:02:50.969Z [modules/statistics/statistics.js] <Function.jn.sendAnalyticsAndLog>: {“type”:“operational”,“action”:“connection.failed”,“attributes”:{“error_type”:“connection.otherError”,“shard_changed”:true,“suspend_time”:1,“time_since_last_success”:17409}}
VM2162 app.bundle.min.js:138 2022-01-21T20:02:50.970Z [features/base/conference] JWT parsing error: (2) [’- invalid nbf value’, ‘- context object is missing from the payload’]
2022-01-21T20:04:29.722Z [features/overlay] <TV.componentDidMount>: The conference will be reloaded after 24 seconds.
VM2211 app.bundle.min.js:138 2022-01-21T20:04:29.746Z [features/large-video] <f$>: selectLocalParticipantInLargeVideo
VM2211 app.bundle.min.js:138 2022-01-21T20:04:29.747Z [features/large-video] selectLocalParticipantInLargeVideo-participantId-local
VM2211 app.bundle.min.js:138 2022-01-21T20:04:29.749Z [features/large-video] selectLocalParticipantInLargeVideo-largeVideo.participantId-f6326660
VM2210 lib-jitsi-meet.min.js:2 2022-01-21T20:04:29.764Z [modules/e2eping/e2eping.js] <Qa.stop>: Stopping e2eping
VM2210 lib-jitsi-meet.min.js:2 2022-01-21T20:04:29.765Z [modules/xmpp/JingleSessionPC.js] <br.close>: JingleSessionPC[session=JVB,initiator=false,sid=dfag5ell0bo1r] Clearing modificationQueue
VM2210 lib-jitsi-meet.min.js:2 2022-01-21T20:04:29.765Z [modules/xmpp/JingleSessionPC.js] <br.close>: JingleSessionPC[session=JVB,initiator=false,sid=dfag5ell0bo1r] Queued PC close task
And restart

If I disable the WS everything works fine
Any Idea?
jitsi-meet-prosody 1.0.5764-1
jitsi-meet-tokens 1.0.5764-1
jitsi-meet-web 1.0.5764-1
jitsi-meet-web-config 1.0.5764-1
jicofo 1.0-840-1
prosody 0.11.4-1
jitsi-videobridge2 2.1-607-g153f7e4e-1

#2

So your request are moving from one shard to another. So your initial config.js is loaded from shard1 then your xmpp connection goes to shard2 when the error is detected we reload, as this is what we call a split brain in the haproxies.

#3

Also, make sure you enable websocketKeepAliveUrl, and make that URL stick using the room param, this will keep the haproxy entries from expiring.

#4

Thanks @damencho
I add
// Websocket URL

websocket: ‘wss://meet.xi.com/xmpp-websocket’ ,

websocketKeepAliveUrl: ‘https:/meet.xi.com//_unlock’ ,

Still seeing the issue.
“make that URL stick using the room param”
Can you please give me an example and where should I configure it.

Also, do I need to set
crossRegion:
in deploymentInfo:

#5

The websocket keepalive link seems broken.

Sorry, I’m not very familiar with haproxy configs, bit make sure it follows the rest of the rules for room param.

#6

@Hamid_Narimani haproxy config for “make that URL stick using the room param” can look like this

backend jitsi-shards
        ...
        stick-table type string len 256 size 200k expire 120m peers haproxy-jitsi-peers
        stick on urlp(room) table jitsi-shards
        ...

where haproxy-jitsi-peers is a definition of other haproxy instances (peers), if you have them, in your infrastructure with which you should sync the stick-table