Graceful Shutdown is not stopping at all

Hiello

we have two working Videobriges, and want to stop one of it, on this VB is a conference. We use the script: /usr/share/jitsi-videobridge/graceful_shutdown.sh to trigger a gracefull shutdown.
Output:

Graceful shutdown started
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
There are still 2 participants
...

(it looks as this go endlees)
I have looked in the script the loop is only stopping if participantCount == 0.

When we stop the vb (systemctl status jitsi-videobridge2.service) the shutdown is not graceful, the image freece for ~20 Seconds.

We see in the jvb.log where we start the shutdown:

JVB 2021-12-31 12:48:46.945 INFO: [17] VideobridgeExpireThread.expire#140: Running expire()
JVB 2021-12-31 12:48:47.026 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000012S. Sticky failure: false
JVB 2021-12-31 12:48:57.025 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000014S. Sticky failure: false
JVB 2021-12-31 12:49:03.724 WARNING: [45] Videobridge.shutdown#483: Received shutdown request, graceful=true
JVB 2021-12-31 12:49:03.724 INFO: [45] Videobridge.enableGracefulShutdownMode#274: Entered graceful shutdown mode
JVB 2021-12-31 12:49:07.025 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000035S. Sticky failure: false
JVB 2021-12-31 12:49:17.025 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000009S. Sticky failure: false
JVB 2021-12-31 12:49:27.026 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000137S. Sticky failure: false
JVB 2021-12-31 12:49:37.026 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000018S. Sticky failure: false
JVB 2021-12-31 12:49:46.945 INFO: [17] VideobridgeExpireThread.expire#140: Running expire()
JVB 2021-12-31 12:49:47.026 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000021S. Sticky failure: false
JVB 2021-12-31 12:49:57.026 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000016S. Sticky failure: false
JVB 2021-12-31 12:50:07.026 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000037S. Sticky failure: false
JVB 2021-12-31 12:50:17.026 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000018S. Sticky failure: false
JVB 2021-12-31 12:50:27.026 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000012S. Sticky failure: false
JVB 2021-12-31 12:50:37.026 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000026S. Sticky failure: false
JVB 2021-12-31 12:50:46.945 INFO: [17] VideobridgeExpireThread.expire#140: Running expire()
JVB 2021-12-31 12:50:47.026 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000028S. Sticky failure: false
JVB 2021-12-31 12:50:57.025 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000034S. Sticky failure: false
JVB 2021-12-31 12:51:07.025 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000012S. Sticky failure: false
JVB 2021-12-31 12:51:17.025 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000011S. Sticky failure: false
JVB 2021-12-31 12:51:27.025 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000045S. Sticky failure: false
JVB 2021-12-31 12:51:37.026 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000009S. Sticky failure: false
JVB 2021-12-31 12:51:46.945 INFO: [17] VideobridgeExpireThread.expire#140: Running expire()
JVB 2021-12-31 12:51:47.025 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000017S. Sticky failure: false
JVB 2021-12-31 12:51:57.026 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000063S. Sticky failure: false
JVB 2021-12-31 12:52:07.026 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000047S. Sticky failure: false
JVB 2021-12-31 12:52:17.025 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.00001S. Sticky failure: false
JVB 2021-12-31 12:52:27.026 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000027S. Sticky failure: false
JVB 2021-12-31 12:52:37.026 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000012S. Sticky failure: false
JVB 2021-12-31 12:52:46.946 INFO: [17] VideobridgeExpireThread.expire#140: Running expire()
JVB 2021-12-31 12:52:47.026 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000039S. Sticky failure: false
JVB 2021-12-31 12:52:57.025 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000022S. Sticky failure: false
JVB 2021-12-31 12:53:07.025 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000019S. Sticky failure: false
JVB 2021-12-31 12:53:17.025 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000015S. Sticky failure: false
JVB 2021-12-31 12:53:27.026 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000019S. Sticky failure: false
JVB 2021-12-31 12:53:37.025 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000013S. Sticky failure: false
JVB 2021-12-31 12:53:46.945 INFO: [17] VideobridgeExpireThread.expire#140: Running expire()
JVB 2021-12-31 12:53:47.026 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000014S. Sticky failure: false
JVB 2021-12-31 12:53:57.025 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000015S. Sticky failure: false
JVB 2021-12-31 12:54:07.025 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.00001S. Sticky failure: false
JVB 2021-12-31 12:54:17.025 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000035S. Sticky failure: false
JVB 2021-12-31 12:54:27.025 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000008S. Sticky failure: false
JVB 2021-12-31 12:54:37.025 INFO: [18] HealthChecker.run#171: Performed a successful health check in PT0.000024S. Sticky failure: false

Jicofo log:

Jicofo 2021-12-31 12:49:05.420 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012091S. Sticky failure: false
Jicofo 2021-12-31 12:49:15.421 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012602S. Sticky failure: false
Jicofo 2021-12-31 12:49:25.421 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012677S. Sticky failure: false
Jicofo 2021-12-31 12:49:35.421 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012801S. Sticky failure: false
Jicofo 2021-12-31 12:49:45.422 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012789S. Sticky failure: false
Jicofo 2021-12-31 12:49:55.421 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.01317S. Sticky failure: false
Jicofo 2021-12-31 12:50:05.420 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012217S. Sticky failure: false
Jicofo 2021-12-31 12:50:15.426 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.018011S. Sticky failure: false
Jicofo 2021-12-31 12:50:25.421 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012692S. Sticky failure: false
Jicofo 2021-12-31 12:50:35.422 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.013667S. Sticky failure: false
Jicofo 2021-12-31 12:50:45.421 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012938S. Sticky failure: false
Jicofo 2021-12-31 12:50:55.421 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.013222S. Sticky failure: false
Jicofo 2021-12-31 12:51:05.420 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012511S. Sticky failure: false
Jicofo 2021-12-31 12:51:15.420 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012328S. Sticky failure: false
Jicofo 2021-12-31 12:51:25.420 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012557S. Sticky failure: false
Jicofo 2021-12-31 12:51:35.421 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012855S. Sticky failure: false
Jicofo 2021-12-31 12:51:45.423 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.015479S. Sticky failure: false
Jicofo 2021-12-31 12:51:55.421 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012574S. Sticky failure: false
Jicofo 2021-12-31 12:52:05.421 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.013367S. Sticky failure: false
Jicofo 2021-12-31 12:52:15.421 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012764S. Sticky failure: false
Jicofo 2021-12-31 12:52:25.421 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.013335S. Sticky failure: false
Jicofo 2021-12-31 12:52:35.421 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012448S. Sticky failure: false
Jicofo 2021-12-31 12:52:45.421 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012534S. Sticky failure: false
Jicofo 2021-12-31 12:52:55.420 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012191S. Sticky failure: false
Jicofo 2021-12-31 12:53:05.420 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012152S. Sticky failure: false
Jicofo 2021-12-31 12:53:15.420 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.01248S. Sticky failure: false
Jicofo 2021-12-31 12:53:25.420 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012183S. Sticky failure: false
Jicofo 2021-12-31 12:53:35.421 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012712S. Sticky failure: false
Jicofo 2021-12-31 12:53:45.420 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012306S. Sticky failure: false
Jicofo 2021-12-31 12:53:55.420 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012481S. Sticky failure: false
Jicofo 2021-12-31 12:54:05.421 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012464S. Sticky failure: false
Jicofo 2021-12-31 12:54:15.421 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012107S. Sticky failure: false
Jicofo 2021-12-31 12:54:25.420 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012349S. Sticky failure: false
Jicofo 2021-12-31 12:54:35.421 INFO: [22] HealthChecker.run#171: Performed a successful health check in PT0.012763S. Sticky failure: false
Jicofo 2021-12-31 12:54:42.631 INFO: [19] [type=bridge brewery=jvbbrewery] BaseBrewery.removeInstance#353: Removed brewery instance: jvbbrewery@internal.auth.meet.adfc-intern.de/vbdh01.meet.adfc-intern.de
Jicofo 2021-12-31 12:54:42.631 INFO: [19] BridgeSelector.removeJvbAddress#150: Removing JVB: jvbbrewery@internal.auth.meet.adfc-intern.de/vbdh01.meet.adfc-intern.de
Jicofo 2021-12-31 12:54:42.631 INFO: [19] JvbDoctor.removeBridge#152: Stopping health-check task for: jvbbrewery@internal.auth.meet.adfc-intern.de/vbdh01.meet.adfc-intern.de
Jicofo 2021-12-31 12:54:42.631 SEVERE: [61] [room=s@conference.meet.adfc-intern.de meeting_id=9f1f9b58-61d6-490f-9239-3a2968d0b8cd] JitsiMeetConferenceImpl.lambda$onMultipleBridgesDown$19#2108: One of our bridges 
failed: jvbbrewery@internal.auth.meet.adfc-intern.de/vbdh01.meet.adfc-intern.de
Jicofo 2021-12-31 12:54:42.632 INFO: [61] [room=s@conference.meet.adfc-intern.de meeting_id=9f1f9b58-61d6-490f-9239-3a2968d0b8cd bs_id=b18b5f] BridgeSession.terminate#199: Expiring channels for: Participant[s@conf
erence.meet.adfc-intern.de/c42cd6d6]@1365131413 on: Bridge[jid=jvbbrewery@internal.auth.meet.adfc-intern.de/vbdh01.meet.adfc-intern.de, relayId=157.90.226.172:4096, region=region1, stress=0.01]
Jicofo 2021-12-31 12:54:42.632 INFO: [61] [room=s@conference.meet.adfc-intern.de meeting_id=9f1f9b58-61d6-490f-9239-3a2968d0b8cd bs_id=b18b5f] BridgeSession.terminate#199: Expiring channels for: Participant[s@conf
erence.meet.adfc-intern.de/ca7b3bb8]@1726192016 on: Bridge[jid=jvbbrewery@internal.auth.meet.adfc-intern.de/vbdh01.meet.adfc-intern.de, relayId=157.90.226.172:4096, region=region1, stress=0.01]
Jicofo 2021-12-31 12:54:42.633 INFO: [61] BridgeSelectionStrategy.select#104: Selected initial bridge Bridge[jid=jvbbrewery@internal.auth.meet.adfc-intern.de/meet.adfc-intern.de, relayId=88.198.32.212:4096, region
=region1, stress=0.00] with reported stress=0.0 for participantRegion=region1 using strategy SplitBridgeSelectionStrategy
Jicofo 2021-12-31 12:54:42.633 INFO: [61] [room=s@conference.meet.adfc-intern.de meeting_id=9f1f9b58-61d6-490f-9239-3a2968d0b8cd] JitsiMeetConferenceImpl.inviteParticipant#789: Added participant id=c42cd6d6, bridg
e=jvbbrewery@internal.auth.meet.adfc-intern.de/meet.adfc-intern.de
Jicofo 2021-12-31 12:54:42.634 INFO: [41] DiscoveryUtil.discoverParticipantFeatures#154: Doing feature discovery for s@conference.meet.adfc-intern.de/c42cd6d6
Jicofo 2021-12-31 12:54:42.634 INFO: [61] [room=s@conference.meet.adfc-intern.de meeting_id=9f1f9b58-61d6-490f-9239-3a2968d0b8cd] JitsiMeetConferenceImpl.inviteParticipant#789: Added participant id=ca7b3bb8, bridg
e=jvbbrewery@internal.auth.meet.adfc-intern.de/meet.adfc-intern.de
Jicofo 2021-12-31 12:54:42.634 INFO: [41] DiscoveryUtil.discoverParticipantFeatures#194: Successfully discovered features for s@conference.meet.adfc-intern.de/c42cd6d6 in 0
Jicofo 2021-12-31 12:54:42.634 INFO: [41] [room=s@conference.meet.adfc-intern.de meeting_id=9f1f9b58-61d6-490f-9239-3a2968d0b8cd] AbstractChannelAllocator.allocateChannels#248: Using jvbbrewery@internal.auth.meet.
adfc-intern.de/meet.adfc-intern.de to allocate channels for: Participant[s@conference.meet.adfc-intern.de/c42cd6d6]@1365131413
Jicofo 2021-12-31 12:54:42.634 INFO: [167] DiscoveryUtil.discoverParticipantFeatures#154: Doing feature discovery for s@conference.meet.adfc-intern.de/ca7b3bb8
Jicofo 2021-12-31 12:54:42.635 INFO: [167] DiscoveryUtil.discoverParticipantFeatures#194: Successfully discovered features for s@conference.meet.adfc-intern.de/ca7b3bb8 in 1
Jicofo 2021-12-31 12:54:42.635 INFO: [167] [room=s@conference.meet.adfc-intern.de meeting_id=9f1f9b58-61d6-490f-9239-3a2968d0b8cd] AbstractChannelAllocator.allocateChannels#248: Using jvbbrewery@internal.auth.meet

VB Config:

videobridge {
    apis {
        xmpp-client {
           stats-filter {
              enabled = false
           }
        }
        rest {
            enabled = true
        }
        jvb-api {
            enabled = true
        }
    }
    rest {
            debug {
                enabled = true
            }
            health {
                enabled = true
            }
            shutdown {
                enabled = true
            }
            version {
                enabled = true
            }
    }
    http-servers {
        public {
            port = 9090
        }
        private {
            host = 127.0.0.1
        }
    }
    octo {
        enabled=true
        region="region1"
        bind_address="157.x.y.z"
        public_address="157.x.y.z"
    }
    websockets {
        enabled = true
        domain = "meet.adfc-intern.de:443"
        tls = true
    }
  graceful-shutdown-delay = 1 minute
}

Debian bullseye with openjdk-11 and:

root@meet:~# dpkg -l |grep jitsi
ii  jitsi-meet                     2.0.6726-1                     all          WebRTC JavaScript video conferences
ii  jitsi-meet-prosody             1.0.5675-1                     all          Prosody configuration for Jitsi Meet
ii  jitsi-meet-web                 1.0.5675-1                     all          WebRTC JavaScript video conferences
ii  jitsi-meet-web-config          1.0.5675-1                     all          Configuration for web serving of Jitsi Meet
ii  jitsi-videobridge2             2.1-595-g3637fda4-1            all          WebRTC compatible Selective Forwarding Unit (SFU)

I have also found the script:

which i do not find on debian and no instractions how to use this.

Have a good new year!

This is how graceful shutdown is supposed to work — it puts the JVB in graceful shutdown mode so that it doesn’t accept new channel allocations, and then the script doesn’t exit until there are no users on the bridge any more. So once all existing participants on the bridge have left, it will complete shutdown. This way you can shut down or restart a bridge without any impact to users, with the downside that if your users stay in the conference for a long time, it can take a long time for the bridge to shut down.

Hi jbg, thank you for clearify that.

What doese the jicofo/blob/master/script/graceful_shutdown.py script is doing?

It seams for me that it has nothing to do with the other script.

I also do not understand why there is a

setting in the vb. If there are no users on the VB why should I wait one minute to shutdown the bridge?

This looks similar, but for Jicofo. So if you want to shut down a whole Jicofo+Prosody shard, you could use this script to wait until there are no conferences on that shard.

1 Like

I wondered the same. There’s a wonderful feature in Git called “blame” which shows you the commit that last modified a line of code. You can access it in GitHub by clicking the “Blame” button at the top right when viewing a source file. If we do that for JVB’s reference.conf, we find that this configuration option was introduced by this commit with this message:

* fix: Delay shutdown in graceful mode.

When a graceful shutdown is requested while there are no conferences on
the bridge it would shutdown immediately, without updating its presence
with the "graceful-shutdown" flag. This resulted in jicofo wrongly
detecting the bridge as having gone down non-gracefully.

This adds a delay to give time for presence to be updated.
1 Like