Health Check for Jitsi Video Bridge

M_Asif · December 16, 2020, 1:26pm

I have to check health for video bridge so that if it is not working for some reason, I could take appropriate steps ( debugging/ restart instances)

Steps to reproduce:

Enabled the Rest API in config
JVB_OPTS="--apis=rest"
Enabled the statistics in Sip communicator
org.jitsi.videobridge.ENABLE_STATISTICS=true
Then I called the APIs for health check
curl http://localhost:8080/about/health

HTTP ERROR 404

Problem accessing /colibri/stats. Reason:
    Not Found
Powered by Jetty:// 9.4.15.v20190215

But stats are showing up for Colibri Stats
curl http://localhost:8080/colibri/stats

I am new to this platform, guide me how to check health of the Video bridge. Any help would be appreciated.

deepakangadi · December 16, 2020, 1:36pm

nano /etc/jitsi/videobridge/config
- Make sure to configure the jvb options:
  JVB_OPTS="--apis=rest,xmpp"

nano /etc/jitsi/videobridge/sip-communicator.properties

Here we configure colibri statistics:

    org.jitsi.videobridge.ENABLE_STATISTICS=true
    org.jitsi.videobridge.STATISTICS_TRANSPORT=muc,colibri
    org.jitsi.videobridge.rest.private.jetty.host=0.0.0.0

curl http://127.0.0.1:8080/colibri/stats

My reference: How-to to setup Grafana dashboards to monitor Jitsi, my comprehensive tutorial for the beginner

M_Asif · December 16, 2020, 1:38pm

@deepakangadi, I am able to see the stats but cannot see the health check( unhealthy/healthy).

curl http://localhost:8080/about/health

deepakangadi · December 16, 2020, 1:39pm

Please do this based on stress_level.

M_Asif · December 16, 2020, 1:45pm

Getting stress_level -> 2.2E-4

What is the value range for stress level?

deepakangadi · December 16, 2020, 1:52pm

github.com

jitsi/jicofo/blob/5dd66fea472d23ef2a663f0bacf8d27eb88b16a9/src/main/resources/reference.conf#L30


// The assumed maximum packet rate that a bridge can handle.
max-bridge-packet-rate = 50000
// The assumed average packet rate per participant.
average-participant-packet-rate-pps = 500
// The assumed average stress per participant.
average-participant-stress = 0.01
// The assumed time that an endpoint takes to start contributing fully to the load on a bridge. To avoid allocating
// a burst of endpoints to the same bridge, the bridge stress is adjusted by adding the number of new endpoints
// in the last [participant-rampup-time] multiplied by [average-participant-stress].
participant-rampup-interval = 20 seconds
// The stress level above which a bridge is considered overstressed.
stress-threshold = 0.8
// The amount of to wait before retrying using a failed bridge.
failure-reset-threshold = 1 minute
// The bridge selection strategy. The built-in strategies are:
// SingleBridgeSelectionStrategy: Use the least loaded bridge, do not split a conference between bridges (Octo).
// SplitBridgeSelectionStrategy: Use a separate bridge for each participant (for testing).
// RegionBasedBridgeSelectionStrategy: Attempt to put each participant in a bridge in their local region (i.e. use
//    Octo for geo-location).
// IntraRegionBridgeSelectionStrategy: Use additional bridges when a bridge becomes overloaded (i.e. use Octo for
//    load balancing).

gpatel-fr · December 16, 2020, 1:58pm

are you sure of pasting the error message correctly ? something does not look right here.

M_Asif · December 16, 2020, 2:05pm

@gpatel-fr, Here is the full error message:

Error 404 Not Found
HTTP ERROR 404

Problem accessing /about/health. Reason:
    Not Found
Powered by Jetty:// 9.4.15.v20190215

Yes, I think I copied the old error message in the original answer.

M_Asif · December 16, 2020, 2:21pm

@deepakangadi, I don’t think, only stress levels will be enough.
Last time, participants were able to join the meeting but unable to view each other. And CPU utilization were also moderate. So, we restarted the Videobridge instance and it started working. How can we detect such situation and handle it wisely?

M_Asif · December 16, 2020, 2:25pm

Will this work on Separate Video bridge?
Because I want to mark the instance unhealthy if that instance out of Video Bridge pool is not working properly.

gpatel-fr · December 16, 2020, 2:51pm

maybe the jvb.log file has more info ? possibly by forcing up the debug level in logging.properties.

bbaldino · December 16, 2020, 4:52pm

Can you include your (censored) sip-communicator.properties, jvb.conf and jvb.log?

M_Asif · December 16, 2020, 7:27pm

@bbaldino, @gpatel-fr
Here are the attached files that might help.
Sip communicator properties
sip-communicator.txt (1.1 KB)

JVB conf
jvb.txt (207 Bytes)

JVB config
config.txt (837 Bytes)

JVB logs
jvb.log (48.6 KB)

gpatel-fr · December 16, 2020, 8:32pm

you have only one videobridge at the moment right ? if not could you have different configs ? because I don’t see in your config what could explain this problem.

And the jvb.log is useless to me. At INFO log level nothing is logged about the curl access. Maybe @bbaldino will read something in the rest but I can’t.
I’d set the log level to FINE in the logging.properties, but beware of not doing that in prod since it produces huge logs: stop the videobridge immediately after running the curl command and turn the debug level to INFO before restarting jvb. If you post the log at this level here, try to cut it to only show the part concerned by the health access.
I just tried it and it should produce 5 pages at most.

deepakangadi · December 17, 2020, 9:05am

Note that for /about/health the response body is empty.

200 Response status code: Healthy
500 Response status code: Unhealthy

Refer: https://github.com/jitsi/jitsi-videobridge/blob/master/doc/health-checks.md

M_Asif · December 17, 2020, 10:42am

@deepakangadi,
For /about/health , I am getting 404.
Can you guide me how to implement it?

M_Asif · December 18, 2020, 5:02am

I am able to get the stats(/colibri/stats) but not health check(/about/health). Can I know whether the video bridge is healthy or not by colibri stats?
If yes, which properties should I consider?
If no, then how can I get the health check API working?

emrah · December 18, 2020, 6:58am

What are the outputs?

curl http://localhost:8080/stats

netstat -taunp | grep java

M_Asif · December 22, 2020, 11:23am

With the new setup /about/health, it is giving 200 OK.
How to test if /about/health is working for 500(internal server error) condition?

emrah · December 22, 2020, 5:20pm

Check non 200 status

systemctl stop jitsi-videobridge2.service
curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/about/health