Scaling Jibri on K8s using HPA

Install & Config Record & stream - Jibri
#1

Hello All, Is anyone able to scale jibri using HPA based on the metrics offered by jibri itself? Metrics like BUSY and IDLE status. So I want to scale up if the status is BUSY and scale down when it’s IDLE.

@kpeiruza @Arthur_Morales How are you scaling jibri on k8s?

#2

Scrape jicofo metrics into prometheus, there you can get the total number of jibris and how many are spare.

Then use HPA against custom metrics and you have it :slight_smile:

#3

Thanks @kpeiruza for responding.

But what’s the purpose of HPA if I’m already running more than a couple of jibris. It would be a great help if you can explain it a bit more.

My Current scenario is as below:

  1. I’m running a sidecar container with jibri which scrapes jibri stats on the /metrics URL to Prometheus.
  2. I don’t have jicofo metrics exporter as of now to scrape Jicofo metrics to the Prometheus but it isn’t an issue. I’ll create one.
  3. I’m facing some issues with hpa. Below is the config I’m using

jibri stats offered by /metrics URL

# HELP jibri_busystatus It checks the status of the jibri whether BUSY, IDLE.
# TYPE jibri_busystatus gauge
jibri_busystatus 0     <-- for IDLE it is zero and BUSY it is 1
# HELP jibri_healthstatus It checks the health status of the jibri whether HEALTHY or not.
# TYPE jibri_healthstatus gauge
jibri_healthstatus 1  <--- for HEALTHY it is 1 and UNHEALTHY it is zero

Prometheus adapter config

rules:
    - seriesQuery: 'jibri_busystatus{namespace="jitsi"}'
      resources:
        overrides:
          namespace: {resource: "namespace"}
          pod: {resource: "pod"}
      name:
        matches: "jibri_busystatus"
        as: "jibri_busystatus"
      metricsQuery: 'jibri_busystatus'

jibri-hpa

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: jibri-hpa
  namespace : jitsi
spec:
  maxReplicas: 4
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: jibri
  metrics:
    - type: Pods
      pods:
        metric:
          name: jibri_busystatus
        target:
          type: AverageValue  <--- I have doubt here. I think I'm not doing it correctly
          averageValue: 1

Metric Value List

command--> kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta2/namespaces/jitsi/pod/*/jibri_busystatus" | jq .
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta2",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta2/namespaces/jitsi/pod/%2A/jibri_busystatus"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "jitsi",
        "name": "jibri-0",
        "apiVersion": "/v1"
      },
      "metric": {
        "name": "jibri_busystatus",
        "selector": null
      },
      "timestamp": "2021-04-18T18:27:01Z",
      "value": "0"
    }
  ]
}

please help