[Solved]Scaling Jibri

Hi,

I have a docker setup and have successfully integrated jibri in my setup. I can record videos although only one at a time. So, I tried scaling up my jibri containers and ended up having 5 of them running simultaneously, which kills my free tier ec2 server. So, that got me wondering about my use case of having 100s of jibris running together to record calls simultaneously. Is it viable. If so, i found that jibri consumes a lot of resources. It spawns a chrome and ffmpeg together. I don’t know how my 1 GB system handled one of them, but even if i can get an instance with 8 GB ram, what’s the approximate maximum number of jibris i can spawn without killing my system? If jibri seems impractical for my use case, is there any other way i can save my recordings, i was thinking i can stream them to youtube and save there.

This uses the same jibri you have for recording.

I would say for running chrome you need at least 2GB, and running the rest you also need RAM, so I would say 4 as a minimum.

You can always create auto-scaling rules and spin up jibri instances on-demand and scale them down when they are not used, not sure how this is done with containers though … but this is what we do with VM jibri instances in aws.

4 GB per jibri instance seems a lot. I can auto scale with containers as well, but this process seems much costly than i figured it would be. Isn’t there a way where we can use the same chrome and ffmpeg to record multiple sessions? Also, does autoscaling mean that for every call i will have a jibri instance, and with a minimum 4GB required per instance, for 100 calls i will end up having 100 instances :no_mouth:

Nope.

Yes.

Ok, thanks for the help @damencho

@damencho can u elaborate more on this?

we are using jibri on aws gpu accelerated instances (running multiple jibris per one ec2 instance) with somehow manual scaling… how do you get the number of idle/busy jibri instances? do you count idle jibris in jibri brewery room and based on that number you spin more VMs? if possible, can you share some of the code that is behind this?

we basically need to know whether there is some meeting running and how many jibris are currently occupied. i’ll be glad if u can point me somewhere…

You can do a local http query for the status of jibri and report it to cloudwatch and base your autoscale logic based on the number of used/free jibris.

Thanks a lot, your tip to use api request to jibri http api simplifies whole thing a lot…

I guess than, that basically you have some cloudwatch agent present on each jibri instance (i suppose one jibri instance per ec2 instance) and you collect results from jibri api call via this agent to cloudwatch and then you have defined autoscaling(?) group in aws, that you scale up/down based on count of used/free jibris through cloudwatch thresholds or something like that. Am i guessing right?

Correct.

@nosmo I saw a post earlier about jibri using a lot of resources and recommended using gpu, but didn’t know how to go about implementing it, would you please elaborate more on that :slight_smile: I know it’s done in the Jellyfin docker containers but don’t know enough to port it over to the jibri containers.

we found out, that most of the cpu is eaten by basically two things:

  1. chrome rendering the conference session in x11 session
  2. encoding in ffmpeg
    minor part of cpu is eaten by x11grab part and the rest.

gpu offloading can help with both: x11 and chrome accelerated on gpu and definitely offloading whole encoding to gpu (nvenc). we decided to do 1920x1080 recordings instead of standard 1280x720 recordings, but that hit another performance threshold - even if fully offloaded encoding to gpu (basically ffmpeg is showing only mild usage for x11grab) the problem is now chrome rendering which is eating so much cpu, that is cancelling all the gains from offloading work to gpu. we ended with higher resolution recordings with almost no performance gain.
we are doing it on kubernetes so we can have multiple instances of jibri sharing single gpu at once on aws g3 instances (tesla m60)
i can write something like use-case of how we did it if you are still interested. @damencho in that case, should i write that here in forum as new article?

1 Like

Yep, that will be helpful for the community. We can tag it or make new category for howto topics.

Thanks

Thank you @nosmo. @dimm0 might have some ideas. He wrote up a k8s example for the rest of the jitsi stack, and I know was working to add in Jibri.

The github page

And his Rocket.Chat forum to collaborate on
https://rocket.nautilus.optiputer.net/