Virtual backgrounds with Tensorflow BodyPix

So the background blur feature is already using the Tensorflow BodyPix library.

Here you can read a lot more about this BodyPix.

Has anyone tried using image backgrounds instead of the blur?
I’m not a developer myself but I’m thinking of asking my development team to add this option for our users. We’re using Jitsi in Healthcare and it would be great if our users can select from a few professional backgrounds like doctor offices or just a nice living room.

Another question is if anyone tried the ResNet50 architecture instead of the current default MobileNetV1 architecture? It seems that the ResNet50 is newer and more accurate.

virtualBackgrounds++… also video backgrounds!

The challenge will be to make it perform fast enough and not use too much CPU on the client. From my previous tests with the blur it was using a lot of CPU. Hopefully we’ll find ways to optimize this. Think it would be a killer feature.

The blur algorithm is surely more cpu intensive than background replacement. So if CPU is an issue with blur, maybe background replacement is a better way to go.

I second this. Having a static image in the background should require lower CPU usage. Currently the blur is unusable since it makes the video very slow and jerky.

Providing the background replacement will be a big improvement to Jitsi. Please consider making it a higher priority than improving the blur.


So google is currently releasing change your background feature inside Google Meet.

Here is a high level overview of how it works

Basically they use a tuned version of MobileNetV3-small.
They were able to have very good performance through the use of web assembly SIMD, released in chrome 84.

I hope we soon will be able to have such implementation in Jitsi


Here the github of MEDIAPIPE

I dont know if you have to get any google API Key to get it works ?