Adding Computer Vision Models in Jitsi Meet

Hello! Before I start I just wanted to say I am a great fan of this project as well as the community, very supportive to say the least.

Now on to the problem, is it possible to embed computer vision models within Jitsi? My goal is to create a video chat that can recognize humans, their emotions and their eye movements, etc. It is for a school project so rest assured I just want to test if that is possible or not. Something like this in the real world would be quite dystopian I have to say.

Again, thanks for all the support in advance.

Hi @AjeelAhmed1998,

it’s definitely possible, but I think it’s too hard for a (high-)school project. That’s more an university or collage project. Probably it efforts you 100 hours to get a to work, if you’re not a total beginner. Therefore I recommend you to ask your teacher, if this is too ambitious :wink:

After that you have two option:

  1. Clone the Repository and modify the code. You can add AI libraries to Jitsi’s video bridge.
  2. I’m not sure if a lib exists for this case, but you could use the iFrame API of Jitsi and analyze it in the frontend.
1 Like

Thank for the quick reply! I actually am in university haha. I just wanted to know where should one start from to modify the codebase and if there are any other alternatives that could be much easier to work with besides Jitsi.

So I have to go look at Videobridge for starters is that right?

For reference I have worked with Jitsi before to some extent, I have set it up with docker but I have never looked inside the source code or really studied its architectural design.

Nope, the bridge manages only encoded data and don’t hove access to decoded streams, but this is what you want.
You can look at the blur effect, it already does part of what you want - recognizes human and blurs the background. You need to look in jitsi-meet and lib-jitsi-meet.

1 Like

Alright! Thanks a lot Damian, I always appreciate your support!