Architectural question regarding processing video frames

I’m working on a project to put filters/avatars on individual meeting attendee face. I’m in the process of understanding the architecture. please can you point me to modules, i need to look into to tap the data/video for processing from individual attendee or any community links would be useful.

A birds eye view:
Camera -> “A” Camera driver -> “B” on the sending client Web-browser using webrtc and javascript -> “C” on the server inside jitsi VideoBridge -> “D” at the remote viewer

A: Some people add filters to their jitsi meet video today by using a camera driver that can process the image.

B: at the sending client web-browser jitsi’s javascript react webinterface can use some webrtc extensions to access the webrtc camera videoframes, this is today used to encode jitsi’s end to end enctyption. https://jitsi.org/blog/e2ee/

C: you can tap into the video stream on the Jitsi Video Bridge, JVB, as the server have access to all attendants video streams.

D: at the receiving clients web-browser jitsi’s javascript react webinterface can use some webrtc extensions to access the webrtc camera videoframes, this is used to decode jitsi’s end to end video encryption.

1 Like

So if you want to extend jitsi meet web interface at point “B” to add filters then focus on what you can do with webrtc:

http://www.agilityfeat.com/blog/2015/08/did-you-know-using-webgl-shaders-on-webrtc-video

1 Like