How to stream a HTML canvas content using video stream?


My goal is to stream a canvas content using the Jitsi video stream.
I am using JitsiMeetExternalAPI to setup the video session.

An option would be to create a new command and call the conference.replaceTrack() but I cannot find a way to expose the instance of API.js to my code, so I can take the content of const stream = canvas.captureStream(); and assign to the existing video stream

Ways I tried to access the APP

  • window.APP - not available before the video session, not available after the video session starts
  • via _transport - transport has no idea about the subscribers, so not a good idea, but revealed a way to get inner elements from JitsiMeetExternalAPI, using JitsiMeetExternalAPI.prototype.getInternalAPI = function () { return this._transport }
  • via - only has setExternalTransportBackend method, is an instance of JitsiMeetExternalAPI, nothing to see here…

Any ideas?


Related topics

Another related topic Using external API and lib-meet.js for video and object read

Considering this option to create a custom application, Customizing lib-jitsi-meet for SDK - #2 by damencho but it looks like TOO much when I need only to add a custom command to APP

It’s not possible to send a stream through the postMessage API. This would need to be baked into jitsi for it to work.

1 Like

What about exposing the instance of API.js to MeetExternalAPI? I know it would violate the current design, which is oriented by commands, but that would allow lots of custom behaviors to be built on top of MeetExternalAPI.

You’d still not be able to do that.

Please elaborate, as far as I can see in the documentation if I have an instance of JitsiConference, I can manipulate the stream or even replace it with another stream, isn’t how background blur works?

You cannot get ahold of a JitsiConference instance from outside of the iframe.

1 Like

That is true, no access to the JitsiConference from outside, what about add a command “registerVideoStreamFilter” where a pipeline to process the stream before is sent is defined, to be used with Window.postMessage() - Web APIs | MDN

  • input event name - the event where the parent will receive the content of the stream
  • output event name - the event where the Iframe will receive an updated version of the stream to finally send it

That would enable other extensions of the video, all sort of effects or updates to the video stream without recreating the existing video UI, that is very cool btw :smiley:

I’m not sure that would be enough since a pipeline would likely be a function, which cannot be serialized over postMessage. There is some movement in the screen-sharing area at the w3c. Maybe some new APIs will help with this.

I was able to capture the canvas from other Iframe using windows events, will try to send the data thru the text stream instead of the video as it is available.

For reference the capture canvas from other Iframe is here: GitHub - hamilton-lima/poc-iframe-capture-screen