Thanks for chatting with me Monday about supporting multiple video tracks
per endpoint/participant in lib-jitsi-meet and jitsi-videobridge. Before
doing any development, I want to get consensus on a couple of changes
(outlined below) to the lib-jitsi-meet api and how the videobridge handles
pinning and selecting. I will be grateful for your feedback.
I have previously implemented most of these features in an old version of
lib-jitsi-meet and the videobridge from last July. That code is working
fairly reliably for my users, but I would like to get the multi-track
features into the official build. I realize that there is a lot of
excellent work that goes on to maintain jitsi, and I think these changes
could significantly benefit other users.
Adding Tracks [lib-jitsi-meet]
JitsiConference.addTrack() will no longer reject adding a local video track
if one is already added.
Accessing Tracks (new calls) [lib-jitsi-meet]
Each of these will return an array of tracks, instead of a single track. I
propose leaving the getLocalVideoTrack and getLocalAudioTrack methods. They
will return the first video/audio track, respectively. This will maintain
Selecting and Pinning Tracks [lib-jitsi-meet / jitsi-videobridge]
Given that one participant will be able to have multiple video tracks, I
propose supporting selecting tracks instead of participants/endpoints. We
can optionally keep selectParticipant(), for backward compatibility, which
will then just select that participant's first track. Alternatively, it
could select all of that participant's tracks.
I propose the same treatment for JitsiConference.pinParticipant(). It will
pin the first or all tracks of the participant. I welcome your thoughts.
Both selectTrack and pinTrack will require changing the client to
videobridge datachannel messages to send track ids instead of participant
ids. The videobridge will then need to be updated to handle selection and
pinning on a per track basis instead of per endpoint. I expect that this
will require updating the Endpoint, VideoChannel, and BitrateController
data structures to handle the new datachannel messages, changing simulcast
layers, and adaptivity computations appropriately. There may need to be
some changes to how the adaptive lastN videobridge implementation works,
but at first glance it seems that it handles multiple video streams per
This is a first pass on what I expect will need to change. I expect there
might be some needed changes that I am not yet aware of. I will appreciate
any feedback or ideas that people have about supporting this functionality.
Interface Ecology Lab | Computer Science and Engineering Department
Texas A&M University
College Station, Texas, USA