[jitsi-dev] Proposed Changes for Supporting Multiple Tracks Per Endpoint


#1

Hi everyone,

Thanks for chatting with me Monday about supporting multiple video tracks
per endpoint/participant in lib-jitsi-meet and jitsi-videobridge. Before
doing any development, I want to get consensus on a couple of changes
(outlined below) to the lib-jitsi-meet api and how the videobridge handles
pinning and selecting. I will be grateful for your feedback.

I have previously implemented most of these features in an old version of
lib-jitsi-meet and the videobridge from last July. That code is working
fairly reliably for my users, but I would like to get the multi-track
features into the official build. I realize that there is a lot of
excellent work that goes on to maintain jitsi, and I think these changes
could significantly benefit other users.

Adding Tracks [lib-jitsi-meet]
JitsiConference.addTrack() will no longer reject adding a local video track
if one is already added.

Accessing Tracks (new calls) [lib-jitsi-meet]
JitsiConference.getLocalVideoTracks();
JitsiConference.getLocalAudioTracks();
JitsiConference.rtc.getLocalVideoTracks();
JitsiConference.rtc.getLocalAudioTracks();

Each of these will return an array of tracks, instead of a single track. I
propose leaving the getLocalVideoTrack and getLocalAudioTrack methods. They
will return the first video/audio track, respectively. This will maintain
backward compatibility.

Selecting and Pinning Tracks [lib-jitsi-meet / jitsi-videobridge]
JitsiConference.selectTrack()
Given that one participant will be able to have multiple video tracks, I
propose supporting selecting tracks instead of participants/endpoints. We
can optionally keep selectParticipant(), for backward compatibility, which
will then just select that participant's first track. Alternatively, it
could select all of that participant's tracks.

JitsiConference.pinTrack()
I propose the same treatment for JitsiConference.pinParticipant(). It will
pin the first or all tracks of the participant. I welcome your thoughts.

Both selectTrack and pinTrack will require changing the client to
videobridge datachannel messages to send track ids instead of participant
ids. The videobridge will then need to be updated to handle selection and
pinning on a per track basis instead of per endpoint. I expect that this
will require updating the Endpoint, VideoChannel, and BitrateController
data structures to handle the new datachannel messages, changing simulcast
layers, and adaptivity computations appropriately. There may need to be
some changes to how the adaptive lastN videobridge implementation works,
but at first glance it seems that it handles multiple video streams per
endpoint already.

This is a first pass on what I expect will need to change. I expect there
might be some needed changes that I am not yet aware of. I will appreciate
any feedback or ideas that people have about supporting this functionality.

Thank you,
Bill Hamilton

Ph.D. Candidate
Interface Ecology Lab | Computer Science and Engineering Department
Texas A&M University
College Station, Texas, USA


#2

Hello Bill,

Thanks for detailed description of the plan ! All the API changes
you've mentioned sound reasonable to me. I know that there will be
many places in lib-jitsi-meet where some adjustments will be
necessary, but this should be doable. I can't say anything about the
JVB part.

Regards,
Pawel

···

On Thu, Jun 8, 2017 at 1:24 PM, Bill Hamilton <bill@ecologylab.net> wrote:

Hi everyone,

Thanks for chatting with me Monday about supporting multiple video tracks
per endpoint/participant in lib-jitsi-meet and jitsi-videobridge. Before
doing any development, I want to get consensus on a couple of changes
(outlined below) to the lib-jitsi-meet api and how the videobridge handles
pinning and selecting. I will be grateful for your feedback.

I have previously implemented most of these features in an old version of
lib-jitsi-meet and the videobridge from last July. That code is working
fairly reliably for my users, but I would like to get the multi-track
features into the official build. I realize that there is a lot of excellent
work that goes on to maintain jitsi, and I think these changes could
significantly benefit other users.

Adding Tracks [lib-jitsi-meet]
JitsiConference.addTrack() will no longer reject adding a local video track
if one is already added.

Accessing Tracks (new calls) [lib-jitsi-meet]
JitsiConference.getLocalVideoTracks();
JitsiConference.getLocalAudioTracks();
JitsiConference.rtc.getLocalVideoTracks();
JitsiConference.rtc.getLocalAudioTracks();

Each of these will return an array of tracks, instead of a single track. I
propose leaving the getLocalVideoTrack and getLocalAudioTrack methods. They
will return the first video/audio track, respectively. This will maintain
backward compatibility.

Selecting and Pinning Tracks [lib-jitsi-meet / jitsi-videobridge]
JitsiConference.selectTrack()
Given that one participant will be able to have multiple video tracks, I
propose supporting selecting tracks instead of participants/endpoints. We
can optionally keep selectParticipant(), for backward compatibility, which
will then just select that participant's first track. Alternatively, it
could select all of that participant's tracks.

JitsiConference.pinTrack()
I propose the same treatment for JitsiConference.pinParticipant(). It will
pin the first or all tracks of the participant. I welcome your thoughts.

Both selectTrack and pinTrack will require changing the client to
videobridge datachannel messages to send track ids instead of participant
ids. The videobridge will then need to be updated to handle selection and
pinning on a per track basis instead of per endpoint. I expect that this
will require updating the Endpoint, VideoChannel, and BitrateController data
structures to handle the new datachannel messages, changing simulcast
layers, and adaptivity computations appropriately. There may need to be some
changes to how the adaptive lastN videobridge implementation works, but at
first glance it seems that it handles multiple video streams per endpoint
already.

This is a first pass on what I expect will need to change. I expect there
might be some needed changes that I am not yet aware of. I will appreciate
any feedback or ideas that people have about supporting this functionality.

Thank you,
Bill Hamilton

Ph.D. Candidate
Interface Ecology Lab | Computer Science and Engineering Department
Texas A&M University
College Station, Texas, USA

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#3

Hi Bill,

Apologies for the late reply, see more inline.

Hi everyone,
Thanks for chatting with me Monday about supporting multiple video tracks per endpoint/participant in lib-jitsi-meet and jitsi-videobridge. Before doing any development, I want to get consensus on a couple of changes (outlined below) to the lib-jitsi-meet api and how the videobridge handles pinning and selecting. I will be grateful for your feedback.
I have previously implemented most of these features in an old version of lib-jitsi-meet and the videobridge from last July. That code is working fairly reliably for my users, but I would like to get the multi-track features into the official build. I realize that there is a lot of excellent work that goes on to maintain jitsi, and I think these changes could significantly benefit other users.

Adding Tracks [lib-jitsi-meet]
JitsiConference.addTrack() will no longer reject adding a local video track if one is already added.
Accessing Tracks (new calls) [lib-jitsi-meet]
JitsiConference.getLocalVideoTracks();
JitsiConference.getLocalAudioTracks();
JitsiConference.rtc.getLocalVideoTracks();
JitsiConference.rtc.getLocalAudioTracks();
Each of these will return an array of tracks, instead of a single track. I propose leaving the getLocalVideoTrack and getLocalAudioTrack methods. They will return the first video/audio track, respectively. This will maintain backward compatibility.
Selecting and Pinning Tracks [lib-jitsi-meet / jitsi-videobridge]
JitsiConference.selectTrack()
Given that one participant will be able to have multiple video tracks, I propose supporting selecting tracks instead of participants/endpoints.

This makes sense. How do you suggest we identify tracks in a way which both lib-jitsi-meet and jitsi-videobridge understand?

We can optionally keep selectParticipant(), for backward compatibility, which will then just select that participant's first track. Alternatively, it could select all of that participant's tracks.
JitsiConference.pinTrack()
I propose the same treatment for JitsiConference.pinParticipant(). It will pin the first or all tracks of the participant. I welcome your thoughts.

Sounds good.

Both selectTrack and pinTrack will require changing the client to videobridge datachannel messages to send track ids instead of participant ids.

Makes sense, and it should be simple to do.

The videobridge will then need to be updated to handle selection and pinning on a per track basis instead of per endpoint. I expect that this will require updating the Endpoint, VideoChannel, and BitrateController data structures to handle the new datachannel messages, changing simulcast layers, and adaptivity computations appropriately. There may need to be some changes to how the adaptive lastN videobridge implementation works, but at first glance it seems that it handles multiple video streams per endpoint already.
This is a first pass on what I expect will need to change. I expect there might be some needed changes that I am not yet aware of. I will appreciate any feedback or ideas that people have about supporting this functionality.

This also makes sense. I can think of only one thing missing from the list, which is adding the track identifiers (unless it happens that they are already present?).

Regards,
Boris

···

On 08/06/2017 13:24, Bill Hamilton wrote: