[jitsi-dev] The Videobridge and the Rest interface


#1

Sadly I have still not yet managed to conference with myself using the Rest interface. I have a number of problems:

a) The Video Stream.

Having looked at the chrome real time logs and the internals in chrome I find that chrome thinks it is sending a lot of video data (which goes up all the time) to the bridge. However, it does not think it is getting anything back from the bridge. The remote stream is attached to the connection and the connection reports its existance. When the stream is added it attaches it to the HTML element and asks it to play and it thinks it is playing (when I try again I get an error because it is being interrupted that gives rise to the 150ms problem for stopping playing and starting again, but I have not fixed that.)

b) The Audio stream

In the first patch I get a source number for the audio stream, in the second patch I get a different number for the audio stream. Chrome logs complain that the second number is "not associated with a receiving track" - a message I don't get for the video stream. Obviously I can and will try repatching the SDP on the peer connection with a different source number for the audio. However, I would like to know what is going on.

c) Multichannels and conferencing.

It appears that to have more than one connection to the bridge in the same conference I have to define an adequate number of empty channels right at the start. Because I have not yet managed to get a video stream I don't really know what is going on although the video stream source number is the same for both sessions with the bridge (Which are on the same machine).

Question 1.

What is likely to be the cause of the video not receving any data?

Question 2.

Why is there a second different source number on the audio patch.

Question 3.

If patching in more than one conference participant how are the patches structured. Do you have to send effectively an empty object for each patch that you are not part of and remember which in the sequence you are dealing with?


#2

Sadly I have still not yet managed to conference with myself using the
Rest interface. I have a number of problems:

a) The Video Stream.

Having looked at the chrome real time logs and the internals in chrome I
find that chrome thinks it is sending a lot of video data (which goes up
all the time) to the bridge. However, it does not think it is getting
anything back from the bridge. The remote stream is attached to the
connection and the connection reports its existance. When the stream is
added it attaches it to the HTML element and asks it to play and it
thinks it is playing (when I try again I get an error because it is
being interrupted that gives rise to the 150ms problem for stopping
playing and starting again, but I have not fixed that.)

b) The Audio stream

In the first patch I get a source number for the audio stream, in the
second patch I get a different number for the audio stream. Chrome logs
complain that the second number is "not associated with a receiving
track" - a message I don't get for the video stream. Obviously I can
and will try repatching the SDP on the peer connection with a different
source number for the audio. However, I would like to know what is going on.

c) Multichannels and conferencing.

It appears that to have more than one connection to the bridge in the
same conference I have to define an adequate number of empty channels
right at the start. Because I have not yet managed to get a video
stream I don't really know what is going on although the video stream
source number is the same for both sessions with the bridge (Which are
on the same machine).

Question 1.

What is likely to be the cause of the video not receving any data?

Hard to tell. Take a look at the network traffic in order to determine whether video packets leave the bridge as expected. Also take a look at the bridge logs for clues.

Question 2.

Why is there a second different source number on the audio patch.

Are you using "translator" or "mixer" mode? With mixer this is what you expect, since the bridge creates a separate mix for everyone.

Question 3.

If patching in more than one conference participant how are the patches
structured. Do you have to send effectively an empty object for each
patch that you are not part of and remember which in the sequence you
are dealing with?

No, you just include the "channel"s or "channelBundle"s that you want to change.

Boris

···

On 20/01/2017 08:28, John Hemming wrote:


#3

Thank you for this.

I am slightly mystified about what the videobridge is doing although it should become clear.

I believe what happens is that all the particpants send an audio and a video stream to the bridge. It mixes all the audio streams and sends the video stream of the speaker with focus plus the last-n speakers out to each of the participants. The separate video streams are sent as separate sources within an audio and video bundle (or do they have a separate peerconnection).

My belief (which I won't be able to check until Monday unless I sort it through looking through the code) is that the bridge is currently not sending a video stream.

Question 1.

Is it possible that the focus has not been set and without setting the focus the bridge won't send out a video stream?

Question 2.

If there are video streams for the current focus plus last n and at the time someone joins there are not a last n number of video streams how canthey find out what they are. Does the client have to ask the bridge periodically if there are new streams to attach somewhere?

Question 3.

I presume there is no "chairman" function as yet which enables someone to specify who the speaker is. Am I right.

···

On 20/01/2017 16:14, Boris Grozev wrote:

On 20/01/2017 08:28, John Hemming wrote:

Sadly I have still not yet managed to conference with myself using the
Rest interface. I have a number of problems:

a) The Video Stream.

Having looked at the chrome real time logs and the internals in chrome I
find that chrome thinks it is sending a lot of video data (which goes up
all the time) to the bridge. However, it does not think it is getting
anything back from the bridge. The remote stream is attached to the
connection and the connection reports its existance. When the stream is
added it attaches it to the HTML element and asks it to play and it
thinks it is playing (when I try again I get an error because it is
being interrupted that gives rise to the 150ms problem for stopping
playing and starting again, but I have not fixed that.)

b) The Audio stream

In the first patch I get a source number for the audio stream, in the
second patch I get a different number for the audio stream. Chrome logs
complain that the second number is "not associated with a receiving
track" - a message I don't get for the video stream. Obviously I can
and will try repatching the SDP on the peer connection with a different
source number for the audio. However, I would like to know what is going on.

c) Multichannels and conferencing.

It appears that to have more than one connection to the bridge in the
same conference I have to define an adequate number of empty channels
right at the start. Because I have not yet managed to get a video
stream I don't really know what is going on although the video stream
source number is the same for both sessions with the bridge (Which are
on the same machine).

Question 1.

What is likely to be the cause of the video not receving any data?

Hard to tell. Take a look at the network traffic in order to determine whether video packets leave the bridge as expected. Also take a look at the bridge logs for clues.

Question 2.

Why is there a second different source number on the audio patch.

Are you using "translator" or "mixer" mode? With mixer this is what you expect, since the bridge creates a separate mix for everyone.

Question 3.

If patching in more than one conference participant how are the patches
structured. Do you have to send effectively an empty object for each
patch that you are not part of and remember which in the sequence you
are dealing with?

No, you just include the "channel"s or "channelBundle"s that you want to change.

Boris

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#4

Hi,

I'm having trouble keeping up with the emails, so this may be outdated.

Thank you for this.

I am slightly mystified about what the videobridge is doing although it
should become clear.

I believe what happens is that all the particpants send an audio and a
video stream to the bridge. It mixes all the audio streams and sends
the video stream of the speaker with focus plus the last-n speakers out
to each of the participants. The separate video streams are sent as
separate sources within an audio and video bundle (or do they have a
separate peerconnection).

My belief (which I won't be able to check until Monday unless I sort it
through looking through the code) is that the bridge is currently not
sending a video stream.

Question 1.

Is it possible that the focus has not been set and without setting the
focus the bridge won't send out a video stream?

No, the focus field is optional.

Question 2.

If there are video streams for the current focus plus last n and at the
time someone joins there are not a last n number of video streams how
canthey find out what they are. Does the client have to ask the bridge
periodically if there are new streams to attach somewhere?

The receiving client must all be notified of remote streams via signalling (e.g. if a new participant joins the conference, any receivers must do a round of setLocalDescription/setRemoteDescription to add the new source).

The bridge will notify receivers when the set of last-n endpoints changes, so they can modify the user interface accordingly, but the streams to attach video elements ultimately come from the browser based on signalling.

Question 3.

I presume there is no "chairman" function as yet which enables someone
to specify who the speaker is. Am I right.

The bridge doesn't have such a feature, but it allows clients to implement it. Jitsi-meet has (had?) a mode in which the moderator (first to join) would manually select the "on-stage" participant, and everyone else would follow.

Boris

···

On 21/01/2017 02:23, John Hemming wrote:


#5

It wasn't outdated and I thank you for it.

What I am trying to get to the bottom of is why I am not getting a video return from the bridge in the video element. I think I know why now.

The first question is what happens when people join the conference. Obviously when there is only one conference member the bridge does not send the video back to that member as it is the video from that member.

When another one joins then the bridge will send the video from each conference member to the other one. What seems to be the case is that it uses the same SSRC to send back the video stream that is used to send the video stream to the Bridge. This would be a sensible solution, but would explain why it seemed to be using an SSRC for a remote stream that I thought was one of the local streams.

Question 1.

Is the Bridge sending back the video stream with the same SSRC as it is sent to the bridge.

Question 1(a)

Is anything ever sent as video using the initial remote (from the client's perpective) SSRC?

I have got partially into the JSON datastream that is created using the label "default" by the bridge once the SCPT connection is open.

Question 2.

What is sent via this datastream?

Obviously the bridge can send out notifications to all endpoints that someone has joined or left the conference.

I would think that this is where I should look for the new stream sources. I assume they are multiplexed into the same channel bundle. Hence we have a channel bundle with a DTLS/SRTP notification. That has on the same port and IP address multiplexed not only RTP and RTCP, but also a number of video channels and potentially a number of audio channels as well as the datachannel (and in fact potentially more than one data channel).

Question 3.

Am I right.

There is some difficulty negotiating a second client/server link as far as DTLS is concerned from the same client. It works unreliably. I am not really worried about that as it is not necessarily a real world problem.

···

On 22/01/2017 22:10, Boris Grozev wrote:

Hi,

I'm having trouble keeping up with the emails, so this may be outdated.

On 21/01/2017 02:23, John Hemming wrote:

Thank you for this.

I am slightly mystified about what the videobridge is doing although it
should become clear.

I believe what happens is that all the particpants send an audio and a
video stream to the bridge. It mixes all the audio streams and sends
the video stream of the speaker with focus plus the last-n speakers out
to each of the participants. The separate video streams are sent as
separate sources within an audio and video bundle (or do they have a
separate peerconnection).

My belief (which I won't be able to check until Monday unless I sort it
through looking through the code) is that the bridge is currently not
sending a video stream.

Question 1.

Is it possible that the focus has not been set and without setting the
focus the bridge won't send out a video stream?

No, the focus field is optional.

Question 2.

If there are video streams for the current focus plus last n and at the
time someone joins there are not a last n number of video streams how
canthey find out what they are. Does the client have to ask the bridge
periodically if there are new streams to attach somewhere?

The receiving client must all be notified of remote streams via signalling (e.g. if a new participant joins the conference, any receivers must do a round of setLocalDescription/setRemoteDescription to add the new source).

The bridge will notify receivers when the set of last-n endpoints changes, so they can modify the user interface accordingly, but the streams to attach video elements ultimately come from the browser based on signalling.

Question 3.

I presume there is no "chairman" function as yet which enables someone
to specify who the speaker is. Am I right.

The bridge doesn't have such a feature, but it allows clients to implement it. Jitsi-meet has (had?) a mode in which the moderator (first to join) would manually select the "on-stage" participant, and everyone else would follow.

Boris

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#6

It wasn't outdated and I thank you for it.

What I am trying to get to the bottom of is why I am not getting a video
return from the bridge in the video element. I think I know why now.

The first question is what happens when people join the conference.
Obviously when there is only one conference member the bridge does not
send the video back to that member as it is the video from that member.

When another one joins then the bridge will send the video from each
conference member to the other one. What seems to be the case is that
it uses the same SSRC to send back the video stream that is used to send
the video stream to the Bridge. This would be a sensible solution, but
would explain why it seemed to be using an SSRC for a remote stream that
I thought was one of the local streams.

Question 1.

Is the Bridge sending back the video stream with the same SSRC as it is
sent to the bridge.

Yes.

With a little caveat for simulcast. Ignore this if you don't care about simulcast. In this case the bridge receives multiple streams, say SSRCs A, B and C and uses only SSRC A (the first from the signaled "SIM" group) to receivers. If it forwards the payload of stream B, then it rewrites the packets' SSRC from B to A.

Question 1(a)

Is anything ever sent as video using the initial remote (from the
client's perpective) SSRC?

The bridge generates an SSRC of its own, but doesn't send any video with it. It only uses it for RTCP, e.g. to request that a sender generate a keyframe. This SSRC should be added to the remote description of receivers, so they honor the keyframe requests for example, but it should not be rendered.

I have got partially into the JSON datastream that is created using the
label "default" by the bridge once the SCPT connection is open.

Note that SCTP is entirely optional, and you might want to start without it.

Question 2.

What is sent via this datastream?

Currently this is used for:
1. Endpoint to endpoint messages
2. Dominant speaker change notifications: the bridge tells clients when it detects a new dominant speaker in a conference.
3. Last-n notifications: the bridge notifies clients of the subset of streams that is being forwarded.
4. Simulcast: clients "select" and "pin" endpoints to control which endpoints should be received and at what quality.

Obviously the bridge can send out notifications to all endpoints that
someone has joined or left the conference.

It doesn't, all this happens through signaling.

I would think that this is where I should look for the new stream
sources. I assume they are multiplexed into the same channel bundle.
Hence we have a channel bundle with a DTLS/SRTP notification. That has
on the same port and IP address multiplexed not only RTP and RTCP, but
also a number of video channels and potentially a number of audio
channels as well as the datachannel (and in fact potentially more than
one data channel).

Question 3.

Am I right.

Yes.

Regards,
Boris

···

On 23/01/2017 05:47, John Hemming wrote:


#7

In terms of the problems negotiating the connection (after the first one) I have encountered this also when trying to connect to the bridge from a computer which is not the same as the bridge (but on the same Class C network)

Question

Is there a known issue about having to make a number of attempts to connect to the bridge?

It now appears to be a real world problem so I am going to give it some attention.

···

On 23/01/2017 11:47, John Hemming wrote:

It wasn't outdated and I thank you for it.

What I am trying to get to the bottom of is why I am not getting a video return from the bridge in the video element. I think I know why now.

The first question is what happens when people join the conference. Obviously when there is only one conference member the bridge does not send the video back to that member as it is the video from that member.

When another one joins then the bridge will send the video from each conference member to the other one. What seems to be the case is that it uses the same SSRC to send back the video stream that is used to send the video stream to the Bridge. This would be a sensible solution, but would explain why it seemed to be using an SSRC for a remote stream that I thought was one of the local streams.

Question 1.

Is the Bridge sending back the video stream with the same SSRC as it is sent to the bridge.

Question 1(a)

Is anything ever sent as video using the initial remote (from the client's perpective) SSRC?

I have got partially into the JSON datastream that is created using the label "default" by the bridge once the SCPT connection is open.

Question 2.

What is sent via this datastream?

Obviously the bridge can send out notifications to all endpoints that someone has joined or left the conference.

I would think that this is where I should look for the new stream sources. I assume they are multiplexed into the same channel bundle. Hence we have a channel bundle with a DTLS/SRTP notification. That has on the same port and IP address multiplexed not only RTP and RTCP, but also a number of video channels and potentially a number of audio channels as well as the datachannel (and in fact potentially more than one data channel).

Question 3.

Am I right.

There is some difficulty negotiating a second client/server link as far as DTLS is concerned from the same client. It works unreliably. I am not really worried about that as it is not necessarily a real world problem.

On 22/01/2017 22:10, Boris Grozev wrote:

Hi,

I'm having trouble keeping up with the emails, so this may be outdated.

On 21/01/2017 02:23, John Hemming wrote:

Thank you for this.

I am slightly mystified about what the videobridge is doing although it
should become clear.

I believe what happens is that all the particpants send an audio and a
video stream to the bridge. It mixes all the audio streams and sends
the video stream of the speaker with focus plus the last-n speakers out
to each of the participants. The separate video streams are sent as
separate sources within an audio and video bundle (or do they have a
separate peerconnection).

My belief (which I won't be able to check until Monday unless I sort it
through looking through the code) is that the bridge is currently not
sending a video stream.

Question 1.

Is it possible that the focus has not been set and without setting the
focus the bridge won't send out a video stream?

No, the focus field is optional.

Question 2.

If there are video streams for the current focus plus last n and at the
time someone joins there are not a last n number of video streams how
canthey find out what they are. Does the client have to ask the bridge
periodically if there are new streams to attach somewhere?

The receiving client must all be notified of remote streams via signalling (e.g. if a new participant joins the conference, any receivers must do a round of setLocalDescription/setRemoteDescription to add the new source).

The bridge will notify receivers when the set of last-n endpoints changes, so they can modify the user interface accordingly, but the streams to attach video elements ultimately come from the browser based on signalling.

Question 3.

I presume there is no "chairman" function as yet which enables someone
to specify who the speaker is. Am I right.

The bridge doesn't have such a feature, but it allows clients to implement it. Jitsi-meet has (had?) a mode in which the moderator (first to join) would manually select the "on-stage" participant, and everyone else would follow.

Boris

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev