[jitsi-dev] Takes nearly a minute for frames to start coming in


#1

I'm still trying to get to the bottom of the issues I am seeing with video
frame decoding from the JVB using the native C++ webrtc library.

Using WebRTC remotes/branch-heads/57:
0a71dc836641acdaaa63b2724f83c822dbb58ce7

My bridge is set to use ulpfec and red, and has rtx disabled.

Everything works fine on a mobile version that performs nearly the same
step, but isn't written in C++.

VideoReceiveStream: {decoders: [{decoder: (VideoDecoder), payload_type:
100, payload_name: VP8, codec_params: {}}], rtp: {remote_ssrc: 4006849248,
local_ssrc: 1, rtcp_mode: RtcpMode::kCompound, rtcp_xr:
{receiver_reference_time_report: off}, remb: on, transport_cc: off, nack:
{rtp_history_ms: 1000}, ulpfec: {ulpfec_payload_type: 117,
red_payload_type: 116, red_rtx_payload_type: -1}, rtx: {}, extensions:
[{uri: http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time, id:
3}]}, renderer: (renderer), render_delay_ms: 10, sync_group: mixedmslabel,
pre_decode_callback: nullptr, pre_render_callback: nullptr,
target_delay_ms: 0}

channel.cc:904): Channel writable (video) for the first time
(channel.cc:966): Installing keys from DTLS-SRTP on video RTP
(webrtcvideoengine2.cc:1450): OnReadyToSend: Ready.
(webrtcvideoengine2.cc:994): SetSend: false
(channel.cc:1970): Changing video state, send=0
(rtp_receiver_video.cc:74): Received first video RTP packet

1 Minute later:

(video_receiver.cc:285): Received first complete decodable video frame
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a

After this I push the frames to ffmpeg, and they encode and look
perfect.... This long delay between receiving the first RTP packet and
being able to decode a full frame seems like the issue.

Again, any ideas or help would be greatly appreciated.

I would be willing to open source a large part of this native recorder
if/when I am able to finish it.

···

--
- Jason Thomas
   http://jasonthom.as


#2

I've seen this when a complete frame arrived, but the decoder couldn't
start decoding yet because it hadn't received an IDR. Maybe you can add
some logs to detect when/if a keyframe has been received?

···

On Tue, Mar 7, 2017 at 2:04 PM, Jason Thomas <mail@jasonthom.as> wrote:

I'm still trying to get to the bottom of the issues I am seeing with video
frame decoding from the JVB using the native C++ webrtc library.

Using WebRTC remotes/branch-heads/57:
0a71dc836641acdaaa63b2724f83c822dbb58ce7

My bridge is set to use ulpfec and red, and has rtx disabled.

Everything works fine on a mobile version that performs nearly the same
step, but isn't written in C++.

VideoReceiveStream: {decoders: [{decoder: (VideoDecoder), payload_type:
100, payload_name: VP8, codec_params: {}}], rtp: {remote_ssrc: 4006849248,
local_ssrc: 1, rtcp_mode: RtcpMode::kCompound, rtcp_xr:
{receiver_reference_time_report: off}, remb: on, transport_cc: off, nack:
{rtp_history_ms: 1000}, ulpfec: {ulpfec_payload_type: 117,
red_payload_type: 116, red_rtx_payload_type: -1}, rtx: {}, extensions:
[{uri: http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time, id:
3}]}, renderer: (renderer), render_delay_ms: 10, sync_group: mixedmslabel,
pre_decode_callback: nullptr, pre_render_callback: nullptr,
target_delay_ms: 0}

channel.cc:904): Channel writable (video) for the first time
(channel.cc:966): Installing keys from DTLS-SRTP on video RTP
(webrtcvideoengine2.cc:1450): OnReadyToSend: Ready.
(webrtcvideoengine2.cc:994): SetSend: false
(channel.cc:1970): Changing video state, send=0
(rtp_receiver_video.cc:74): Received first video RTP packet

1 Minute later:

(video_receiver.cc:285): Received first complete decodable video frame
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a

After this I push the frames to ffmpeg, and they encode and look
perfect.... This long delay between receiving the first RTP packet and
being able to decode a full frame seems like the issue.

Again, any ideas or help would be greatly appreciated.

I would be willing to open source a large part of this native recorder
if/when I am able to finish it.

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#3

Hi Brian,

   I thought perhaps that was the issue too.

looking in webrtc/modules/video_coding/video_receiver.cc the culprit code is located on line 257:

if( drop_frames_until_keyframe_ ) …

Unfortunately it’s happening even before this, in _receiver.FrameForDecoding:
webrtc/modules/video_coding/receiver.cc <http://receiver.cc/>:
140: VCMEncodedFrame *found_frame = jitter_buffer_.NextCompleteFrame(max_wait_time_ms);

Like it isn’t able to understand the RTP format of the incoming video packets… Or maybe it’s something to do with encryption.

I’m going to check over the SDP again, it’s possible my parser did something wrong still and maybe it’s causing the media to be clobbered somehow?

- Jason.

···

On Mar 7, 2017, at 3:15 PM, Brian Baldino <brian@jitsi.org> wrote:

I've seen this when a complete frame arrived, but the decoder couldn't start decoding yet because it hadn't received an IDR. Maybe you can add some logs to detect when/if a keyframe has been received?

On Tue, Mar 7, 2017 at 2:04 PM, Jason Thomas <mail@jasonthom.as <mailto:mail@jasonthom.as>> wrote:
I'm still trying to get to the bottom of the issues I am seeing with video frame decoding from the JVB using the native C++ webrtc library.

Using WebRTC remotes/branch-heads/57:
0a71dc836641acdaaa63b2724f83c822dbb58ce7

My bridge is set to use ulpfec and red, and has rtx disabled.

Everything works fine on a mobile version that performs nearly the same step, but isn't written in C++.

VideoReceiveStream: {decoders: [{decoder: (VideoDecoder), payload_type: 100, payload_name: VP8, codec_params: {}}], rtp: {remote_ssrc: 4006849248, local_ssrc: 1, rtcp_mode: RtcpMode::kCompound, rtcp_xr: {receiver_reference_time_report: off}, remb: on, transport_cc: off, nack: {rtp_history_ms: 1000}, ulpfec: {ulpfec_payload_type: 117, red_payload_type: 116, red_rtx_payload_type: -1}, rtx: {}, extensions: [{uri: http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time, id: 3}]}, renderer: (renderer), render_delay_ms: 10, sync_group: mixedmslabel, pre_decode_callback: nullptr, pre_render_callback: nullptr, target_delay_ms: 0}

channel.cc:904): Channel writable (video) for the first time
(channel.cc:966): Installing keys from DTLS-SRTP on video RTP
(webrtcvideoengine2.cc:1450): OnReadyToSend: Ready.
(webrtcvideoengine2.cc:994): SetSend: false
(channel.cc:1970): Changing video state, send=0
(rtp_receiver_video.cc:74): Received first video RTP packet

1 Minute later:

(video_receiver.cc:285): Received first complete decodable video frame
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a

After this I push the frames to ffmpeg, and they encode and look perfect.... This long delay between receiving the first RTP packet and being able to decode a full frame seems like the issue.

Again, any ideas or help would be greatly appreciated.

I would be willing to open source a large part of this native recorder if/when I am able to finish it.

--
- Jason Thomas
   http://jasonthom.as <http://jasonthom.as/>

_______________________________________________
dev mailing list
dev@jitsi.org <mailto:dev@jitsi.org>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#4

Hey Jason,
Am I understanding you correctly that you're getting blocked at line 140 in
receiver.cc for 1 minute? I.e., 'NextCompleteFrame' isn't returning
anything valid for a minute?

If so, I wonder...do you see anything about dtls negotiation completing?
Maybe that is taking a long time for some reason and packets can't be
decrypted correctly for a while? Otherwise it seems odd that packets would
all of a sudden become understandable.

Since you're in the native code already, you might add some prints lower
down to see if things are flowing as expected...for example, is
InsertPacket being called on the jitter buffer? If not, must be something
lower than that, etc..

-brian

···

On Wed, Mar 8, 2017 at 9:27 AM, Jason Thomas <mail@jasonthom.as> wrote:

Hi Brian,

   I thought perhaps that was the issue too.

looking in webrtc/modules/video_coding/video_receiver.cc the culprit code
is located on line 257:

if( drop_frames_until_keyframe_ ) …

Unfortunately it’s happening even before this, in
_receiver.FrameForDecoding:
webrtc/modules/video_coding/receiver.cc:
140: VCMEncodedFrame *found_frame = jitter_buffer_.
NextCompleteFrame(max_wait_time_ms);

Like it isn’t able to understand the RTP format of the incoming video
packets… Or maybe it’s something to do with encryption.

I’m going to check over the SDP again, it’s possible my parser did
something wrong still and maybe it’s causing the media to be clobbered
somehow?

- Jason.

On Mar 7, 2017, at 3:15 PM, Brian Baldino <brian@jitsi.org> wrote:

I've seen this when a complete frame arrived, but the decoder couldn't
start decoding yet because it hadn't received an IDR. Maybe you can add
some logs to detect when/if a keyframe has been received?

On Tue, Mar 7, 2017 at 2:04 PM, Jason Thomas <mail@jasonthom.as> wrote:

I'm still trying to get to the bottom of the issues I am seeing with
video frame decoding from the JVB using the native C++ webrtc library.

Using WebRTC remotes/branch-heads/57:
0a71dc836641acdaaa63b2724f83c822dbb58ce7

My bridge is set to use ulpfec and red, and has rtx disabled.

Everything works fine on a mobile version that performs nearly the same
step, but isn't written in C++.

VideoReceiveStream: {decoders: [{decoder: (VideoDecoder), payload_type:
100, payload_name: VP8, codec_params: {}}], rtp: {remote_ssrc: 4006849248,
local_ssrc: 1, rtcp_mode: RtcpMode::kCompound, rtcp_xr:
{receiver_reference_time_report: off}, remb: on, transport_cc: off,
nack: {rtp_history_ms: 1000}, ulpfec: {ulpfec_payload_type: 117,
red_payload_type: 116, red_rtx_payload_type: -1}, rtx: {}, extensions:
[{uri: http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time, id:
3}]}, renderer: (renderer), render_delay_ms: 10, sync_group: mixedmslabel,
pre_decode_callback: nullptr, pre_render_callback: nullptr,
target_delay_ms: 0}

channel.cc:904): Channel writable (video) for the first time
(channel.cc:966): Installing keys from DTLS-SRTP on video RTP
(webrtcvideoengine2.cc:1450): OnReadyToSend: Ready.
(webrtcvideoengine2.cc:994): SetSend: false
(channel.cc:1970): Changing video state, send=0
(rtp_receiver_video.cc:74): Received first video RTP packet

1 Minute later:

(video_receiver.cc:285): Received first complete decodable video frame
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a

After this I push the frames to ffmpeg, and they encode and look
perfect.... This long delay between receiving the first RTP packet and
being able to decode a full frame seems like the issue.

Again, any ideas or help would be greatly appreciated.

I would be willing to open source a large part of this native recorder
if/when I am able to finish it.

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#5

Hi Brian,

   It seems like it's falling into:
src/webrtc/modules/video_coding/jitter_buffer.cc: 782

Basically falling into this 'incomplete_frames' section, and nack_mode is
set to kNack, so it just falls into incomplete_frames_.InsertFrame(frame)
continuously.

It seems that the packets that are getting put into it are getting drained
out somewhere.

I am continuously receiving a message: (rtp_utility.cc:342) Failed to find
extension id: 3

Which seems to refer to the rtp-hdrext/abs-send-time

So maybe this is the issue, maybe it can't parse the RTP header for some
reason?

Regarding DTLS, it seems as if all of the encryption stuff is working
appropriately, although sometimes I am getting messages on failed
decryption- but pretty normal for network loss.

- Jason.

···

On Wed, Mar 8, 2017 at 10:58 AM, Brian Baldino <brian@jitsi.org> wrote:

Hey Jason,
Am I understanding you correctly that you're getting blocked at line 140
in receiver.cc for 1 minute? I.e., 'NextCompleteFrame' isn't returning
anything valid for a minute?

If so, I wonder...do you see anything about dtls negotiation completing?
Maybe that is taking a long time for some reason and packets can't be
decrypted correctly for a while? Otherwise it seems odd that packets would
all of a sudden become understandable.

Since you're in the native code already, you might add some prints lower
down to see if things are flowing as expected...for example, is
InsertPacket being called on the jitter buffer? If not, must be something
lower than that, etc..

-brian

On Wed, Mar 8, 2017 at 9:27 AM, Jason Thomas <mail@jasonthom.as> wrote:

Hi Brian,

   I thought perhaps that was the issue too.

looking in webrtc/modules/video_coding/video_receiver.cc the culprit
code is located on line 257:

if( drop_frames_until_keyframe_ ) …

Unfortunately it’s happening even before this, in
_receiver.FrameForDecoding:
webrtc/modules/video_coding/receiver.cc:
140: VCMEncodedFrame *found_frame = jitter_buffer_.NextCompleteFra
me(max_wait_time_ms);

Like it isn’t able to understand the RTP format of the incoming video
packets… Or maybe it’s something to do with encryption.

I’m going to check over the SDP again, it’s possible my parser did
something wrong still and maybe it’s causing the media to be clobbered
somehow?

- Jason.

On Mar 7, 2017, at 3:15 PM, Brian Baldino <brian@jitsi.org> wrote:

I've seen this when a complete frame arrived, but the decoder couldn't
start decoding yet because it hadn't received an IDR. Maybe you can add
some logs to detect when/if a keyframe has been received?

On Tue, Mar 7, 2017 at 2:04 PM, Jason Thomas <mail@jasonthom.as> wrote:

I'm still trying to get to the bottom of the issues I am seeing with
video frame decoding from the JVB using the native C++ webrtc library.

Using WebRTC remotes/branch-heads/57:
0a71dc836641acdaaa63b2724f83c822dbb58ce7

My bridge is set to use ulpfec and red, and has rtx disabled.

Everything works fine on a mobile version that performs nearly the same
step, but isn't written in C++.

VideoReceiveStream: {decoders: [{decoder: (VideoDecoder), payload_type:
100, payload_name: VP8, codec_params: {}}], rtp: {remote_ssrc: 4006849248,
local_ssrc: 1, rtcp_mode: RtcpMode::kCompound, rtcp_xr:
{receiver_reference_time_report: off}, remb: on, transport_cc: off,
nack: {rtp_history_ms: 1000}, ulpfec: {ulpfec_payload_type: 117,
red_payload_type: 116, red_rtx_payload_type: -1}, rtx: {}, extensions:
[{uri: http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time, id:
3}]}, renderer: (renderer), render_delay_ms: 10, sync_group: mixedmslabel,
pre_decode_callback: nullptr, pre_render_callback: nullptr,
target_delay_ms: 0}

channel.cc:904): Channel writable (video) for the first time
(channel.cc:966): Installing keys from DTLS-SRTP on video RTP
(webrtcvideoengine2.cc:1450): OnReadyToSend: Ready.
(webrtcvideoengine2.cc:994): SetSend: false
(channel.cc:1970): Changing video state, send=0
(rtp_receiver_video.cc:74): Received first video RTP packet

1 Minute later:

(video_receiver.cc:285): Received first complete decodable video frame
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a

After this I push the frames to ffmpeg, and they encode and look
perfect.... This long delay between receiving the first RTP packet and
being able to decode a full frame seems like the issue.

Again, any ideas or help would be greatly appreciated.

I would be willing to open source a large part of this native recorder
if/when I am able to finish it.

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as


#6

Sorry, I misspoke, it is falling into kIncomplete, and then into
kDecodableSession, where it is inserted into incomplete_frames_, and
drained somewhere else.

Basically RTP packets are coming in, I am going to try to focus on figuring
out what changes in here that causes frames to suddenly start decoding
correctly (and continue to do so for the duration of the session.)

Thanks for all of your help and incite so far.

- Jason.

···

On Wed, Mar 8, 2017 at 2:23 PM, Jason Thomas <mail@jasonthom.as> wrote:

Hi Brian,

   It seems like it's falling into: src/webrtc/modules/video_coding/jitter_buffer.cc:
782

Basically falling into this 'incomplete_frames' section, and nack_mode is
set to kNack, so it just falls into incomplete_frames_.InsertFrame(frame)
continuously.

It seems that the packets that are getting put into it are getting drained
out somewhere.

I am continuously receiving a message: (rtp_utility.cc:342) Failed to find
extension id: 3

Which seems to refer to the rtp-hdrext/abs-send-time

So maybe this is the issue, maybe it can't parse the RTP header for some
reason?

Regarding DTLS, it seems as if all of the encryption stuff is working
appropriately, although sometimes I am getting messages on failed
decryption- but pretty normal for network loss.

- Jason.

On Wed, Mar 8, 2017 at 10:58 AM, Brian Baldino <brian@jitsi.org> wrote:

Hey Jason,
Am I understanding you correctly that you're getting blocked at line 140
in receiver.cc for 1 minute? I.e., 'NextCompleteFrame' isn't returning
anything valid for a minute?

If so, I wonder...do you see anything about dtls negotiation completing?
Maybe that is taking a long time for some reason and packets can't be
decrypted correctly for a while? Otherwise it seems odd that packets would
all of a sudden become understandable.

Since you're in the native code already, you might add some prints lower
down to see if things are flowing as expected...for example, is
InsertPacket being called on the jitter buffer? If not, must be something
lower than that, etc..

-brian

On Wed, Mar 8, 2017 at 9:27 AM, Jason Thomas <mail@jasonthom.as> wrote:

Hi Brian,

   I thought perhaps that was the issue too.

looking in webrtc/modules/video_coding/video_receiver.cc the culprit
code is located on line 257:

if( drop_frames_until_keyframe_ ) …

Unfortunately it’s happening even before this, in
_receiver.FrameForDecoding:
webrtc/modules/video_coding/receiver.cc:
140: VCMEncodedFrame *found_frame = jitter_buffer_.NextCompleteFra
me(max_wait_time_ms);

Like it isn’t able to understand the RTP format of the incoming video
packets… Or maybe it’s something to do with encryption.

I’m going to check over the SDP again, it’s possible my parser did
something wrong still and maybe it’s causing the media to be clobbered
somehow?

- Jason.

On Mar 7, 2017, at 3:15 PM, Brian Baldino <brian@jitsi.org> wrote:

I've seen this when a complete frame arrived, but the decoder couldn't
start decoding yet because it hadn't received an IDR. Maybe you can add
some logs to detect when/if a keyframe has been received?

On Tue, Mar 7, 2017 at 2:04 PM, Jason Thomas <mail@jasonthom.as> wrote:

I'm still trying to get to the bottom of the issues I am seeing with
video frame decoding from the JVB using the native C++ webrtc library.

Using WebRTC remotes/branch-heads/57:
0a71dc836641acdaaa63b2724f83c822dbb58ce7

My bridge is set to use ulpfec and red, and has rtx disabled.

Everything works fine on a mobile version that performs nearly the same
step, but isn't written in C++.

VideoReceiveStream: {decoders: [{decoder: (VideoDecoder), payload_type:
100, payload_name: VP8, codec_params: {}}], rtp: {remote_ssrc: 4006849248,
local_ssrc: 1, rtcp_mode: RtcpMode::kCompound, rtcp_xr:
{receiver_reference_time_report: off}, remb: on, transport_cc: off,
nack: {rtp_history_ms: 1000}, ulpfec: {ulpfec_payload_type: 117,
red_payload_type: 116, red_rtx_payload_type: -1}, rtx: {}, extensions:
[{uri: http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time, id:
3}]}, renderer: (renderer), render_delay_ms: 10, sync_group: mixedmslabel,
pre_decode_callback: nullptr, pre_render_callback: nullptr,
target_delay_ms: 0}

channel.cc:904): Channel writable (video) for the first time
(channel.cc:966): Installing keys from DTLS-SRTP on video RTP
(webrtcvideoengine2.cc:1450): OnReadyToSend: Ready.
(webrtcvideoengine2.cc:994): SetSend: false
(channel.cc:1970): Changing video state, send=0
(rtp_receiver_video.cc:74): Received first video RTP packet

1 Minute later:

(video_receiver.cc:285): Received first complete decodable video frame
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a

After this I push the frames to ffmpeg, and they encode and look
perfect.... This long delay between receiving the first RTP packet and
being able to decode a full frame seems like the issue.

Again, any ideas or help would be greatly appreciated.

I would be willing to open source a large part of this native recorder
if/when I am able to finish it.

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

--
- Jason Thomas
   http://jasonthom.as


#7

Hey Jason,
No worries...sounds like you are on the right track. I don't know if I've
seen that header extension print before. I assume by that point it has
received an SDP that included signaling of abs-send-time, so I wouldn't
think it would choke on it, but I'm not sure. Falling into kIncomplete is
normal until the frame gets complete, but I know it's a pretty tricky
adding logging to that area to print when a frame does become complete, and
even then I think the results you get may not be straightforward...but you
might be able to add something to the 'have-complete-frame' detection logic
that might give some clue. If you've got verbose logging on and can see
when things started working, sometimes you can look at the logs a bit
before that point and they might give some hints as to what occurred to
make things start working.

-brian

···

On Wed, Mar 8, 2017 at 1:32 PM, Jason Thomas <mail@jasonthom.as> wrote:

Sorry, I misspoke, it is falling into kIncomplete, and then into
kDecodableSession, where it is inserted into incomplete_frames_, and
drained somewhere else.

Basically RTP packets are coming in, I am going to try to focus on
figuring out what changes in here that causes frames to suddenly start
decoding correctly (and continue to do so for the duration of the session.)

Thanks for all of your help and incite so far.

- Jason.

On Wed, Mar 8, 2017 at 2:23 PM, Jason Thomas <mail@jasonthom.as> wrote:

Hi Brian,

   It seems like it's falling into: src/webrtc/modules/video_coding/jitter_buffer.cc:
782

Basically falling into this 'incomplete_frames' section, and nack_mode is
set to kNack, so it just falls into incomplete_frames_.InsertFrame(frame)
continuously.

It seems that the packets that are getting put into it are getting
drained out somewhere.

I am continuously receiving a message: (rtp_utility.cc:342) Failed to
find extension id: 3

Which seems to refer to the rtp-hdrext/abs-send-time

So maybe this is the issue, maybe it can't parse the RTP header for some
reason?

Regarding DTLS, it seems as if all of the encryption stuff is working
appropriately, although sometimes I am getting messages on failed
decryption- but pretty normal for network loss.

- Jason.

On Wed, Mar 8, 2017 at 10:58 AM, Brian Baldino <brian@jitsi.org> wrote:

Hey Jason,
Am I understanding you correctly that you're getting blocked at line 140
in receiver.cc for 1 minute? I.e., 'NextCompleteFrame' isn't returning
anything valid for a minute?

If so, I wonder...do you see anything about dtls negotiation
completing? Maybe that is taking a long time for some reason and packets
can't be decrypted correctly for a while? Otherwise it seems odd that
packets would all of a sudden become understandable.

Since you're in the native code already, you might add some prints lower
down to see if things are flowing as expected...for example, is
InsertPacket being called on the jitter buffer? If not, must be something
lower than that, etc..

-brian

On Wed, Mar 8, 2017 at 9:27 AM, Jason Thomas <mail@jasonthom.as> wrote:

Hi Brian,

   I thought perhaps that was the issue too.

looking in webrtc/modules/video_coding/video_receiver.cc the culprit
code is located on line 257:

if( drop_frames_until_keyframe_ ) …

Unfortunately it’s happening even before this, in
_receiver.FrameForDecoding:
webrtc/modules/video_coding/receiver.cc:
140: VCMEncodedFrame *found_frame = jitter_buffer_.NextCompleteFra
me(max_wait_time_ms);

Like it isn’t able to understand the RTP format of the incoming video
packets… Or maybe it’s something to do with encryption.

I’m going to check over the SDP again, it’s possible my parser did
something wrong still and maybe it’s causing the media to be clobbered
somehow?

- Jason.

On Mar 7, 2017, at 3:15 PM, Brian Baldino <brian@jitsi.org> wrote:

I've seen this when a complete frame arrived, but the decoder couldn't
start decoding yet because it hadn't received an IDR. Maybe you can add
some logs to detect when/if a keyframe has been received?

On Tue, Mar 7, 2017 at 2:04 PM, Jason Thomas <mail@jasonthom.as> wrote:

I'm still trying to get to the bottom of the issues I am seeing with
video frame decoding from the JVB using the native C++ webrtc library.

Using WebRTC remotes/branch-heads/57:
0a71dc836641acdaaa63b2724f83c822dbb58ce7

My bridge is set to use ulpfec and red, and has rtx disabled.

Everything works fine on a mobile version that performs nearly the
same step, but isn't written in C++.

VideoReceiveStream: {decoders: [{decoder: (VideoDecoder),
payload_type: 100, payload_name: VP8, codec_params: {}}], rtp:
{remote_ssrc: 4006849248, local_ssrc: 1, rtcp_mode: RtcpMode::kCompound,
rtcp_xr: {receiver_reference_time_report: off}, remb: on,
transport_cc: off, nack: {rtp_history_ms: 1000}, ulpfec:
{ulpfec_payload_type: 117, red_payload_type: 116, red_rtx_payload_type:
-1}, rtx: {}, extensions: [{uri: http://www.webrtc.org/experime
nts/rtp-hdrext/abs-send-time, id: 3}]}, renderer: (renderer),
render_delay_ms: 10, sync_group: mixedmslabel, pre_decode_callback:
nullptr, pre_render_callback: nullptr, target_delay_ms: 0}

channel.cc:904): Channel writable (video) for the first time
(channel.cc:966): Installing keys from DTLS-SRTP on video RTP
(webrtcvideoengine2.cc:1450): OnReadyToSend: Ready.
(webrtcvideoengine2.cc:994): SetSend: false
(channel.cc:1970): Changing video state, send=0
(rtp_receiver_video.cc:74): Received first video RTP packet

1 Minute later:

(video_receiver.cc:285): Received first complete decodable video frame
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a

After this I push the frames to ffmpeg, and they encode and look
perfect.... This long delay between receiving the first RTP packet and
being able to decode a full frame seems like the issue.

Again, any ideas or help would be greatly appreciated.

I would be willing to open source a large part of this native recorder
if/when I am able to finish it.

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#8

Okay, after a hellish night of debugging the jitter-buffer, I determined that the RTP packets were somehow getting clobbered.

I’m on an AMD processor, and behind a VPN tunnel- so maybe something along the lines is causing this, although the problem doesn’t occur on Chrome running in the same environment.

I have moved it over to an Intel based machine on AWS- and it works perfectly without even being recompiled…

I’m wondering if some odd combination of network conditions is causing it, but regardless- this is just my development environment and not
where I will be running the recorder.

Thanks for helping out with this Brian, I really appreciate your advice around an issue that turned out to have nearly nothing to do with the JVB itself :slight_smile:

- Jason.

···

On Mar 8, 2017, at 4:56 PM, Brian Baldino <brian@jitsi.org> wrote:

Hey Jason,
No worries...sounds like you are on the right track. I don't know if I've seen that header extension print before. I assume by that point it has received an SDP that included signaling of abs-send-time, so I wouldn't think it would choke on it, but I'm not sure. Falling into kIncomplete is normal until the frame gets complete, but I know it's a pretty tricky adding logging to that area to print when a frame does become complete, and even then I think the results you get may not be straightforward...but you might be able to add something to the 'have-complete-frame' detection logic that might give some clue. If you've got verbose logging on and can see when things started working, sometimes you can look at the logs a bit before that point and they might give some hints as to what occurred to make things start working.

-brian

On Wed, Mar 8, 2017 at 1:32 PM, Jason Thomas <mail@jasonthom.as <mailto:mail@jasonthom.as>> wrote:
Sorry, I misspoke, it is falling into kIncomplete, and then into kDecodableSession, where it is inserted into incomplete_frames_, and drained somewhere else.

Basically RTP packets are coming in, I am going to try to focus on figuring out what changes in here that causes frames to suddenly start decoding correctly (and continue to do so for the duration of the session.)

Thanks for all of your help and incite so far.

- Jason.

On Wed, Mar 8, 2017 at 2:23 PM, Jason Thomas <mail@jasonthom.as <mailto:mail@jasonthom.as>> wrote:
Hi Brian,

   It seems like it's falling into: src/webrtc/modules/video_coding/jitter_buffer.cc: 782

Basically falling into this 'incomplete_frames' section, and nack_mode is set to kNack, so it just falls into incomplete_frames_.InsertFrame(frame) continuously.

It seems that the packets that are getting put into it are getting drained out somewhere.

I am continuously receiving a message: (rtp_utility.cc:342) Failed to find extension id: 3

Which seems to refer to the rtp-hdrext/abs-send-time

So maybe this is the issue, maybe it can't parse the RTP header for some reason?

Regarding DTLS, it seems as if all of the encryption stuff is working appropriately, although sometimes I am getting messages on failed decryption- but pretty normal for network loss.

- Jason.

On Wed, Mar 8, 2017 at 10:58 AM, Brian Baldino <brian@jitsi.org <mailto:brian@jitsi.org>> wrote:
Hey Jason,
Am I understanding you correctly that you're getting blocked at line 140 in receiver.cc for 1 minute? I.e., 'NextCompleteFrame' isn't returning anything valid for a minute?

If so, I wonder...do you see anything about dtls negotiation completing? Maybe that is taking a long time for some reason and packets can't be decrypted correctly for a while? Otherwise it seems odd that packets would all of a sudden become understandable.

Since you're in the native code already, you might add some prints lower down to see if things are flowing as expected...for example, is InsertPacket being called on the jitter buffer? If not, must be something lower than that, etc..

-brian

On Wed, Mar 8, 2017 at 9:27 AM, Jason Thomas <mail@jasonthom.as <mailto:mail@jasonthom.as>> wrote:
Hi Brian,

   I thought perhaps that was the issue too.

looking in webrtc/modules/video_coding/video_receiver.cc the culprit code is located on line 257:

if( drop_frames_until_keyframe_ ) …

Unfortunately it’s happening even before this, in _receiver.FrameForDecoding:
webrtc/modules/video_coding/receiver.cc <http://receiver.cc/>:
140: VCMEncodedFrame *found_frame = jitter_buffer_.NextCompleteFrame(max_wait_time_ms);

Like it isn’t able to understand the RTP format of the incoming video packets… Or maybe it’s something to do with encryption.

I’m going to check over the SDP again, it’s possible my parser did something wrong still and maybe it’s causing the media to be clobbered somehow?

- Jason.

On Mar 7, 2017, at 3:15 PM, Brian Baldino <brian@jitsi.org <mailto:brian@jitsi.org>> wrote:

I've seen this when a complete frame arrived, but the decoder couldn't start decoding yet because it hadn't received an IDR. Maybe you can add some logs to detect when/if a keyframe has been received?

On Tue, Mar 7, 2017 at 2:04 PM, Jason Thomas <mail@jasonthom.as <mailto:mail@jasonthom.as>> wrote:
I'm still trying to get to the bottom of the issues I am seeing with video frame decoding from the JVB using the native C++ webrtc library.

Using WebRTC remotes/branch-heads/57:
0a71dc836641acdaaa63b2724f83c822dbb58ce7

My bridge is set to use ulpfec and red, and has rtx disabled.

Everything works fine on a mobile version that performs nearly the same step, but isn't written in C++.

VideoReceiveStream: {decoders: [{decoder: (VideoDecoder), payload_type: 100, payload_name: VP8, codec_params: {}}], rtp: {remote_ssrc: 4006849248, local_ssrc: 1, rtcp_mode: RtcpMode::kCompound, rtcp_xr: {receiver_reference_time_report: off}, remb: on, transport_cc: off, nack: {rtp_history_ms: 1000}, ulpfec: {ulpfec_payload_type: 117, red_payload_type: 116, red_rtx_payload_type: -1}, rtx: {}, extensions: [{uri: http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time, id: 3}]}, renderer: (renderer), render_delay_ms: 10, sync_group: mixedmslabel, pre_decode_callback: nullptr, pre_render_callback: nullptr, target_delay_ms: 0}

channel.cc:904 <http://channel.cc:904/>): Channel writable (video) for the first time
(channel.cc:966 <http://channel.cc:966/>): Installing keys from DTLS-SRTP on video RTP
(webrtcvideoengine2.cc:1450 <http://webrtcvideoengine2.cc:1450/>): OnReadyToSend: Ready.
(webrtcvideoengine2.cc:994 <http://webrtcvideoengine2.cc:994/>): SetSend: false
(channel.cc:1970 <http://channel.cc:1970/>): Changing video state, send=0
(rtp_receiver_video.cc:74): Received first video RTP packet

1 Minute later:

(video_receiver.cc:285): Received first complete decodable video frame
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a

After this I push the frames to ffmpeg, and they encode and look perfect.... This long delay between receiving the first RTP packet and being able to decode a full frame seems like the issue.

Again, any ideas or help would be greatly appreciated.

I would be willing to open source a large part of this native recorder if/when I am able to finish it.

--
- Jason Thomas
   http://jasonthom.as <http://jasonthom.as/>

_______________________________________________
dev mailing list
dev@jitsi.org <mailto:dev@jitsi.org>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org <mailto:dev@jitsi.org>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org <mailto:dev@jitsi.org>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org <mailto:dev@jitsi.org>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as <http://jasonthom.as/>

--
- Jason Thomas
   http://jasonthom.as <http://jasonthom.as/>

_______________________________________________
dev mailing list
dev@jitsi.org <mailto:dev@jitsi.org>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#9

So, actually the tunnel and architecture seems to be a red herring too.

The real difference was how I was testing, in that I was creating a room
with nobody else in it, and then connecting to it.

Basically what happens is the original session-initiate has nothing but the
mixed video/audio synchronization streams.

Then a source-add is sent, and the SDP is amended and the remoteDescription
is set again.

As a result I get the video stream just as it is starting up, I assume, and
BINGO I immediately start decoding frames.

The issue is if I join a room where the videos are already ongoing.

It gets in the original session-initiate the streams, creates them, grabs
the video tracks, and I am still seeing this issue where it sits and takes
forever to find the first decodable stream.

It seems almost like there is some step that will trigger the bridge to
request an new iFrame from all of the clients (Maybe the session-accept)
that isn't being triggered?

Any ideas or help would be very very appreciated.

···

On Thu, Mar 9, 2017 at 8:45 AM, Jason Thomas <mail@jasonthom.as> wrote:

Okay, after a hellish night of debugging the jitter-buffer, I determined
that the RTP packets were somehow getting clobbered.

I’m on an AMD processor, and behind a VPN tunnel- so maybe something along
the lines is causing this, although the problem doesn’t occur on Chrome
running in the same environment.

I have moved it over to an Intel based machine on AWS- and it works
perfectly without even being recompiled…

I’m wondering if some odd combination of network conditions is causing it,
but regardless- this is just my development environment and not
where I will be running the recorder.

Thanks for helping out with this Brian, I really appreciate your advice
around an issue that turned out to have nearly nothing to do with the JVB
itself :slight_smile:

- Jason.

On Mar 8, 2017, at 4:56 PM, Brian Baldino <brian@jitsi.org> wrote:

Hey Jason,
No worries...sounds like you are on the right track. I don't know if I've
seen that header extension print before. I assume by that point it has
received an SDP that included signaling of abs-send-time, so I wouldn't
think it would choke on it, but I'm not sure. Falling into kIncomplete is
normal until the frame gets complete, but I know it's a pretty tricky
adding logging to that area to print when a frame does become complete, and
even then I think the results you get may not be straightforward...but you
might be able to add something to the 'have-complete-frame' detection logic
that might give some clue. If you've got verbose logging on and can see
when things started working, sometimes you can look at the logs a bit
before that point and they might give some hints as to what occurred to
make things start working.

-brian

On Wed, Mar 8, 2017 at 1:32 PM, Jason Thomas <mail@jasonthom.as> wrote:

Sorry, I misspoke, it is falling into kIncomplete, and then into
kDecodableSession, where it is inserted into incomplete_frames_, and
drained somewhere else.

Basically RTP packets are coming in, I am going to try to focus on
figuring out what changes in here that causes frames to suddenly start
decoding correctly (and continue to do so for the duration of the session.)

Thanks for all of your help and incite so far.

- Jason.

On Wed, Mar 8, 2017 at 2:23 PM, Jason Thomas <mail@jasonthom.as> wrote:

Hi Brian,

   It seems like it's falling into: src/webrtc/modules/video_coding/jitter_buffer.cc:
782

Basically falling into this 'incomplete_frames' section, and nack_mode
is set to kNack, so it just falls into incomplete_frames_.InsertFrame(frame)
continuously.

It seems that the packets that are getting put into it are getting
drained out somewhere.

I am continuously receiving a message: (rtp_utility.cc:342) Failed to
find extension id: 3

Which seems to refer to the rtp-hdrext/abs-send-time

So maybe this is the issue, maybe it can't parse the RTP header for some
reason?

Regarding DTLS, it seems as if all of the encryption stuff is working
appropriately, although sometimes I am getting messages on failed
decryption- but pretty normal for network loss.

- Jason.

On Wed, Mar 8, 2017 at 10:58 AM, Brian Baldino <brian@jitsi.org> wrote:

Hey Jason,
Am I understanding you correctly that you're getting blocked at line
140 in receiver.cc for 1 minute? I.e., 'NextCompleteFrame' isn't
returning anything valid for a minute?

If so, I wonder...do you see anything about dtls negotiation
completing? Maybe that is taking a long time for some reason and packets
can't be decrypted correctly for a while? Otherwise it seems odd that
packets would all of a sudden become understandable.

Since you're in the native code already, you might add some prints
lower down to see if things are flowing as expected...for example, is
InsertPacket being called on the jitter buffer? If not, must be something
lower than that, etc..

-brian

On Wed, Mar 8, 2017 at 9:27 AM, Jason Thomas <mail@jasonthom.as> wrote:

Hi Brian,

   I thought perhaps that was the issue too.

looking in webrtc/modules/video_coding/video_receiver.cc the culprit
code is located on line 257:

if( drop_frames_until_keyframe_ ) …

Unfortunately it’s happening even before this, in
_receiver.FrameForDecoding:
webrtc/modules/video_coding/receiver.cc:
140: VCMEncodedFrame *found_frame = jitter_buffer_.NextCompleteFra
me(max_wait_time_ms);

Like it isn’t able to understand the RTP format of the incoming video
packets… Or maybe it’s something to do with encryption.

I’m going to check over the SDP again, it’s possible my parser did
something wrong still and maybe it’s causing the media to be clobbered
somehow?

- Jason.

On Mar 7, 2017, at 3:15 PM, Brian Baldino <brian@jitsi.org> wrote:

I've seen this when a complete frame arrived, but the decoder couldn't
start decoding yet because it hadn't received an IDR. Maybe you can add
some logs to detect when/if a keyframe has been received?

On Tue, Mar 7, 2017 at 2:04 PM, Jason Thomas <mail@jasonthom.as> >>>>> wrote:

I'm still trying to get to the bottom of the issues I am seeing with
video frame decoding from the JVB using the native C++ webrtc library.

Using WebRTC remotes/branch-heads/57:
0a71dc836641acdaaa63b2724f83c822dbb58ce7

My bridge is set to use ulpfec and red, and has rtx disabled.

Everything works fine on a mobile version that performs nearly the
same step, but isn't written in C++.

VideoReceiveStream: {decoders: [{decoder: (VideoDecoder),
payload_type: 100, payload_name: VP8, codec_params: {}}], rtp:
{remote_ssrc: 4006849248, local_ssrc: 1, rtcp_mode: RtcpMode::kCompound,
rtcp_xr: {receiver_reference_time_report: off}, remb: on,
transport_cc: off, nack: {rtp_history_ms: 1000}, ulpfec:
{ulpfec_payload_type: 117, red_payload_type: 116, red_rtx_payload_type:
-1}, rtx: {}, extensions: [{uri: http://www.webrtc.org/experime
nts/rtp-hdrext/abs-send-time, id: 3}]}, renderer: (renderer),
render_delay_ms: 10, sync_group: mixedmslabel, pre_decode_callback:
nullptr, pre_render_callback: nullptr, target_delay_ms: 0}

channel.cc:904): Channel writable (video) for the first time
(channel.cc:966): Installing keys from DTLS-SRTP on video RTP
(webrtcvideoengine2.cc:1450): OnReadyToSend: Ready.
(webrtcvideoengine2.cc:994): SetSend: false
(channel.cc:1970): Changing video state, send=0
(rtp_receiver_video.cc:74): Received first video RTP packet

1 Minute later:

(video_receiver.cc:285): Received first complete decodable video frame
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a

After this I push the frames to ffmpeg, and they encode and look
perfect.... This long delay between receiving the first RTP packet and
being able to decode a full frame seems like the issue.

Again, any ideas or help would be greatly appreciated.

I would be willing to open source a large part of this native
recorder if/when I am able to finish it.

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as


#10

Hey Jason,
Glad you figured some things out. When switching a stream, the bridge will
request an IDR automatically so there should be one coming there. It
won't, however, block forwarding the stream until the IDR comes (that is
only done when the bridge is doing simulcast layer switches), so I would
expect you'd receive some packets before being able to render (though not
long). When you join a conference with other participants already in it,
how long is it taking to start showing video? How does it compare to a web
client doing the same thing? There shouldn't be any difference between the
two. How customized is your client? Is the webrtc stack still handling
all the rtcp feedback for you?

···

On Thu, Mar 9, 2017 at 2:08 PM, Jason Thomas <mail@jasonthom.as> wrote:

So, actually the tunnel and architecture seems to be a red herring too.

The real difference was how I was testing, in that I was creating a room
with nobody else in it, and then connecting to it.

Basically what happens is the original session-initiate has nothing but
the mixed video/audio synchronization streams.

Then a source-add is sent, and the SDP is amended and the
remoteDescription is set again.

As a result I get the video stream just as it is starting up, I assume,
and BINGO I immediately start decoding frames.

The issue is if I join a room where the videos are already ongoing.

It gets in the original session-initiate the streams, creates them, grabs
the video tracks, and I am still seeing this issue where it sits and takes
forever to find the first decodable stream.

It seems almost like there is some step that will trigger the bridge to
request an new iFrame from all of the clients (Maybe the session-accept)
that isn't being triggered?

Any ideas or help would be very very appreciated.

On Thu, Mar 9, 2017 at 8:45 AM, Jason Thomas <mail@jasonthom.as> wrote:

Okay, after a hellish night of debugging the jitter-buffer, I determined
that the RTP packets were somehow getting clobbered.

I’m on an AMD processor, and behind a VPN tunnel- so maybe something
along the lines is causing this, although the problem doesn’t occur on
Chrome running in the same environment.

I have moved it over to an Intel based machine on AWS- and it works
perfectly without even being recompiled…

I’m wondering if some odd combination of network conditions is causing
it, but regardless- this is just my development environment and not
where I will be running the recorder.

Thanks for helping out with this Brian, I really appreciate your advice
around an issue that turned out to have nearly nothing to do with the JVB
itself :slight_smile:

- Jason.

On Mar 8, 2017, at 4:56 PM, Brian Baldino <brian@jitsi.org> wrote:

Hey Jason,
No worries...sounds like you are on the right track. I don't know if
I've seen that header extension print before. I assume by that point it
has received an SDP that included signaling of abs-send-time, so I wouldn't
think it would choke on it, but I'm not sure. Falling into kIncomplete is
normal until the frame gets complete, but I know it's a pretty tricky
adding logging to that area to print when a frame does become complete, and
even then I think the results you get may not be straightforward...but you
might be able to add something to the 'have-complete-frame' detection logic
that might give some clue. If you've got verbose logging on and can see
when things started working, sometimes you can look at the logs a bit
before that point and they might give some hints as to what occurred to
make things start working.

-brian

On Wed, Mar 8, 2017 at 1:32 PM, Jason Thomas <mail@jasonthom.as> wrote:

Sorry, I misspoke, it is falling into kIncomplete, and then into
kDecodableSession, where it is inserted into incomplete_frames_, and
drained somewhere else.

Basically RTP packets are coming in, I am going to try to focus on
figuring out what changes in here that causes frames to suddenly start
decoding correctly (and continue to do so for the duration of the session.)

Thanks for all of your help and incite so far.

- Jason.

On Wed, Mar 8, 2017 at 2:23 PM, Jason Thomas <mail@jasonthom.as> wrote:

Hi Brian,

   It seems like it's falling into: src/webrtc/modules/video_coding/jitter_buffer.cc:
782

Basically falling into this 'incomplete_frames' section, and nack_mode
is set to kNack, so it just falls into incomplete_frames_.InsertFrame(frame)
continuously.

It seems that the packets that are getting put into it are getting
drained out somewhere.

I am continuously receiving a message: (rtp_utility.cc:342) Failed to
find extension id: 3

Which seems to refer to the rtp-hdrext/abs-send-time

So maybe this is the issue, maybe it can't parse the RTP header for
some reason?

Regarding DTLS, it seems as if all of the encryption stuff is working
appropriately, although sometimes I am getting messages on failed
decryption- but pretty normal for network loss.

- Jason.

On Wed, Mar 8, 2017 at 10:58 AM, Brian Baldino <brian@jitsi.org> wrote:

Hey Jason,
Am I understanding you correctly that you're getting blocked at line
140 in receiver.cc for 1 minute? I.e., 'NextCompleteFrame' isn't
returning anything valid for a minute?

If so, I wonder...do you see anything about dtls negotiation
completing? Maybe that is taking a long time for some reason and packets
can't be decrypted correctly for a while? Otherwise it seems odd that
packets would all of a sudden become understandable.

Since you're in the native code already, you might add some prints
lower down to see if things are flowing as expected...for example, is
InsertPacket being called on the jitter buffer? If not, must be something
lower than that, etc..

-brian

On Wed, Mar 8, 2017 at 9:27 AM, Jason Thomas <mail@jasonthom.as> >>>>> wrote:

Hi Brian,

   I thought perhaps that was the issue too.

looking in webrtc/modules/video_coding/video_receiver.cc the culprit
code is located on line 257:

if( drop_frames_until_keyframe_ ) …

Unfortunately it’s happening even before this, in
_receiver.FrameForDecoding:
webrtc/modules/video_coding/receiver.cc:
140: VCMEncodedFrame *found_frame = jitter_buffer_.NextCompleteFra
me(max_wait_time_ms);

Like it isn’t able to understand the RTP format of the incoming video
packets… Or maybe it’s something to do with encryption.

I’m going to check over the SDP again, it’s possible my parser did
something wrong still and maybe it’s causing the media to be clobbered
somehow?

- Jason.

On Mar 7, 2017, at 3:15 PM, Brian Baldino <brian@jitsi.org> wrote:

I've seen this when a complete frame arrived, but the decoder
couldn't start decoding yet because it hadn't received an IDR. Maybe you
can add some logs to detect when/if a keyframe has been received?

On Tue, Mar 7, 2017 at 2:04 PM, Jason Thomas <mail@jasonthom.as> >>>>>> wrote:

I'm still trying to get to the bottom of the issues I am seeing with
video frame decoding from the JVB using the native C++ webrtc library.

Using WebRTC remotes/branch-heads/57:
0a71dc836641acdaaa63b2724f83c822dbb58ce7

My bridge is set to use ulpfec and red, and has rtx disabled.

Everything works fine on a mobile version that performs nearly the
same step, but isn't written in C++.

VideoReceiveStream: {decoders: [{decoder: (VideoDecoder),
payload_type: 100, payload_name: VP8, codec_params: {}}], rtp:
{remote_ssrc: 4006849248, local_ssrc: 1, rtcp_mode: RtcpMode::kCompound,
rtcp_xr: {receiver_reference_time_report: off}, remb: on,
transport_cc: off, nack: {rtp_history_ms: 1000}, ulpfec:
{ulpfec_payload_type: 117, red_payload_type: 116, red_rtx_payload_type:
-1}, rtx: {}, extensions: [{uri: http://www.webrtc.org/experime
nts/rtp-hdrext/abs-send-time, id: 3}]}, renderer: (renderer),
render_delay_ms: 10, sync_group: mixedmslabel, pre_decode_callback:
nullptr, pre_render_callback: nullptr, target_delay_ms: 0}

channel.cc:904): Channel writable (video) for the first time
(channel.cc:966): Installing keys from DTLS-SRTP on video RTP
(webrtcvideoengine2.cc:1450): OnReadyToSend: Ready.
(webrtcvideoengine2.cc:994): SetSend: false
(channel.cc:1970): Changing video state, send=0
(rtp_receiver_video.cc:74): Received first video RTP packet

1 Minute later:

(video_receiver.cc:285): Received first complete decodable video
frame
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a

After this I push the frames to ffmpeg, and they encode and look
perfect.... This long delay between receiving the first RTP packet and
being able to decode a full frame seems like the issue.

Again, any ideas or help would be greatly appreciated.

I would be willing to open source a large part of this native
recorder if/when I am able to finish it.

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#11

So, in the scenario where my client adds no additional streams to the
bridge, I am assuming this probably does not trigger the above condition?

Basically I am sending no audio or video feeds, since the purpose of this
is simply to record.

Is there perhaps an XMPP way to trigger an iFrame request or some other
mechanism?

- Jason.

···

On Mar 9, 2017 5:43 PM, "Brian Baldino" <brian@jitsi.org> wrote:

Hey Jason,
Glad you figured some things out. When switching a stream, the bridge
will request an IDR automatically so there should be one coming there. It
won't, however, block forwarding the stream until the IDR comes (that is
only done when the bridge is doing simulcast layer switches), so I would
expect you'd receive some packets before being able to render (though not
long). When you join a conference with other participants already in it,
how long is it taking to start showing video? How does it compare to a web
client doing the same thing? There shouldn't be any difference between the
two. How customized is your client? Is the webrtc stack still handling
all the rtcp feedback for you?

On Thu, Mar 9, 2017 at 2:08 PM, Jason Thomas <mail@jasonthom.as> wrote:

So, actually the tunnel and architecture seems to be a red herring too.

The real difference was how I was testing, in that I was creating a room
with nobody else in it, and then connecting to it.

Basically what happens is the original session-initiate has nothing but
the mixed video/audio synchronization streams.

Then a source-add is sent, and the SDP is amended and the
remoteDescription is set again.

As a result I get the video stream just as it is starting up, I assume,
and BINGO I immediately start decoding frames.

The issue is if I join a room where the videos are already ongoing.

It gets in the original session-initiate the streams, creates them, grabs
the video tracks, and I am still seeing this issue where it sits and takes
forever to find the first decodable stream.

It seems almost like there is some step that will trigger the bridge to
request an new iFrame from all of the clients (Maybe the session-accept)
that isn't being triggered?

Any ideas or help would be very very appreciated.

On Thu, Mar 9, 2017 at 8:45 AM, Jason Thomas <mail@jasonthom.as> wrote:

Okay, after a hellish night of debugging the jitter-buffer, I determined
that the RTP packets were somehow getting clobbered.

I’m on an AMD processor, and behind a VPN tunnel- so maybe something
along the lines is causing this, although the problem doesn’t occur on
Chrome running in the same environment.

I have moved it over to an Intel based machine on AWS- and it works
perfectly without even being recompiled…

I’m wondering if some odd combination of network conditions is causing
it, but regardless- this is just my development environment and not
where I will be running the recorder.

Thanks for helping out with this Brian, I really appreciate your advice
around an issue that turned out to have nearly nothing to do with the JVB
itself :slight_smile:

- Jason.

On Mar 8, 2017, at 4:56 PM, Brian Baldino <brian@jitsi.org> wrote:

Hey Jason,
No worries...sounds like you are on the right track. I don't know if
I've seen that header extension print before. I assume by that point it
has received an SDP that included signaling of abs-send-time, so I wouldn't
think it would choke on it, but I'm not sure. Falling into kIncomplete is
normal until the frame gets complete, but I know it's a pretty tricky
adding logging to that area to print when a frame does become complete, and
even then I think the results you get may not be straightforward...but you
might be able to add something to the 'have-complete-frame' detection logic
that might give some clue. If you've got verbose logging on and can see
when things started working, sometimes you can look at the logs a bit
before that point and they might give some hints as to what occurred to
make things start working.

-brian

On Wed, Mar 8, 2017 at 1:32 PM, Jason Thomas <mail@jasonthom.as> wrote:

Sorry, I misspoke, it is falling into kIncomplete, and then into
kDecodableSession, where it is inserted into incomplete_frames_, and
drained somewhere else.

Basically RTP packets are coming in, I am going to try to focus on
figuring out what changes in here that causes frames to suddenly start
decoding correctly (and continue to do so for the duration of the session.)

Thanks for all of your help and incite so far.

- Jason.

On Wed, Mar 8, 2017 at 2:23 PM, Jason Thomas <mail@jasonthom.as> wrote:

Hi Brian,

   It seems like it's falling into: src/webrtc/modules/video_coding/jitter_buffer.cc:
782

Basically falling into this 'incomplete_frames' section, and nack_mode
is set to kNack, so it just falls into incomplete_frames_.InsertFrame(frame)
continuously.

It seems that the packets that are getting put into it are getting
drained out somewhere.

I am continuously receiving a message: (rtp_utility.cc:342) Failed to
find extension id: 3

Which seems to refer to the rtp-hdrext/abs-send-time

So maybe this is the issue, maybe it can't parse the RTP header for
some reason?

Regarding DTLS, it seems as if all of the encryption stuff is working
appropriately, although sometimes I am getting messages on failed
decryption- but pretty normal for network loss.

- Jason.

On Wed, Mar 8, 2017 at 10:58 AM, Brian Baldino <brian@jitsi.org> >>>>> wrote:

Hey Jason,
Am I understanding you correctly that you're getting blocked at line
140 in receiver.cc for 1 minute? I.e., 'NextCompleteFrame' isn't
returning anything valid for a minute?

If so, I wonder...do you see anything about dtls negotiation
completing? Maybe that is taking a long time for some reason and packets
can't be decrypted correctly for a while? Otherwise it seems odd that
packets would all of a sudden become understandable.

Since you're in the native code already, you might add some prints
lower down to see if things are flowing as expected...for example, is
InsertPacket being called on the jitter buffer? If not, must be something
lower than that, etc..

-brian

On Wed, Mar 8, 2017 at 9:27 AM, Jason Thomas <mail@jasonthom.as> >>>>>> wrote:

Hi Brian,

   I thought perhaps that was the issue too.

looking in webrtc/modules/video_coding/video_receiver.cc the
culprit code is located on line 257:

if( drop_frames_until_keyframe_ ) …

Unfortunately it’s happening even before this, in
_receiver.FrameForDecoding:
webrtc/modules/video_coding/receiver.cc:
140: VCMEncodedFrame *found_frame = jitter_buffer_.NextCompleteFra
me(max_wait_time_ms);

Like it isn’t able to understand the RTP format of the incoming
video packets… Or maybe it’s something to do with encryption.

I’m going to check over the SDP again, it’s possible my parser did
something wrong still and maybe it’s causing the media to be clobbered
somehow?

- Jason.

On Mar 7, 2017, at 3:15 PM, Brian Baldino <brian@jitsi.org> wrote:

I've seen this when a complete frame arrived, but the decoder
couldn't start decoding yet because it hadn't received an IDR. Maybe you
can add some logs to detect when/if a keyframe has been received?

On Tue, Mar 7, 2017 at 2:04 PM, Jason Thomas <mail@jasonthom.as> >>>>>>> wrote:

I'm still trying to get to the bottom of the issues I am seeing
with video frame decoding from the JVB using the native C++ webrtc library.

Using WebRTC remotes/branch-heads/57:
0a71dc836641acdaaa63b2724f83c822dbb58ce7

My bridge is set to use ulpfec and red, and has rtx disabled.

Everything works fine on a mobile version that performs nearly the
same step, but isn't written in C++.

VideoReceiveStream: {decoders: [{decoder: (VideoDecoder),
payload_type: 100, payload_name: VP8, codec_params: {}}], rtp:
{remote_ssrc: 4006849248, local_ssrc: 1, rtcp_mode: RtcpMode::kCompound,
rtcp_xr: {receiver_reference_time_report: off}, remb: on,
transport_cc: off, nack: {rtp_history_ms: 1000}, ulpfec:
{ulpfec_payload_type: 117, red_payload_type: 116, red_rtx_payload_type:
-1}, rtx: {}, extensions: [{uri: http://www.webrtc.org/experime
nts/rtp-hdrext/abs-send-time, id: 3}]}, renderer: (renderer),
render_delay_ms: 10, sync_group: mixedmslabel, pre_decode_callback:
nullptr, pre_render_callback: nullptr, target_delay_ms: 0}

channel.cc:904): Channel writable (video) for the first time
(channel.cc:966): Installing keys from DTLS-SRTP on video RTP
(webrtcvideoengine2.cc:1450): OnReadyToSend: Ready.
(webrtcvideoengine2.cc:994): SetSend: false
(channel.cc:1970): Changing video state, send=0
(rtp_receiver_video.cc:74): Received first video RTP packet

1 Minute later:

(video_receiver.cc:285): Received first complete decodable video
frame
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a

After this I push the frames to ffmpeg, and they encode and look
perfect.... This long delay between receiving the first RTP packet and
being able to decode a full frame seems like the issue.

Again, any ideas or help would be greatly appreciated.

I would be willing to open source a large part of this native
recorder if/when I am able to finish it.

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#12

"When you join a conference with other participants already in it, how long
is it taking to start showing video? "

It takes a very long time, upwards of 50-60 seconds.

If I have another client join, I start getting frames immediately after the
renegotiate.

"How does it compare to a web client doing the same thing? There shouldn't
be any difference between the two. How customized is your client?"

It is exactly the same if I am subscribing to a new stream, or the client
has just joined. But only in the case where I join in the middle of an
established conference do I see this delay.

There is no customization, I am using a vanilla WebRTC, Chrome/57 build.

I use pretty much the same strategy on mobile clients with iOS/Android
linking to the same version of WebRTC natively, and haven't experienced any
issues.

- Jason.

Hey Jason,
Glad you figured some things out. When switching a stream, the bridge will
request an IDR automatically so there should be one coming there. It
won't, however, block forwarding the stream until the IDR comes (that is
only done when the bridge is doing simulcast layer switches), so I would
expect you'd receive some packets before being able to render (though not
long). When you join a conference with other participants already in it,
how long is it taking to start showing video? How does it compare to a web
client doing the same thing? There shouldn't be any difference between the
two. How customized is your client? Is the webrtc stack still handling
all the rtcp feedback for you?

···

On Mar 9, 2017 5:43 PM, "Brian Baldino" <brian@jitsi.org> wrote:

On Thu, Mar 9, 2017 at 2:08 PM, Jason Thomas <mail@jasonthom.as> wrote:

So, actually the tunnel and architecture seems to be a red herring too.

The real difference was how I was testing, in that I was creating a room
with nobody else in it, and then connecting to it.

Basically what happens is the original session-initiate has nothing but
the mixed video/audio synchronization streams.

Then a source-add is sent, and the SDP is amended and the
remoteDescription is set again.

As a result I get the video stream just as it is starting up, I assume,
and BINGO I immediately start decoding frames.

The issue is if I join a room where the videos are already ongoing.

It gets in the original session-initiate the streams, creates them, grabs
the video tracks, and I am still seeing this issue where it sits and takes
forever to find the first decodable stream.

It seems almost like there is some step that will trigger the bridge to
request an new iFrame from all of the clients (Maybe the session-accept)
that isn't being triggered?

Any ideas or help would be very very appreciated.

On Thu, Mar 9, 2017 at 8:45 AM, Jason Thomas <mail@jasonthom.as> wrote:

Okay, after a hellish night of debugging the jitter-buffer, I determined
that the RTP packets were somehow getting clobbered.

I’m on an AMD processor, and behind a VPN tunnel- so maybe something
along the lines is causing this, although the problem doesn’t occur on
Chrome running in the same environment.

I have moved it over to an Intel based machine on AWS- and it works
perfectly without even being recompiled…

I’m wondering if some odd combination of network conditions is causing
it, but regardless- this is just my development environment and not
where I will be running the recorder.

Thanks for helping out with this Brian, I really appreciate your advice
around an issue that turned out to have nearly nothing to do with the JVB
itself :slight_smile:

- Jason.

On Mar 8, 2017, at 4:56 PM, Brian Baldino <brian@jitsi.org> wrote:

Hey Jason,
No worries...sounds like you are on the right track. I don't know if
I've seen that header extension print before. I assume by that point it
has received an SDP that included signaling of abs-send-time, so I wouldn't
think it would choke on it, but I'm not sure. Falling into kIncomplete is
normal until the frame gets complete, but I know it's a pretty tricky
adding logging to that area to print when a frame does become complete, and
even then I think the results you get may not be straightforward...but you
might be able to add something to the 'have-complete-frame' detection logic
that might give some clue. If you've got verbose logging on and can see
when things started working, sometimes you can look at the logs a bit
before that point and they might give some hints as to what occurred to
make things start working.

-brian

On Wed, Mar 8, 2017 at 1:32 PM, Jason Thomas <mail@jasonthom.as> wrote:

Sorry, I misspoke, it is falling into kIncomplete, and then into
kDecodableSession, where it is inserted into incomplete_frames_, and
drained somewhere else.

Basically RTP packets are coming in, I am going to try to focus on
figuring out what changes in here that causes frames to suddenly start
decoding correctly (and continue to do so for the duration of the session.)

Thanks for all of your help and incite so far.

- Jason.

On Wed, Mar 8, 2017 at 2:23 PM, Jason Thomas <mail@jasonthom.as> wrote:

Hi Brian,

   It seems like it's falling into: src/webrtc/modules/video_coding/jitter_buffer.cc:
782

Basically falling into this 'incomplete_frames' section, and nack_mode
is set to kNack, so it just falls into incomplete_frames_.InsertFrame(frame)
continuously.

It seems that the packets that are getting put into it are getting
drained out somewhere.

I am continuously receiving a message: (rtp_utility.cc:342) Failed to
find extension id: 3

Which seems to refer to the rtp-hdrext/abs-send-time

So maybe this is the issue, maybe it can't parse the RTP header for
some reason?

Regarding DTLS, it seems as if all of the encryption stuff is working
appropriately, although sometimes I am getting messages on failed
decryption- but pretty normal for network loss.

- Jason.

On Wed, Mar 8, 2017 at 10:58 AM, Brian Baldino <brian@jitsi.org> wrote:

Hey Jason,
Am I understanding you correctly that you're getting blocked at line
140 in receiver.cc for 1 minute? I.e., 'NextCompleteFrame' isn't
returning anything valid for a minute?

If so, I wonder...do you see anything about dtls negotiation
completing? Maybe that is taking a long time for some reason and packets
can't be decrypted correctly for a while? Otherwise it seems odd that
packets would all of a sudden become understandable.

Since you're in the native code already, you might add some prints
lower down to see if things are flowing as expected...for example, is
InsertPacket being called on the jitter buffer? If not, must be something
lower than that, etc..

-brian

On Wed, Mar 8, 2017 at 9:27 AM, Jason Thomas <mail@jasonthom.as> >>>>> wrote:

Hi Brian,

   I thought perhaps that was the issue too.

looking in webrtc/modules/video_coding/video_receiver.cc the culprit
code is located on line 257:

if( drop_frames_until_keyframe_ ) …

Unfortunately it’s happening even before this, in
_receiver.FrameForDecoding:
webrtc/modules/video_coding/receiver.cc:
140: VCMEncodedFrame *found_frame = jitter_buffer_.NextCompleteFra
me(max_wait_time_ms);

Like it isn’t able to understand the RTP format of the incoming video
packets… Or maybe it’s something to do with encryption.

I’m going to check over the SDP again, it’s possible my parser did
something wrong still and maybe it’s causing the media to be clobbered
somehow?

- Jason.

On Mar 7, 2017, at 3:15 PM, Brian Baldino <brian@jitsi.org> wrote:

I've seen this when a complete frame arrived, but the decoder
couldn't start decoding yet because it hadn't received an IDR. Maybe you
can add some logs to detect when/if a keyframe has been received?

On Tue, Mar 7, 2017 at 2:04 PM, Jason Thomas <mail@jasonthom.as> >>>>>> wrote:

I'm still trying to get to the bottom of the issues I am seeing with
video frame decoding from the JVB using the native C++ webrtc library.

Using WebRTC remotes/branch-heads/57:
0a71dc836641acdaaa63b2724f83c822dbb58ce7

My bridge is set to use ulpfec and red, and has rtx disabled.

Everything works fine on a mobile version that performs nearly the
same step, but isn't written in C++.

VideoReceiveStream: {decoders: [{decoder: (VideoDecoder),
payload_type: 100, payload_name: VP8, codec_params: {}}], rtp:
{remote_ssrc: 4006849248, local_ssrc: 1, rtcp_mode: RtcpMode::kCompound,
rtcp_xr: {receiver_reference_time_report: off}, remb: on,
transport_cc: off, nack: {rtp_history_ms: 1000}, ulpfec:
{ulpfec_payload_type: 117, red_payload_type: 116, red_rtx_payload_type:
-1}, rtx: {}, extensions: [{uri: http://www.webrtc.org/experime
nts/rtp-hdrext/abs-send-time, id: 3}]}, renderer: (renderer),
render_delay_ms: 10, sync_group: mixedmslabel, pre_decode_callback:
nullptr, pre_render_callback: nullptr, target_delay_ms: 0}

channel.cc:904): Channel writable (video) for the first time
(channel.cc:966): Installing keys from DTLS-SRTP on video RTP
(webrtcvideoengine2.cc:1450): OnReadyToSend: Ready.
(webrtcvideoengine2.cc:994): SetSend: false
(channel.cc:1970): Changing video state, send=0
(rtp_receiver_video.cc:74): Received first video RTP packet

1 Minute later:

(video_receiver.cc:285): Received first complete decodable video
frame
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a

After this I push the frames to ffmpeg, and they encode and look
perfect.... This long delay between receiving the first RTP packet and
being able to decode a full frame seems like the issue.

Again, any ideas or help would be greatly appreciated.

I would be willing to open source a large part of this native
recorder if/when I am able to finish it.

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#13

Ok, so none of your other tests involve a client joining that doesn't send
any data, is that right? What does your native client's answer look like?
Are you signaling any video ssrcs? If there are no ssrcs in the video
mline, chrome will use a media ssrc of "1" by default in rtcp. The bridge
won't accept these unless they are signaled (it will automatically accept
rtp packets from ssrcs it doesn't recognize, but not rtcp). So there are a
couple options:

1) You could put a recvonly ssrc in your answer. This involves just an
a=ssrc line, no msid or anything like that
2) You could add the ssrc '1' to the video channel manually somewhere (in
signaling, for example) so that these packets will be accepted by the
bridge for this client.

Hopefully something there pans out.

···

On Thu, Mar 9, 2017 at 4:52 PM, Jason Thomas <mail@jasonthom.as> wrote:

"When you join a conference with other participants already in it, how
long is it taking to start showing video? "

It takes a very long time, upwards of 50-60 seconds.

If I have another client join, I start getting frames immediately after
the renegotiate.

"How does it compare to a web client doing the same thing? There
shouldn't be any difference between the two. How customized is your
client?"

It is exactly the same if I am subscribing to a new stream, or the client
has just joined. But only in the case where I join in the middle of an
established conference do I see this delay.

There is no customization, I am using a vanilla WebRTC, Chrome/57 build.

I use pretty much the same strategy on mobile clients with iOS/Android
linking to the same version of WebRTC natively, and haven't experienced any
issues.

- Jason.

On Mar 9, 2017 5:43 PM, "Brian Baldino" <brian@jitsi.org> wrote:

Hey Jason,
Glad you figured some things out. When switching a stream, the bridge
will request an IDR automatically so there should be one coming there. It
won't, however, block forwarding the stream until the IDR comes (that is
only done when the bridge is doing simulcast layer switches), so I would
expect you'd receive some packets before being able to render (though not
long). When you join a conference with other participants already in it,
how long is it taking to start showing video? How does it compare to a web
client doing the same thing? There shouldn't be any difference between the
two. How customized is your client? Is the webrtc stack still handling
all the rtcp feedback for you?

On Thu, Mar 9, 2017 at 2:08 PM, Jason Thomas <mail@jasonthom.as> wrote:

So, actually the tunnel and architecture seems to be a red herring too.

The real difference was how I was testing, in that I was creating a room
with nobody else in it, and then connecting to it.

Basically what happens is the original session-initiate has nothing but
the mixed video/audio synchronization streams.

Then a source-add is sent, and the SDP is amended and the
remoteDescription is set again.

As a result I get the video stream just as it is starting up, I assume,
and BINGO I immediately start decoding frames.

The issue is if I join a room where the videos are already ongoing.

It gets in the original session-initiate the streams, creates them, grabs
the video tracks, and I am still seeing this issue where it sits and takes
forever to find the first decodable stream.

It seems almost like there is some step that will trigger the bridge to
request an new iFrame from all of the clients (Maybe the session-accept)
that isn't being triggered?

Any ideas or help would be very very appreciated.

On Thu, Mar 9, 2017 at 8:45 AM, Jason Thomas <mail@jasonthom.as> wrote:

Okay, after a hellish night of debugging the jitter-buffer, I determined
that the RTP packets were somehow getting clobbered.

I’m on an AMD processor, and behind a VPN tunnel- so maybe something
along the lines is causing this, although the problem doesn’t occur on
Chrome running in the same environment.

I have moved it over to an Intel based machine on AWS- and it works
perfectly without even being recompiled…

I’m wondering if some odd combination of network conditions is causing
it, but regardless- this is just my development environment and not
where I will be running the recorder.

Thanks for helping out with this Brian, I really appreciate your advice
around an issue that turned out to have nearly nothing to do with the JVB
itself :slight_smile:

- Jason.

On Mar 8, 2017, at 4:56 PM, Brian Baldino <brian@jitsi.org> wrote:

Hey Jason,
No worries...sounds like you are on the right track. I don't know if
I've seen that header extension print before. I assume by that point it
has received an SDP that included signaling of abs-send-time, so I wouldn't
think it would choke on it, but I'm not sure. Falling into kIncomplete is
normal until the frame gets complete, but I know it's a pretty tricky
adding logging to that area to print when a frame does become complete, and
even then I think the results you get may not be straightforward...but you
might be able to add something to the 'have-complete-frame' detection logic
that might give some clue. If you've got verbose logging on and can see
when things started working, sometimes you can look at the logs a bit
before that point and they might give some hints as to what occurred to
make things start working.

-brian

On Wed, Mar 8, 2017 at 1:32 PM, Jason Thomas <mail@jasonthom.as> wrote:

Sorry, I misspoke, it is falling into kIncomplete, and then into
kDecodableSession, where it is inserted into incomplete_frames_, and
drained somewhere else.

Basically RTP packets are coming in, I am going to try to focus on
figuring out what changes in here that causes frames to suddenly start
decoding correctly (and continue to do so for the duration of the session.)

Thanks for all of your help and incite so far.

- Jason.

On Wed, Mar 8, 2017 at 2:23 PM, Jason Thomas <mail@jasonthom.as> wrote:

Hi Brian,

   It seems like it's falling into: src/webrtc/modules/video_coding/jitter_buffer.cc:
782

Basically falling into this 'incomplete_frames' section, and nack_mode
is set to kNack, so it just falls into incomplete_frames_.InsertFrame(frame)
continuously.

It seems that the packets that are getting put into it are getting
drained out somewhere.

I am continuously receiving a message: (rtp_utility.cc:342) Failed to
find extension id: 3

Which seems to refer to the rtp-hdrext/abs-send-time

So maybe this is the issue, maybe it can't parse the RTP header for
some reason?

Regarding DTLS, it seems as if all of the encryption stuff is working
appropriately, although sometimes I am getting messages on failed
decryption- but pretty normal for network loss.

- Jason.

On Wed, Mar 8, 2017 at 10:58 AM, Brian Baldino <brian@jitsi.org> >>>>> wrote:

Hey Jason,
Am I understanding you correctly that you're getting blocked at line
140 in receiver.cc for 1 minute? I.e., 'NextCompleteFrame' isn't
returning anything valid for a minute?

If so, I wonder...do you see anything about dtls negotiation
completing? Maybe that is taking a long time for some reason and packets
can't be decrypted correctly for a while? Otherwise it seems odd that
packets would all of a sudden become understandable.

Since you're in the native code already, you might add some prints
lower down to see if things are flowing as expected...for example, is
InsertPacket being called on the jitter buffer? If not, must be something
lower than that, etc..

-brian

On Wed, Mar 8, 2017 at 9:27 AM, Jason Thomas <mail@jasonthom.as> >>>>>> wrote:

Hi Brian,

   I thought perhaps that was the issue too.

looking in webrtc/modules/video_coding/video_receiver.cc the
culprit code is located on line 257:

if( drop_frames_until_keyframe_ ) …

Unfortunately it’s happening even before this, in
_receiver.FrameForDecoding:
webrtc/modules/video_coding/receiver.cc:
140: VCMEncodedFrame *found_frame = jitter_buffer_.NextCompleteFra
me(max_wait_time_ms);

Like it isn’t able to understand the RTP format of the incoming
video packets… Or maybe it’s something to do with encryption.

I’m going to check over the SDP again, it’s possible my parser did
something wrong still and maybe it’s causing the media to be clobbered
somehow?

- Jason.

On Mar 7, 2017, at 3:15 PM, Brian Baldino <brian@jitsi.org> wrote:

I've seen this when a complete frame arrived, but the decoder
couldn't start decoding yet because it hadn't received an IDR. Maybe you
can add some logs to detect when/if a keyframe has been received?

On Tue, Mar 7, 2017 at 2:04 PM, Jason Thomas <mail@jasonthom.as> >>>>>>> wrote:

I'm still trying to get to the bottom of the issues I am seeing
with video frame decoding from the JVB using the native C++ webrtc library.

Using WebRTC remotes/branch-heads/57:
0a71dc836641acdaaa63b2724f83c822dbb58ce7

My bridge is set to use ulpfec and red, and has rtx disabled.

Everything works fine on a mobile version that performs nearly the
same step, but isn't written in C++.

VideoReceiveStream: {decoders: [{decoder: (VideoDecoder),
payload_type: 100, payload_name: VP8, codec_params: {}}], rtp:
{remote_ssrc: 4006849248, local_ssrc: 1, rtcp_mode: RtcpMode::kCompound,
rtcp_xr: {receiver_reference_time_report: off}, remb: on,
transport_cc: off, nack: {rtp_history_ms: 1000}, ulpfec:
{ulpfec_payload_type: 117, red_payload_type: 116, red_rtx_payload_type:
-1}, rtx: {}, extensions: [{uri: http://www.webrtc.org/experime
nts/rtp-hdrext/abs-send-time, id: 3}]}, renderer: (renderer),
render_delay_ms: 10, sync_group: mixedmslabel, pre_decode_callback:
nullptr, pre_render_callback: nullptr, target_delay_ms: 0}

channel.cc:904): Channel writable (video) for the first time
(channel.cc:966): Installing keys from DTLS-SRTP on video RTP
(webrtcvideoengine2.cc:1450): OnReadyToSend: Ready.
(webrtcvideoengine2.cc:994): SetSend: false
(channel.cc:1970): Changing video state, send=0
(rtp_receiver_video.cc:74): Received first video RTP packet

1 Minute later:

(video_receiver.cc:285): Received first complete decodable video
frame
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a

After this I push the frames to ffmpeg, and they encode and look
perfect.... This long delay between receiving the first RTP packet and
being able to decode a full frame seems like the issue.

Again, any ideas or help would be greatly appreciated.

I would be willing to open source a large part of this native
recorder if/when I am able to finish it.

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#14

Brian,

   Exactly, all of the other clients send both video and audio.

Here are some links from the original post with the Jingle->SDP parser, and
the generated SDP, just a they're all in one place.

The local SDP is generated by WebRTC, and generates a recvonly
content/description without an ssrc for both video and audio.

"I have created a parser that converts from the XMPP session-initiate
messages to an SDP that can be placed in a PeerConnection:

Source:http://pastebin.com/g2wcqkmC

SDP generated from session-initiate (IPs changed):http://pastebin.com/fBMzBg4n"

The local SDP is here:

http://pastebin.com/7NGPyyTn

And here is a sample of the session-accept XML I generate from it:

http://pastebin.com/B3wzQSmp

So senders is set to responder only, I believe this is the same as
recvonly, and no sources are generated.

Thanks again.

- Jason.

Ok, so none of your other tests involve a client joining that doesn't send
any data, is that right? What does your native client's answer look like?
Are you signaling any video ssrcs? If there are no ssrcs in the video
mline, chrome will use a media ssrc of "1" by default in rtcp. The bridge
won't accept these unless they are signaled (it will automatically accept
rtp packets from ssrcs it doesn't recognize, but not rtcp). So there are a
couple options:

1) You could put a recvonly ssrc in your answer. This involves just an
a=ssrc line, no msid or anything like that
2) You could add the ssrc '1' to the video channel manually somewhere (in
signaling, for example) so that these packets will be accepted by the
bridge for this client.

Hopefully something there pans out.

···

On Mar 9, 2017 6:42 PM, "Brian Baldino" <brian@jitsi.org> wrote:

On Thu, Mar 9, 2017 at 4:52 PM, Jason Thomas <mail@jasonthom.as> wrote:

"When you join a conference with other participants already in it, how
long is it taking to start showing video? "

It takes a very long time, upwards of 50-60 seconds.

If I have another client join, I start getting frames immediately after
the renegotiate.

"How does it compare to a web client doing the same thing? There
shouldn't be any difference between the two. How customized is your
client?"

It is exactly the same if I am subscribing to a new stream, or the client
has just joined. But only in the case where I join in the middle of an
established conference do I see this delay.

There is no customization, I am using a vanilla WebRTC, Chrome/57 build.

I use pretty much the same strategy on mobile clients with iOS/Android
linking to the same version of WebRTC natively, and haven't experienced any
issues.

- Jason.

On Mar 9, 2017 5:43 PM, "Brian Baldino" <brian@jitsi.org> wrote:

Hey Jason,
Glad you figured some things out. When switching a stream, the bridge
will request an IDR automatically so there should be one coming there. It
won't, however, block forwarding the stream until the IDR comes (that is
only done when the bridge is doing simulcast layer switches), so I would
expect you'd receive some packets before being able to render (though not
long). When you join a conference with other participants already in it,
how long is it taking to start showing video? How does it compare to a web
client doing the same thing? There shouldn't be any difference between the
two. How customized is your client? Is the webrtc stack still handling
all the rtcp feedback for you?

On Thu, Mar 9, 2017 at 2:08 PM, Jason Thomas <mail@jasonthom.as> wrote:

So, actually the tunnel and architecture seems to be a red herring too.

The real difference was how I was testing, in that I was creating a room
with nobody else in it, and then connecting to it.

Basically what happens is the original session-initiate has nothing but
the mixed video/audio synchronization streams.

Then a source-add is sent, and the SDP is amended and the
remoteDescription is set again.

As a result I get the video stream just as it is starting up, I assume,
and BINGO I immediately start decoding frames.

The issue is if I join a room where the videos are already ongoing.

It gets in the original session-initiate the streams, creates them, grabs
the video tracks, and I am still seeing this issue where it sits and takes
forever to find the first decodable stream.

It seems almost like there is some step that will trigger the bridge to
request an new iFrame from all of the clients (Maybe the session-accept)
that isn't being triggered?

Any ideas or help would be very very appreciated.

On Thu, Mar 9, 2017 at 8:45 AM, Jason Thomas <mail@jasonthom.as> wrote:

Okay, after a hellish night of debugging the jitter-buffer, I determined
that the RTP packets were somehow getting clobbered.

I’m on an AMD processor, and behind a VPN tunnel- so maybe something
along the lines is causing this, although the problem doesn’t occur on
Chrome running in the same environment.

I have moved it over to an Intel based machine on AWS- and it works
perfectly without even being recompiled…

I’m wondering if some odd combination of network conditions is causing
it, but regardless- this is just my development environment and not
where I will be running the recorder.

Thanks for helping out with this Brian, I really appreciate your advice
around an issue that turned out to have nearly nothing to do with the JVB
itself :slight_smile:

- Jason.

On Mar 8, 2017, at 4:56 PM, Brian Baldino <brian@jitsi.org> wrote:

Hey Jason,
No worries...sounds like you are on the right track. I don't know if
I've seen that header extension print before. I assume by that point it
has received an SDP that included signaling of abs-send-time, so I wouldn't
think it would choke on it, but I'm not sure. Falling into kIncomplete is
normal until the frame gets complete, but I know it's a pretty tricky
adding logging to that area to print when a frame does become complete, and
even then I think the results you get may not be straightforward...but you
might be able to add something to the 'have-complete-frame' detection logic
that might give some clue. If you've got verbose logging on and can see
when things started working, sometimes you can look at the logs a bit
before that point and they might give some hints as to what occurred to
make things start working.

-brian

On Wed, Mar 8, 2017 at 1:32 PM, Jason Thomas <mail@jasonthom.as> wrote:

Sorry, I misspoke, it is falling into kIncomplete, and then into
kDecodableSession, where it is inserted into incomplete_frames_, and
drained somewhere else.

Basically RTP packets are coming in, I am going to try to focus on
figuring out what changes in here that causes frames to suddenly start
decoding correctly (and continue to do so for the duration of the session.)

Thanks for all of your help and incite so far.

- Jason.

On Wed, Mar 8, 2017 at 2:23 PM, Jason Thomas <mail@jasonthom.as> wrote:

Hi Brian,

   It seems like it's falling into: src/webrtc/modules/video_coding/jitter_buffer.cc:
782

Basically falling into this 'incomplete_frames' section, and nack_mode
is set to kNack, so it just falls into incomplete_frames_.InsertFrame(frame)
continuously.

It seems that the packets that are getting put into it are getting
drained out somewhere.

I am continuously receiving a message: (rtp_utility.cc:342) Failed to
find extension id: 3

Which seems to refer to the rtp-hdrext/abs-send-time

So maybe this is the issue, maybe it can't parse the RTP header for
some reason?

Regarding DTLS, it seems as if all of the encryption stuff is working
appropriately, although sometimes I am getting messages on failed
decryption- but pretty normal for network loss.

- Jason.

On Wed, Mar 8, 2017 at 10:58 AM, Brian Baldino <brian@jitsi.org> >>>>> wrote:

Hey Jason,
Am I understanding you correctly that you're getting blocked at line
140 in receiver.cc for 1 minute? I.e., 'NextCompleteFrame' isn't
returning anything valid for a minute?

If so, I wonder...do you see anything about dtls negotiation
completing? Maybe that is taking a long time for some reason and packets
can't be decrypted correctly for a while? Otherwise it seems odd that
packets would all of a sudden become understandable.

Since you're in the native code already, you might add some prints
lower down to see if things are flowing as expected...for example, is
InsertPacket being called on the jitter buffer? If not, must be something
lower than that, etc..

-brian

On Wed, Mar 8, 2017 at 9:27 AM, Jason Thomas <mail@jasonthom.as> >>>>>> wrote:

Hi Brian,

   I thought perhaps that was the issue too.

looking in webrtc/modules/video_coding/video_receiver.cc the
culprit code is located on line 257:

if( drop_frames_until_keyframe_ ) …

Unfortunately it’s happening even before this, in
_receiver.FrameForDecoding:
webrtc/modules/video_coding/receiver.cc:
140: VCMEncodedFrame *found_frame = jitter_buffer_.NextCompleteFra
me(max_wait_time_ms);

Like it isn’t able to understand the RTP format of the incoming
video packets… Or maybe it’s something to do with encryption.

I’m going to check over the SDP again, it’s possible my parser did
something wrong still and maybe it’s causing the media to be clobbered
somehow?

- Jason.

On Mar 7, 2017, at 3:15 PM, Brian Baldino <brian@jitsi.org> wrote:

I've seen this when a complete frame arrived, but the decoder
couldn't start decoding yet because it hadn't received an IDR. Maybe you
can add some logs to detect when/if a keyframe has been received?

On Tue, Mar 7, 2017 at 2:04 PM, Jason Thomas <mail@jasonthom.as> >>>>>>> wrote:

I'm still trying to get to the bottom of the issues I am seeing
with video frame decoding from the JVB using the native C++ webrtc library.

Using WebRTC remotes/branch-heads/57:
0a71dc836641acdaaa63b2724f83c822dbb58ce7

My bridge is set to use ulpfec and red, and has rtx disabled.

Everything works fine on a mobile version that performs nearly the
same step, but isn't written in C++.

VideoReceiveStream: {decoders: [{decoder: (VideoDecoder),
payload_type: 100, payload_name: VP8, codec_params: {}}], rtp:
{remote_ssrc: 4006849248, local_ssrc: 1, rtcp_mode: RtcpMode::kCompound,
rtcp_xr: {receiver_reference_time_report: off}, remb: on,
transport_cc: off, nack: {rtp_history_ms: 1000}, ulpfec:
{ulpfec_payload_type: 117, red_payload_type: 116, red_rtx_payload_type:
-1}, rtx: {}, extensions: [{uri: http://www.webrtc.org/experime
nts/rtp-hdrext/abs-send-time, id: 3}]}, renderer: (renderer),
render_delay_ms: 10, sync_group: mixedmslabel, pre_decode_callback:
nullptr, pre_render_callback: nullptr, target_delay_ms: 0}

channel.cc:904): Channel writable (video) for the first time
(channel.cc:966): Installing keys from DTLS-SRTP on video RTP
(webrtcvideoengine2.cc:1450): OnReadyToSend: Ready.
(webrtcvideoengine2.cc:994): SetSend: false
(channel.cc:1970): Changing video state, send=0
(rtp_receiver_video.cc:74): Received first video RTP packet

1 Minute later:

(video_receiver.cc:285): Received first complete decodable video
frame
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a

After this I push the frames to ffmpeg, and they encode and look
perfect.... This long delay between receiving the first RTP packet and
being able to decode a full frame seems like the issue.

Again, any ideas or help would be greatly appreciated.

I would be willing to open source a large part of this native
recorder if/when I am able to finish it.

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#15

Ok, so it's definitely possible you're being affected by what I described,
so you'll need to do one of the two options I mentioned before. The
easiest thing will be to add a line like this to your video mline:

a=ssrc:83277272 cname:recvonly-83277272

For reference, here's the entire video mline in that example:

m=video 9 RTP/SAVPF 100 107 96
c=IN IP4 0.0.0.0
a=rtpmap:100 VP8/90000
a=rtpmap:107 h264/90000
a=rtpmap:96 rtx/90000
a=fmtp:107 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=fmtp:96 apt=100
a=rtcp:9 IN IP4 0.0.0.0
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=rtcp-fb:100 goog-remb
a=rtcp-fb:107 ccm fir
a=rtcp-fb:107 nack
a=rtcp-fb:107 nack pli
a=rtcp-fb:107 goog-remb
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=setup:passive
a=mid:video
a=recvonly
a=ice-ufrag:XQXl
a=ice-pwd:14sG9fh3/a3GN5LX1Yq+vfK5
a=fingerprint:sha-256
23:63:82:2A:20:0B:AB:E2:18:AB:EA:84:18:40:C5:B4:D2:A2:EE:21:21:40:AA:0C:18:A9:01:AD:A4:17:61:68
a=ssrc:83277272 cname:recvonly-83277272
a=rtcp-mux

The ssrc itself should just be a randomly generated ssrc, and the cname can
be anything (doesn't have to have the same format as what I pasted there).
Make sure this ssrc gets communicated over xmpp as well. This way, chrome
will use that ssrc as the media ssrc for its rtcp feedback and, because you
signaled it, the bridge will accept the incoming rtcp with that ssrc.

···

On Thu, Mar 9, 2017 at 9:35 PM, Jason Thomas <mail@jasonthom.as> wrote:

Brian,

   Exactly, all of the other clients send both video and audio.

Here are some links from the original post with the Jingle->SDP parser,
and the generated SDP, just a they're all in one place.

The local SDP is generated by WebRTC, and generates a recvonly
content/description without an ssrc for both video and audio.

"I have created a parser that converts from the XMPP session-initiate
messages to an SDP that can be placed in a PeerConnection:

Source:http://pastebin.com/g2wcqkmC

SDP generated from session-initiate (IPs changed):http://pastebin.com/fBMzBg4n"

The local SDP is here:

http://pastebin.com/7NGPyyTn

And here is a sample of the session-accept XML I generate from it:

http://pastebin.com/B3wzQSmp

So senders is set to responder only, I believe this is the same as
recvonly, and no sources are generated.

Thanks again.

- Jason.

On Mar 9, 2017 6:42 PM, "Brian Baldino" <brian@jitsi.org> wrote:

Ok, so none of your other tests involve a client joining that doesn't send
any data, is that right? What does your native client's answer look like?
Are you signaling any video ssrcs? If there are no ssrcs in the video
mline, chrome will use a media ssrc of "1" by default in rtcp. The bridge
won't accept these unless they are signaled (it will automatically accept
rtp packets from ssrcs it doesn't recognize, but not rtcp). So there are a
couple options:

1) You could put a recvonly ssrc in your answer. This involves just an
a=ssrc line, no msid or anything like that
2) You could add the ssrc '1' to the video channel manually somewhere (in
signaling, for example) so that these packets will be accepted by the
bridge for this client.

Hopefully something there pans out.

On Thu, Mar 9, 2017 at 4:52 PM, Jason Thomas <mail@jasonthom.as> wrote:

"When you join a conference with other participants already in it, how
long is it taking to start showing video? "

It takes a very long time, upwards of 50-60 seconds.

If I have another client join, I start getting frames immediately after
the renegotiate.

"How does it compare to a web client doing the same thing? There
shouldn't be any difference between the two. How customized is your
client?"

It is exactly the same if I am subscribing to a new stream, or the client
has just joined. But only in the case where I join in the middle of an
established conference do I see this delay.

There is no customization, I am using a vanilla WebRTC, Chrome/57 build.

I use pretty much the same strategy on mobile clients with iOS/Android
linking to the same version of WebRTC natively, and haven't experienced any
issues.

- Jason.

On Mar 9, 2017 5:43 PM, "Brian Baldino" <brian@jitsi.org> wrote:

Hey Jason,
Glad you figured some things out. When switching a stream, the bridge
will request an IDR automatically so there should be one coming there. It
won't, however, block forwarding the stream until the IDR comes (that is
only done when the bridge is doing simulcast layer switches), so I would
expect you'd receive some packets before being able to render (though not
long). When you join a conference with other participants already in it,
how long is it taking to start showing video? How does it compare to a web
client doing the same thing? There shouldn't be any difference between the
two. How customized is your client? Is the webrtc stack still handling
all the rtcp feedback for you?

On Thu, Mar 9, 2017 at 2:08 PM, Jason Thomas <mail@jasonthom.as> wrote:

So, actually the tunnel and architecture seems to be a red herring too.

The real difference was how I was testing, in that I was creating a room
with nobody else in it, and then connecting to it.

Basically what happens is the original session-initiate has nothing but
the mixed video/audio synchronization streams.

Then a source-add is sent, and the SDP is amended and the
remoteDescription is set again.

As a result I get the video stream just as it is starting up, I assume,
and BINGO I immediately start decoding frames.

The issue is if I join a room where the videos are already ongoing.

It gets in the original session-initiate the streams, creates them,
grabs the video tracks, and I am still seeing this issue where it sits and
takes forever to find the first decodable stream.

It seems almost like there is some step that will trigger the bridge to
request an new iFrame from all of the clients (Maybe the session-accept)
that isn't being triggered?

Any ideas or help would be very very appreciated.

On Thu, Mar 9, 2017 at 8:45 AM, Jason Thomas <mail@jasonthom.as> wrote:

Okay, after a hellish night of debugging the jitter-buffer, I
determined that the RTP packets were somehow getting clobbered.

I’m on an AMD processor, and behind a VPN tunnel- so maybe something
along the lines is causing this, although the problem doesn’t occur on
Chrome running in the same environment.

I have moved it over to an Intel based machine on AWS- and it works
perfectly without even being recompiled…

I’m wondering if some odd combination of network conditions is causing
it, but regardless- this is just my development environment and not
where I will be running the recorder.

Thanks for helping out with this Brian, I really appreciate your advice
around an issue that turned out to have nearly nothing to do with the JVB
itself :slight_smile:

- Jason.

On Mar 8, 2017, at 4:56 PM, Brian Baldino <brian@jitsi.org> wrote:

Hey Jason,
No worries...sounds like you are on the right track. I don't know if
I've seen that header extension print before. I assume by that point it
has received an SDP that included signaling of abs-send-time, so I wouldn't
think it would choke on it, but I'm not sure. Falling into kIncomplete is
normal until the frame gets complete, but I know it's a pretty tricky
adding logging to that area to print when a frame does become complete, and
even then I think the results you get may not be straightforward...but you
might be able to add something to the 'have-complete-frame' detection logic
that might give some clue. If you've got verbose logging on and can see
when things started working, sometimes you can look at the logs a bit
before that point and they might give some hints as to what occurred to
make things start working.

-brian

On Wed, Mar 8, 2017 at 1:32 PM, Jason Thomas <mail@jasonthom.as> wrote:

Sorry, I misspoke, it is falling into kIncomplete, and then into
kDecodableSession, where it is inserted into incomplete_frames_, and
drained somewhere else.

Basically RTP packets are coming in, I am going to try to focus on
figuring out what changes in here that causes frames to suddenly start
decoding correctly (and continue to do so for the duration of the session.)

Thanks for all of your help and incite so far.

- Jason.

On Wed, Mar 8, 2017 at 2:23 PM, Jason Thomas <mail@jasonthom.as> >>>>> wrote:

Hi Brian,

   It seems like it's falling into: src/webrtc/modules/video_coding/jitter_buffer.cc:
782

Basically falling into this 'incomplete_frames' section, and
nack_mode is set to kNack, so it just falls into
incomplete_frames_.InsertFrame(frame) continuously.

It seems that the packets that are getting put into it are getting
drained out somewhere.

I am continuously receiving a message: (rtp_utility.cc:342) Failed to
find extension id: 3

Which seems to refer to the rtp-hdrext/abs-send-time

So maybe this is the issue, maybe it can't parse the RTP header for
some reason?

Regarding DTLS, it seems as if all of the encryption stuff is working
appropriately, although sometimes I am getting messages on failed
decryption- but pretty normal for network loss.

- Jason.

On Wed, Mar 8, 2017 at 10:58 AM, Brian Baldino <brian@jitsi.org> >>>>>> wrote:

Hey Jason,
Am I understanding you correctly that you're getting blocked at line
140 in receiver.cc for 1 minute? I.e., 'NextCompleteFrame' isn't
returning anything valid for a minute?

If so, I wonder...do you see anything about dtls negotiation
completing? Maybe that is taking a long time for some reason and packets
can't be decrypted correctly for a while? Otherwise it seems odd that
packets would all of a sudden become understandable.

Since you're in the native code already, you might add some prints
lower down to see if things are flowing as expected...for example, is
InsertPacket being called on the jitter buffer? If not, must be something
lower than that, etc..

-brian

On Wed, Mar 8, 2017 at 9:27 AM, Jason Thomas <mail@jasonthom.as> >>>>>>> wrote:

Hi Brian,

   I thought perhaps that was the issue too.

looking in webrtc/modules/video_coding/video_receiver.cc the
culprit code is located on line 257:

if( drop_frames_until_keyframe_ ) …

Unfortunately it’s happening even before this, in
_receiver.FrameForDecoding:
webrtc/modules/video_coding/receiver.cc:
140: VCMEncodedFrame *found_frame = jitter_buffer_.NextCompleteFra
me(max_wait_time_ms);

Like it isn’t able to understand the RTP format of the incoming
video packets… Or maybe it’s something to do with encryption.

I’m going to check over the SDP again, it’s possible my parser did
something wrong still and maybe it’s causing the media to be clobbered
somehow?

- Jason.

On Mar 7, 2017, at 3:15 PM, Brian Baldino <brian@jitsi.org> wrote:

I've seen this when a complete frame arrived, but the decoder
couldn't start decoding yet because it hadn't received an IDR. Maybe you
can add some logs to detect when/if a keyframe has been received?

On Tue, Mar 7, 2017 at 2:04 PM, Jason Thomas <mail@jasonthom.as> >>>>>>>> wrote:

I'm still trying to get to the bottom of the issues I am seeing
with video frame decoding from the JVB using the native C++ webrtc library.

Using WebRTC remotes/branch-heads/57:
0a71dc836641acdaaa63b2724f83c822dbb58ce7

My bridge is set to use ulpfec and red, and has rtx disabled.

Everything works fine on a mobile version that performs nearly the
same step, but isn't written in C++.

VideoReceiveStream: {decoders: [{decoder: (VideoDecoder),
payload_type: 100, payload_name: VP8, codec_params: {}}], rtp:
{remote_ssrc: 4006849248, local_ssrc: 1, rtcp_mode: RtcpMode::kCompound,
rtcp_xr: {receiver_reference_time_report: off}, remb: on,
transport_cc: off, nack: {rtp_history_ms: 1000}, ulpfec:
{ulpfec_payload_type: 117, red_payload_type: 116, red_rtx_payload_type:
-1}, rtx: {}, extensions: [{uri: http://www.webrtc.org/experime
nts/rtp-hdrext/abs-send-time, id: 3}]}, renderer: (renderer),
render_delay_ms: 10, sync_group: mixedmslabel, pre_decode_callback:
nullptr, pre_render_callback: nullptr, target_delay_ms: 0}

channel.cc:904): Channel writable (video) for the first time
(channel.cc:966): Installing keys from DTLS-SRTP on video RTP
(webrtcvideoengine2.cc:1450): OnReadyToSend: Ready.
(webrtcvideoengine2.cc:994): SetSend: false
(channel.cc:1970): Changing video state, send=0
(rtp_receiver_video.cc:74): Received first video RTP packet

1 Minute later:

(video_receiver.cc:285): Received first complete decodable video
frame
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a

After this I push the frames to ffmpeg, and they encode and look
perfect.... This long delay between receiving the first RTP packet and
being able to decode a full frame seems like the issue.

Again, any ideas or help would be greatly appreciated.

I would be willing to open source a large part of this native
recorder if/when I am able to finish it.

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#16

I hope I'm not declaring premature victory again, but preliminary results
seem to show that that did the trick! Thanks so much Brian!

- Jason.

···

On Fri, Mar 10, 2017 at 11:25 AM, Brian Baldino <brian@jitsi.org> wrote:

Ok, so it's definitely possible you're being affected by what I described,
so you'll need to do one of the two options I mentioned before. The
easiest thing will be to add a line like this to your video mline:

a=ssrc:83277272 cname:recvonly-83277272

For reference, here's the entire video mline in that example:

m=video 9 RTP/SAVPF 100 107 96
c=IN IP4 0.0.0.0
a=rtpmap:100 VP8/90000
a=rtpmap:107 h264/90000
a=rtpmap:96 rtx/90000
a=fmtp:107 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=fmtp:96 apt=100
a=rtcp:9 IN IP4 0.0.0.0
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=rtcp-fb:100 goog-remb
a=rtcp-fb:107 ccm fir
a=rtcp-fb:107 nack
a=rtcp-fb:107 nack pli
a=rtcp-fb:107 goog-remb
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=setup:passive
a=mid:video
a=recvonly
a=ice-ufrag:XQXl
a=ice-pwd:14sG9fh3/a3GN5LX1Yq+vfK5
a=fingerprint:sha-256 <http://www.webrtc.org/experiments/rtp-hdrext/abs-send-timea=setup:passivea=mid:videoa=recvonlya=ice-ufrag:XQXla=ice-pwd:14sG9fh3/a3GN5LX1Yq+vfK5a=fingerprint:sha-256> 23:63:82:2A:20:0B:AB:E2:18:AB:EA:84:18:40:C5:B4:D2:A2:EE:21:21:40:AA:0C:18:A9:01:AD:A4:17:61:68
a=ssrc:83277272 cname:recvonly-83277272
a=rtcp-mux

The ssrc itself should just be a randomly generated ssrc, and the cname
can be anything (doesn't have to have the same format as what I pasted
there). Make sure this ssrc gets communicated over xmpp as well. This
way, chrome will use that ssrc as the media ssrc for its rtcp feedback and,
because you signaled it, the bridge will accept the incoming rtcp with that
ssrc.

On Thu, Mar 9, 2017 at 9:35 PM, Jason Thomas <mail@jasonthom.as> wrote:

Brian,

   Exactly, all of the other clients send both video and audio.

Here are some links from the original post with the Jingle->SDP parser,
and the generated SDP, just a they're all in one place.

The local SDP is generated by WebRTC, and generates a recvonly
content/description without an ssrc for both video and audio.

"I have created a parser that converts from the XMPP session-initiate
messages to an SDP that can be placed in a PeerConnection:

Source:http://pastebin.com/g2wcqkmC

SDP generated from session-initiate (IPs changed):http://pastebin.com/fBMzBg4n"

The local SDP is here:

http://pastebin.com/7NGPyyTn

And here is a sample of the session-accept XML I generate from it:

http://pastebin.com/B3wzQSmp

So senders is set to responder only, I believe this is the same as
recvonly, and no sources are generated.

Thanks again.

- Jason.

On Mar 9, 2017 6:42 PM, "Brian Baldino" <brian@jitsi.org> wrote:

Ok, so none of your other tests involve a client joining that doesn't
send any data, is that right? What does your native client's answer look
like? Are you signaling any video ssrcs? If there are no ssrcs in the
video mline, chrome will use a media ssrc of "1" by default in rtcp. The
bridge won't accept these unless they are signaled (it will automatically
accept rtp packets from ssrcs it doesn't recognize, but not rtcp). So
there are a couple options:

1) You could put a recvonly ssrc in your answer. This involves just an
a=ssrc line, no msid or anything like that
2) You could add the ssrc '1' to the video channel manually somewhere (in
signaling, for example) so that these packets will be accepted by the
bridge for this client.

Hopefully something there pans out.

On Thu, Mar 9, 2017 at 4:52 PM, Jason Thomas <mail@jasonthom.as> wrote:

"When you join a conference with other participants already in it, how
long is it taking to start showing video? "

It takes a very long time, upwards of 50-60 seconds.

If I have another client join, I start getting frames immediately after
the renegotiate.

"How does it compare to a web client doing the same thing? There
shouldn't be any difference between the two. How customized is your
client?"

It is exactly the same if I am subscribing to a new stream, or the
client has just joined. But only in the case where I join in the middle of
an established conference do I see this delay.

There is no customization, I am using a vanilla WebRTC, Chrome/57 build.

I use pretty much the same strategy on mobile clients with iOS/Android
linking to the same version of WebRTC natively, and haven't experienced any
issues.

- Jason.

On Mar 9, 2017 5:43 PM, "Brian Baldino" <brian@jitsi.org> wrote:

Hey Jason,
Glad you figured some things out. When switching a stream, the bridge
will request an IDR automatically so there should be one coming there. It
won't, however, block forwarding the stream until the IDR comes (that is
only done when the bridge is doing simulcast layer switches), so I would
expect you'd receive some packets before being able to render (though not
long). When you join a conference with other participants already in it,
how long is it taking to start showing video? How does it compare to a web
client doing the same thing? There shouldn't be any difference between the
two. How customized is your client? Is the webrtc stack still handling
all the rtcp feedback for you?

On Thu, Mar 9, 2017 at 2:08 PM, Jason Thomas <mail@jasonthom.as> wrote:

So, actually the tunnel and architecture seems to be a red herring too.

The real difference was how I was testing, in that I was creating a
room with nobody else in it, and then connecting to it.

Basically what happens is the original session-initiate has nothing but
the mixed video/audio synchronization streams.

Then a source-add is sent, and the SDP is amended and the
remoteDescription is set again.

As a result I get the video stream just as it is starting up, I assume,
and BINGO I immediately start decoding frames.

The issue is if I join a room where the videos are already ongoing.

It gets in the original session-initiate the streams, creates them,
grabs the video tracks, and I am still seeing this issue where it sits and
takes forever to find the first decodable stream.

It seems almost like there is some step that will trigger the bridge to
request an new iFrame from all of the clients (Maybe the session-accept)
that isn't being triggered?

Any ideas or help would be very very appreciated.

On Thu, Mar 9, 2017 at 8:45 AM, Jason Thomas <mail@jasonthom.as> wrote:

Okay, after a hellish night of debugging the jitter-buffer, I
determined that the RTP packets were somehow getting clobbered.

I’m on an AMD processor, and behind a VPN tunnel- so maybe something
along the lines is causing this, although the problem doesn’t occur on
Chrome running in the same environment.

I have moved it over to an Intel based machine on AWS- and it works
perfectly without even being recompiled…

I’m wondering if some odd combination of network conditions is causing
it, but regardless- this is just my development environment and not
where I will be running the recorder.

Thanks for helping out with this Brian, I really appreciate your
advice around an issue that turned out to have nearly nothing to do with
the JVB itself :slight_smile:

- Jason.

On Mar 8, 2017, at 4:56 PM, Brian Baldino <brian@jitsi.org> wrote:

Hey Jason,
No worries...sounds like you are on the right track. I don't know if
I've seen that header extension print before. I assume by that point it
has received an SDP that included signaling of abs-send-time, so I wouldn't
think it would choke on it, but I'm not sure. Falling into kIncomplete is
normal until the frame gets complete, but I know it's a pretty tricky
adding logging to that area to print when a frame does become complete, and
even then I think the results you get may not be straightforward...but you
might be able to add something to the 'have-complete-frame' detection logic
that might give some clue. If you've got verbose logging on and can see
when things started working, sometimes you can look at the logs a bit
before that point and they might give some hints as to what occurred to
make things start working.

-brian

On Wed, Mar 8, 2017 at 1:32 PM, Jason Thomas <mail@jasonthom.as> >>>>> wrote:

Sorry, I misspoke, it is falling into kIncomplete, and then into
kDecodableSession, where it is inserted into incomplete_frames_, and
drained somewhere else.

Basically RTP packets are coming in, I am going to try to focus on
figuring out what changes in here that causes frames to suddenly start
decoding correctly (and continue to do so for the duration of the session.)

Thanks for all of your help and incite so far.

- Jason.

On Wed, Mar 8, 2017 at 2:23 PM, Jason Thomas <mail@jasonthom.as> >>>>>> wrote:

Hi Brian,

   It seems like it's falling into: src/webrtc/modules/video_coding/jitter_buffer.cc:
782

Basically falling into this 'incomplete_frames' section, and
nack_mode is set to kNack, so it just falls into
incomplete_frames_.InsertFrame(frame) continuously.

It seems that the packets that are getting put into it are getting
drained out somewhere.

I am continuously receiving a message: (rtp_utility.cc:342) Failed
to find extension id: 3

Which seems to refer to the rtp-hdrext/abs-send-time

So maybe this is the issue, maybe it can't parse the RTP header for
some reason?

Regarding DTLS, it seems as if all of the encryption stuff is
working appropriately, although sometimes I am getting messages on failed
decryption- but pretty normal for network loss.

- Jason.

On Wed, Mar 8, 2017 at 10:58 AM, Brian Baldino <brian@jitsi.org> >>>>>>> wrote:

Hey Jason,
Am I understanding you correctly that you're getting blocked at
line 140 in receiver.cc for 1 minute? I.e., 'NextCompleteFrame'
isn't returning anything valid for a minute?

If so, I wonder...do you see anything about dtls negotiation
completing? Maybe that is taking a long time for some reason and packets
can't be decrypted correctly for a while? Otherwise it seems odd that
packets would all of a sudden become understandable.

Since you're in the native code already, you might add some prints
lower down to see if things are flowing as expected...for example, is
InsertPacket being called on the jitter buffer? If not, must be something
lower than that, etc..

-brian

On Wed, Mar 8, 2017 at 9:27 AM, Jason Thomas <mail@jasonthom.as> >>>>>>>> wrote:

Hi Brian,

   I thought perhaps that was the issue too.

looking in webrtc/modules/video_coding/video_receiver.cc the
culprit code is located on line 257:

if( drop_frames_until_keyframe_ ) …

Unfortunately it’s happening even before this, in
_receiver.FrameForDecoding:
webrtc/modules/video_coding/receiver.cc:
140: VCMEncodedFrame *found_frame = jitter_buffer_.NextCompleteFra
me(max_wait_time_ms);

Like it isn’t able to understand the RTP format of the incoming
video packets… Or maybe it’s something to do with encryption.

I’m going to check over the SDP again, it’s possible my parser did
something wrong still and maybe it’s causing the media to be clobbered
somehow?

- Jason.

On Mar 7, 2017, at 3:15 PM, Brian Baldino <brian@jitsi.org> wrote:

I've seen this when a complete frame arrived, but the decoder
couldn't start decoding yet because it hadn't received an IDR. Maybe you
can add some logs to detect when/if a keyframe has been received?

On Tue, Mar 7, 2017 at 2:04 PM, Jason Thomas <mail@jasonthom.as> >>>>>>>>> wrote:

I'm still trying to get to the bottom of the issues I am seeing
with video frame decoding from the JVB using the native C++ webrtc library.

Using WebRTC remotes/branch-heads/57:
0a71dc836641acdaaa63b2724f83c822dbb58ce7

My bridge is set to use ulpfec and red, and has rtx disabled.

Everything works fine on a mobile version that performs nearly
the same step, but isn't written in C++.

VideoReceiveStream: {decoders: [{decoder: (VideoDecoder),
payload_type: 100, payload_name: VP8, codec_params: {}}], rtp:
{remote_ssrc: 4006849248, local_ssrc: 1, rtcp_mode: RtcpMode::kCompound,
rtcp_xr: {receiver_reference_time_report: off}, remb: on,
transport_cc: off, nack: {rtp_history_ms: 1000}, ulpfec:
{ulpfec_payload_type: 117, red_payload_type: 116, red_rtx_payload_type:
-1}, rtx: {}, extensions: [{uri: http://www.webrtc.org/experime
nts/rtp-hdrext/abs-send-time, id: 3}]}, renderer: (renderer),
render_delay_ms: 10, sync_group: mixedmslabel, pre_decode_callback:
nullptr, pre_render_callback: nullptr, target_delay_ms: 0}

channel.cc:904): Channel writable (video) for the first time
(channel.cc:966): Installing keys from DTLS-SRTP on video RTP
(webrtcvideoengine2.cc:1450): OnReadyToSend: Ready.
(webrtcvideoengine2.cc:994): SetSend: false
(channel.cc:1970): Changing video state, send=0
(rtp_receiver_video.cc:74): Received first video RTP packet

1 Minute later:

(video_receiver.cc:285): Received first complete decodable video
frame
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a

After this I push the frames to ffmpeg, and they encode and look
perfect.... This long delay between receiving the first RTP packet and
being able to decode a full frame seems like the issue.

Again, any ideas or help would be greatly appreciated.

I would be willing to open source a large part of this native
recorder if/when I am able to finish it.

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as


#17

No prob...glad it seems to be working!

···

On Sun, Mar 12, 2017 at 8:55 PM, Jason Thomas <mail@jasonthom.as> wrote:

I hope I'm not declaring premature victory again, but preliminary results
seem to show that that did the trick! Thanks so much Brian!

- Jason.

On Fri, Mar 10, 2017 at 11:25 AM, Brian Baldino <brian@jitsi.org> wrote:

Ok, so it's definitely possible you're being affected by what I
described, so you'll need to do one of the two options I mentioned before.
The easiest thing will be to add a line like this to your video mline:

a=ssrc:83277272 cname:recvonly-83277272

For reference, here's the entire video mline in that example:

m=video 9 RTP/SAVPF 100 107 96
c=IN IP4 0.0.0.0
a=rtpmap:100 VP8/90000
a=rtpmap:107 h264/90000
a=rtpmap:96 rtx/90000
a=fmtp:107 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=fmtp:96 apt=100
a=rtcp:9 IN IP4 0.0.0.0
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=rtcp-fb:100 goog-remb
a=rtcp-fb:107 ccm fir
a=rtcp-fb:107 nack
a=rtcp-fb:107 nack pli
a=rtcp-fb:107 goog-remb
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=setup:passive
a=mid:video
a=recvonly
a=ice-ufrag:XQXl
a=ice-pwd:14sG9fh3/a3GN5LX1Yq+vfK5
a=fingerprint:sha-256 <http://www.webrtc.org/experiments/rtp-hdrext/abs-send-timea=setup:passivea=mid:videoa=recvonlya=ice-ufrag:XQXla=ice-pwd:14sG9fh3/a3GN5LX1Yq+vfK5a=fingerprint:sha-256> 23:63:82:2A:20:0B:AB:E2:18:AB:EA:84:18:40:C5:B4:D2:A2:EE:21:21:40:AA:0C:18:A9:01:AD:A4:17:61:68
a=ssrc:83277272 cname:recvonly-83277272
a=rtcp-mux

The ssrc itself should just be a randomly generated ssrc, and the cname
can be anything (doesn't have to have the same format as what I pasted
there). Make sure this ssrc gets communicated over xmpp as well. This
way, chrome will use that ssrc as the media ssrc for its rtcp feedback and,
because you signaled it, the bridge will accept the incoming rtcp with that
ssrc.

On Thu, Mar 9, 2017 at 9:35 PM, Jason Thomas <mail@jasonthom.as> wrote:

Brian,

   Exactly, all of the other clients send both video and audio.

Here are some links from the original post with the Jingle->SDP parser,
and the generated SDP, just a they're all in one place.

The local SDP is generated by WebRTC, and generates a recvonly
content/description without an ssrc for both video and audio.

"I have created a parser that converts from the XMPP session-initiate
messages to an SDP that can be placed in a PeerConnection:

Source:http://pastebin.com/g2wcqkmC

SDP generated from session-initiate (IPs changed):http://pastebin.com/fBMzBg4n"

The local SDP is here:

http://pastebin.com/7NGPyyTn

And here is a sample of the session-accept XML I generate from it:

http://pastebin.com/B3wzQSmp

So senders is set to responder only, I believe this is the same as
recvonly, and no sources are generated.

Thanks again.

- Jason.

On Mar 9, 2017 6:42 PM, "Brian Baldino" <brian@jitsi.org> wrote:

Ok, so none of your other tests involve a client joining that doesn't
send any data, is that right? What does your native client's answer look
like? Are you signaling any video ssrcs? If there are no ssrcs in the
video mline, chrome will use a media ssrc of "1" by default in rtcp. The
bridge won't accept these unless they are signaled (it will automatically
accept rtp packets from ssrcs it doesn't recognize, but not rtcp). So
there are a couple options:

1) You could put a recvonly ssrc in your answer. This involves just an
a=ssrc line, no msid or anything like that
2) You could add the ssrc '1' to the video channel manually somewhere
(in signaling, for example) so that these packets will be accepted by the
bridge for this client.

Hopefully something there pans out.

On Thu, Mar 9, 2017 at 4:52 PM, Jason Thomas <mail@jasonthom.as> wrote:

"When you join a conference with other participants already in it, how
long is it taking to start showing video? "

It takes a very long time, upwards of 50-60 seconds.

If I have another client join, I start getting frames immediately after
the renegotiate.

"How does it compare to a web client doing the same thing? There
shouldn't be any difference between the two. How customized is your
client?"

It is exactly the same if I am subscribing to a new stream, or the
client has just joined. But only in the case where I join in the middle of
an established conference do I see this delay.

There is no customization, I am using a vanilla WebRTC, Chrome/57 build.

I use pretty much the same strategy on mobile clients with iOS/Android
linking to the same version of WebRTC natively, and haven't experienced any
issues.

- Jason.

On Mar 9, 2017 5:43 PM, "Brian Baldino" <brian@jitsi.org> wrote:

Hey Jason,
Glad you figured some things out. When switching a stream, the bridge
will request an IDR automatically so there should be one coming there. It
won't, however, block forwarding the stream until the IDR comes (that is
only done when the bridge is doing simulcast layer switches), so I would
expect you'd receive some packets before being able to render (though not
long). When you join a conference with other participants already in it,
how long is it taking to start showing video? How does it compare to a web
client doing the same thing? There shouldn't be any difference between the
two. How customized is your client? Is the webrtc stack still handling
all the rtcp feedback for you?

On Thu, Mar 9, 2017 at 2:08 PM, Jason Thomas <mail@jasonthom.as> wrote:

So, actually the tunnel and architecture seems to be a red herring too.

The real difference was how I was testing, in that I was creating a
room with nobody else in it, and then connecting to it.

Basically what happens is the original session-initiate has nothing
but the mixed video/audio synchronization streams.

Then a source-add is sent, and the SDP is amended and the
remoteDescription is set again.

As a result I get the video stream just as it is starting up, I
assume, and BINGO I immediately start decoding frames.

The issue is if I join a room where the videos are already ongoing.

It gets in the original session-initiate the streams, creates them,
grabs the video tracks, and I am still seeing this issue where it sits and
takes forever to find the first decodable stream.

It seems almost like there is some step that will trigger the bridge
to request an new iFrame from all of the clients (Maybe the session-accept)
that isn't being triggered?

Any ideas or help would be very very appreciated.

On Thu, Mar 9, 2017 at 8:45 AM, Jason Thomas <mail@jasonthom.as> >>>>> wrote:

Okay, after a hellish night of debugging the jitter-buffer, I
determined that the RTP packets were somehow getting clobbered.

I’m on an AMD processor, and behind a VPN tunnel- so maybe something
along the lines is causing this, although the problem doesn’t occur on
Chrome running in the same environment.

I have moved it over to an Intel based machine on AWS- and it works
perfectly without even being recompiled…

I’m wondering if some odd combination of network conditions is
causing it, but regardless- this is just my development environment and not
where I will be running the recorder.

Thanks for helping out with this Brian, I really appreciate your
advice around an issue that turned out to have nearly nothing to do with
the JVB itself :slight_smile:

- Jason.

On Mar 8, 2017, at 4:56 PM, Brian Baldino <brian@jitsi.org> wrote:

Hey Jason,
No worries...sounds like you are on the right track. I don't know if
I've seen that header extension print before. I assume by that point it
has received an SDP that included signaling of abs-send-time, so I wouldn't
think it would choke on it, but I'm not sure. Falling into kIncomplete is
normal until the frame gets complete, but I know it's a pretty tricky
adding logging to that area to print when a frame does become complete, and
even then I think the results you get may not be straightforward...but you
might be able to add something to the 'have-complete-frame' detection logic
that might give some clue. If you've got verbose logging on and can see
when things started working, sometimes you can look at the logs a bit
before that point and they might give some hints as to what occurred to
make things start working.

-brian

On Wed, Mar 8, 2017 at 1:32 PM, Jason Thomas <mail@jasonthom.as> >>>>>> wrote:

Sorry, I misspoke, it is falling into kIncomplete, and then into
kDecodableSession, where it is inserted into incomplete_frames_, and
drained somewhere else.

Basically RTP packets are coming in, I am going to try to focus on
figuring out what changes in here that causes frames to suddenly start
decoding correctly (and continue to do so for the duration of the session.)

Thanks for all of your help and incite so far.

- Jason.

On Wed, Mar 8, 2017 at 2:23 PM, Jason Thomas <mail@jasonthom.as> >>>>>>> wrote:

Hi Brian,

   It seems like it's falling into: src/webrtc/modules/video_coding/jitter_buffer.cc:
782

Basically falling into this 'incomplete_frames' section, and
nack_mode is set to kNack, so it just falls into
incomplete_frames_.InsertFrame(frame) continuously.

It seems that the packets that are getting put into it are getting
drained out somewhere.

I am continuously receiving a message: (rtp_utility.cc:342) Failed
to find extension id: 3

Which seems to refer to the rtp-hdrext/abs-send-time

So maybe this is the issue, maybe it can't parse the RTP header for
some reason?

Regarding DTLS, it seems as if all of the encryption stuff is
working appropriately, although sometimes I am getting messages on failed
decryption- but pretty normal for network loss.

- Jason.

On Wed, Mar 8, 2017 at 10:58 AM, Brian Baldino <brian@jitsi.org> >>>>>>>> wrote:

Hey Jason,
Am I understanding you correctly that you're getting blocked at
line 140 in receiver.cc for 1 minute? I.e., 'NextCompleteFrame'
isn't returning anything valid for a minute?

If so, I wonder...do you see anything about dtls negotiation
completing? Maybe that is taking a long time for some reason and packets
can't be decrypted correctly for a while? Otherwise it seems odd that
packets would all of a sudden become understandable.

Since you're in the native code already, you might add some prints
lower down to see if things are flowing as expected...for example, is
InsertPacket being called on the jitter buffer? If not, must be something
lower than that, etc..

-brian

On Wed, Mar 8, 2017 at 9:27 AM, Jason Thomas <mail@jasonthom.as> >>>>>>>>> wrote:

Hi Brian,

   I thought perhaps that was the issue too.

looking in webrtc/modules/video_coding/video_receiver.cc the
culprit code is located on line 257:

if( drop_frames_until_keyframe_ ) …

Unfortunately it’s happening even before this, in
_receiver.FrameForDecoding:
webrtc/modules/video_coding/receiver.cc:
140: VCMEncodedFrame *found_frame = jitter_buffer_.NextCompleteFra
me(max_wait_time_ms);

Like it isn’t able to understand the RTP format of the incoming
video packets… Or maybe it’s something to do with encryption.

I’m going to check over the SDP again, it’s possible my parser
did something wrong still and maybe it’s causing the media to be clobbered
somehow?

- Jason.

On Mar 7, 2017, at 3:15 PM, Brian Baldino <brian@jitsi.org> >>>>>>>>>> wrote:

I've seen this when a complete frame arrived, but the decoder
couldn't start decoding yet because it hadn't received an IDR. Maybe you
can add some logs to detect when/if a keyframe has been received?

On Tue, Mar 7, 2017 at 2:04 PM, Jason Thomas <mail@jasonthom.as> >>>>>>>>>> wrote:

I'm still trying to get to the bottom of the issues I am seeing
with video frame decoding from the JVB using the native C++ webrtc library.

Using WebRTC remotes/branch-heads/57:
0a71dc836641acdaaa63b2724f83c822dbb58ce7

My bridge is set to use ulpfec and red, and has rtx disabled.

Everything works fine on a mobile version that performs nearly
the same step, but isn't written in C++.

VideoReceiveStream: {decoders: [{decoder: (VideoDecoder),
payload_type: 100, payload_name: VP8, codec_params: {}}], rtp:
{remote_ssrc: 4006849248, local_ssrc: 1, rtcp_mode: RtcpMode::kCompound,
rtcp_xr: {receiver_reference_time_report: off}, remb: on,
transport_cc: off, nack: {rtp_history_ms: 1000}, ulpfec:
{ulpfec_payload_type: 117, red_payload_type: 116, red_rtx_payload_type:
-1}, rtx: {}, extensions: [{uri: http://www.webrtc.org/experime
nts/rtp-hdrext/abs-send-time, id: 3}]}, renderer: (renderer),
render_delay_ms: 10, sync_group: mixedmslabel, pre_decode_callback:
nullptr, pre_render_callback: nullptr, target_delay_ms: 0}

channel.cc:904): Channel writable (video) for the first time
(channel.cc:966): Installing keys from DTLS-SRTP on video RTP
(webrtcvideoengine2.cc:1450): OnReadyToSend: Ready.
(webrtcvideoengine2.cc:994): SetSend: false
(channel.cc:1970): Changing video state, send=0
(rtp_receiver_video.cc:74): Received first video RTP packet

1 Minute later:

(video_receiver.cc:285): Received first complete decodable video
frame
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a
video FEED FRAME 79a4ec9c-45d2-49f4-ba40-e65664d1200a

After this I push the frames to ffmpeg, and they encode and look
perfect.... This long delay between receiving the first RTP packet and
being able to decode a full frame seems like the issue.

Again, any ideas or help would be greatly appreciated.

I would be willing to open source a large part of this native
recorder if/when I am able to finish it.

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
- Jason Thomas
   http://jasonthom.as

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev