[sip-comm-dev] Video call delays


#1

Hi all

I (and probably others) notice that on video call (encoded with H264), video is delayed arround 5-6 seconds. My goal is to reduce as much as possible this delay to provide more "instant" video calls.

I search through codebase and I notice impl.neomedia.codec.video.h264.Packetize class.

After establishing video call, add and start streams (MediaStreamImpl::startSendStreams()), SC "packetizes" H264 data in Packetize::process() and the encoded data is sent on the network in RTPConnectorOutputStream::write. But the first "write" happens 5-6 seconds after streams have been started. It appears that during 5-6 seconds each call to Packetizer::process(inBuffer, outBuffer), inBuffer.getLength() == 0 and so method ends up in if(inputLength < 10) {... return BUFFER_PROCESSED_OK) }.

Finally after 5-6 seconds, inBuffer have data (length > 0) and process method ends up with INPUT_BUFFER_NOT_CONSUMED | OUTPUT_BUFFER_NOT_FILLED and the first RTP is sent over the network.

If someone has an idea why data (I would say 5 seconds delayed data) takes times to go to Packetizer (why an empty inBuffer in Packetizer::process()).

I will keep you updated if I find something.

Regards,

···

--
Seb

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#2

Hi,

you can get net.sf.fmj.media.protocol.civil.DataSource from fmj
sources put a print in MyCaptureObserver::onNewImage for buffer length
comming from lti-civil and replace the class in the fmj.jar we use
just for tests. So you can see when actual buffers start to come from
capture device into jmf and see is the capture started later or for
some reason jmf discards some of the data for 5-6 seconds or buffers
it or something else.

damencho

···

On Mon, Dec 21, 2009 at 5:06 PM, Sebastien Vincent <seb@sip-communicator.org> wrote:

Hi all

I (and probably others) notice that on video call (encoded with H264), video
is delayed arround 5-6 seconds. My goal is to reduce as much as possible
this delay to provide more "instant" video calls.

I search through codebase and I notice
impl.neomedia.codec.video.h264.Packetize class.

After establishing video call, add and start streams
(MediaStreamImpl::startSendStreams()), SC "packetizes" H264 data in
Packetize::process() and the encoded data is sent on the network in
RTPConnectorOutputStream::write. But the first "write" happens 5-6 seconds
after streams have been started. It appears that during 5-6 seconds each
call to Packetizer::process(inBuffer, outBuffer), inBuffer.getLength() == 0
and so method ends up in if(inputLength < 10) {... return
BUFFER_PROCESSED_OK) }.

Finally after 5-6 seconds, inBuffer have data (length > 0) and process
method ends up with INPUT_BUFFER_NOT_CONSUMED | OUTPUT_BUFFER_NOT_FILLED and
the first RTP is sent over the network.

If someone has an idea why data (I would say 5 seconds delayed data) takes
times to go to Packetizer (why an empty inBuffer in Packetizer::process()).

I will keep you updated if I find something.

Regards,
--
Seb

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#3

Hi Damian,

Thanks for your suggestion.

It appears that fmj's buffer (on newImage) has data (921600 bytes) and packets are not dropped (handler.transferData(this) always happen). I will investigate more.

···

--
Seb

Damian Minkov a �crit :

Hi,

you can get net.sf.fmj.media.protocol.civil.DataSource from fmj
sources put a print in MyCaptureObserver::onNewImage for buffer length
comming from lti-civil and replace the class in the fmj.jar we use
just for tests. So you can see when actual buffers start to come from
capture device into jmf and see is the capture started later or for
some reason jmf discards some of the data for 5-6 seconds or buffers
it or something else.

damencho

On Mon, Dec 21, 2009 at 5:06 PM, Sebastien Vincent > <seb@sip-communicator.org> wrote:
  

Hi all

I (and probably others) notice that on video call (encoded with H264), video
is delayed arround 5-6 seconds. My goal is to reduce as much as possible
this delay to provide more "instant" video calls.

I search through codebase and I notice
impl.neomedia.codec.video.h264.Packetize class.

After establishing video call, add and start streams
(MediaStreamImpl::startSendStreams()), SC "packetizes" H264 data in
Packetize::process() and the encoded data is sent on the network in
RTPConnectorOutputStream::write. But the first "write" happens 5-6 seconds
after streams have been started. It appears that during 5-6 seconds each
call to Packetizer::process(inBuffer, outBuffer), inBuffer.getLength() == 0
and so method ends up in if(inputLength < 10) {... return
BUFFER_PROCESSED_OK) }.

Finally after 5-6 seconds, inBuffer have data (length > 0) and process
method ends up with INPUT_BUFFER_NOT_CONSUMED | OUTPUT_BUFFER_NOT_FILLED and
the first RTP is sent over the network.

If someone has an idea why data (I would say 5 seconds delayed data) takes
times to go to Packetizer (why an empty inBuffer in Packetizer::process()).

I will keep you updated if I find something.

Regards,
--
Seb

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#4

Hi all,

After debug/tracing in SC/JMF, I found the cause of video delay at start (with H264). It is generated by JNIEncoder::process, more specifically by native call FFMPEG.avcodec_encode_video which always returns 0 during these 5-6 seconds.

I study FFMPEG initialization parameters in JNIEncoder.open() and by changing
FFMPEG.avcodeccontext_set_rc_buffer_size(avcontext, 10000000);
to
FFMPEG.avcodeccontext_set_rc_buffer_size(avcontext, 0);

I have more "instant" video calls, it still happens some freeze and video acceleration some time but this is another story I will take care.

So now my question is why setting 10000000 for rc_buffer ? is it wanted ?

Regards,

···

--
Seb

Damian Minkov a �crit :

Hi,

you can get net.sf.fmj.media.protocol.civil.DataSource from fmj
sources put a print in MyCaptureObserver::onNewImage for buffer length
comming from lti-civil and replace the class in the fmj.jar we use
just for tests. So you can see when actual buffers start to come from
capture device into jmf and see is the capture started later or for
some reason jmf discards some of the data for 5-6 seconds or buffers
it or something else.

damencho

On Mon, Dec 21, 2009 at 5:06 PM, Sebastien Vincent > <seb@sip-communicator.org> wrote:
  

Hi all

I (and probably others) notice that on video call (encoded with H264), video
is delayed arround 5-6 seconds. My goal is to reduce as much as possible
this delay to provide more "instant" video calls.

I search through codebase and I notice
impl.neomedia.codec.video.h264.Packetize class.

After establishing video call, add and start streams
(MediaStreamImpl::startSendStreams()), SC "packetizes" H264 data in
Packetize::process() and the encoded data is sent on the network in
RTPConnectorOutputStream::write. But the first "write" happens 5-6 seconds
after streams have been started. It appears that during 5-6 seconds each
call to Packetizer::process(inBuffer, outBuffer), inBuffer.getLength() == 0
and so method ends up in if(inputLength < 10) {... return
BUFFER_PROCESSED_OK) }.

Finally after 5-6 seconds, inBuffer have data (length > 0) and process
method ends up with INPUT_BUFFER_NOT_CONSUMED | OUTPUT_BUFFER_NOT_FILLED and
the first RTP is sent over the network.

If someone has an idea why data (I would say 5 seconds delayed data) takes
times to go to Packetizer (why an empty inBuffer in Packetizer::process()).

I will keep you updated if I find something.

Regards,
--
Seb

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#5

Hi,

good catch :slight_smile: When we were making the codec all these parameter values
that are used now are trying to adjust the encoder to a state that was
common and for other clients using this codec.
As we recently updated the encoder and the decoder, you can try and
find better settings. For the parameter meanings you can try looking
-decoder bitstream buffer size
    * encoding: Set by user.
    * decoding: unused
Maybe by setting it to 0 it uses some default value.

Cheers
damencho

···

at the ffmpeg api. http://cekirdek.pardus.org.tr/~ismail/ffmpeg-docs/structAVCodecContext.html But sometimes the doc is not very useful for example for this rc_buffer_size it says :

On Tue, Dec 22, 2009 at 5:04 PM, Sebastien Vincent <seb@sip-communicator.org> wrote:

Hi all,

After debug/tracing in SC/JMF, I found the cause of video delay at start
(with H264). It is generated by JNIEncoder::process, more specifically by
native call FFMPEG.avcodec_encode_video which always returns 0 during these
5-6 seconds.

I study FFMPEG initialization parameters in JNIEncoder.open() and by
changing
FFMPEG.avcodeccontext_set_rc_buffer_size(avcontext, 10000000);
to
FFMPEG.avcodeccontext_set_rc_buffer_size(avcontext, 0);

I have more "instant" video calls, it still happens some freeze and video
acceleration some time but this is another story I will take care.

So now my question is why setting 10000000 for rc_buffer ? is it wanted ?

Regards,
--
Seb

Damian Minkov a écrit :

Hi,

you can get net.sf.fmj.media.protocol.civil.DataSource from fmj
sources put a print in MyCaptureObserver::onNewImage for buffer length
comming from lti-civil and replace the class in the fmj.jar we use
just for tests. So you can see when actual buffers start to come from
capture device into jmf and see is the capture started later or for
some reason jmf discards some of the data for 5-6 seconds or buffers
it or something else.

damencho

On Mon, Dec 21, 2009 at 5:06 PM, Sebastien Vincent >> <seb@sip-communicator.org> wrote:

Hi all

I (and probably others) notice that on video call (encoded with H264),
video
is delayed arround 5-6 seconds. My goal is to reduce as much as possible
this delay to provide more "instant" video calls.

I search through codebase and I notice
impl.neomedia.codec.video.h264.Packetize class.

After establishing video call, add and start streams
(MediaStreamImpl::startSendStreams()), SC "packetizes" H264 data in
Packetize::process() and the encoded data is sent on the network in
RTPConnectorOutputStream::write. But the first "write" happens 5-6
seconds
after streams have been started. It appears that during 5-6 seconds each
call to Packetizer::process(inBuffer, outBuffer), inBuffer.getLength() ==
0
and so method ends up in if(inputLength < 10) {... return
BUFFER_PROCESSED_OK) }.

Finally after 5-6 seconds, inBuffer have data (length > 0) and process
method ends up with INPUT_BUFFER_NOT_CONSUMED | OUTPUT_BUFFER_NOT_FILLED
and
the first RTP is sent over the network.

If someone has an idea why data (I would say 5 seconds delayed data)
takes
times to go to Packetizer (why an empty inBuffer in
Packetizer::process()).

I will keep you updated if I find something.

Regards,
--
Seb

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net