[jitsi-dev] RTP sessions


#1

Hi Lubomir,

I wanted to ask you about RTP sessions in Libjitsi. We need to pause
data flow for the time when Android device is being rotated, but
without the need of restarting RTP session and performing ICE
negotiations again. It would be fine if whole codec chain could be
recreated when it happens. This is especially required for hardware
codecs that rely on Surface objects which are invalid after rotate. Is
it possible to do it ? Where should I start looking ?

Best regards,
Pawel


#2

Hello, Pawel!

I wanted to ask you about RTP sessions in Libjitsi. We need to pause
data flow for the time when Android device is being rotated, but
without the need of restarting RTP session and performing ICE
negotiations again. It would be fine if whole codec chain could be
recreated when it happens. This is especially required for hardware
codecs that rely on Surface objects which are invalid after rotate. Is
it possible to do it ? Where should I start looking ?

As far as my understanding goes, (1) there is nothing particular in
FMJ which disallows the changing of the Format parameters midstream
and (2) problems could arise because of CaptureDevice/DataSource,
Codec and Renderer implementation specifics.

I presume that we could easily take care of (2) because I expect that
we/libjitsi implement every node of the flow graph on Android.

With (2) properly handled (if there is anything to be handled in the
current implementations at all), we could implement (1) by making the
CaptureDevice and the Renderer implementations aware of the rotation.
Each of them could destroy their internal state relying on a Surface
expected to be invalidated at the beginning of the rotation and
re-create that internal state at the end of the rotation. During the
rotation, (a) the CaptureDevice implementation will naturally not
provide any data because its internal Surface-related state will be
non-existent and (b) the Renderer implementation will ignore any video
frames submitted for processing.

How does that sound?

Best regards,
Lyubomir

···

2014/1/2 Paweł Domas <pawel.domas@jitsi.org>:


#3

Hi Lubomir,

Hello, Pawel!

I wanted to ask you about RTP sessions in Libjitsi. We need to pause
data flow for the time when Android device is being rotated, but
without the need of restarting RTP session and performing ICE
negotiations again. It would be fine if whole codec chain could be
recreated when it happens. This is especially required for hardware
codecs that rely on Surface objects which are invalid after rotate. Is
it possible to do it ? Where should I start looking ?

As far as my understanding goes, (1) there is nothing particular in
FMJ which disallows the changing of the Format parameters midstream
and (2) problems could arise because of CaptureDevice/DataSource,
Codec and Renderer implementation specifics.

I presume that we could easily take care of (2) because I expect that
we/libjitsi implement every node of the flow graph on Android.

With (2) properly handled (if there is anything to be handled in the
current implementations at all), we could implement (1) by making the
CaptureDevice and the Renderer implementations aware of the rotation.
Each of them could destroy their internal state relying on a Surface
expected to be invalidated at the beginning of the rotation and
re-create that internal state at the end of the rotation. During the
rotation, (a) the CaptureDevice implementation will naturally not
provide any data because its internal Surface-related state will be
non-existent and (b) the Renderer implementation will ignore any video
frames submitted for processing.

How does that sound?

Best regards,
Lyubomir

Thanks for the answer. Yes, we can (1)make the DataSource and the
Renderer rotation aware. This was the first solution I've been
thinking of, but it will result in even more hacks included to the
current code.

What I was asking about is if there is (2)an operation available in
Libjitsi that will destroy whole codec chain(DataSource, Codec,
Packetizer), but will keep the ICE connection alive. Another one that
would recreate it back. If there is no such facility right now then
I'm going to go for the (1). (1) will probably work faster than (2)
and easier for me to implement without making any harm to Libjitsi.

Best regards,
Pawel

···

On Thu, Jan 2, 2014 at 5:43 PM, Lyubomir Marinov <lyubomir.marinov@jitsi.org> wrote:

2014/1/2 Paweł Domas <pawel.domas@jitsi.org>:


#4

What I was asking about is if there is (2)an operation available in
Libjitsi that will destroy whole codec chain(DataSource, Codec,
Packetizer), but will keep the ICE connection alive.

I think MediaStream#stop() will destroy all codec chains within the stream.

Another one that
would recreate it back.

I expect MediaStream#start() after #stop() to re-create all codec
chains within the stream.

Have you looked at these two methods?

···

2014/1/2 Paweł Domas <pawel.domas@jitsi.org>:


#5

I really think that re-creating the whole codec chain is not always
desirable. If you are streaming video to me and I rotate my device, it
will not be very good for me to recreate my Renderer the newly created
codec chain to that Renderer will have to wait for the next keyframe
to display anything at all.

Otherwise, I presume you're right that a rotation should recreate the
codec chain from the CaptureDevice because RTP likely says something
about changing the SSRC of a video stream when its resolution/size
changes. I suppose MediaStream#setDirection(MediaDirection) with a
MediaDirection for which !MediaDirection.allowsSending() holds true
will re-create the video codec chain from the CaptureDevice. Please
take a look at that method as well.

···

2014/1/2 Paweł Domas <pawel.domas@jitsi.org>:

Thanks for the answer. Yes, we can (1)make the DataSource and the
Renderer rotation aware. This was the first solution I've been
thinking of, but it will result in even more hacks included to the
current code.


#6

Hi Lubomir,

···

On Thu, Jan 2, 2014 at 6:20 PM, Lyubomir Marinov <lyubomir.marinov@jitsi.org> wrote:

2014/1/2 Paweł Domas <pawel.domas@jitsi.org>:

What I was asking about is if there is (2)an operation available in
Libjitsi that will destroy whole codec chain(DataSource, Codec,
Packetizer), but will keep the ICE connection alive.

I think MediaStream#stop() will destroy all codec chains within the stream.

Another one that
would recreate it back.

I expect MediaStream#start() after #stop() to re-create all codec
chains within the stream.

Have you looked at these two methods?

No, but they indeed look promising. Thanks !

Best regards,
Pawel


#7

Thanks for the answer. Yes, we can (1)make the DataSource and the
Renderer rotation aware. This was the first solution I've been
thinking of, but it will result in even more hacks included to the
current code.

I really think that re-creating the whole codec chain is not always
desirable. If you are streaming video to me and I rotate my device, it
will not be very good for me to recreate my Renderer the newly created
codec chain to that Renderer will have to wait for the next keyframe
to display anything at all.

Yes and I will avoid that when AWTRenderer is used as it works goods
when rotated. The problems is only when the MediaCodec is involved it
will loose it's state anyway and a new keyframe will be required. I'm
not sure yet how this will be resolved. Maybe it's possible to cache
last keyframe ? But probably at least some artifacts will appear.

There is another option with manual rotation handling. Maybe this will
prevent from loosing the Surface, so I'm going to try that first.

···

On Thu, Jan 2, 2014 at 6:40 PM, Lyubomir Marinov <lyubomir.marinov@jitsi.org> wrote:

2014/1/2 Paweł Domas <pawel.domas@jitsi.org>:

Otherwise, I presume you're right that a rotation should recreate the
codec chain from the CaptureDevice because RTP likely says something
about changing the SSRC of a video stream when its resolution/size
changes. I suppose MediaStream#setDirection(MediaDirection) with a
MediaDirection for which !MediaDirection.allowsSending() holds true
will re-create the video codec chain from the CaptureDevice. Please
take a look at that method as well.