[jitsi-dev] VP8 recording when PT != 100


#1

We're using Jitsi Videobridge in a context where it accepts incoming offers from client endpoints. When accepting an offer from Firefox, VP8 ends up using the Firefox default payload type of 120, instead of the Chrome/Videobridge default of 100. In this case, if we enable recording on the conference, we get mp3 files for the Firefox audio, but no webm files for the Firefox video.

Doing some digging in libjitsi, I found that RecorderRtpImpl appears to use some hard-coded PTs that match Chrome's defaults. With further digging, I'm pretty sure the RTP translator handles PT modifications properly, as long as they have been properly configured; if not configured then the PT is not modified. It also looks like RecorderRtpImpl acts like another endpoint as far as the translator is concerned, and can register its own PT-to-format mapping. However, it currently only does this for the Opus PT [1], and code that would add mappings for vp8/red/ulpfec is commented out.

I modified the code to add the recorder's preference for PT 100, and Firefox video recording started working. However, since I'm not very familiar with the code base, and can see that similar code is currently commented out, I thought I should check whether this has any unexpected consequences. :slight_smile:

Regards,
Gavin

[1] https://github.com/jitsi/libjitsi/blob/master/src/org/jitsi/impl/neomedia/recording/RecorderRtpImpl.java#L327

Gavin Llewellyn
Lead Software Engineer
t +44 1189 308895
e gavin.llewellyn@xura.com

<mailto:gavin.llewellyn@xura.com%0b%0b>
[cid:image010.png@01D0EBB8.A3083BC0]

<http://www.xura.com/>
[cid:image011.png@01D0EBB8.A3083BC0]<https://twitter.com/i_am_xura> [cid:image012.png@01D0EBB8.A3083BC0] <https://www.linkedin.com/company/1995> [cid:image013.png@01D0EBB8.A3083BC0] <https://www.facebook.com/Xura-163308093686483/> [cid:image014.png@01D0EBB8.A3083BC0] <https://plus.google.com/117205588788549930025/about> [cid:image015.png@01D0EBB8.A3083BC0] <https://www.youtube.com/channel/UCQDtmvjvYomuITYo1ZIyMpg>

···

________________________________
This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Xura, Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to security@xura.com. Thank You.


#2

Hi,

We’re using Jitsi Videobridge in a context where it accepts incoming
offers from client endpoints. When accepting an offer from Firefox, VP8
ends up using the Firefox default payload type of 120, instead of the
Chrome/Videobridge default of 100. In this case, if we enable recording
on the conference, we get mp3 files for the Firefox audio, but no webm
files for the Firefox video.

Doing some digging in libjitsi, I found that RecorderRtpImpl appears to
use some hard-coded PTs that match Chrome’s defaults.

Yes, unfortunately. These were meant to be made configurable, but we never got to it.

With further
digging, I’m pretty sure the RTP translator handles PT modifications
properly, as long as they have been properly configured; if not
configured then the PT is not modified. It also looks like
RecorderRtpImpl acts like another endpoint as far as the translator is
concerned, and can register its own PT-to-format mapping. However, it
currently only does this for the Opus PT [1], and code that would add
mappings for vp8/red/ulpfec is commented out.

When RED is in use (usually with chrome), the translator will not be able to re-write the VP8 PT since packets will be RED-encapsulated.

RecorderRtpImpl handles RED itself (with the chain of transformers here[0]) after it gets a packet from the translator.

I modified the code to add the recorder’s preference for PT 100, and
Firefox video recording started working. However, since I’m not very
familiar with the code base, and can see that similar code is currently
commented out, I thought I should check whether this has any unexpected
consequences. J

In some scenarios (i.e. jitsi-meet) some endpoints send RED, while others send VP8, and I would expect things to brake in this case.

I think that making the vp8PayloadType field configurable (and of course setting it to agreed-upon value for VP8) should make the recorder work in both scenarios.

Regards,
Boris

[0]https://github.com/jitsi/libjitsi/blob/master/src/org/jitsi/impl/neomedia/recording/RecorderRtpImpl.java#L1226

···

On 12/11/15 07:02, Llewellyn, Gavin wrote:


#3

Thanks Boris, that's made things a lot clearer.

It looks like Firefox is not in any particular rush to support red/ulpfec[0]. So if I understand correctly, to get cross-browser conferences working consistently, you have to either:
1) Enhance the translator implementation to understand red, re-write PTs within the compound packet, and extract the vp8 payload for endpoints that don't support red.
2) Disable red/ulpfec in the SDP negotiation for all clients (lowest common denominator). Quick fix, but may impact quality on flaky connections.

With either of the above options, it sounds like RecorderRtpImpl can get away with hard-coded PTs; as long as it informs the translator of the mapping. If you instead make the vp8PayloadType field configurable, you have to make sure *all* endpoints in the conference use the same value, which you don't necessarily have control over (you do with Jitsi-meet, since it only generates SDP offers, and answerers generally fall into line with PT selection).

Regards,
Gavin

[0] https://bugzilla.mozilla.org/show_bug.cgi?id=875922

···

-----Original Message-----
From: dev [mailto:dev-bounces@jitsi.org] On Behalf Of Boris Grozev
Sent: 12 November 2015 16:56
To: Jitsi Developers <dev@jitsi.org>
Subject: Re: [jitsi-dev] VP8 recording when PT != 100

Hi,

On 12/11/15 07:02, Llewellyn, Gavin wrote:

We’re using Jitsi Videobridge in a context where it accepts incoming
offers from client endpoints. When accepting an offer from Firefox,
VP8 ends up using the Firefox default payload type of 120, instead of
the Chrome/Videobridge default of 100. In this case, if we enable
recording on the conference, we get mp3 files for the Firefox audio,
but no webm files for the Firefox video.

Doing some digging in libjitsi, I found that RecorderRtpImpl appears
to use some hard-coded PTs that match Chrome’s defaults.

Yes, unfortunately. These were meant to be made configurable, but we never got to it.

With further
digging, I’m pretty sure the RTP translator handles PT modifications
properly, as long as they have been properly configured; if not
configured then the PT is not modified. It also looks like
RecorderRtpImpl acts like another endpoint as far as the translator is
concerned, and can register its own PT-to-format mapping. However, it
currently only does this for the Opus PT [1], and code that would add
mappings for vp8/red/ulpfec is commented out.

When RED is in use (usually with chrome), the translator will not be able to re-write the VP8 PT since packets will be RED-encapsulated.

RecorderRtpImpl handles RED itself (with the chain of transformers
here[0]) after it gets a packet from the translator.

I modified the code to add the recorder’s preference for PT 100, and
Firefox video recording started working. However, since I’m not very
familiar with the code base, and can see that similar code is
currently commented out, I thought I should check whether this has any
unexpected consequences. J

In some scenarios (i.e. jitsi-meet) some endpoints send RED, while others send VP8, and I would expect things to brake in this case.

I think that making the vp8PayloadType field configurable (and of course setting it to agreed-upon value for VP8) should make the recorder work in both scenarios.

Regards,
Boris

[0]http://cp.mcafee.com/d/5fHCN8p3zqb30VBwsNssrKrjhpKCOyyCYrhhhsKYUM-qejqqbdSknxPP9J554sCYUeKyr8WpB3zP_cuuumw2wZKeYKr3SUXOVLXcmoosvW_9EzA-LsKCPt7C1RTemKzp55mWrDaxVZicHs3jq9JcTvxMVyWrXOrapKVIDeqR4IMzapdIxO-2cFQBxYy0zat9ogfGgNnqj-ekzr00e2cFQBABlzYxZlvgBKh-RoEiwhM5_qIkfy2VH1GGN_2eCt0MaEFFK6QfcBO5mUm-wa4pjFbw09JVxNUS2_id41Fr4pjF8KxVAQKCy0cxpUQgh-RoEiwhd41wDY3h05wiq83h2sq84rTdIc6XCDTKJ8z

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://cp.mcafee.com/d/avndz8O86Qm61Pb0VyUUTsSCyPtdB55dUSyyyVtVNxYQsCQQmrIEL3DCjqaa8VdVMtt4ShQPa77D-oYYYJ051XstVsS7JNTBPvSoIMMU_R-jh79ZuVtdCWfc3HKsJt6OaaJQTel3PWApmU6CQPqpK_3xP5QTTASkPtPo0c-l9QUhBeAK00U9GX33VkDa3JsgYOrgYOn8lrxrW0EhBeAK00CTC77zobZ8Qg6BIhBeAyW7CjiWq80O5Dzh17Xlyxa14Qg62vMd40m19Ewd49NEwhLsSMMrfG6u
________________________________
This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Xura, Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to security@xura.com. Thank You.


#4

Hi Gavin,

Thanks Boris, that's made things a lot clearer.

It looks like Firefox is not in any particular rush to support
red/ulpfec[0]. So if I understand correctly, to get cross-browser
conferences working consistently, you have to either: 1) Enhance the
translator implementation to understand red, re-write PTs within the
compound packet, and extract the vp8 payload for endpoints that don't
support red. 2) Disable red/ulpfec in the SDP negotiation for all
clients (lowest common denominator). Quick fix, but may impact
quality on flaky connections.

We do strip RED for endpoints that don't support it, but not in the translator. It happens here:
https://github.com/jitsi/jitsi-videobridge/blob/master/src/main/java/org/jitsi/videobridge/transform/RtpChannelTransformEngine.java#L145

My understanding of the RTP translator is limited, but AFAIK it writes the very same data to each receiving endpoint, so any endpoint-specific transformations need to be done elsewhere (i.e. in the corresponding MediaStream's transformer chain*).

*Just yesterday we discussed the need to document this chain and the way it's used in videobridge, and I'll try to write a description in the next couple of days.

Regards,
Boris

···

On 12/11/15 12:13, Llewellyn, Gavin wrote:


#5

Hi Boris,

OK, I hadn't spotted the RED stripping. I found the translator turning an incoming PT into a Format in [0], then rewriting PTs for registered receiver formats in OutputDataStreamImpl[1]. This is how my Firefox fix in RecorderRtpImpl works[2].

Stripping RED in a stream transformer is fine, but only if the embedded payload types have already been adjusted for the receiver, and the stream transformer has no knowledge of the PTs used in the source stream (as mentioned in a comment in RtpChannelTransformEngine, where there is another hard-coded PT[3]). This suggests to me that the translator needs to be RED-aware, as it has access to both the incoming and outgoing PTs.

Regards,
Gavin

[0] https://github.com/jitsi/libjitsi/blob/master/src/org/jitsi/impl/neomedia/rtp/translator/RTPTranslatorImpl.java#L591
[1] https://github.com/jitsi/libjitsi/blob/master/src/org/jitsi/impl/neomedia/rtp/translator/OutputDataStreamImpl.java#L549
[2] adding "translator.addFormat(streamRTPManager, vp8RtpFormat, vp8PayloadType)" to the start() method
[3] https://github.com/jitsi/jitsi-videobridge/blob/master/src/main/java/org/jitsi/videobridge/transform/RtpChannelTransformEngine.java#L37

···

-----Original Message-----
From: dev [mailto:dev-bounces@jitsi.org] On Behalf Of Boris Grozev
Sent: 12 November 2015 19:09
To: Jitsi Developers <dev@jitsi.org>
Subject: Re: [jitsi-dev] VP8 recording when PT != 100

Hi Gavin,

On 12/11/15 12:13, Llewellyn, Gavin wrote:

Thanks Boris, that's made things a lot clearer.

It looks like Firefox is not in any particular rush to support
red/ulpfec[0]. So if I understand correctly, to get cross-browser
conferences working consistently, you have to either: 1) Enhance the
translator implementation to understand red, re-write PTs within the
compound packet, and extract the vp8 payload for endpoints that don't
support red. 2) Disable red/ulpfec in the SDP negotiation for all
clients (lowest common denominator). Quick fix, but may impact
quality on flaky connections.

We do strip RED for endpoints that don't support it, but not in the translator. It happens here:
http://cp.mcafee.com/d/5fHCN0i4xESyMMepvvosKrKrjhpKCOyyCYrhhhsKYUM-qejqqbdSknxPP9J554sCYUeKyr8WpB3zP_cuuumw2wZKeYKr3SUXOVJBxzS677-LOrXzXMXHTbFFKfs-MMMCZORQr8EGT7cYG7DR8OJMddECQPt-7f3xObP35T3tPpesRG9px6kOrp3BY4pjFb16kWibCp8vkx6i17Y87R8oHJ9_7ahJyKOxwzFDs00U8ODim6p8vkx6i17YD7oktH5n0ndoi9X27DwCeMEXmaER0za7Y8WpQ30GyM-rgYOn8lrxrW0EhBeAK00CNPbPNI5-Aq83iS8ODiht3P9Ftd40p2PNEwzZGNgB0yq831fU6y0b0AQg6y4UQg8TKroodGuAJjt2-2lIu

My understanding of the RTP translator is limited, but AFAIK it writes the very same data to each receiving endpoint, so any endpoint-specific transformations need to be done elsewhere (i.e. in the corresponding MediaStream's transformer chain*).

*Just yesterday we discussed the need to document this chain and the way it's used in videobridge, and I'll try to write a description in the next couple of days.

Regards,
Boris

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://cp.mcafee.com/d/avndzgA938srhoo7cLLIendTdFEITjphhjudEEEKnusovd79Jd5CXabMVVASyyyejus7nhdAtcOxNV_Cfffbg1guT7undxXstVsSOMNX33z_nVdZNZUtRXBQQT7KvooojuVqWdAklrzCul3PWApmU6CQPqpK_3DxMV5VxyXxKVI06vaAWs8ODin00s4RtxxYGjB1SK8updEupbAaJMJZ0k8ODin00joVBVUS2_id41Fr4pjF8KxVAQKCy0cxpUQgh-RoEiwhd41wDY3h05wiq83h2sq84rTdIc6_f4TMYKu
________________________________
This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Xura, Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to security@xura.com. Thank You.


#6

Hi Gavin,

···

On 13/11/15 05:16, Llewellyn, Gavin wrote:

Hi Boris,

OK, I hadn't spotted the RED stripping. I found the translator
turning an incoming PT into a Format in [0], then rewriting PTs for
registered receiver formats in OutputDataStreamImpl[1]. This is how
my Firefox fix in RecorderRtpImpl works[2].

Stripping RED in a stream transformer is fine, but only if the
embedded payload types have already been adjusted for the receiver,
and the stream transformer has no knowledge of the PTs used in the
source stream (as mentioned in a comment in
RtpChannelTransformEngine, where there is another hard-coded PT[3]).
This suggests to me that the translator needs to be RED-aware, as it
has access to both the incoming and outgoing PTs.

Agreed, this seems like a better approach.

Regards,
Boris