[jitsi-dev] [Patches] Linux VAAPI support (finally)


#1

Hi all,

I have finally found some time to rewrote from scratch the VAAPI stuff I made 2 years ago.

Basically I have something that works good (on my Thinkpad T410 with nvidia NV3100M)! Here is a summary of the modifications I made.

Java side:
     - Refactor a little bit the native video renderer (Lyubomir feel free to review and comment this part);
     - Adds an HWRenderer/HWRendererVideoComponent class;
     - Modify output format for H.264/H.263 JNIDecoder;
     - Adds parsing for multiple FU-A in H.264 depacketizer (because we cannot use CODEC_FLAG2_CHUNKS, see below)
     - Property to enable/disable use of hardware decoding (net.java.sip.communicator.impl.neomedia.codec.video.ENABLE_VIDEO_HARDWARE_DECODING);
     - Avoid adding filters or other codec in the codec chain when we are in hardware decoding mode (VideoMediaDeviceSession);
     - Additionnal native methods in FFmpeg.

Native side:
     - Adds some native method to detect hardware decoding for a specific codec and one more to set opaque field;
     - Defines somes API functions to be used with FFmpeg (for a DXVA2 or VDA these have to be implemented) and the implementation for VAAPI;
     - The hardware renderer code (close to the JAWTRenderer, maybe make more refactoring for this) with implementation for VAAPI => new library jnhwrenderer.

However there are some limitations:
- VAAPI does not work very well with H.264 packetization mode 0 (bad image), I don't know where it comes from. So to be that it will work, enable packetization mode 1 => net.java.sip.communicator.impl.neomedia.codec.video.h264.packetization-mode-1.enabled=true;
- We must not use CODEC_FLAG2_CHUNKS in H.264 JNIDecoder otherwise decoding always failed;
- Dynamic link to libva and libva-x1, in other words libva, libva-x11 and the va driver (intel, vdpau or xvba) have to be installed on the computer;
- Have only tested with FFmpeg and not with Debian libav (we have an issue related to avfilter_graph_parse prototype that differs from FFmpeg ones).

Things that I have not tested:
- I don't have a graphic card that supports hardware decoding for H.263 so I cannot test it for this codec :/;
- I don't have tested on x86 (32-bit) Linux.

Please note that to use VAAPI you have to have a compatible graphic card and the associated va driver installed (libva-intel-vaapi-driver/i965-va-driver for intel based, libva-vdpau-driver for NVidia cards, xvba-va-driver for ATI/AMD cards, ...). The best way to see if you can use VAAPI is to installed libva, the va driver and vainfo program then type vainfo and see what it returns.

Also you have to build a "VAAPI aware" FFmpeg (see src/native/ffmpeg/README), rebuild libjnffmpeg and build the new libjnhwrenderer JNIs (via ant ffmpeg / hwrenderer, see src/native/build.xml).

The best way to continue its development (and to make easier than handle patch) is, I think, to create a git branch (since sources are now in git :). Emil tell me if you are OK with that.

(PS : I plan to begin DXVA2 implementation as well, the only stuff that will take time is rendering with EVR...).

Anyway do not hesitate to give feedback.

Regards,

libjitsi-vaapi-support-10825.diff (110 KB)

···

--
Seb


#2

Hey Seb,

Thank you for your patch! Yes, pushing that in a branch on GitHub should
be fine, so feel free to do so.

A few more comments inline:

Hi all,

I have finally found some time to rewrote from scratch the VAAPI stuff I
made 2 years ago.

Basically I have something that works good (on my Thinkpad T410 with
nvidia NV3100M)! Here is a summary of the modifications I made.

Java side:
     - Refactor a little bit the native video renderer (Lyubomir feel
free to review and comment this part);
     - Adds an HWRenderer/HWRendererVideoComponent class;
     - Modify output format for H.264/H.263 JNIDecoder;
     - Adds parsing for multiple FU-A in H.264 depacketizer (because we
cannot use CODEC_FLAG2_CHUNKS, see below)
     - Property to enable/disable use of hardware decoding
(net.java.sip.communicator.impl.neomedia.codec.video.ENABLE_VIDEO_HARDWARE_DECODING);
     - Avoid adding filters or other codec in the codec chain when we
are in hardware decoding mode (VideoMediaDeviceSession);
     - Additionnal native methods in FFmpeg.

Note that in order to satisfy Debian policies we are switching toward
the vanilla FFmpeg libs and are no longer shipping any changes on those.
If you believe they are necessary, you should probably send them upstream.

Native side:
     - Adds some native method to detect hardware decoding for a
specific codec and one more to set opaque field;
     - Defines somes API functions to be used with FFmpeg (for a DXVA2
or VDA these have to be implemented) and the implementation for VAAPI;
     - The hardware renderer code (close to the JAWTRenderer, maybe make
more refactoring for this) with implementation for VAAPI => new library
jnhwrenderer.

However there are some limitations:
- VAAPI does not work very well with H.264 packetization mode 0 (bad
image), I don't know where it comes from. So to be that it will work,
enable packetization mode 1 =>
net.java.sip.communicator.impl.neomedia.codec.video.h264.packetization-mode-1.enabled=true;
- We must not use CODEC_FLAG2_CHUNKS in H.264 JNIDecoder otherwise
decoding always failed;
- Dynamic link to libva and libva-x1, in other words libva, libva-x11
and the va driver (intel, vdpau or xvba) have to be installed on the
computer;
- Have only tested with FFmpeg and not with Debian libav (we have an
issue related to avfilter_graph_parse prototype that differs from FFmpeg
ones).

Things that I have not tested:
- I don't have a graphic card that supports hardware decoding for H.263
so I cannot test it for this codec :/;
- I don't have tested on x86 (32-bit) Linux.

Please note that to use VAAPI you have to have a compatible graphic card
and the associated va driver installed
(libva-intel-vaapi-driver/i965-va-driver for intel based,
libva-vdpau-driver for NVidia cards, xvba-va-driver for ATI/AMD cards,
...). The best way to see if you can use VAAPI is to installed libva,
the va driver and vainfo program then type vainfo and see what it returns.

Also you have to build a "VAAPI aware" FFmpeg (see
src/native/ffmpeg/README), rebuild libjnffmpeg and build the new
libjnhwrenderer JNIs (via ant ffmpeg / hwrenderer, see
src/native/build.xml).

The best way to continue its development (and to make easier than handle
patch) is, I think, to create a git branch (since sources are now in git
:). Emil tell me if you are OK with that.

I am.

(PS : I plan to begin DXVA2 implementation as well, the only stuff that
will take time is rendering with EVR...).

Anyway do not hesitate to give feedback.

Looking forward to windows and maybe OS X one day?

Thanks Seb!
Emil

···

On 25.04.13, 18:42, Sebastien Vincent wrote:

Regards,
--
Seb

--
https://jitsi.org


#3

Hi,

Yes dxva2 is on the way and I hope to give a try to OSX VDA as well.

For the FFmpeg modifications it is related to FFmpeg JNI not the library.
Sorry for confusion.

Regards,

···

--
Seb

Le mardi 30 avril 2013, Emil Ivov <emcho@jitsi.org> a écrit :

Hey Seb,

Thank you for your patch! Yes, pushing that in a branch on GitHub should
be fine, so feel free to do so.

A few more comments inline:

On 25.04.13, 18:42, Sebastien Vincent wrote:

Hi all,

I have finally found some time to rewrote from scratch the VAAPI stuff I
made 2 years ago.

Basically I have something that works good (on my Thinkpad T410 with
nvidia NV3100M)! Here is a summary of the modifications I made.

Java side:
     - Refactor a little bit the native video renderer (Lyubomir feel
free to review and comment this part);
     - Adds an HWRenderer/HWRendererVideoComponent class;
     - Modify output format for H.264/H.263 JNIDecoder;
     - Adds parsing for multiple FU-A in H.264 depacketizer (because we
cannot use CODEC_FLAG2_CHUNKS, see below)
     - Property to enable/disable use of hardware decoding

(net.java.sip.communicator.impl.neomedia.codec.video.ENABLE_VIDEO_HARDWARE_DECODING);

     - Avoid adding filters or other codec in the codec chain when we
are in hardware decoding mode (VideoMediaDeviceSession);
     - Additionnal native methods in FFmpeg.

Note that in order to satisfy Debian policies we are switching toward
the vanilla FFmpeg libs and are no longer shipping any changes on those.
If you believe they are necessary, you should probably send them upstream.

Native side:
     - Adds some native method to detect hardware decoding for a
specific codec and one more to set opaque field;
     - Defines somes API functions to be used with FFmpeg (for a DXVA2
or VDA these have to be implemented) and the implementation for VAAPI;
     - The hardware renderer code (close to the JAWTRenderer, maybe make
more refactoring for this) with implementation for VAAPI => new library
jnhwrenderer.

However there are some limitations:
- VAAPI does not work very well with H.264 packetization mode 0 (bad
image), I don't know where it comes from. So to be that it will work,
enable packetization mode 1 =>

net.java.sip.communicator.impl.neomedia.codec.video.h264.packetization-mode-1.enabled=true;

- We must not use CODEC_FLAG2_CHUNKS in H.264 JNIDecoder otherwise
decoding always failed;
- Dynamic link to libva and libva-x1, in other words libva, libva-x11
and the va driver (intel, vdpau or xvba) have to be installed on the
computer;
- Have only tested with FFmpeg and not with Debian libav (we have an
issue related to avfilter_graph_parse prototype that differs from FFmpeg
ones).

Things that I have not tested:
- I don't have a graphic card that supports hardware decoding for H.263
so I cannot test it for this codec :/;
- I don't have tested on x86 (32-bit) Linux.

Please note that to use VAAPI you have to have a compatible graphic card
and the associated va driver installed
(libva-intel-vaapi-driver/i965-va-driver for intel based,
libva-vdpau-driver for NVidia cards, xvba-va-driver for ATI/AMD cards,
...). The best way to see if you can use VAAPI is to installed libva,
the va driver and vainfo program then type vainfo and see what it

returns.

Also you have to build a "VAAPI aware" FFmpeg (see
src/native/ffmpeg/README), rebuild libjnffmpeg and build the new
libjnhwrenderer JNIs (via ant ffmpeg / hwrenderer, see
src/native/build.xml).

The best way to continue its development (and to make easier than handle
patch) is, I think, to create a git branch (since sources are now in git
:). Emil tell me if you are OK with that.

I am.

(PS : I plan to begin DXVA2 implementation as well, the only stuff that
will take time is rendering with EVR...).

Anyway do not hesitate to give feedback.

Looking forward to windows and maybe OS X one day?

Thanks Seb!
Emil

Regards,
--
Seb

--
https://jitsi.org