[sip-comm-dev] Video, H.264


#1

Hi guys interested in video streaming,

I'm running the latest of trunk with a few local optimizations to the
JNIEncoder on Ubuntu Karmic with Apple iSight. I'm testing a P2P call
with H.264 video streaming in both directions, the call window is in
its default size. The CPU utilization of SIP Communicator is roughly
150%, most of the time more.

Looking at a breakdown of the system load shows that:
1. 46% of the time was spent in the kernel. Not surprising I presume.
I guess it also has to do with the V4L2 driver utilized by the video
capture.
2. 12% was given to libcivil.so and, more specifically, to its
uyvy2bgr. A bit of an unexpected entry for the second place. While SIP
Communicator takes more than 150% of the CPU, Cheese manages to
capture and display the video frames with only 10% CPU utilization!
3. 6% was allocated to libffmpeg.so i.e. the native counterpart of our
H.264 encoder and decoder. While H.264 encoding is kind of supposed to
be heavy on the CPU, LTI-CIVIL still managed to beat it.

The top of the breakdown of the java process looks like:
1. 47% for com.sun.media.codec.video.colorspace.JavaRGBToYUV.convert24.
Taking in account the result of libcivil.so from the system-wide
profile, I guess converting between RGB and YUV is pretty intensive,
more than encoding and decoding H.264 at the same time.
2. 25% of the Java time is in
com.sun.media.codec.video.colorspace.JavaRGBConverter.componentToComponent.
Whatever it's converting, it still sounds an awful lot to me.
3. 0.1989% for the first pure SC entry in the profile
net.java.sip.communicator.impl.neomedia.codec.video.h264.Packetizer.process.
Only included here for the purposes of comparison.

I think it's worth looking into what LTI-CIVIL and JMF know (or rather
don't know) about converting between RGB and YUV. I've already entered
the following issues in our issue tracker which should track the
progress of finding answers to these questions:

- https://sip-communicator.dev.java.net/issues/show_bug.cgi?id=771:
Optimize ImageScaler. It seems like the best for us is to find a good
way to convert in a single go (read JMF Codec) between as many RGB and
YUV types as possible. Is FFmpeg's libswscale the answer?

- https://sip-communicator.dev.java.net/issues/show_bug.cgi?id=770:
video4linux2 video CaptureDevice. I guess one could fix LTI-CIVIL
instead, I don't personally plan on digging into LTI-CIVIL and I'd
rather go with our own, JNI video4linux2 JMF CaptureDevice
implementation.

Best regards,
Lubomir

···

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#2

Which version of JVM were you using? 1.6? I don't know how exactly
RGBToYUV conversion is handled but I wouldn't be surprised if 1.7
would be considerably faster on that. Would be interesting to know if
you get noticeably less CPU utilization on the newer JVM.

Kalle

···

On Sun, Jan 24, 2010 at 12:00 PM, Lubomir Marinov <lubo@sip-communicator.org> wrote:

Hi guys interested in video streaming,

I'm running the latest of trunk with a few local optimizations to the
JNIEncoder on Ubuntu Karmic with Apple iSight. I'm testing a P2P call
with H.264 video streaming in both directions, the call window is in
its default size. The CPU utilization of SIP Communicator is roughly
150%, most of the time more.

Looking at a breakdown of the system load shows that:
1. 46% of the time was spent in the kernel. Not surprising I presume.
I guess it also has to do with the V4L2 driver utilized by the video
capture.
2. 12% was given to libcivil.so and, more specifically, to its
uyvy2bgr. A bit of an unexpected entry for the second place. While SIP
Communicator takes more than 150% of the CPU, Cheese manages to
capture and display the video frames with only 10% CPU utilization!
3. 6% was allocated to libffmpeg.so i.e. the native counterpart of our
H.264 encoder and decoder. While H.264 encoding is kind of supposed to
be heavy on the CPU, LTI-CIVIL still managed to beat it.

The top of the breakdown of the java process looks like:
1. 47% for com.sun.media.codec.video.colorspace.JavaRGBToYUV.convert24.
Taking in account the result of libcivil.so from the system-wide
profile, I guess converting between RGB and YUV is pretty intensive,
more than encoding and decoding H.264 at the same time.
2. 25% of the Java time is in
com.sun.media.codec.video.colorspace.JavaRGBConverter.componentToComponent.
Whatever it's converting, it still sounds an awful lot to me.
3. 0.1989% for the first pure SC entry in the profile
net.java.sip.communicator.impl.neomedia.codec.video.h264.Packetizer.process.
Only included here for the purposes of comparison.

I think it's worth looking into what LTI-CIVIL and JMF know (or rather
don't know) about converting between RGB and YUV. I've already entered
the following issues in our issue tracker which should track the
progress of finding answers to these questions:

- https://sip-communicator.dev.java.net/issues/show_bug.cgi?id=771:
Optimize ImageScaler. It seems like the best for us is to find a good
way to convert in a single go (read JMF Codec) between as many RGB and
YUV types as possible. Is FFmpeg's libswscale the answer?

- https://sip-communicator.dev.java.net/issues/show_bug.cgi?id=770:
video4linux2 video CaptureDevice. I guess one could fix LTI-CIVIL
instead, I don't personally plan on digging into LTI-CIVIL and I'd
rather go with our own, JNI video4linux2 JMF CaptureDevice
implementation.

Best regards,
Lubomir

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#3

Hi Lubomir and all,

I try to optimise video streaming by playing with FFMPEG parameters.

I have not yet try your optimisation patch you post on dev, but I have never see SIP Communicator running at 150% on Linux.
With latest trunk IIRC it runs between 50-65%. With some tweaking I have manage to reach ~36% on a desktop intel dual core 2 GHz.
For information I run Linux Debian 64-bit on both peers (I think 64-bit improve H264 encoding/decoding speed).

For information, what tool do you use to get percentage of time consumed by java class/jni on Linux ?

Regards,

···

--
Seb

Lubomir Marinov a �crit :

Hi guys interested in video streaming,

I'm running the latest of trunk with a few local optimizations to the
JNIEncoder on Ubuntu Karmic with Apple iSight. I'm testing a P2P call
with H.264 video streaming in both directions, the call window is in
its default size. The CPU utilization of SIP Communicator is roughly
150%, most of the time more.

Looking at a breakdown of the system load shows that:
1. 46% of the time was spent in the kernel. Not surprising I presume.
I guess it also has to do with the V4L2 driver utilized by the video
capture.
2. 12% was given to libcivil.so and, more specifically, to its
uyvy2bgr. A bit of an unexpected entry for the second place. While SIP
Communicator takes more than 150% of the CPU, Cheese manages to
capture and display the video frames with only 10% CPU utilization!
3. 6% was allocated to libffmpeg.so i.e. the native counterpart of our
H.264 encoder and decoder. While H.264 encoding is kind of supposed to
be heavy on the CPU, LTI-CIVIL still managed to beat it.

The top of the breakdown of the java process looks like:
1. 47% for com.sun.media.codec.video.colorspace.JavaRGBToYUV.convert24.
Taking in account the result of libcivil.so from the system-wide
profile, I guess converting between RGB and YUV is pretty intensive,
more than encoding and decoding H.264 at the same time.
2. 25% of the Java time is in
com.sun.media.codec.video.colorspace.JavaRGBConverter.componentToComponent.
Whatever it's converting, it still sounds an awful lot to me.
3. 0.1989% for the first pure SC entry in the profile
net.java.sip.communicator.impl.neomedia.codec.video.h264.Packetizer.process.
Only included here for the purposes of comparison.

I think it's worth looking into what LTI-CIVIL and JMF know (or rather
don't know) about converting between RGB and YUV. I've already entered
the following issues in our issue tracker which should track the
progress of finding answers to these questions:

- https://sip-communicator.dev.java.net/issues/show_bug.cgi?id=771:
Optimize ImageScaler. It seems like the best for us is to find a good
way to convert in a single go (read JMF Codec) between as many RGB and
YUV types as possible. Is FFmpeg's libswscale the answer?

- https://sip-communicator.dev.java.net/issues/show_bug.cgi?id=770:
video4linux2 video CaptureDevice. I guess one could fix LTI-CIVIL
instead, I don't personally plan on digging into LTI-CIVIL and I'd
rather go with our own, JNI video4linux2 JMF CaptureDevice
implementation.

Best regards,
Lubomir

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#4

For information I run Linux Debian 64-bit on both peers (I think 64-bit
improve H264 encoding/decoding speed).

I'm also running 64-bit Linux, namely Ubuntu Karmic. When looking at
SIP Communicator in System Monitor, 50-65% is what I get by just
previewing the video CaptureDevice in the "Media (New)" configuration
form. As I said, Cheese Webcam Booth consumes only 10%.

For information, what tool do you use to get percentage of time consumed by
java class/jni on Linux ?

For the comparison of the relative performance of Java code, I usually
use NetBeans Profiler and YourKit Java Profiler. As these impose
(noticeable) increase in the CPU utilization of the process, this time
I went with the system-wide profiler OProfile with enabled JIT
profiling to see how the native methods (and/or their respective
modules) relate to the Java code.

···

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net