[sip-comm-dev] Re: svn commit: r6787 - trunk: lib/native/freebsd lib/native/freebsd-64 lib/native/linux lib/native/linux-64 lib/native/mac lib/native/windows lib/nati...


#1

Hello Seb,

Have you been able to make a rough evaluation of the performance
differences between the JMF converter+scaler and the libswscale
implementation? I remember the quality of the scaled image was
improved by libswscale in comparison to JMF scaler and I'm quite
interested in the execution times and the garbage collection.

Regards,
Lubo

···

On Mon, Feb 22, 2010 at 7:02 PM, <s_vincent@dev.java.net> wrote:

Author: s_vincent
Date: 2010-02-22 17:02:32+0000
New Revision: 6787

Added:
trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/SwScaler.java
Modified:
trunk/lib/native/freebsd-64/libffmpeg.so
trunk/lib/native/freebsd/libffmpeg.so
trunk/lib/native/linux-64/libffmpeg.so
trunk/lib/native/linux/libffmpeg.so
trunk/lib/native/mac/libffmpeg.jnilib
trunk/lib/native/windows-64/ffmpeg.dll
trunk/lib/native/windows/ffmpeg.dll
trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c
trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h
trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java
trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java
trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java
trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java
trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java

Log:
- Add new scaler that uses libswscale and can change format (i.e. RGB to YUV, ...) and scale in one pass;
- Add the possibility to change output video size of a stream in VideoMediaDeviceSession;
- Add recompiled ffmpeg JNI for all supported platforms.

Modified: trunk/lib/native/freebsd-64/libffmpeg.so
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/freebsd-64/libffmpeg.so?view=diff&rev=6787&p1=trunk/lib/native/freebsd-64/libffmpeg.so&p2=trunk/lib/native/freebsd-64/libffmpeg.so&r1=6786&r2=6787

Binary files. No diff available.

Modified: trunk/lib/native/freebsd/libffmpeg.so
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/freebsd/libffmpeg.so?view=diff&rev=6787&p1=trunk/lib/native/freebsd/libffmpeg.so&p2=trunk/lib/native/freebsd/libffmpeg.so&r1=6786&r2=6787

Binary files. No diff available.

Modified: trunk/lib/native/linux-64/libffmpeg.so
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/linux-64/libffmpeg.so?view=diff&rev=6787&p1=trunk/lib/native/linux-64/libffmpeg.so&p2=trunk/lib/native/linux-64/libffmpeg.so&r1=6786&r2=6787

Binary files. No diff available.

Modified: trunk/lib/native/linux/libffmpeg.so
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/linux/libffmpeg.so?view=diff&rev=6787&p1=trunk/lib/native/linux/libffmpeg.so&p2=trunk/lib/native/linux/libffmpeg.so&r1=6786&r2=6787

Binary files. No diff available.

Modified: trunk/lib/native/mac/libffmpeg.jnilib
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/mac/libffmpeg.jnilib?view=diff&rev=6787&p1=trunk/lib/native/mac/libffmpeg.jnilib&p2=trunk/lib/native/mac/libffmpeg.jnilib&r1=6786&r2=6787

Binary files. No diff available.

Modified: trunk/lib/native/windows-64/ffmpeg.dll
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/windows-64/ffmpeg.dll?view=diff&rev=6787&p1=trunk/lib/native/windows-64/ffmpeg.dll&p2=trunk/lib/native/windows-64/ffmpeg.dll&r1=6786&r2=6787

Binary files. No diff available.

Modified: trunk/lib/native/windows/ffmpeg.dll
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/windows/ffmpeg.dll?view=diff&rev=6787&p1=trunk/lib/native/windows/ffmpeg.dll&p2=trunk/lib/native/windows/ffmpeg.dll&r1=6786&r2=6787

Binary files. No diff available.

Modified: trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c?view=diff&rev=6787&p1=trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c&p2=trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c&r1=6786&r2=6787

--- trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c (original)
+++ trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c 2010-02-22 17:02:32+0000
@@ -438,15 +438,15 @@

static int image_convert(
AVPicture* dst, int dst_pix_fmt,
-const AVPicture* src, int pix_fmt, int width, int height)
+const AVPicture* src, int pix_fmt, int width, int height, jint newWidth, jint newHeight)
{
struct SwsContext *img_convert_ctx =
sws_getContext(
width, height, pix_fmt,
- width, height, dst_pix_fmt, SWS_BICUBIC, NULL, NULL, NULL);
+ newWidth, newHeight, dst_pix_fmt, SWS_BICUBIC, NULL, NULL, NULL);

int result = sws_scale(img_convert_ctx,
- src->data, src->linesize, 0, height,
+ (uint8_t**)src->data, (int*)src->linesize, 0, height,
dst->data, dst->linesize);
sws_freeContext(img_convert_ctx);

@@ -454,7 +454,7 @@
}

JNIEXPORT jint JNICALL
-Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_img_1convert (
+Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_img_1convert__JIJIII (
JNIEnv *jniEnv, jclass clazz, jlong dst, jint dst_pix_fmt, jlong src,
jint pix_fmt, jint width, jint height)
{
@@ -465,9 +465,50 @@
(const AVPicture *) src,
(int) pix_fmt,
(int) width,
+ (int) height,
+ (int) width,
(int) height);
}

+JNIEXPORT jint JNICALL
+Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_img_1convert___3BI_3BIIIII
+ (JNIEnv *jniEnv, jclass clazz, jbyteArray dst, jint dst_pix_fmt,
+ jbyteArray src, jint pix_fmt, jint width, jint height, jint newWidth, jint newHeight)
+{
+ AVPicture dst_dummy;
+ AVPicture src_dummy;
+ int size = width * height;
+ jint ret = 0;
+/*
+ uint8_t* dst_buf = (*jniEnv)->GetPrimitiveArrayCritical(jniEnv, dst, 0);
+ uint8_t* src_buf = (*jniEnv)->GetPrimitiveArrayCritical(jniEnv, src, 0);
+*/
+ uint8_t* dst_buf = (*jniEnv)->GetByteArrayElements(jniEnv, dst, 0);
+ uint8_t* src_buf = (*jniEnv)->GetByteArrayElements(jniEnv, src, 0);
+
+ if(!src || !dst)
+ {
+ return -1;
+ }
+
+ /* assign AVPicture with buffer */
+ avpicture_fill(&dst_dummy, dst_buf, (int)dst_pix_fmt, newWidth, newHeight);
+ avpicture_fill(&src_dummy, src_buf, (int)pix_fmt, width, height);
+
+ ret = image_convert(&dst_dummy, (int)dst_pix_fmt, &src_dummy, (int)pix_fmt,
+ (int)width, (int)height, (int)newWidth, (int)newHeight);
+
+ /* release pointers */
+/*
+ (*jniEnv)->ReleasePrimitiveArrayCritical(jniEnv, dst, dst_buf, 0);
+ (*jniEnv)->ReleasePrimitiveArrayCritical(jniEnv, src, src_buf, 0);
+*/
+ (*jniEnv)->ReleaseByteArrayElements(jniEnv, dst, dst_buf, 0);
+ (*jniEnv)->ReleaseByteArrayElements(jniEnv, src, src_buf, 0);
+ return ret;
+}
+
+
JNIEXPORT void JNICALL
Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_memcpy___3IIIJ (
JNIEnv *jniEnv, jclass clazz, jintArray dst, jint dst_offset,
@@ -492,6 +533,13 @@
}

JNIEXPORT jint JNICALL
+Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_getRGBAFormat
+ (JNIEnv *env, jclass clazz)
+{
+ return PIX_FMT_RGBA;
+}
+
+JNIEXPORT jint JNICALL
Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_getYUV420PFormat
(JNIEnv *env, jclass clazz)
{

Modified: trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h?view=diff&rev=6787&p1=trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h&p2=trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h&r1=6786&r2=6787

--- trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h (original)
+++ trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h 2010-02-22 17:02:32+0000
@@ -31,6 +31,14 @@

/*
* Class: net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG
+ * Method: getRGBAFormat
+ * Signature: ()I
+ */
+JNIEXPORT jint JNICALL Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_getRGBAFormat
+ (JNIEnv *, jclass);
+
+/*
+ * Class: net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG
* Method: getYUV420PFormat
* Signature: ()I
*/
@@ -458,11 +466,19 @@
* Method: img_convert
* Signature: (JIJIII)I
*/
-JNIEXPORT jint JNICALL Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_img_1convert
+JNIEXPORT jint JNICALL Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_img_1convert__JIJIII
(JNIEnv *, jclass, jlong, jint, jlong, jint, jint, jint);

/*
* Class: net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG
+ * Method: img_convert
+ * Signature: ([BI[BIIIII)I
+ */
+JNIEXPORT jint JNICALL Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_img_1convert___3BI_3BIIIII
+ (JNIEnv *, jclass, jbyteArray, jint, jbyteArray, jint, jint, jint, jint, jint);
+
+/*
+ * Class: net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG
* Method: memcpy
* Signature: ([IIIJ)V
*/

Modified: trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java?view=diff&rev=6787&p1=trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java&p2=trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java&r1=6786&r2=6787

--- trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java (original)
+++ trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java 2010-02-22 17:02:32+0000
@@ -62,7 +62,8 @@
"net.java.sip.communicator.impl.neomedia.codec.video.h264.JNIEncoder",
"net.java.sip.communicator.impl.neomedia.codec.video.h264.Packetizer",
"net.java.sip.communicator.impl.neomedia.codec.video.h264.JNIDecoder",
- "net.java.sip.communicator.impl.neomedia.codec.video.ImageScaler",
+ //"net.java.sip.communicator.impl.neomedia.codec.video.ImageScaler",
+ "net.java.sip.communicator.impl.neomedia.codec.video.SwScaler",
"net.java.sip.communicator.impl.neomedia.codec.audio.speex.JavaEncoder",
"net.java.sip.communicator.impl.neomedia.codec.audio.speex.JavaDecoder",
"net.java.sip.communicator.impl.neomedia.codec.audio.ilbc.JavaEncoder",
@@ -328,6 +329,9 @@
.getPlugInList(null, null, PlugInManager.CODEC));
boolean commit = false;

+ /* remove JavaRGBToYUV */
+ PlugInManager.removePlugIn("com.sun.media.codec.video.colorspace.JavaRGBToYUV", PlugInManager.CODEC);
+
for (String className : CUSTOM_CODECS)
{

Modified: trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java?view=diff&rev=6787&p1=trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java&p2=trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java&r1=6786&r2=6787

--- trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java (original)
+++ trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java 2010-02-22 17:02:32+0000
@@ -24,12 +24,16 @@
public static final int FF_MB_DECISION_SIMPLE = 0;

public static final int PIX\_FMT\_RGB32;

+
+ public static final int PIX_FMT_RGBA;

public static final int PIX\_FMT\_YUV420P;

public static final int X264\_RC\_ABR = 2;

public static native int getRGB32Format\(\);

+
+ public static native int getRGBAFormat();

public static native int getYUV420PFormat\(\);

@@ -172,6 +176,10 @@

public static native int img\_convert\(long dst, int dst\_pix\_fmt, long src,
    int pix\_fmt, int width, int height\);

+
+ public static native int img_convert(byte dst[], int dst_pix_fmt,
+ byte src[], int pix_fmt, int width, int height, int newWidth,
+ int newHeight);

public static native void memcpy\(int\[\] dst, int dst\_offset, int dst\_length,
    long src\);

@@ -185,7 +193,8 @@

    av\_register\_all\(\);
    avcodec\_init\(\);

- PIX_FMT_RGB32 = getRGB32Format();
+ PIX_FMT_RGB32 = getRGB32Format(); /* for decoding */
+ PIX_FMT_RGBA = getRGBAFormat(); /* for encoding */
PIX_FMT_YUV420P = getYUV420PFormat();
}
}

Added: trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/SwScaler.java
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/SwScaler.java?view=auto&rev=6787

--- (empty file)
+++ trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/SwScaler.java 2010-02-22 17:02:32+0000
@@ -0,0 +1,196 @@
+/*
+ * SIP Communicator, the OpenSource Java VoIP and Instant Messaging client.
+ *
+ * Distributable under LGPL license.
+ * See terms of license at gnu.org.
+ */
+package net.java.sip.communicator.impl.neomedia.codec.video;
+
+import java.awt.*;
+
+import javax.media.*;
+import javax.media.format.*;
+
+import net.sf.fmj.media.*;
+
+/**
+ * Codec that use libswccale to scale images from one size to
+ * another and change format (typically RGB to YUV).
+ *
+ * @author Sebastien Vincent
+ */
+public class SwScaler
+ extends AbstractCodec
+ implements Codec
+{
+
+ /**
+ * Supported input formats.
+ */
+ private final Format[] supportedInputFormats = new Format[] {
+ new RGBFormat(null, -1, Format.byteArray, -1.0f, 32, -1, -1, -1)
+ };
+
+ /**
+ * Supported output formats.
+ */
+ private Format[] supportedOutputFormats = new Format[] {
+ new YUVFormat(null, -1, Format.byteArray, -1.0f, YUVFormat.YUV_420, -1, -1, 0, -1, -1)
+ };
+
+ /**
+ * Set output size.
+ *
+ * @param size size to set
+ */
+ public void setOutputSize(Dimension size)
+ {
+ if(size == null)
+ {
+ size = new Dimension(640, 480);
+ }
+
+ Format newFormat = new YUVFormat(size, -1, Format.byteArray, -1.0f, YUVFormat.YUV_420, -1, -1, 0, -1, -1);
+ supportedOutputFormats[0] = newFormat;
+ }
+
+ /**
+ * Get the supported input formats.
+ *
+ * @return array of supported input format
+ */
+ @Override
+ public Format[] getSupportedInputFormats()
+ {
+ return supportedInputFormats;
+ }
+
+ /**
+ * Get the supported output formats for an input ones.
+ *
+ * @param input input format to convert
+ * @return array of supported output format
+ */
+ @Override
+ public Format[] getSupportedOutputFormats(Format input)
+ {
+ if(input == null)
+ {
+ return supportedOutputFormats;
+ }
+
+ //System.out.println("input: " + ((VideoFormat)input).getSize());
+ if(((VideoFormat)supportedOutputFormats[0]).getSize() != null)
+ {
+ return supportedOutputFormats;
+ }
+
+ return new Format[] { new YUVFormat(((VideoFormat)input).getSize(), -1, Format.byteArray, -1.0f, YUVFormat.YUV_420, -1, -1, 0, -1, -1)};
+ }
+
+ /**
+ * Set the input format.
+ *
+ * @param format format to set
+ * @return format
+ */
+ @Override
+ public Format setInputFormat(Format format)
+ {
+ //System.out.println("setInput: " + ((VideoFormat)format).getSize());
+ final VideoFormat videoFormat = (VideoFormat) format;
+
+ if (videoFormat.getSize() == null)
+ return null; // must set a size.
+
+ format = super.setInputFormat(format);
+ return format;
+ }
+
+ /**
+ * Set the output format.
+ *
+ * @param format format to set
+ * @return format
+ */
+ @Override
+ public Format setOutputFormat(Format format)
+ {
+ //System.out.println("setOutput: " + ((VideoFormat)format).getSize());
+ format = super.setOutputFormat(format);
+ return format;
+ }
+
+ /**
+ * Process (format convertion, rescale) a buffer.
+ *
+ * @param input input buffer
+ * @param output output buffer
+ * @return BUFFER_PROCESSED_OK if buffer successfully processed
+ */
+ @Override
+ public int process(Buffer input, Buffer output)
+ {
+ VideoFormat vinput = (VideoFormat)input.getFormat();
+ VideoFormat voutput = (VideoFormat)output.getFormat();
+ int inputWidth = (int)vinput.getSize().getWidth();
+ int inputHeight = (int)vinput.getSize().getHeight();
+ int outputWidth = (int)voutput.getSize().getWidth();
+ int outputHeight = (int)voutput.getSize().getHeight();
+ byte src[] = (byte[])input.getData();
+ byte dst[] = (byte[])output.getData();
+ int outputSize = 0;
+ int infmt = 0;
+ int outfmt = 0;
+
+ if (!checkInputBuffer(input))
+ {
+ return BUFFER_PROCESSED_FAILED;
+ }
+
+ if (isEOM(input))
+ {
+ propagateEOM(output); // TODO: what about data? can there be any?
+ return BUFFER_PROCESSED_OK;
+ }
+
+ /* determine output format and output size needed */
+ if(voutput instanceof YUVFormat)
+ {
+ /* YUV420P is 12 bpp (bit per pixel) => 1,5 bytes */
+ outputSize = (int)(outputWidth * outputHeight * 1.5);
+ outfmt = FFMPEG.PIX_FMT_YUV420P;
+ }
+ else /* RGB format */
+ {
+ outputSize = (outputWidth * outputHeight * 4);
+ outfmt = FFMPEG.PIX_FMT_RGBA;
+ }
+
+ /* determine input format */
+ if(vinput instanceof YUVFormat)
+ {
+ infmt = FFMPEG.PIX_FMT_YUV420P;
+ }
+ else /* RGBFormat */
+ {
+ infmt = FFMPEG.PIX_FMT_RGBA;
+ }
+
+ if(dst == null || dst.length < outputSize)
+ {
+ dst = new byte[outputSize];
+ }
+
+ /* convertion! */
+ //System.out.println("Convert: " + inputWidth + "x" + inputHeight + " to " + outputWidth + "x" + outputHeight);
+ FFMPEG.img_convert(dst, outfmt, src, infmt, inputWidth, inputHeight, outputWidth, outputHeight);
+
+ output.setData(dst);
+ output.setLength(dst.length);
+ output.setOffset(0);
+
+ return BUFFER_PROCESSED_OK;
+ }
+}
+

Modified: trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java?view=diff&rev=6787&p1=trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java&p2=trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java&r1=6786&r2=6787

--- trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java (original)
+++ trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java 2010-02-22 17:02:32+0000
@@ -19,6 +19,7 @@
import net.java.sip.communicator.impl.neomedia.*;
import net.java.sip.communicator.impl.neomedia.format.*;
import net.java.sip.communicator.impl.neomedia.protocol.*;
+import net.java.sip.communicator.impl.neomedia.codec.video.*;
import net.java.sip.communicator.service.neomedia.*;
import net.java.sip.communicator.service.neomedia.device.*;
import net.java.sip.communicator.service.neomedia.format.*;
@@ -1226,7 +1227,7 @@
* of
* @param format the JMF <tt>Format</tt> to set to <tt>processor</tt>
*/
- private void setFormat(Processor processor, Format format)
+ protected void setFormat(Processor processor, Format format)
{
TrackControl[] trackControls = processor.getTrackControls();
MediaType mediaType = getMediaType();
@@ -1276,7 +1277,36 @@
case VIDEO:
if (supportedFormats[0] instanceof VideoFormat)
{
- supportedFormat
+ VideoFormat tmp = (VideoFormat)format;
+ Dimension size = tmp.getSize();
+
+ if(size != null)
+ {
+ /* We have been explictely told to use
+ * a specified output size so create a
+ * custom SwScaler that will rescale and
+ * change format in one call
+ */
+ Codec ar[] = new Codec[1];
+ SwScaler scaler = new SwScaler();
+
+ scaler.setOutputSize(size);
+ ar[0] = scaler;
+
+ /* add our custom SwScaler to the codec chain so that
+ * it will be used instead of default SwScaler
+ */
+ try
+ {
+ trackControl.setCodecChain(ar);
+ }
+ catch(Exception e)
+ {
+ System.out.println("Error setCodecChain: " + e);
+ }
+ }
+
+ supportedFormat
= findFirstMatchingFormat(supportedFormats, format);

                /\*

Modified: trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java?view=diff&rev=6787&p1=trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java&p2=trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java&r1=6786&r2=6787

--- trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java (original)
+++ trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java 2010-02-22 17:02:32+0000
@@ -9,6 +9,7 @@
import java.awt.*;

import javax.media.*;
+import javax.media.format.*;
import javax.media.control.*;
import javax.media.protocol.*;

@@ -48,6 +49,16 @@
private Player localPlayer = null;

/\*\*

+ * Output size of the stream.
+ *
+ * It is used to specify a different size (generally lesser ones)
+ * than the capture device provides. Typically one usage can be
+ * in desktop streaming/sharing session when sender desktop is bigger
+ * than remote ones.
+ */
+ private Dimension outputSize = null;
+
+ /**
* Initializes a new <tt>VideoMediaDeviceSession</tt> instance which is to
* represent the work of a <tt>MediaStream</tt> with a specific video
* <tt>MediaDevice</tt>.
@@ -148,6 +159,31 @@
}

/\*\*

+ * Sets the JMF <tt>Format</tt> in which a specific <tt>Processor</tt> is to
+ * output media data.
+ *
+ * @param processor the <tt>Processor</tt> to set the output <tt>Format</tt>
+ * of
+ * @param format the JMF <tt>Format</tt> to set to <tt>processor</tt>
+ */
+ @Override
+ protected void setFormat(Processor processor, Format format)
+ {
+ Format newFormat = null;
+ VideoFormat tmp = (VideoFormat)format;
+
+ /* add a size in the output format, as VideoFormat has no
+ * set accessors, we recreate the object
+ */
+ if(outputSize != null)
+ {
+ newFormat = new VideoFormat(tmp.getEncoding(), outputSize, tmp.getMaxDataLength(), tmp.getDataType(), tmp.getFrameRate());
+ }
+
+ super.setFormat(processor, newFormat != null ? newFormat : format);
+ }
+
+ /**
* Asserts that a specific <tt>MediaDevice</tt> is acceptable to be set as
* the <tt>MediaDevice</tt> of this instance. Makes sure that its
* <tt>MediaType</tt> is {@link MediaType#VIDEO}.

Modified: trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java?view=diff&rev=6787&p1=trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java&p2=trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java&r1=6786&r2=6787

--- trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java (original)
+++ trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java 2010-02-22 17:02:32+0000
@@ -9,9 +9,10 @@
import java.awt.image.*;

/**
- * This class uses native code to capture
- * desktop screen. It should work for Windows and
- * X11-based Unix such as Linux and FreeBSD.
+ * This class uses native code to capture desktop screen.
+ *
+ * It should work for Windows, Mac OS X and X11-based Unix such as Linux
+ * and FreeBSD.
*
* @author Sebastien Vincent
*/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#2

Hi Lubomir,

On GNU/Linux Debian x86-64 with OpenJDK I have the following results for tests:
- Stream 1280x1024 (no resizing), CPU: arround 64% RAM: arround 330 MB;
- Stream resized 1280x1024 to 1000x800, CPU: arround 64% RAM arround 320 MB

I will do tests with pre-SwScaler version of SC today so that we could compare.

···

--
Seb

Lubomir Marinov a �crit :

Hello Seb,

Have you been able to make a rough evaluation of the performance
differences between the JMF converter+scaler and the libswscale
implementation? I remember the quality of the scaled image was
improved by libswscale in comparison to JMF scaler and I'm quite
interested in the execution times and the garbage collection.

Regards,
Lubo

On Mon, Feb 22, 2010 at 7:02 PM, <s_vincent@dev.java.net> wrote:
  

Author: s_vincent
Date: 2010-02-22 17:02:32+0000
New Revision: 6787

Added:
  trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/SwScaler.java
Modified:
  trunk/lib/native/freebsd-64/libffmpeg.so
  trunk/lib/native/freebsd/libffmpeg.so
  trunk/lib/native/linux-64/libffmpeg.so
  trunk/lib/native/linux/libffmpeg.so
  trunk/lib/native/mac/libffmpeg.jnilib
  trunk/lib/native/windows-64/ffmpeg.dll
  trunk/lib/native/windows/ffmpeg.dll
  trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c
  trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h
  trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java
  trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java
  trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java
  trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java
  trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java

Log:
- Add new scaler that uses libswscale and can change format (i.e. RGB to YUV, ...) and scale in one pass;
- Add the possibility to change output video size of a stream in VideoMediaDeviceSession;
- Add recompiled ffmpeg JNI for all supported platforms.

Modified: trunk/lib/native/freebsd-64/libffmpeg.so
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/freebsd-64/libffmpeg.so?view=diff&rev=6787&p1=trunk/lib/native/freebsd-64/libffmpeg.so&p2=trunk/lib/native/freebsd-64/libffmpeg.so&r1=6786&r2=6787

Binary files. No diff available.

Modified: trunk/lib/native/freebsd/libffmpeg.so
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/freebsd/libffmpeg.so?view=diff&rev=6787&p1=trunk/lib/native/freebsd/libffmpeg.so&p2=trunk/lib/native/freebsd/libffmpeg.so&r1=6786&r2=6787

Binary files. No diff available.

Modified: trunk/lib/native/linux-64/libffmpeg.so
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/linux-64/libffmpeg.so?view=diff&rev=6787&p1=trunk/lib/native/linux-64/libffmpeg.so&p2=trunk/lib/native/linux-64/libffmpeg.so&r1=6786&r2=6787

Binary files. No diff available.

Modified: trunk/lib/native/linux/libffmpeg.so
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/linux/libffmpeg.so?view=diff&rev=6787&p1=trunk/lib/native/linux/libffmpeg.so&p2=trunk/lib/native/linux/libffmpeg.so&r1=6786&r2=6787

Binary files. No diff available.

Modified: trunk/lib/native/mac/libffmpeg.jnilib
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/mac/libffmpeg.jnilib?view=diff&rev=6787&p1=trunk/lib/native/mac/libffmpeg.jnilib&p2=trunk/lib/native/mac/libffmpeg.jnilib&r1=6786&r2=6787

Binary files. No diff available.

Modified: trunk/lib/native/windows-64/ffmpeg.dll
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/windows-64/ffmpeg.dll?view=diff&rev=6787&p1=trunk/lib/native/windows-64/ffmpeg.dll&p2=trunk/lib/native/windows-64/ffmpeg.dll&r1=6786&r2=6787

Binary files. No diff available.

Modified: trunk/lib/native/windows/ffmpeg.dll
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/windows/ffmpeg.dll?view=diff&rev=6787&p1=trunk/lib/native/windows/ffmpeg.dll&p2=trunk/lib/native/windows/ffmpeg.dll&r1=6786&r2=6787

Binary files. No diff available.

Modified: trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c?view=diff&rev=6787&p1=trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c&p2=trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c&r1=6786&r2=6787

--- trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c (original)
+++ trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c 2010-02-22 17:02:32+0000
@@ -438,15 +438,15 @@

static int image_convert(
AVPicture* dst, int dst_pix_fmt,
-const AVPicture* src, int pix_fmt, int width, int height)
+const AVPicture* src, int pix_fmt, int width, int height, jint newWidth, jint newHeight)
{
   struct SwsContext *img_convert_ctx =
        sws_getContext(
            width, height, pix_fmt,
- width, height, dst_pix_fmt, SWS_BICUBIC, NULL, NULL, NULL);
+ newWidth, newHeight, dst_pix_fmt, SWS_BICUBIC, NULL, NULL, NULL);

   int result = sws_scale(img_convert_ctx,
- src->data, src->linesize, 0, height,
+ (uint8_t**)src->data, (int*)src->linesize, 0, height,
        dst->data, dst->linesize);
   sws_freeContext(img_convert_ctx);

@@ -454,7 +454,7 @@
}

JNIEXPORT jint JNICALL
-Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_img_1convert (
+Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_img_1convert__JIJIII (
        JNIEnv *jniEnv, jclass clazz, jlong dst, jint dst_pix_fmt, jlong src,
        jint pix_fmt, jint width, jint height)
{
@@ -465,9 +465,50 @@
            (const AVPicture *) src,
            (int) pix_fmt,
            (int) width,
+ (int) height,
+ (int) width,
            (int) height);
}

+JNIEXPORT jint JNICALL
+Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_img_1convert___3BI_3BIIIII
+ (JNIEnv *jniEnv, jclass clazz, jbyteArray dst, jint dst_pix_fmt,
+ jbyteArray src, jint pix_fmt, jint width, jint height, jint newWidth, jint newHeight)
+{
+ AVPicture dst_dummy;
+ AVPicture src_dummy;
+ int size = width * height;
+ jint ret = 0;
+/*
+ uint8_t* dst_buf = (*jniEnv)->GetPrimitiveArrayCritical(jniEnv, dst, 0);
+ uint8_t* src_buf = (*jniEnv)->GetPrimitiveArrayCritical(jniEnv, src, 0);
+*/
+ uint8_t* dst_buf = (*jniEnv)->GetByteArrayElements(jniEnv, dst, 0);
+ uint8_t* src_buf = (*jniEnv)->GetByteArrayElements(jniEnv, src, 0);
+
+ if(!src || !dst)
+ {
+ return -1;
+ }
+
+ /* assign AVPicture with buffer */
+ avpicture_fill(&dst_dummy, dst_buf, (int)dst_pix_fmt, newWidth, newHeight);
+ avpicture_fill(&src_dummy, src_buf, (int)pix_fmt, width, height);
+
+ ret = image_convert(&dst_dummy, (int)dst_pix_fmt, &src_dummy, (int)pix_fmt,
+ (int)width, (int)height, (int)newWidth, (int)newHeight);
+
+ /* release pointers */
+/*
+ (*jniEnv)->ReleasePrimitiveArrayCritical(jniEnv, dst, dst_buf, 0);
+ (*jniEnv)->ReleasePrimitiveArrayCritical(jniEnv, src, src_buf, 0);
+*/
+ (*jniEnv)->ReleaseByteArrayElements(jniEnv, dst, dst_buf, 0);
+ (*jniEnv)->ReleaseByteArrayElements(jniEnv, src, src_buf, 0);
+ return ret;
+}
+
JNIEXPORT void JNICALL
Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_memcpy___3IIIJ (
        JNIEnv *jniEnv, jclass clazz, jintArray dst, jint dst_offset,
@@ -492,6 +533,13 @@
}

JNIEXPORT jint JNICALL
+Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_getRGBAFormat
+ (JNIEnv *env, jclass clazz)
+{
+ return PIX_FMT_RGBA;
+}
+
+JNIEXPORT jint JNICALL
Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_getYUV420PFormat
  (JNIEnv *env, jclass clazz)
{

Modified: trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h?view=diff&rev=6787&p1=trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h&p2=trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h&r1=6786&r2=6787

--- trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h (original)
+++ trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h 2010-02-22 17:02:32+0000
@@ -31,6 +31,14 @@

/*
* Class: net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG
+ * Method: getRGBAFormat
+ * Signature: ()I
+ */
+JNIEXPORT jint JNICALL Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_getRGBAFormat
+ (JNIEnv *, jclass);
+
+/*
+ * Class: net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG
* Method: getYUV420PFormat
* Signature: ()I
*/
@@ -458,11 +466,19 @@
* Method: img_convert
* Signature: (JIJIII)I
*/
-JNIEXPORT jint JNICALL Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_img_1convert
+JNIEXPORT jint JNICALL Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_img_1convert__JIJIII
  (JNIEnv *, jclass, jlong, jint, jlong, jint, jint, jint);

/*
* Class: net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG
+ * Method: img_convert
+ * Signature: ([BI[BIIIII)I
+ */
+JNIEXPORT jint JNICALL Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_img_1convert___3BI_3BIIIII
+ (JNIEnv *, jclass, jbyteArray, jint, jbyteArray, jint, jint, jint, jint, jint);
+
+/*
+ * Class: net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG
* Method: memcpy
* Signature: ([IIIJ)V
*/

Modified: trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java?view=diff&rev=6787&p1=trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java&p2=trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java&r1=6786&r2=6787

--- trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java (original)
+++ trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java 2010-02-22 17:02:32+0000
@@ -62,7 +62,8 @@
            "net.java.sip.communicator.impl.neomedia.codec.video.h264.JNIEncoder",
            "net.java.sip.communicator.impl.neomedia.codec.video.h264.Packetizer",
            "net.java.sip.communicator.impl.neomedia.codec.video.h264.JNIDecoder",
- "net.java.sip.communicator.impl.neomedia.codec.video.ImageScaler",
+ //"net.java.sip.communicator.impl.neomedia.codec.video.ImageScaler",
+ "net.java.sip.communicator.impl.neomedia.codec.video.SwScaler",
            "net.java.sip.communicator.impl.neomedia.codec.audio.speex.JavaEncoder",
            "net.java.sip.communicator.impl.neomedia.codec.audio.speex.JavaDecoder",
            "net.java.sip.communicator.impl.neomedia.codec.audio.ilbc.JavaEncoder",
@@ -328,6 +329,9 @@
                        .getPlugInList(null, null, PlugInManager.CODEC));
        boolean commit = false;

+ /* remove JavaRGBToYUV */
+ PlugInManager.removePlugIn("com.sun.media.codec.video.colorspace.JavaRGBToYUV", PlugInManager.CODEC);
+
        for (String className : CUSTOM_CODECS)
        {

Modified: trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java?view=diff&rev=6787&p1=trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java&p2=trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java&r1=6786&r2=6787

--- trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java (original)
+++ trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java 2010-02-22 17:02:32+0000
@@ -24,12 +24,16 @@
    public static final int FF_MB_DECISION_SIMPLE = 0;

    public static final int PIX_FMT_RGB32;
+
+ public static final int PIX_FMT_RGBA;

    public static final int PIX_FMT_YUV420P;

    public static final int X264_RC_ABR = 2;

    public static native int getRGB32Format();
+
+ public static native int getRGBAFormat();

    public static native int getYUV420PFormat();

@@ -172,6 +176,10 @@

    public static native int img_convert(long dst, int dst_pix_fmt, long src,
        int pix_fmt, int width, int height);
+
+ public static native int img_convert(byte dst[], int dst_pix_fmt,
+ byte src[], int pix_fmt, int width, int height, int newWidth,
+ int newHeight);

    public static native void memcpy(int[] dst, int dst_offset, int dst_length,
        long src);
@@ -185,7 +193,8 @@

        av_register_all();
        avcodec_init();
- PIX_FMT_RGB32 = getRGB32Format();
+ PIX_FMT_RGB32 = getRGB32Format(); /* for decoding */
+ PIX_FMT_RGBA = getRGBAFormat(); /* for encoding */
        PIX_FMT_YUV420P = getYUV420PFormat();
    }
}

Added: trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/SwScaler.java
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/SwScaler.java?view=auto&rev=6787

--- (empty file)
+++ trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/SwScaler.java 2010-02-22 17:02:32+0000
@@ -0,0 +1,196 @@
+/*
+ * SIP Communicator, the OpenSource Java VoIP and Instant Messaging client.
+ *
+ * Distributable under LGPL license.
+ * See terms of license at gnu.org.
+ */
+package net.java.sip.communicator.impl.neomedia.codec.video;
+
+import java.awt.*;
+
+import javax.media.*;
+import javax.media.format.*;
+
+import net.sf.fmj.media.*;
+
+/**
+ * Codec that use libswccale to scale images from one size to
+ * another and change format (typically RGB to YUV).
+ *
+ * @author Sebastien Vincent
+ */
+public class SwScaler
+ extends AbstractCodec
+ implements Codec
+{
+
+ /**
+ * Supported input formats.
+ */
+ private final Format[] supportedInputFormats = new Format[] {
+ new RGBFormat(null, -1, Format.byteArray, -1.0f, 32, -1, -1, -1)
+ };
+
+ /**
+ * Supported output formats.
+ */
+ private Format[] supportedOutputFormats = new Format[] {
+ new YUVFormat(null, -1, Format.byteArray, -1.0f, YUVFormat.YUV_420, -1, -1, 0, -1, -1)
+ };
+
+ /**
+ * Set output size.
+ *
+ * @param size size to set
+ */
+ public void setOutputSize(Dimension size)
+ {
+ if(size == null)
+ {
+ size = new Dimension(640, 480);
+ }
+
+ Format newFormat = new YUVFormat(size, -1, Format.byteArray, -1.0f, YUVFormat.YUV_420, -1, -1, 0, -1, -1);
+ supportedOutputFormats[0] = newFormat;
+ }
+
+ /**
+ * Get the supported input formats.
+ *
+ * @return array of supported input format
+ */
+ @Override
+ public Format[] getSupportedInputFormats()
+ {
+ return supportedInputFormats;
+ }
+
+ /**
+ * Get the supported output formats for an input ones.
+ *
+ * @param input input format to convert
+ * @return array of supported output format
+ */
+ @Override
+ public Format[] getSupportedOutputFormats(Format input)
+ {
+ if(input == null)
+ {
+ return supportedOutputFormats;
+ }
+
+ //System.out.println("input: " + ((VideoFormat)input).getSize());
+ if(((VideoFormat)supportedOutputFormats[0]).getSize() != null)
+ {
+ return supportedOutputFormats;
+ }
+
+ return new Format[] { new YUVFormat(((VideoFormat)input).getSize(), -1, Format.byteArray, -1.0f, YUVFormat.YUV_420, -1, -1, 0, -1, -1)};
+ }
+
+ /**
+ * Set the input format.
+ *
+ * @param format format to set
+ * @return format
+ */
+ @Override
+ public Format setInputFormat(Format format)
+ {
+ //System.out.println("setInput: " + ((VideoFormat)format).getSize());
+ final VideoFormat videoFormat = (VideoFormat) format;
+
+ if (videoFormat.getSize() == null)
+ return null; // must set a size.
+
+ format = super.setInputFormat(format);
+ return format;
+ }
+
+ /**
+ * Set the output format.
+ *
+ * @param format format to set
+ * @return format
+ */
+ @Override
+ public Format setOutputFormat(Format format)
+ {
+ //System.out.println("setOutput: " + ((VideoFormat)format).getSize());
+ format = super.setOutputFormat(format);
+ return format;
+ }
+
+ /**
+ * Process (format convertion, rescale) a buffer.
+ *
+ * @param input input buffer
+ * @param output output buffer
+ * @return BUFFER_PROCESSED_OK if buffer successfully processed
+ */
+ @Override
+ public int process(Buffer input, Buffer output)
+ {
+ VideoFormat vinput = (VideoFormat)input.getFormat();
+ VideoFormat voutput = (VideoFormat)output.getFormat();
+ int inputWidth = (int)vinput.getSize().getWidth();
+ int inputHeight = (int)vinput.getSize().getHeight();
+ int outputWidth = (int)voutput.getSize().getWidth();
+ int outputHeight = (int)voutput.getSize().getHeight();
+ byte src[] = (byte[])input.getData();
+ byte dst[] = (byte[])output.getData();
+ int outputSize = 0;
+ int infmt = 0;
+ int outfmt = 0;
+
+ if (!checkInputBuffer(input))
+ {
+ return BUFFER_PROCESSED_FAILED;
+ }
+
+ if (isEOM(input))
+ {
+ propagateEOM(output); // TODO: what about data? can there be any?
+ return BUFFER_PROCESSED_OK;
+ }
+
+ /* determine output format and output size needed */
+ if(voutput instanceof YUVFormat)
+ {
+ /* YUV420P is 12 bpp (bit per pixel) => 1,5 bytes */
+ outputSize = (int)(outputWidth * outputHeight * 1.5);
+ outfmt = FFMPEG.PIX_FMT_YUV420P;
+ }
+ else /* RGB format */
+ {
+ outputSize = (outputWidth * outputHeight * 4);
+ outfmt = FFMPEG.PIX_FMT_RGBA;
+ }
+
+ /* determine input format */
+ if(vinput instanceof YUVFormat)
+ {
+ infmt = FFMPEG.PIX_FMT_YUV420P;
+ }
+ else /* RGBFormat */
+ {
+ infmt = FFMPEG.PIX_FMT_RGBA;
+ }
+
+ if(dst == null || dst.length < outputSize)
+ {
+ dst = new byte[outputSize];
+ }
+
+ /* convertion! */
+ //System.out.println("Convert: " + inputWidth + "x" + inputHeight + " to " + outputWidth + "x" + outputHeight);
+ FFMPEG.img_convert(dst, outfmt, src, infmt, inputWidth, inputHeight, outputWidth, outputHeight);
+
+ output.setData(dst);
+ output.setLength(dst.length);
+ output.setOffset(0);
+
+ return BUFFER_PROCESSED_OK;
+ }
+}
+

Modified: trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java?view=diff&rev=6787&p1=trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java&p2=trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java&r1=6786&r2=6787

--- trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java (original)
+++ trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java 2010-02-22 17:02:32+0000
@@ -19,6 +19,7 @@
import net.java.sip.communicator.impl.neomedia.*;
import net.java.sip.communicator.impl.neomedia.format.*;
import net.java.sip.communicator.impl.neomedia.protocol.*;
+import net.java.sip.communicator.impl.neomedia.codec.video.*;
import net.java.sip.communicator.service.neomedia.*;
import net.java.sip.communicator.service.neomedia.device.*;
import net.java.sip.communicator.service.neomedia.format.*;
@@ -1226,7 +1227,7 @@
     * of
     * @param format the JMF <tt>Format</tt> to set to <tt>processor</tt>
     */
- private void setFormat(Processor processor, Format format)
+ protected void setFormat(Processor processor, Format format)
    {
        TrackControl[] trackControls = processor.getTrackControls();
        MediaType mediaType = getMediaType();
@@ -1276,7 +1277,36 @@
            case VIDEO:
                if (supportedFormats[0] instanceof VideoFormat)
                {
- supportedFormat
+ VideoFormat tmp = (VideoFormat)format;
+ Dimension size = tmp.getSize();
+
+ if(size != null)
+ {
+ /* We have been explictely told to use
+ * a specified output size so create a
+ * custom SwScaler that will rescale and
+ * change format in one call
+ */
+ Codec ar[] = new Codec[1];
+ SwScaler scaler = new SwScaler();
+
+ scaler.setOutputSize(size);
+ ar[0] = scaler;
+
+ /* add our custom SwScaler to the codec chain so that
+ * it will be used instead of default SwScaler
+ */
+ try
+ {
+ trackControl.setCodecChain(ar);
+ }
+ catch(Exception e)
+ {
+ System.out.println("Error setCodecChain: " + e);
+ }
+ }
+
+ supportedFormat
                        = findFirstMatchingFormat(supportedFormats, format);

                    /*

Modified: trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java?view=diff&rev=6787&p1=trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java&p2=trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java&r1=6786&r2=6787

--- trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java (original)
+++ trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java 2010-02-22 17:02:32+0000
@@ -9,6 +9,7 @@
import java.awt.*;

import javax.media.*;
+import javax.media.format.*;
import javax.media.control.*;
import javax.media.protocol.*;

@@ -48,6 +49,16 @@
    private Player localPlayer = null;

    /**
+ * Output size of the stream.
+ *
+ * It is used to specify a different size (generally lesser ones)
+ * than the capture device provides. Typically one usage can be
+ * in desktop streaming/sharing session when sender desktop is bigger
+ * than remote ones.
+ */
+ private Dimension outputSize = null;
+
+ /**
     * Initializes a new <tt>VideoMediaDeviceSession</tt> instance which is to
     * represent the work of a <tt>MediaStream</tt> with a specific video
     * <tt>MediaDevice</tt>.
@@ -148,6 +159,31 @@
    }

    /**
+ * Sets the JMF <tt>Format</tt> in which a specific <tt>Processor</tt> is to
+ * output media data.
+ *
+ * @param processor the <tt>Processor</tt> to set the output <tt>Format</tt>
+ * of
+ * @param format the JMF <tt>Format</tt> to set to <tt>processor</tt>
+ */
+ @Override
+ protected void setFormat(Processor processor, Format format)
+ {
+ Format newFormat = null;
+ VideoFormat tmp = (VideoFormat)format;
+
+ /* add a size in the output format, as VideoFormat has no
+ * set accessors, we recreate the object
+ */
+ if(outputSize != null)
+ {
+ newFormat = new VideoFormat(tmp.getEncoding(), outputSize, tmp.getMaxDataLength(), tmp.getDataType(), tmp.getFrameRate());
+ }
+
+ super.setFormat(processor, newFormat != null ? newFormat : format);
+ }
+
+ /**
     * Asserts that a specific <tt>MediaDevice</tt> is acceptable to be set as
     * the <tt>MediaDevice</tt> of this instance. Makes sure that its
     * <tt>MediaType</tt> is {@link MediaType#VIDEO}.

Modified: trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java?view=diff&rev=6787&p1=trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java&p2=trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java&r1=6786&r2=6787

--- trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java (original)
+++ trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java 2010-02-22 17:02:32+0000
@@ -9,9 +9,10 @@
import java.awt.image.*;

/**
- * This class uses native code to capture
- * desktop screen. It should work for Windows and
- * X11-based Unix such as Linux and FreeBSD.
+ * This class uses native code to capture desktop screen.
+ *
+ * It should work for Windows, Mac OS X and X11-based Unix such as Linux
+ * and FreeBSD.
*
* @author Sebastien Vincent
*/
    
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#3

Sebastien Vincent a �crit :

Hi Lubomir,

On GNU/Linux Debian x86-64 with OpenJDK I have the following results for tests:
- Stream 1280x1024 (no resizing), CPU: arround 64% RAM: arround 330 MB;
- Stream resized 1280x1024 to 1000x800, CPU: arround 64% RAM arround 320 MB

I will do tests with pre-SwScaler version of SC today so that we could compare.

Without SwScaler I have:
- Stream 1280x1024 (no resizing), CPU: arround 62% RAM: arround 352 MB;
- Stream resized 1280x1024 to 1000x800, CPU: arround 62% RAM arround 345 MB

So apparently there is not really CPU usage reduction with SwScaler (which may be explain by libswscale takes more CPU to do its job or JVM/JNI switching) but we have two benefits. First, less memory usage (amount 20 MB) and second rescaled images (1280x1024 to 1000x800 in my tests) are strongly better with SwScaler.

Regards,

···

--
Seb

--
Seb

Lubomir Marinov a �crit :

Hello Seb,

Have you been able to make a rough evaluation of the performance
differences between the JMF converter+scaler and the libswscale
implementation? I remember the quality of the scaled image was
improved by libswscale in comparison to JMF scaler and I'm quite
interested in the execution times and the garbage collection.

Regards,
Lubo

On Mon, Feb 22, 2010 at 7:02 PM, <s_vincent@dev.java.net> wrote:

Author: s_vincent
Date: 2010-02-22 17:02:32+0000
New Revision: 6787

Added:
  trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/SwScaler.java

Modified:
  trunk/lib/native/freebsd-64/libffmpeg.so
  trunk/lib/native/freebsd/libffmpeg.so
  trunk/lib/native/linux-64/libffmpeg.so
  trunk/lib/native/linux/libffmpeg.so
  trunk/lib/native/mac/libffmpeg.jnilib
  trunk/lib/native/windows-64/ffmpeg.dll
  trunk/lib/native/windows/ffmpeg.dll
  trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c

  trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h

  trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java

  trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java

  trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java

  trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java

  trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java

Log:
- Add new scaler that uses libswscale and can change format (i.e. RGB to YUV, ...) and scale in one pass;
- Add the possibility to change output video size of a stream in VideoMediaDeviceSession;
- Add recompiled ffmpeg JNI for all supported platforms.

Modified: trunk/lib/native/freebsd-64/libffmpeg.so
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/freebsd-64/libffmpeg.so?view=diff&rev=6787&p1=trunk/lib/native/freebsd-64/libffmpeg.so&p2=trunk/lib/native/freebsd-64/libffmpeg.so&r1=6786&r2=6787

==============================================================================

Binary files. No diff available.

Modified: trunk/lib/native/freebsd/libffmpeg.so
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/freebsd/libffmpeg.so?view=diff&rev=6787&p1=trunk/lib/native/freebsd/libffmpeg.so&p2=trunk/lib/native/freebsd/libffmpeg.so&r1=6786&r2=6787

==============================================================================

Binary files. No diff available.

Modified: trunk/lib/native/linux-64/libffmpeg.so
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/linux-64/libffmpeg.so?view=diff&rev=6787&p1=trunk/lib/native/linux-64/libffmpeg.so&p2=trunk/lib/native/linux-64/libffmpeg.so&r1=6786&r2=6787

==============================================================================

Binary files. No diff available.

Modified: trunk/lib/native/linux/libffmpeg.so
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/linux/libffmpeg.so?view=diff&rev=6787&p1=trunk/lib/native/linux/libffmpeg.so&p2=trunk/lib/native/linux/libffmpeg.so&r1=6786&r2=6787

==============================================================================

Binary files. No diff available.

Modified: trunk/lib/native/mac/libffmpeg.jnilib
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/mac/libffmpeg.jnilib?view=diff&rev=6787&p1=trunk/lib/native/mac/libffmpeg.jnilib&p2=trunk/lib/native/mac/libffmpeg.jnilib&r1=6786&r2=6787

==============================================================================

Binary files. No diff available.

Modified: trunk/lib/native/windows-64/ffmpeg.dll
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/windows-64/ffmpeg.dll?view=diff&rev=6787&p1=trunk/lib/native/windows-64/ffmpeg.dll&p2=trunk/lib/native/windows-64/ffmpeg.dll&r1=6786&r2=6787

==============================================================================

Binary files. No diff available.

Modified: trunk/lib/native/windows/ffmpeg.dll
Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/lib/native/windows/ffmpeg.dll?view=diff&rev=6787&p1=trunk/lib/native/windows/ffmpeg.dll&p2=trunk/lib/native/windows/ffmpeg.dll&r1=6786&r2=6787

==============================================================================

Binary files. No diff available.

Modified: trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c

Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c?view=diff&rev=6787&p1=trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c&p2=trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c&r1=6786&r2=6787

==============================================================================

--- trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c (original)
+++ trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.c 2010-02-22 17:02:32+0000
@@ -438,15 +438,15 @@

static int image_convert(
AVPicture* dst, int dst_pix_fmt,
-const AVPicture* src, int pix_fmt, int width, int height)
+const AVPicture* src, int pix_fmt, int width, int height, jint newWidth, jint newHeight)
{
   struct SwsContext *img_convert_ctx =
        sws_getContext(
            width, height, pix_fmt,
- width, height, dst_pix_fmt, SWS_BICUBIC, NULL, NULL, NULL);
+ newWidth, newHeight, dst_pix_fmt, SWS_BICUBIC, NULL, NULL, NULL);

   int result = sws_scale(img_convert_ctx,
- src->data, src->linesize, 0, height,
+ (uint8_t**)src->data, (int*)src->linesize, 0, height,
        dst->data, dst->linesize);
   sws_freeContext(img_convert_ctx);

@@ -454,7 +454,7 @@
}

JNIEXPORT jint JNICALL
-Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_img_1convert (
+Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_img_1convert__JIJIII (
        JNIEnv *jniEnv, jclass clazz, jlong dst, jint dst_pix_fmt, jlong src,
        jint pix_fmt, jint width, jint height)
{
@@ -465,9 +465,50 @@
            (const AVPicture *) src,
            (int) pix_fmt,
            (int) width,
+ (int) height,
+ (int) width,
            (int) height);
}

+JNIEXPORT jint JNICALL
+Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_img_1convert___3BI_3BIIIII

+ (JNIEnv *jniEnv, jclass clazz, jbyteArray dst, jint dst_pix_fmt,
+ jbyteArray src, jint pix_fmt, jint width, jint height, jint newWidth, jint newHeight)
+{
+ AVPicture dst_dummy;
+ AVPicture src_dummy;
+ int size = width * height;
+ jint ret = 0;
+/*
+ uint8_t* dst_buf = (*jniEnv)->GetPrimitiveArrayCritical(jniEnv, dst, 0);
+ uint8_t* src_buf = (*jniEnv)->GetPrimitiveArrayCritical(jniEnv, src, 0);
+*/
+ uint8_t* dst_buf = (*jniEnv)->GetByteArrayElements(jniEnv, dst, 0);
+ uint8_t* src_buf = (*jniEnv)->GetByteArrayElements(jniEnv, src, 0);
+
+ if(!src || !dst)
+ {
+ return -1;
+ }
+
+ /* assign AVPicture with buffer */
+ avpicture_fill(&dst_dummy, dst_buf, (int)dst_pix_fmt, newWidth, newHeight);
+ avpicture_fill(&src_dummy, src_buf, (int)pix_fmt, width, height);
+
+ ret = image_convert(&dst_dummy, (int)dst_pix_fmt, &src_dummy, (int)pix_fmt,
+ (int)width, (int)height, (int)newWidth, (int)newHeight);
+
+ /* release pointers */
+/*
+ (*jniEnv)->ReleasePrimitiveArrayCritical(jniEnv, dst, dst_buf, 0);
+ (*jniEnv)->ReleasePrimitiveArrayCritical(jniEnv, src, src_buf, 0);
+*/
+ (*jniEnv)->ReleaseByteArrayElements(jniEnv, dst, dst_buf, 0);
+ (*jniEnv)->ReleaseByteArrayElements(jniEnv, src, src_buf, 0);
+ return ret;
+}
+
JNIEXPORT void JNICALL
Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_memcpy___3IIIJ (
        JNIEnv *jniEnv, jclass clazz, jintArray dst, jint dst_offset,
@@ -492,6 +533,13 @@
}

JNIEXPORT jint JNICALL
+Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_getRGBAFormat

+ (JNIEnv *env, jclass clazz)
+{
+ return PIX_FMT_RGBA;
+}
+
+JNIEXPORT jint JNICALL
Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_getYUV420PFormat

  (JNIEnv *env, jclass clazz)
{

Modified: trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h

Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h?view=diff&rev=6787&p1=trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h&p2=trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h&r1=6786&r2=6787

==============================================================================

--- trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h (original)
+++ trunk/src/native/ffmpeg/net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG.h 2010-02-22 17:02:32+0000
@@ -31,6 +31,14 @@

/*
* Class: net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG
+ * Method: getRGBAFormat
+ * Signature: ()I
+ */
+JNIEXPORT jint JNICALL Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_getRGBAFormat

+ (JNIEnv *, jclass);
+
+/*
+ * Class: net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG
* Method: getYUV420PFormat
* Signature: ()I
*/
@@ -458,11 +466,19 @@
* Method: img_convert
* Signature: (JIJIII)I
*/
-JNIEXPORT jint JNICALL Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_img_1convert

+JNIEXPORT jint JNICALL Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_img_1convert__JIJIII

  (JNIEnv *, jclass, jlong, jint, jlong, jint, jint, jint);

/*
* Class: net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG
+ * Method: img_convert
+ * Signature: ([BI[BIIIII)I
+ */
+JNIEXPORT jint JNICALL Java_net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG_img_1convert___3BI_3BIIIII

+ (JNIEnv *, jclass, jbyteArray, jint, jbyteArray, jint, jint, jint, jint, jint);
+
+/*
+ * Class: net_java_sip_communicator_impl_neomedia_codec_video_FFMPEG
* Method: memcpy
* Signature: ([IIIJ)V
*/

Modified: trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java

Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java?view=diff&rev=6787&p1=trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java&p2=trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java&r1=6786&r2=6787

==============================================================================

--- trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java (original)
+++ trunk/src/net/java/sip/communicator/impl/neomedia/codec/EncodingConfiguration.java 2010-02-22 17:02:32+0000
@@ -62,7 +62,8 @@
            "net.java.sip.communicator.impl.neomedia.codec.video.h264.JNIEncoder",
            "net.java.sip.communicator.impl.neomedia.codec.video.h264.Packetizer",
            "net.java.sip.communicator.impl.neomedia.codec.video.h264.JNIDecoder",
- "net.java.sip.communicator.impl.neomedia.codec.video.ImageScaler",
+ //"net.java.sip.communicator.impl.neomedia.codec.video.ImageScaler",
+ "net.java.sip.communicator.impl.neomedia.codec.video.SwScaler",
            "net.java.sip.communicator.impl.neomedia.codec.audio.speex.JavaEncoder",

            "net.java.sip.communicator.impl.neomedia.codec.audio.speex.JavaDecoder",

            "net.java.sip.communicator.impl.neomedia.codec.audio.ilbc.JavaEncoder",
@@ -328,6 +329,9 @@
                        .getPlugInList(null, null, PlugInManager.CODEC));
        boolean commit = false;

+ /* remove JavaRGBToYUV */
+ PlugInManager.removePlugIn("com.sun.media.codec.video.colorspace.JavaRGBToYUV", PlugInManager.CODEC);
+
        for (String className : CUSTOM_CODECS)
        {

Modified: trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java

Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java?view=diff&rev=6787&p1=trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java&p2=trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java&r1=6786&r2=6787

==============================================================================

--- trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java (original)
+++ trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/FFMPEG.java 2010-02-22 17:02:32+0000
@@ -24,12 +24,16 @@
    public static final int FF_MB_DECISION_SIMPLE = 0;

    public static final int PIX_FMT_RGB32;
+
+ public static final int PIX_FMT_RGBA;

    public static final int PIX_FMT_YUV420P;

    public static final int X264_RC_ABR = 2;

    public static native int getRGB32Format();
+
+ public static native int getRGBAFormat();

    public static native int getYUV420PFormat();

@@ -172,6 +176,10 @@

    public static native int img_convert(long dst, int dst_pix_fmt, long src,
        int pix_fmt, int width, int height);
+
+ public static native int img_convert(byte dst[], int dst_pix_fmt,
+ byte src[], int pix_fmt, int width, int height, int newWidth,
+ int newHeight);

    public static native void memcpy(int[] dst, int dst_offset, int dst_length,
        long src);
@@ -185,7 +193,8 @@

        av_register_all();
        avcodec_init();
- PIX_FMT_RGB32 = getRGB32Format();
+ PIX_FMT_RGB32 = getRGB32Format(); /* for decoding */
+ PIX_FMT_RGBA = getRGBAFormat(); /* for encoding */
        PIX_FMT_YUV420P = getYUV420PFormat();
    }
}

Added: trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/SwScaler.java

Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/SwScaler.java?view=auto&rev=6787

==============================================================================

--- (empty file)
+++ trunk/src/net/java/sip/communicator/impl/neomedia/codec/video/SwScaler.java 2010-02-22 17:02:32+0000
@@ -0,0 +1,196 @@
+/*
+ * SIP Communicator, the OpenSource Java VoIP and Instant Messaging client.
+ *
+ * Distributable under LGPL license.
+ * See terms of license at gnu.org.
+ */
+package net.java.sip.communicator.impl.neomedia.codec.video;
+
+import java.awt.*;
+
+import javax.media.*;
+import javax.media.format.*;
+
+import net.sf.fmj.media.*;
+
+/**
+ * Codec that use libswccale to scale images from one size to
+ * another and change format (typically RGB to YUV).
+ *
+ * @author Sebastien Vincent
+ */
+public class SwScaler
+ extends AbstractCodec
+ implements Codec
+{
+
+ /**
+ * Supported input formats.
+ */
+ private final Format[] supportedInputFormats = new Format[] {
+ new RGBFormat(null, -1, Format.byteArray, -1.0f, 32, -1, -1, -1)
+ };
+
+ /**
+ * Supported output formats.
+ */
+ private Format[] supportedOutputFormats = new Format[] {
+ new YUVFormat(null, -1, Format.byteArray, -1.0f, YUVFormat.YUV_420, -1, -1, 0, -1, -1)
+ };
+
+ /**
+ * Set output size.
+ *
+ * @param size size to set
+ */
+ public void setOutputSize(Dimension size)
+ {
+ if(size == null)
+ {
+ size = new Dimension(640, 480);
+ }
+
+ Format newFormat = new YUVFormat(size, -1, Format.byteArray, -1.0f, YUVFormat.YUV_420, -1, -1, 0, -1, -1);
+ supportedOutputFormats[0] = newFormat;
+ }
+
+ /**
+ * Get the supported input formats.
+ *
+ * @return array of supported input format
+ */
+ @Override
+ public Format[] getSupportedInputFormats()
+ {
+ return supportedInputFormats;
+ }
+
+ /**
+ * Get the supported output formats for an input ones.
+ *
+ * @param input input format to convert
+ * @return array of supported output format
+ */
+ @Override
+ public Format[] getSupportedOutputFormats(Format input)
+ {
+ if(input == null)
+ {
+ return supportedOutputFormats;
+ }
+
+ //System.out.println("input: " + ((VideoFormat)input).getSize());
+ if(((VideoFormat)supportedOutputFormats[0]).getSize() != null)
+ {
+ return supportedOutputFormats;
+ }
+
+ return new Format[] { new YUVFormat(((VideoFormat)input).getSize(), -1, Format.byteArray, -1.0f, YUVFormat.YUV_420, -1, -1, 0, -1, -1)};
+ }
+
+ /**
+ * Set the input format.
+ *
+ * @param format format to set
+ * @return format
+ */
+ @Override
+ public Format setInputFormat(Format format)
+ {
+ //System.out.println("setInput: " + ((VideoFormat)format).getSize());
+ final VideoFormat videoFormat = (VideoFormat) format;
+
+ if (videoFormat.getSize() == null)
+ return null; // must set a size.
+
+ format = super.setInputFormat(format);
+ return format;
+ }
+
+ /**
+ * Set the output format.
+ *
+ * @param format format to set
+ * @return format
+ */
+ @Override
+ public Format setOutputFormat(Format format)
+ {
+ //System.out.println("setOutput: " + ((VideoFormat)format).getSize());
+ format = super.setOutputFormat(format);
+ return format;
+ }
+
+ /**
+ * Process (format convertion, rescale) a buffer.
+ *
+ * @param input input buffer
+ * @param output output buffer
+ * @return BUFFER_PROCESSED_OK if buffer successfully processed
+ */
+ @Override
+ public int process(Buffer input, Buffer output)
+ {
+ VideoFormat vinput = (VideoFormat)input.getFormat();
+ VideoFormat voutput = (VideoFormat)output.getFormat();
+ int inputWidth = (int)vinput.getSize().getWidth();
+ int inputHeight = (int)vinput.getSize().getHeight();
+ int outputWidth = (int)voutput.getSize().getWidth();
+ int outputHeight = (int)voutput.getSize().getHeight();
+ byte src[] = (byte[])input.getData();
+ byte dst[] = (byte[])output.getData();
+ int outputSize = 0;
+ int infmt = 0;
+ int outfmt = 0;
+
+ if (!checkInputBuffer(input))
+ {
+ return BUFFER_PROCESSED_FAILED;
+ }
+
+ if (isEOM(input))
+ {
+ propagateEOM(output); // TODO: what about data? can there be any?
+ return BUFFER_PROCESSED_OK;
+ }
+
+ /* determine output format and output size needed */
+ if(voutput instanceof YUVFormat)
+ {
+ /* YUV420P is 12 bpp (bit per pixel) => 1,5 bytes */
+ outputSize = (int)(outputWidth * outputHeight * 1.5);
+ outfmt = FFMPEG.PIX_FMT_YUV420P;
+ }
+ else /* RGB format */
+ {
+ outputSize = (outputWidth * outputHeight * 4);
+ outfmt = FFMPEG.PIX_FMT_RGBA;
+ }
+
+ /* determine input format */
+ if(vinput instanceof YUVFormat)
+ {
+ infmt = FFMPEG.PIX_FMT_YUV420P;
+ }
+ else /* RGBFormat */
+ {
+ infmt = FFMPEG.PIX_FMT_RGBA;
+ }
+
+ if(dst == null || dst.length < outputSize)
+ {
+ dst = new byte[outputSize];
+ }
+
+ /* convertion! */
+ //System.out.println("Convert: " + inputWidth + "x" + inputHeight + " to " + outputWidth + "x" + outputHeight);
+ FFMPEG.img_convert(dst, outfmt, src, infmt, inputWidth, inputHeight, outputWidth, outputHeight);
+
+ output.setData(dst);
+ output.setLength(dst.length);
+ output.setOffset(0);
+
+ return BUFFER_PROCESSED_OK;
+ }
+}
+

Modified: trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java

Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java?view=diff&rev=6787&p1=trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java&p2=trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java&r1=6786&r2=6787

==============================================================================

--- trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java (original)
+++ trunk/src/net/java/sip/communicator/impl/neomedia/device/MediaDeviceSession.java 2010-02-22 17:02:32+0000
@@ -19,6 +19,7 @@
import net.java.sip.communicator.impl.neomedia.*;
import net.java.sip.communicator.impl.neomedia.format.*;
import net.java.sip.communicator.impl.neomedia.protocol.*;
+import net.java.sip.communicator.impl.neomedia.codec.video.*;
import net.java.sip.communicator.service.neomedia.*;
import net.java.sip.communicator.service.neomedia.device.*;
import net.java.sip.communicator.service.neomedia.format.*;
@@ -1226,7 +1227,7 @@
     * of
     * @param format the JMF <tt>Format</tt> to set to <tt>processor</tt>
     */
- private void setFormat(Processor processor, Format format)
+ protected void setFormat(Processor processor, Format format)
    {
        TrackControl[] trackControls = processor.getTrackControls();
        MediaType mediaType = getMediaType();
@@ -1276,7 +1277,36 @@
            case VIDEO:
                if (supportedFormats[0] instanceof VideoFormat)
                {
- supportedFormat
+ VideoFormat tmp = (VideoFormat)format;
+ Dimension size = tmp.getSize();
+
+ if(size != null)
+ {
+ /* We have been explictely told to use
+ * a specified output size so create a
+ * custom SwScaler that will rescale and
+ * change format in one call
+ */
+ Codec ar[] = new Codec[1];
+ SwScaler scaler = new SwScaler();
+
+ scaler.setOutputSize(size);
+ ar[0] = scaler;
+
+ /* add our custom SwScaler to the codec chain so that
+ * it will be used instead of default SwScaler
+ */
+ try
+ {
+ trackControl.setCodecChain(ar);
+ }
+ catch(Exception e)
+ {
+ System.out.println("Error setCodecChain: " + e);
+ }
+ }
+
+ supportedFormat
                        = findFirstMatchingFormat(supportedFormats, format);

                    /*

Modified: trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java

Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java?view=diff&rev=6787&p1=trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java&p2=trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java&r1=6786&r2=6787

==============================================================================

--- trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java (original)
+++ trunk/src/net/java/sip/communicator/impl/neomedia/device/VideoMediaDeviceSession.java 2010-02-22 17:02:32+0000
@@ -9,6 +9,7 @@
import java.awt.*;

import javax.media.*;
+import javax.media.format.*;
import javax.media.control.*;
import javax.media.protocol.*;

@@ -48,6 +49,16 @@
    private Player localPlayer = null;

    /**
+ * Output size of the stream.
+ *
+ * It is used to specify a different size (generally lesser ones)
+ * than the capture device provides. Typically one usage can be
+ * in desktop streaming/sharing session when sender desktop is bigger
+ * than remote ones.
+ */
+ private Dimension outputSize = null;
+
+ /**
     * Initializes a new <tt>VideoMediaDeviceSession</tt> instance which is to
     * represent the work of a <tt>MediaStream</tt> with a specific video
     * <tt>MediaDevice</tt>.
@@ -148,6 +159,31 @@
    }

    /**
+ * Sets the JMF <tt>Format</tt> in which a specific <tt>Processor</tt> is to
+ * output media data.
+ *
+ * @param processor the <tt>Processor</tt> to set the output <tt>Format</tt>
+ * of
+ * @param format the JMF <tt>Format</tt> to set to <tt>processor</tt>
+ */
+ @Override
+ protected void setFormat(Processor processor, Format format)
+ {
+ Format newFormat = null;
+ VideoFormat tmp = (VideoFormat)format;
+
+ /* add a size in the output format, as VideoFormat has no
+ * set accessors, we recreate the object
+ */
+ if(outputSize != null)
+ {
+ newFormat = new VideoFormat(tmp.getEncoding(), outputSize, tmp.getMaxDataLength(), tmp.getDataType(), tmp.getFrameRate());
+ }
+
+ super.setFormat(processor, newFormat != null ? newFormat : format);
+ }
+
+ /**
     * Asserts that a specific <tt>MediaDevice</tt> is acceptable to be set as
     * the <tt>MediaDevice</tt> of this instance. Makes sure that its
     * <tt>MediaType</tt> is {@link MediaType#VIDEO}.

Modified: trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java

Url: https://sip-communicator.dev.java.net/source/browse/sip-communicator/trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java?view=diff&rev=6787&p1=trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java&p2=trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java&r1=6786&r2=6787

==============================================================================

--- trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java (original)
+++ trunk/src/net/java/sip/communicator/impl/neomedia/imgstreaming/NativeScreenCapture.java 2010-02-22 17:02:32+0000
@@ -9,9 +9,10 @@
import java.awt.image.*;

/**
- * This class uses native code to capture
- * desktop screen. It should work for Windows and
- * X11-based Unix such as Linux and FreeBSD.
+ * This class uses native code to capture desktop screen.
+ *
+ * It should work for Windows, Mac OS X and X11-based Unix such as Linux
+ * and FreeBSD.
*
* @author Sebastien Vincent
*/
    
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#4

Seb,

Do you have any idea why instances of
com.sun.media.codec.video.colorspace.JavaRGBConverter appear in the
heap dump of SIP Communicator?

I remember I shared with you that I think it's best to implement
SwScaler as accepting RGB and YUV input and output format regardless
of their dataType. Do you think it could be because you defined
SwScaler to only accept RGB with byteArray?

Regards,
Lubo

···

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#5

Lubomir Marinov a �crit :

Seb,

Do you have any idea why instances of
com.sun.media.codec.video.colorspace.JavaRGBConverter appear in the
heap dump of SIP Communicator?

I remember I shared with you that I think it's best to implement
SwScaler as accepting RGB and YUV input and output format regardless
of their dataType. Do you think it could be because you defined
SwScaler to only accept RGB with byteArray?
  
Yes it is because of byteArray only input format, my bad! So if the capture device produces intArray it will first pass throught JavaRGBConvert.

If we do multi datatype input/output SwScaler we will have to handle allocation for data type conversion (libswcale takes and returns byte array) and reinvent the wheel (we just move what are done in JavaRGBConverter for dataType conversion into SwScaler) or we could add other FFMPEG native method img_convert() to accept int or short array as parameters (we just have to take pointers their pointers with appropriate JNI functions and pass them to libswscale).

WDYT ?

···

--
Seb

Regards,
Lubo

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#6

If we do multi datatype input/output SwScaler we will have to handle
allocation for data type conversion (libswcale takes and returns byte array)
and reinvent the wheel (we just move what are done in JavaRGBConverter for
dataType conversion into SwScaler) or we could add other FFMPEG native
method img_convert() to accept int or short array as parameters (we just
have to take pointers their pointers with appropriate JNI functions and pass
them to libswscale).

I think the question is really whether libswscale can handle
conversions from (and to?) formats such as PIX_FMT_ARGB, PIX_FMT_RGBA,
PIX_FMT_ABGR and PIX_FMT_BGRA. Because if it can, then it's a question
of taking into account endianness and casting int * to byte *.

···

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#7

Lubomir Marinov a �crit :

If we do multi datatype input/output SwScaler we will have to handle
allocation for data type conversion (libswcale takes and returns byte array)
and reinvent the wheel (we just move what are done in JavaRGBConverter for
dataType conversion into SwScaler) or we could add other FFMPEG native
method img_convert() to accept int or short array as parameters (we just
have to take pointers their pointers with appropriate JNI functions and pass
them to libswscale).
    
I think the question is really whether libswscale can handle
conversions from (and to?) formats such as PIX_FMT_ARGB, PIX_FMT_RGBA,
PIX_FMT_ABGR and PIX_FMT_BGRA. Because if it can, then it's a question
of taking into account endianness and casting int * to byte *.
  
I think library can handle all these formats. So I will see to make SwScaler handle intArray/shortArray in addition to byteArray.

···

--
Seb

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#8

I think library can handle all these formats. So I will see to make SwScaler
handle intArray/shortArray in addition to byteArray.

Thank you! If we have to iterate the arrays and do the conversion
ourselves, then we don't want to do it in SwScaler because
JavaRGBConverter will do it. But if libswscale handles it itself, then
I think it's good to have it do it because we'll not have to pass
through another codec and have another allocation.

···

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#9

Hi,

I have done some tests with SwScaler as RGB/YUV only converter (I remove JavaRGBConverter and JavaRGBToYUV from JMF).

And here results with webcam which now use SwScaler to convert from RGB24 to RGB32 or YUV:
CPU: 40-54% RAM: 265MB

Current version of trunk:
CPU: 44-58% RAM: 277MB

Code will be commited tonight or tomorrow morning, I have some endianess stuff to solve (webcam is good in preview (and not Desktop streaming) but not in peer screen (but Desktop streaming is good) :().

Regards,

···

--
Seb

Lubomir Marinov a �crit :

I think library can handle all these formats. So I will see to make SwScaler
handle intArray/shortArray in addition to byteArray.
    
Thank you! If we have to iterate the arrays and do the conversion
ourselves, then we don't want to do it in SwScaler because
JavaRGBConverter will do it. But if libswscale handles it itself, then
I think it's good to have it do it because we'll not have to pass
through another codec and have another allocation.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net