[sip-comm-dev] [gsoc - wideband codec] work log 1


#1

Hi all,

I am the gsoc project "wideband codec" student Dai Jing, from Beijing,
China. This is my first work log. I'm planning to post my work log
every week, and hope something naive won't annoy you :slight_smile:

In the past few days I read some CELT papers, and now I have got a
main clue of the codec. I am currently reading the c code. This task
is a little tougher than I have thought, for there are a lot of macro
definations, and many "#ifdef", "#ifndef" even in the API functions at
the purpose of adapting to different platforms(both hardware and OS),
which is sometimes confusing and has really slown down my reading
speed. This reminds me of the cross-platform advantage of java :slight_smile:

About the integration work, I generally want to refer to ILBC and G729
in SC source code. ILBC extends
com.ibm.media.codec.audio.AudioCodec(JMF), while G729 extends
net.sf.fmj.media.AbstractCodec(FMJ). I found some information saying
JMF and FMJ are API-compatible. I plan to arrange my work in the same
way. In the interface classes, e.g. JavaEncoder, process(Buffer input,
Buffer output) calls the method encode(short[] data, byte[]
compressed) of class CLETEncoder. The codec algorithm is in
CELTEncoder.encode(), which is my main work during the first period.

Here I have some questions.

Questions:
1. What is the main differences between FMJ and JMF? By knowing their
characteristics I'll decide which is more suitable for the compressed
data of CELT.
2. As far as I know, the audio codec interface classes are only
JavaEncoder and JavaDecoder(both ILBC and G729), right? And in G729
"Coder.java" or "Decoder.java" have nothing to do with upper layer--no
one would call them, they are just used for test, am I right?

Thanks in advance & cheers.

Jing

路路路

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#2

Hi Dai,

I'm pleased to read about your progress!

in SC source code. ILBC extends
com.ibm.media.codec.audio.AudioCodec(JMF), while G729 extends
net.sf.fmj.media.AbstractCodec(FMJ).

We are not employing FMJ as a replacement of JMF at this time, we're
just using an abstract class from FMJ so that we don't have to write
it ourselves.

Questions:
1. What is the main differences between FMJ and JMF? By knowing their
characteristics I'll decide which is more suitable for the compressed
data of CELT.

Since we don't run with FMJ and we only run with JMF, I think the
differences between the two are irrelevant. Besides, I don't have an
idea :wink: The only "suitable" here is JMF.

2. As far as I know, the audio codec interface classes are only
JavaEncoder and JavaDecoder(both ILBC and G729), right? And in G729
"Coder.java" or "Decoder.java" have nothing to do with upper layer--no
one would call them, they are just used for test, am I right?

The JavaEncoder and JavaDecoder classes are the classes which
implement the interfaces of JMF so that the actual G.729
implementation given by Coder and Decoder can be used inside of JMF.
Consequently, JMF sees only JavaEncoder and JavaDecoder, JavaEncoder
and JavaDecoder call Coder and Decoder and noone else calls Coder and
Decoder. For what it's worth, the code of Coder and Decoder could be
moved inside JavaEncoder and JavaDecoder but that would only make the
latter larger and mix the JMF specifics with the G.729 specifics.

I hope I've been able to clearly answer your questions. Please don't
hesitate to request further information.

Best regards,
Lubomir

路路路

On Wed, May 12, 2010 at 2:38 PM, Dai Jing <daijing09@gmail.com> wrote:

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#3

Dai, I'm sorry I forgot to mention it earlier: please be sure to only
look at the codec implementations in neomedia and not in media! The
media bundle is no longer in use, only the neomedia bundle is
maintained and thus it contains the latest versions of the codecs.

路路路

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#4

Hey Lubo, Jean,

First thanks to Lubo's detailed and helpful explanations.

Here's another question about reading the c code. When you
"translated" the G729 or ILBC c code into java, how did you make sure
a certain step realizes the exact same function as in the c code? To
compare the output? This is not easy in the CELT c code, for I cann't
compile it in an IDE(any suggestions about this?)

Or you just get an overview of the existing c code, then follow the
main clue of the codec algorithm and work it out yourself, some kind
of independent from the existing c code?

路路路

2010/5/12 Lubomir Marinov <lubo@sip-communicator.org>:

Dai, I'm sorry I forgot to mention it earlier: please be sure to only
look at the codec implementations in neomedia and not in media! The
media bundle is no longer in use, only the neomedia bundle is
maintained and thus it contains the latest versions of the codecs.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#5

This is not easy in the CELT c code, for I cann't
compile it in an IDE(any suggestions about this?)

I'm not sure how IDEs are related to verifying that your Java
implementation is compatible with the C implementation but... Since
the CELT library is built with a (GNU) Makefile, both Eclipse and
NetBeans should be able to create projects from it and build it
provided that you've installed the respective modules for C/C++
support (e.g. CDT for Eclipse).

Or you just get an overview of the existing c code, then follow the
main clue of the codec algorithm and work it out yourself, some kind
of independent from the existing c code?

Since CELT is still a work in progress and neither the API/ABI nor the
bit-stream are stable, I would personally rather not go with an
implementation of mine which is independent from the C code.

Here's another question about reading the c code. When you
"translated" the G729 or ILBC c code into java, how did you make sure
a certain step realizes the exact same function as in the c code? To
compare the output?

In accord with the work-in-progress statement above, I'm only going to
talk about how I would personally approach translating the C code of
the CELT library to Java.

Regardless of how I translate the library from C to Java, it shouldn't
be too hard to translate the test applications in the tests directory.
I'd expect them to successfully work with my Java implementation.

It will also be useful if there are samples with reference output for
a given input. I don't know whether there are such for the latest
version of the CELT library but it shouldn't hurt to google for them.

To sum the translation up, I'd try to (1) automate it as much as
possible and (2) make sure that I can reproduce it. Obviously, (1)
should minimize the chances for errors in manual changes. (2) is
supposed to at the very least allow me to update to new releases of
the CELT library.

As to how I'd approach (1), I'd go with something that kick-starts me
such as shell and/or Perl scripting. (Your mileage may vary, of
course.) This should allow me to automatically rename and merge files,
place license headers and package statements at the top, define
classes and whatnot.

If there are doubts about differences in the precision of the C and
Java types, these better be taken care of early. The same applies to
bitwise operations such as shifts I guess.

It should be safe at first to translate the C files, global variables
and functions into Java classes and static fields and methods. Because
we'll want to be able to run multiple instances of the encoder and the
decoder at the same time though, the global variables i.e. static
fields may later force me to convert them to instance once. But it's
usually one of the final steps of the development after I'm sure the
encoder and the decoder work as expected without running multiple
instances.

One of the candidates for manual editing (and I'd say a careful one)
will most certainly be related to pointers such as pointer arithmetic.
For example, in the context of array traversal.

I presume I'm missing a lot but if you agree with my approach, please
don't hesitate to ask.

路路路

On Thu, May 13, 2010 at 11:23 AM, Dai Jing <daijing09@gmail.com> wrote:

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#6

Hi guys,

I tend to agree with Lubo. The work made to port the ilbc codec
roughly matches the picture he just sketched.

Just a little note on codec testing, the nice thing when doing this
kind of conversion is that you can do some limited interoperability
testing with both implementations. When doing ilbc, I had both encoder
and decoder stand-alone programs written in both Java and C. Then I
could feed each other's coded stream in each other's decoder and
compare the output.

Regarding the conversion process itself, I agree that the most
important points (i.e. those that might be the culprit in a failed
conversion or interoperability incompatibility) are pointer and FP
arithmetic.

Hope this helps,
jean

路路路

On Thu, May 13, 2010 at 6:51 PM, Lubomir Marinov <lubo@sip-communicator.org> wrote:

On Thu, May 13, 2010 at 11:23 AM, Dai Jing <daijing09@gmail.com> wrote:

This is not easy in the CELT c code, for I cann't
compile it in an IDE(any suggestions about this?)

I'm not sure how IDEs are related to verifying that your Java
implementation is compatible with the C implementation but... Since
the CELT library is built with a (GNU) Makefile, both Eclipse and
NetBeans should be able to create projects from it and build it
provided that you've installed the respective modules for C/C++
support (e.g. CDT for Eclipse).

Or you just get an overview of the existing c code, then follow the
main clue of the codec algorithm and work it out yourself, some kind
of independent from the existing c code?

Since CELT is still a work in progress and neither the API/ABI nor the
bit-stream are stable, I would personally rather not go with an
implementation of mine which is independent from the C code.

Here's another question about reading the c code. When you
"translated" the G729 or ILBC c code into java, how did you make sure
a certain step realizes the exact same function as in the c code? To
compare the output?

In accord with the work-in-progress statement above, I'm only going to
talk about how I would personally approach translating the C code of
the CELT library to Java.

Regardless of how I translate the library from C to Java, it shouldn't
be too hard to translate the test applications in the tests directory.
I'd expect them to successfully work with my Java implementation.

It will also be useful if there are samples with reference output for
a given input. I don't know whether there are such for the latest
version of the CELT library but it shouldn't hurt to google for them.

To sum the translation up, I'd try to (1) automate it as much as
possible and (2) make sure that I can reproduce it. Obviously, (1)
should minimize the chances for errors in manual changes. (2) is
supposed to at the very least allow me to update to new releases of
the CELT library.

As to how I'd approach (1), I'd go with something that kick-starts me
such as shell and/or Perl scripting. (Your mileage may vary, of
course.) This should allow me to automatically rename and merge files,
place license headers and package statements at the top, define
classes and whatnot.

If there are doubts about differences in the precision of the C and
Java types, these better be taken care of early. The same applies to
bitwise operations such as shifts I guess.

It should be safe at first to translate the C files, global variables
and functions into Java classes and static fields and methods. Because
we'll want to be able to run multiple instances of the encoder and the
decoder at the same time though, the global variables i.e. static
fields may later force me to convert them to instance once. But it's
usually one of the final steps of the development after I'm sure the
encoder and the decoder work as expected without running multiple
instances.

One of the candidates for manual editing (and I'd say a careful one)
will most certainly be related to pointers such as pointer arithmetic.
For example, in the context of array traversal.

I presume I'm missing a lot but if you agree with my approach, please
don't hesitate to ask.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net