[jitsi-dev] cpu usage of current alpha

Hello,

I have a problem with audio distortion. It is like PC can't keep up with
the stream (it's similar to what I hear when I'm recording audio but
some other IO operation takes place at the same time).

I dropped a mail to user list but since there was no answer I decided to
look at it by myself. So sampled CPU usage with visualvm.

First thing I found was that ICQ is eating more CPU than anything else.
Please look at http://imagebin.org/135529 (btw is there any restriction
in your list for attaching images?)

I disabled ICQ and tried again. This time packet logging was eating more
cycles that other things. http://imagebin.org/135530

So disabled packet logging. Now it looks like RTP audio is the biggest
CPU user. http://imagebin.org/135531

Besides that I looked at top output and see that CPU usage of
sip-communicator is over 50% where twinkle uses 9-10%. I don't hear any
such artifacts with twinkle. Hence my assumption there is some
inefficiency in handling audio by sip-communicator.

Another FYI is that I hear the same problem when using the native java
audio backend. Enabling/disabling AEC and noise suppression has no
audible effect. I have thought about bad handling of jitter buffer as an
alternative cause of the problem, but I'm not sure how to verify what's
the real cause.

Please advice what can I look at next?

Hallo Aleksandar,

I do extensive audio testing on different machines. In general, audio quality is excellent. The same is the case for the echo canceller.

Unfortunately there are these conditions, you write about. Sometimes both CPU usage and signal latency go up. The most obvious consequence then is the complete failure of the echo canceller (you normally do not realize signal latency in a production environment unless you interrupt your partner or vice versa)

Changing the JVM from OpenJDK to Sun JRE improves the situation dramatically. On OpenJDK turning on either camera or desktop streaming reproducabily triggers the latency / CPU usage surge.

Hard to say what's going on. Its obvious that turning off features brings others in the frontline of CPU usage.

I haven't used VirtualVM to look into the Java processes (nice to learn about that :slight_smile: )

My observations suggest me, that the audio process has a somewhat lower priority over other processes (I wonder if you can adjust priorities within Java - some nice command for the VM.)

Also there are native threads involved - such as in the video encoding. (needless to say that video encoding is the big showstopper for audio - especially on weaker machines - video is always excellent while audio drops off or trails)

Do you have any idea if with the JVM Scheduler, a Java process can interrupt a running native thread - or would the native thread have to collaborerate (such as in Windows 3.1 times)

That might be an explanation. What do you think?

Greetings - nice to feel less alone with my problems :slight_smile:

Conrad

Conrad Beckert wrote, On 02/01/2011 08:57 PM (EEST):

Hallo Aleksandar,

I do extensive audio testing on different machines. In general, audio quality is excellent. The same is the case for the echo canceller.

Unfortunately there are these conditions, you write about. Sometimes both CPU usage and signal latency go up. The most obvious consequence then is the complete failure of the echo canceller (you normally do not realize signal latency in a production environment unless you interrupt your partner or vice versa)

Changing the JVM from OpenJDK to Sun JRE improves the situation dramatically. On OpenJDK turning on either camera or desktop streaming reproducabily triggers the latency / CPU usage surge.

I am trying sun java 1.6.0_22 already with no success. Openjdk is really
somehow unstable with sip-communicator.

Hard to say what's going on. Its obvious that turning off features brings others in the frontline of CPU usage.

I'm almost certain CPU usage is the problem because when I open/close
the DTMF window during call, the sound becomes worse. If I don't touch
anything distortions are less. At least in a few tries I think this is
the case.

I haven't used VirtualVM to look into the Java processes (nice to learn about that :slight_smile: )

My observations suggest me, that the audio process has a somewhat lower priority over other processes (I wonder if you can adjust priorities within Java - some nice command for the VM.)

http://download.oracle.com/javase/1.4.2/docs/api/java/lang/Thread.html
You can set thread priorities. But it should be done by the application
creating the threads. Other trickery would be harder to implement (for
example with the byteman project).

Also there are native threads involved - such as in the video encoding. (needless to say that video encoding is the big showstopper for audio - especially on weaker machines - video is always excellent while audio drops off or trails)

I know that JNI has some overhead but I believe what I experience is too
much to be inly that. I'm not using video.

Do you have any idea if with the JVM Scheduler, a Java process can interrupt a running native thread - or would the native thread have to collaborerate (such as in Windows 3.1 times)

Threads can be given priorities. See the API doc above. There are
particular points in code where thread can be stopped and control given
to another thread. A thread can receive interrupt but AFAIK this is not
used by the JAVA scheduler. Also when IO is involved, sometimes thread
could be stuck waiting for input. I don't think this is the case here
though.

That might be an explanation. What do you think?

I don't think there is such thing as native thread (although I believe
java threads are backed by real thread whatever the OS supports). Rather
there are native calls which can be performed by whatever java thread.
I'm not sure about portaudio. It is maybe also creating some threads
that are not managed by the JVM but live in the JVM process?

Anyways in this case I don't think this is a problem because I'm only
doing audio and it seems threads handling audio are not given enough
priority or work very inefficiently or both. Moreover I'm running on
dual-core so 2 threads can execute simultaneously.

Greetings - nice to feel less alone with my problems :slight_smile:

I wonder what CPU usage are others seeing while talking. Could it be a
problem with some native libraries of my linux distribution?

···

Conrad

Aleksandar Kostadinov wrote, On 02/01/2011 09:54 PM (EEST):

Conrad Beckert wrote, On 02/01/2011 08:57 PM (EEST):

Hallo Aleksandar,

I do extensive audio testing on different machines. In general, audio quality is excellent. The same is the case for the echo canceller.

Unfortunately there are these conditions, you write about. Sometimes both CPU usage and signal latency go up. The most obvious consequence then is the complete failure of the echo canceller (you normally do not realize signal latency in a production environment unless you interrupt your partner or vice versa)

Changing the JVM from OpenJDK to Sun JRE improves the situation dramatically. On OpenJDK turning on either camera or desktop streaming reproducabily triggers the latency / CPU usage surge.

I am trying sun java 1.6.0_22 already with no success. Openjdk is really
somehow unstable with sip-communicator.

Hard to say what's going on. Its obvious that turning off features brings others in the frontline of CPU usage.

I'm almost certain CPU usage is the problem because when I open/close
the DTMF window during call, the sound becomes worse. If I don't touch
anything distortions are less. At least in a few tries I think this is
the case.

I haven't used VirtualVM to look into the Java processes (nice to learn about that :slight_smile: )

My observations suggest me, that the audio process has a somewhat lower priority over other processes (I wonder if you can adjust priorities within Java - some nice command for the VM.)

http://download.oracle.com/javase/1.4.2/docs/api/java/lang/Thread.html

Too optimistic...
http://stackoverflow.com/questions/1662185/do-linux-jvms-actually-implement-thread-priorities

So priorities are not actually very usable on linux.

···

You can set thread priorities. But it should be done by the application
creating the threads. Other trickery would be harder to implement (for
example with the byteman project).

Also there are native threads involved - such as in the video encoding. (needless to say that video encoding is the big showstopper for audio - especially on weaker machines - video is always excellent while audio drops off or trails)

I know that JNI has some overhead but I believe what I experience is too
much to be inly that. I'm not using video.

Do you have any idea if with the JVM Scheduler, a Java process can interrupt a running native thread - or would the native thread have to collaborerate (such as in Windows 3.1 times)

Threads can be given priorities. See the API doc above. There are
particular points in code where thread can be stopped and control given
to another thread. A thread can receive interrupt but AFAIK this is not
used by the JAVA scheduler. Also when IO is involved, sometimes thread
could be stuck waiting for input. I don't think this is the case here
though.

That might be an explanation. What do you think?

I don't think there is such thing as native thread (although I believe
java threads are backed by real thread whatever the OS supports). Rather
there are native calls which can be performed by whatever java thread.
I'm not sure about portaudio. It is maybe also creating some threads
that are not managed by the JVM but live in the JVM process?

Anyways in this case I don't think this is a problem because I'm only
doing audio and it seems threads handling audio are not given enough
priority or work very inefficiently or both. Moreover I'm running on
dual-core so 2 threads can execute simultaneously.

Greetings - nice to feel less alone with my problems :slight_smile:

I wonder what CPU usage are others seeing while talking. Could it be a
problem with some native libraries of my linux distribution?

Conrad

It is a known fact that Sip Communicator uses many CPU cycles in some conditions. I for example use Debian Squeeze with PulseAudio installed and Sip Communicator's voice is very unstable with it - i sometimes get 100% CPU (Athlon II x2 @ 3.00 GHz) sometimes about 40-50% with PCMU codec and no echo cancellation (other SIP clients like Sflphone use max 10% in the same conditions with echo cancellation). The sound sometimes stutters or cuts off for a few seconds. I disabled PulseAudio and the sound quality is good but sometimes i have a few seconds lag at the beginning of calls.
Something is definitely not optimized with the bundled PortAudio.

···

On Tue, 01 Feb 2011 22:07:32 +0200, Aleksandar Kostadinov <akostadi@redhat.com> wrote:

Aleksandar Kostadinov wrote, On 02/01/2011 09:54 PM (EEST):

Conrad Beckert wrote, On 02/01/2011 08:57 PM (EEST):

Hallo Aleksandar,

I do extensive audio testing on different machines. In general, audio quality is excellent. The same is the case for the echo canceller.

Unfortunately there are these conditions, you write about. Sometimes both CPU usage and signal latency go up. The most obvious consequence then is the complete failure of the echo canceller (you normally do not realize signal latency in a production environment unless you interrupt your partner or vice versa)

Changing the JVM from OpenJDK to Sun JRE improves the situation dramatically. On OpenJDK turning on either camera or desktop streaming reproducabily triggers the latency / CPU usage surge.

I am trying sun java 1.6.0_22 already with no success. Openjdk is really
somehow unstable with sip-communicator.

Hard to say what's going on. Its obvious that turning off features brings others in the frontline of CPU usage.

I'm almost certain CPU usage is the problem because when I open/close
the DTMF window during call, the sound becomes worse. If I don't touch
anything distortions are less. At least in a few tries I think this is
the case.

I haven't used VirtualVM to look into the Java processes (nice to learn about that :slight_smile: )

My observations suggest me, that the audio process has a somewhat lower priority over other processes (I wonder if you can adjust priorities within Java - some nice command for the VM.)

http://download.oracle.com/javase/1.4.2/docs/api/java/lang/Thread.html

Too optimistic...
http://stackoverflow.com/questions/1662185/do-linux-jvms-actually-implement-thread-priorities

So priorities are not actually very usable on linux.

You can set thread priorities. But it should be done by the application
creating the threads. Other trickery would be harder to implement (for
example with the byteman project).

Also there are native threads involved - such as in the video encoding. (needless to say that video encoding is the big showstopper for audio - especially on weaker machines - video is always excellent while audio drops off or trails)

I know that JNI has some overhead but I believe what I experience is too
much to be inly that. I'm not using video.

Do you have any idea if with the JVM Scheduler, a Java process can interrupt a running native thread - or would the native thread have to collaborerate (such as in Windows 3.1 times)

Threads can be given priorities. See the API doc above. There are
particular points in code where thread can be stopped and control given
to another thread. A thread can receive interrupt but AFAIK this is not
used by the JAVA scheduler. Also when IO is involved, sometimes thread
could be stuck waiting for input. I don't think this is the case here
though.

That might be an explanation. What do you think?

I don't think there is such thing as native thread (although I believe
java threads are backed by real thread whatever the OS supports). Rather
there are native calls which can be performed by whatever java thread.
I'm not sure about portaudio. It is maybe also creating some threads
that are not managed by the JVM but live in the JVM process?

Anyways in this case I don't think this is a problem because I'm only
doing audio and it seems threads handling audio are not given enough
priority or work very inefficiently or both. Moreover I'm running on
dual-core so 2 threads can execute simultaneously.

Greetings - nice to feel less alone with my problems :slight_smile:

I wonder what CPU usage are others seeing while talking. Could it be a
problem with some native libraries of my linux distribution?

Conrad

--
O zi buna,

Kertesz Laszlo

Using Opera's revolutionary email client: http://www.opera.com/mail/

Kertesz Laszlo wrote, On 02/01/2011 11:05 PM (EEST):

It is a known fact that Sip Communicator uses many CPU cycles in some
conditions. I for example use Debian Squeeze with PulseAudio installed

Is there any bug report and/or thread with some findings? I couldn't
find anything searching yesterday. Thanks!

...

Hello,

There were discussions on the sip communicator mailing lists, but with no conclusive results.
The thing is that PulseAudio (and some reported similar issues on Windows if i remember correctly) and the PortAudio from Sip Communicator dont seem to mix well (at least on Debian). I have crackling sound and high CPU usage (100%) ocasionally, but the occurence is totally random, except the first call after i launch the program, that has a 90% chance to use 100% CPU (other than that is about 60-40, bigger is the high CPU usage scenario).

I recently disabled PulseAudio (using PortAudio+ALSA) and i have no very high CPU usage related sound issues (at least with SIP + PCMU codec). Although 40% is a bit high for PCMU with no echo cancellation (~10-15% on other clients). And if i minimize the call window, i have a ~10% decrease...

This issue came and gone a few times in the old builds but the most recent PortAudio changes (at least i know of) brought it back.

···

On Wed, 02 Feb 2011 08:29:09 +0200, Aleksandar Kostadinov <akostadi@redhat.com> wrote:

Kertesz Laszlo wrote, On 02/01/2011 11:05 PM (EEST):

It is a known fact that Sip Communicator uses many CPU cycles in some
conditions. I for example use Debian Squeeze with PulseAudio installed

Is there any bug report and/or thread with some findings? I couldn't
find anything searching yesterday. Thanks!

...

--
O zi buna,

Kertesz Laszlo

Using Opera's revolutionary email client: http://www.opera.com/mail/

Hey Aleksandar,

На 02.02.11 08:29, Aleksandar Kostadinov написа:

Kertesz Laszlo wrote, On 02/01/2011 11:05 PM (EEST):

It is a known fact that Sip Communicator uses many CPU cycles in some
conditions. I for example use Debian Squeeze with PulseAudio installed

Is there any bug report and/or

thread

There are a few threads on the ML but the archives haven't been migrated
from the old java.net yet which is probably why you couldn't find them.
Here's one:

http://goo.gl/riOFB

As Kertesz mentioned, there weren't any specific findings other than the
fact it seems to be a problem between PortAudio and PulseAudio.

We (BlueJimp) haven't been able to reproduce the issue on any of our
machines (possibly because we all have similar configs) which makes it
hard to fix.

Other projects seem to also be affected by the issue so if anyone on
this list has experience with [Port|Pulse]Audio, help is most welcome.

Cheers,
Emil

P.S. I don't think we have an issue describing this particular problem.
Could you please open one and point this thread?

···

with some findings? I couldn't
find anything searching yesterday. Thanks!

...

--
Emil Ivov, Ph.D. 67000 Strasbourg,
Project Lead France
SIP Communicator
emcho@sip-communicator.org PHONE: +33.1.77.62.43.30
http://sip-communicator.org FAX: +33.1.77.62.47.31

Hi Kertesz,

I can reproduce the CPU usage on 100% all the time just make a Jingle call from a Box with Java 1.6_0 version to another box with Java 1.5_0 version, the Resident Private Memory of the Box with Java 1.6_0 version growth without stop until the App start using 100% of the CPU, usually it happens in about one minute of call.
I don't know if it is related tp the issue you're reporting, but you can reproduce this issue all the time, check the Java version of the Box you're testing and see if you're in this case.

Thanks
Fabio

···

Il giorno 02/feb/2011, alle ore 08.03, Kertesz Laszlo ha scritto:

On Wed, 02 Feb 2011 08:29:09 +0200, Aleksandar Kostadinov <akostadi@redhat.com> wrote:

Kertesz Laszlo wrote, On 02/01/2011 11:05 PM (EEST):

It is a known fact that Sip Communicator uses many CPU cycles in some
conditions. I for example use Debian Squeeze with PulseAudio installed

Is there any bug report and/or thread with some findings? I couldn't
find anything searching yesterday. Thanks!

...

Hello,

There were discussions on the sip communicator mailing lists, but with no conclusive results.
The thing is that PulseAudio (and some reported similar issues on Windows if i remember correctly) and the PortAudio from Sip Communicator dont seem to mix well (at least on Debian). I have crackling sound and high CPU usage (100%) ocasionally, but the occurence is totally random, except the first call after i launch the program, that has a 90% chance to use 100% CPU (other than that is about 60-40, bigger is the high CPU usage scenario).

I recently disabled PulseAudio (using PortAudio+ALSA) and i have no very high CPU usage related sound issues (at least with SIP + PCMU codec). Although 40% is a bit high for PCMU with no echo cancellation (~10-15% on other clients). And if i minimize the call window, i have a ~10% decrease...

This issue came and gone a few times in the old builds but the most recent PortAudio changes (at least i know of) brought it back.

--
O zi buna,

Kertesz Laszlo

Using Opera's revolutionary email client: http://www.opera.com/mail/

Fabio,

For the record, I think the issue you have been describing here is not
related to the PortAudio/PusleAudio problems that have been observed on
Linux.

Emil
На 02.02.11 09:54, Fabio Telme написа:

···

Hi Kertesz,

I can reproduce the CPU usage on 100% all the time just make a
Jingle call from a Box with Java 1.6_0 version to another box with
Java 1.5_0 version, the Resident Private Memory of the Box with Java
1.6_0 version growth without stop until the App start using 100% of
the CPU, usually it happens in about one minute of call. I don't know
if it is related tp the issue you're reporting, but you can reproduce
this issue all the time, check the Java version of the Box you're
testing and see if you're in this case.

Thanks Fabio

Il giorno 02/feb/2011, alle ore 08.03, Kertesz Laszlo ha scritto:

On Wed, 02 Feb 2011 08:29:09 +0200, Aleksandar Kostadinov >> <akostadi@redhat.com> wrote:

Kertesz Laszlo wrote, On 02/01/2011 11:05 PM (EEST):

It is a known fact that Sip Communicator uses many CPU cycles
in some conditions. I for example use Debian Squeeze with
PulseAudio installed

Is there any bug report and/or thread with some findings? I
couldn't find anything searching yesterday. Thanks!

...

Hello,

There were discussions on the sip communicator mailing lists, but
with no conclusive results. The thing is that PulseAudio (and some
reported similar issues on Windows if i remember correctly) and the
PortAudio from Sip Communicator dont seem to mix well (at least on
Debian). I have crackling sound and high CPU usage (100%)
ocasionally, but the occurence is totally random, except the first
call after i launch the program, that has a 90% chance to use 100%
CPU (other than that is about 60-40, bigger is the high CPU usage
scenario).

I recently disabled PulseAudio (using PortAudio+ALSA) and i have no
very high CPU usage related sound issues (at least with SIP + PCMU
codec). Although 40% is a bit high for PCMU with no echo
cancellation (~10-15% on other clients). And if i minimize the call
window, i have a ~10% decrease...

This issue came and gone a few times in the old builds but the most
recent PortAudio changes (at least i know of) brought it back.

-- O zi buna,

Kertesz Laszlo

Using Opera's revolutionary email client:
http://www.opera.com/mail/

--
Emil Ivov, Ph.D. 67000 Strasbourg,
Project Lead France
SIP Communicator
emcho@sip-communicator.org PHONE: +33.1.77.62.43.30
http://sip-communicator.org FAX: +33.1.77.62.47.31

Emil Ivov wrote, On 02/02/2011 09:53 AM (EEST):
...

As Kertesz mentioned, there weren't any specific findings other than the
fact it seems to be a problem between PortAudio and PulseAudio.

I'm using plain ALSA.

We (BlueJimp) haven't been able to reproduce the issue on any of our
machines (possibly because we all have similar configs) which makes it
hard to fix.

I don't have experience with these. This morning I tok a quick look at
PortAudioStream.java
Am I correct that SC uses a static length buffer of 20ms? In my case CPU
is not on 100% usage still it can't keep up with the stream. In this case
I thought using a larger buffer (perhaps make it user configurable), or
even better use an adaptive buffer size should result in less JNI calls
and better latency tolerance.
Algorithm could be something along the lines of send to portaudio
whatever sound we already have and increase receive buffer size with 50%
each time we get a full buffer of data ready from portaudio at the time
we try to get it.

I didn't have time to get a good idea of the code so please let me know
your opinion.

...

P.S. I don't think we have an issue describing this particular problem.
Could you please open one and point this thread?

Will file one.

Kertesz Laszlo wrote, On 02/02/2011 09:03 AM (EEST):
...

I recently disabled PulseAudio (using PortAudio+ALSA) and i have no very
high CPU usage related sound issues (at least with SIP + PCMU codec).
Although 40% is a bit high for PCMU with no echo cancellation (~10-15%
on other clients). And if i minimize the call window, i have a ~10%
decrease...

How do you minimize the call window? What do you mean by that?

Thank you,
Aleksandar

I mean minimizing the active call window, putting it to the taskbar. This decreases CPU usage with around 10%. Probably the resources are needed for drawing the window (kinda high).
I use Debian Squeeze 32-bit.

Java version:

$ java -version
java version "1.6.0_22"
Java(TM) SE Runtime Environment (build 1.6.0_22-b04)
Java HotSpot(TM) Server VM (build 17.1-b03, mixed mode)

···

On Wed, 02 Feb 2011 13:36:17 +0200, Aleksandar Kostadinov <akostadi@redhat.com> wrote:

Kertesz Laszlo wrote, On 02/02/2011 09:03 AM (EEST):
...

I recently disabled PulseAudio (using PortAudio+ALSA) and i have no very
high CPU usage related sound issues (at least with SIP + PCMU codec).
Although 40% is a bit high for PCMU with no echo cancellation (~10-15%
on other clients). And if i minimize the call window, i have a ~10%
decrease...

How do you minimize the call window? What do you mean by that?

Thank you,
Aleksandar

--
Thank you,

Kertesz Laszlo

Using Opera's revolutionary email client: http://www.opera.com/mail/

Am I correct that SC uses a static length buffer of 20ms?

Yes.

In my case CPU
is not on 100% usage still it can't keep up with the stream. In this case
I thought using a larger buffer (perhaps make it user configurable), or
even better use an adaptive buffer size should result in less JNI calls
and better latency tolerance.
Algorithm could be something along the lines of send to portaudio
whatever sound we already have and increase receive buffer size with 50%
each time we get a full buffer of data ready from portaudio at the time
we try to get it.

Please feel free to contribute patches implementing the modifications
described above. As a lot more is happening in SIP Communicator with
audio than vanilla capture and playback with PortAudio (e.g. tuning to
multiple platforms, various audio-quality improvements with their own
sensitivity to latency), please keep in mind the whole big rest of
affected functionality.

BTW, I see you mentioning JNI calls multiple times as the cause of the
problem at hand. I'd personally suggest creating profiles of the
execution (e.g. OProfile, YourKit Java Profiler) which prove your
point prior to pursuing any actual implementation, just to make sure
you're on the right track to a sizable bang for the associated effort.

···

On Wed, Feb 2, 2011 at 1:35 PM, Aleksandar Kostadinov <akostadi@redhat.com> wrote:

I would suggest driving SC in the bad state and then take a a couple of
thread dumps and backtraces. Most of the time the threads would be in the
most time consuming place. No need for debug symbols, just to pin-point the
bad module. There is a fedora article
http://fedoraproject.org/wiki/StackTraces + the linked article for java
stack traces inside.

Also the profiler implies you are using speex and zrtp, both are
computationally expensive. Turning off zrtp and switching to g711 should
further improve the situation.

···

On Wed, Feb 2, 2011 at 2:07 PM, Lyubomir Marinov <lubo@sip-communicator.org>wrote:

On Wed, Feb 2, 2011 at 1:35 PM, Aleksandar Kostadinov > <akostadi@redhat.com> wrote:
> Am I correct that SC uses a static length buffer of 20ms?

Yes.

> In my case CPU
> is not on 100% usage still it can't keep up with the stream. In this case
> I thought using a larger buffer (perhaps make it user configurable), or
> even better use an adaptive buffer size should result in less JNI calls
> and better latency tolerance.
> Algorithm could be something along the lines of send to portaudio
> whatever sound we already have and increase receive buffer size with 50%
> each time we get a full buffer of data ready from portaudio at the time
> we try to get it.

Please feel free to contribute patches implementing the modifications
described above. As a lot more is happening in SIP Communicator with
audio than vanilla capture and playback with PortAudio (e.g. tuning to
multiple platforms, various audio-quality improvements with their own
sensitivity to latency), please keep in mind the whole big rest of
affected functionality.

BTW, I see you mentioning JNI calls multiple times as the cause of the
problem at hand. I'd personally suggest creating profiles of the
execution (e.g. OProfile, YourKit Java Profiler) which prove your
point prior to pursuing any actual implementation, just to make sure
you're on the right track to a sizable bang for the associated effort.