[sip-comm-dev] Wideband PortAudio


#1

Hi devs,

As you know, we're working on wideband codecs during this GSoC.
However, the current PortAudio integration in SIP Communicator doesn't
support wideband capture and playback. The attached patch is supposed
to modify the PortAudio implementation in order to overcome this
limitation. (The .patch file begins with instructions on how to apply
it.)

The patch is not being committed because it requires rebuilding the
JNI PortAudio libraries.

Author: lubomir_m
Date: 2010-06-14 17:47:31+0000
New Revision: 7245
Log:
Implements Speex encoder, decoder and resampler using the native Speex library. Since the JNI binaries are not available, the new functionality shouldn't cause regressions.

The new Speex resampler is also part of the wideband-supporting
effort. And it also lacks its JNI library binaries in the repository.

Seb Vincent and Damencho, as you've been so generous to rebuild the
binaries for JNI libraries on numerous occasions, could I please ask
you to take care of them for src/native/portaudio and src/native/speex
once again? (Damencho, I know I'm asking a lot but could you please
also build the binaries for src/native/jawtrenderer on Linux?) Thank
you very much!

Best regards,
Lubo

wideband-portaudio.patch (189 KB)

···

On Mon, Jun 14, 2010 at 8:47 PM, <lubomir_m@dev.java.net> wrote:


#2

And the patch causes a segfault (discovered by Seb on 64-bit Windows).
As I find it hard to prepare the patch, I'm attaching the (hopefully)
fixed AudioQualityImprovement.c which should be placed after applying
the patch.

AudioQualityImprovement.c (19 KB)

···

On Tue, Jun 15, 2010 at 6:10 AM, Lubomir Marinov <lubo@sip-communicator.org> wrote:

The attached patch is supposed
to modify the PortAudio implementation in order to overcome this
limitation.


#3

Hi again devs,

wideband-portaudio.patch and AudioQualityImprovement.c have been
committed in trunk as revision 7261. The Linux binaries were prepared
by Damencho and the Windows binaries are courtesy of Seb Vincent.
(Thank you very much, Damencho and Seb!) Supposedly, wideband capture
and playback through PortAudio should be supported now. Please feel
free to put it to the test and report any issues you (are almost bound
to) stumble upon.

Regards,
Lubomir

···

On Tue, Jun 15, 2010 at 3:56 PM, Lubomir Marinov <lubo@sip-communicator.org> wrote:

On Tue, Jun 15, 2010 at 6:10 AM, Lubomir Marinov > <lubo@sip-communicator.org> wrote:

The attached patch is supposed
to modify the PortAudio implementation in order to overcome this
limitation.

And the patch causes a segfault (discovered by Seb on 64-bit Windows).
As I find it hard to prepare the patch, I'm attaching the (hopefully)
fixed AudioQualityImprovement.c which should be placed after applying
the patch.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#4

Lubo,

the latest portaudio modifications disable the gathering of random
data to seed the pseudo-random generator. Without a proper seed
the PRNG cannot provide real good random numbers. The function
I need:

- read some data from the audio capture device right after initializing
  the audio subsystem

- reading of the data must be in raw format, not processed by echo
  cancel, de-noise, codec compression or what else can be done to the
  data

Is there any way to do it straight forward? Any ideas? I've seen that
the PortAudioStream (Master and Input) are gone, the PortAudioManager
does not create these streams anymore, etc.

Until the PRNG stuff is fixed the crypto stuff is not production ready.

Regards,
Werner

···

Am 15.06.2010 19:31, schrieb Lubomir Marinov:

On Tue, Jun 15, 2010 at 3:56 PM, Lubomir Marinov > <lubo@sip-communicator.org> wrote:

On Tue, Jun 15, 2010 at 6:10 AM, Lubomir Marinov >> <lubo@sip-communicator.org> wrote:

The attached patch is supposed
to modify the PortAudio implementation in order to overcome this
limitation.

And the patch causes a segfault (discovered by Seb on 64-bit Windows).
As I find it hard to prepare the patch, I'm attaching the (hopefully)
fixed AudioQualityImprovement.c which should be placed after applying
the patch.

Hi again devs,

wideband-portaudio.patch and AudioQualityImprovement.c have been
committed in trunk as revision 7261. The Linux binaries were prepared
by Damencho and the Windows binaries are courtesy of Seb Vincent.
(Thank you very much, Damencho and Seb!) Supposedly, wideband capture
and playback through PortAudio should be supported now. Please feel
free to put it to the test and report any issues you (are almost bound
to) stumble upon.

Regards,
Lubomir

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#5

Hi Werner,

I'm terribly sorry, I completely forgot about it. As a try at an
excuse, I can just say that I was under a lot of stress (induced by
myself alone) while refactoring the PortAudio support and adding the
Speex-related functionality. I'm sitting right now to try to remedy
the regression.

Best regards,
Lubo

···

On Tue, Jun 15, 2010 at 9:02 PM, Werner Dittmann <Werner.Dittmann@t-online.de> wrote:

the latest portaudio modifications disable the gathering of random
data to seed the pseudo-random generator.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#6

Lubo,

I just need an idea how to do it - I'm not so familiar
with the JMF format stuff that is required to start
a portaudio stream in the new version. After getting
some idea/clue from you I can modify the stuff.

Regards,
Werner

···

Am 15.06.2010 20:20, schrieb Lubomir Marinov:

On Tue, Jun 15, 2010 at 9:02 PM, Werner Dittmann > <Werner.Dittmann@t-online.de> wrote:

the latest portaudio modifications disable the gathering of random
data to seed the pseudo-random generator.

Hi Werner,

I'm terribly sorry, I completely forgot about it. As a try at an
excuse, I can just say that I was under a lot of stress (induced by
myself alone) while refactoring the PortAudio support and adding the
Speex-related functionality. I'm sitting right now to try to remedy
the regression.

Best regards,
Lubo

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#7

Lubo,

I just need an idea how to do it - I'm not so familiar
with the JMF format stuff that is required to start
a portaudio stream in the new version. After getting
some idea/clue from you I can modify the stuff.

Hi again Werner,

Thank you very much for your understanding and patience! As I said,
I'm currently trying to restore the functionality and I hope to have
it ready soon.

I have a few thoughts now though and I remember I had them at the
beginning of the PortAudio refactorying. (Admittedly, it's still my
mistake that I failed to refactor GatherEntroy.)

- read some data from the audio capture device right after initializing
the audio subsystem

The neomedia package tries to assume full control of the JMF
CaptureDevices because there are intricacies to their usage so
gathering this knowledge in one place is better than having them here
and there. That's why neomedia has MediaDevice and MediaDeviceSession.
For example, the MediaDevice-related functionality knows that a
JavaSound CaptureDevice cannot be shared and knows how to share it,
knows what BufferControl values are necessary for a CaptureDevice to
work on a particular operating system and whatnot.

And since there are multiple details that have to be taken into
account, opening a capture device in a separate Thread may one day
prevent a device from functioning if another Thread wants to open the
same device.

Additionally, opening a capture device and actually reading from it
can be costly and may be obvious to the user. For example, opening the
camera on my Macbook turns on its light.

- reading of the data must be in raw format, not processed by echo
cancel, de-noise, codec compression or what else can be done to the
data

What one gets from a PortAudio stream isn't necessarily the rawest of
data. For what it's worth, an operating system or driver software or
the very hardware with a sanely usable implementation will likely have
already allowed the user to enable echo cancellation and/or de-noise
at the system level, prior to SIP Communicator. So requiring the data
to not be denoised on the side of SIP Communicator seems unnecessary
to me (though I'm trying to add functionality to disable it for
GatherEntropy) - a Mac OS X user will likely have already denoised it
prior to reaching us.

Until the PRNG stuff is fixed the crypto stuff is not production ready.

We currently support two audio systems: PortAudio and JavaSound. While
I was writing the Speex encoder and decoder, I had successful calls
with JavaSound as well. As far as I understand GatherEntropy, it
doesn't support JavaSound. And I haven't seen a warning in the
application telling me that I'm at risk when not using PortAudio.

Best regards,
Lubo

···

On Tue, Jun 15, 2010 at 10:52 PM, Werner Dittmann <Werner.Dittmann@t-online.de> wrote:

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#8

Hi Lubo,

thanks for taking care of GatherEntropy - I'll do some tests
during the weekend. Javasound was (still is) on my TODO list :slight_smile:

While testing your new PortAudio implementation with my system
(direkt standard ALSA devices withou specific ALSA config file,
no Pulseaudio) I got some issues:

- when I enable echo supression I always hear some noise, it's
  constantly clicking.

- when I disable denoise I have a basic noise (I've a sensitive mic),
  with denoise enabled it seems better when testing with iptel's
  echo service. Need to test it with a real human :slight_smile: .

- a more serious problem: when using ALSA "dmix" as output device
  for audio autput I get an exception:

     [java] Expression 'SetApproximateSampleRate( pcm, hwParams, sr )' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 1307

     [java] Expression 'PaAlsaStreamComponent_InitialConfigure( &self->playback, outParams, self->primeBuffers, hwParamsPlayback, &realSr )' failed in
'src/hostapi/alsa/pa_linux_alsa.c', line: 1790
     [java] Expression 'PaAlsaStream_Configure( stream, inputParameters, outputParameters, sampleRate, framesPerBuffer, &inputLatency, &outputLatency,
&hostBufferSizeMode )' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 1911
     [java] 20:01:37.843 SCHWERWIEGEND: impl.neomedia.notify.PortAudioClipImpl.runOnceInPlayThread().255 Failed to open PortAudioRenderer.

     [java] javax.media.ResourceUnavailableException: Invalid sample rate
     [java] at net.java.sip.communicator.impl.neomedia.jmfext.media.renderer.audio.PortAudioRenderer.open(PortAudioRenderer.java:265)

     [java] at net.java.sip.communicator.impl.neomedia.notify.PortAudioClipImpl.runOnceInPlayThread(PortAudioClipImpl.java:234)

     [java] at net.java.sip.communicator.impl.neomedia.notify.PortAudioClipImpl.runInPlayThread(PortAudioClipImpl.java:131)

     [java] at net.java.sip.communicator.impl.neomedia.notify.PortAudioClipImpl.access$000(PortAudioClipImpl.java:24)

     [java] at net.java.sip.communicator.impl.neomedia.notify.PortAudioClipImpl$1.run(PortAudioClipImpl.java:70)
     [java] BasicTrackControl:prefetchTrack():96 4 bm = com.sun.media.BasicFilterModule@72bb2acf

My assumption is: the notification audio file uses 44.100 or 20.500 Hz but the dmix
device has a fixed smaple rate of 48.000Hz. The same happens then to the normal
audio renderer that uses 8000Hz (PCM aLAW)? and dmix expects 48.000Hz

The previous implementation normalized all sample rates to the device's sample
rate (for dmix it's 48.000Hz) and opened Portaudio with the device's sample
rate. Now it seems that you open the device with the stream's sample rate.

I need to use dmix to get audio notifcations parallel to the normal audio.
Otherwise I need to setup a alsa.conf file - but this was not regarded as
an option for normal SC installation and use.

IMHO there should be a mechanism to check if the device supports the
stream's sample rate and if it does not re-sample the stream to a supported
sample rate.

WDYT?

Regards,
Werner

···

Am 15.06.2010 23:41, schrieb Lubomir Marinov:

We currently support two audio systems: PortAudio and JavaSound. While
I was writing the Speex encoder and decoder, I had successful calls
with JavaSound as well. As far as I understand GatherEntropy, it
doesn't support JavaSound. And I haven't seen a warning in the
application telling me that I'm at risk when not using PortAudio.

Best regards,
Lubo

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#9

If Pa_IsFormatSupported tells me what the device actually supports, I'll try
to get most of it working tonight. Of course, GatherEntropy will double a
good deal of the effort because it's still not JMF generic... but I'll get
there.

Hi Lubo,

thanks for taking care of GatherEntropy - I'll do some tests
during the weekend. Javasound was (still is) on my TODO list :slight_smile:

While testing your new PortAudio implementation with my system
(direkt standard ALSA devices withou specific ALSA config file,
no Pulseaudio) I got some issues:

- when I enable echo supression I always hear some noise, it's
constantly clicking.

- when I disable denoise I have a basic noise (I've a sensitive mic),
with denoise enabled it seems better when testing with iptel's
echo service. Need to test it with a real human :slight_smile: .

- a more serious problem: when using ALSA "dmix" as output device
for audio autput I get an exception:

    [java] Expression 'SetApproximateSampleRate( pcm, hwParams, sr )' failed
in 'src/hostapi/alsa/pa_linux_alsa.c', line: 1307

    [java] Expression 'PaAlsaStreamComponent_InitialConfigure(
&self->playback, outParams, self->primeBuffers, hwParamsPlayback, &realSr )'
failed in
'src/hostapi/alsa/pa_linux_alsa.c', line: 1790
    [java] Expression 'PaAlsaStream_Configure( stream, inputParameters,
outputParameters, sampleRate, framesPerBuffer, &inputLatency,
&outputLatency,
&hostBufferSizeMode )' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line:
1911
    [java] 20:01:37.843 SCHWERWIEGEND:
impl.neomedia.notify.PortAudioClipImpl.runOnceInPlayThread().255 Failed to
open PortAudioRenderer.

    [java] javax.media.ResourceUnavailableException: Invalid sample rate
    [java] at
net.java.sip.communicator.impl.neomedia.jmfext.media.renderer.audio.PortAudioRenderer.open(PortAudioRenderer.java:265)

    [java] at
net.java.sip.communicator.impl.neomedia.notify.PortAudioClipImpl.runOnceInPlayThread(PortAudioClipImpl.java:234)

    [java] at
net.java.sip.communicator.impl.neomedia.notify.PortAudioClipImpl.runInPlayThread(PortAudioClipImpl.java:131)

    [java] at
net.java.sip.communicator.impl.neomedia.notify.PortAudioClipImpl.access$000(PortAudioClipImpl.java:24)

    [java] at
net.java.sip.communicator.impl.neomedia.notify.PortAudioClipImpl$1.run(PortAudioClipImpl.java:70)
    [java] BasicTrackControl:prefetchTrack():96 4 bm =
com.sun.media.BasicFilterModule@72bb2acf

My assumption is: the notification audio file uses 44.100 or 20.500 Hz but
the dmix
device has a fixed smaple rate of 48.000Hz. The same happens then to the
normal
audio renderer that uses 8000Hz (PCM aLAW)? and dmix expects 48.000Hz

The previous implementation normalized all sample rates to the device's
sample
rate (for dmix it's 48.000Hz) and opened Portaudio with the device's sample
rate. Now it seems that you open the device with the stream's sample rate.

I need to use dmix to get audio notifcations parallel to the normal audio.
Otherwise I need to setup a alsa.conf file - but this was not regarded as
an option for normal SC installation and use.

IMHO there should be a mechanism to check if the device supports the
stream's sample rate and if it does not re-sample the stream to a supported
sample rate.

WDYT?

Regards,
Werner

···

On 17 Jun 2010 21:33, "Werner Dittmann" <Werner.Dittmann@t-online.de> wrote:

Am 15.06.2010 23:41, schrieb Lubomir Marinov:

We currently support two audio systems: PortAudio and JavaSound. While
I was writing the Spee...

---------------------------------------------------------------------
To unsubscribe, e-mail: de...


#10

Hi Lubo,

thanks for the fast fix (btw, when do you sleep? :slight_smile: ).

Some report about the latest version r7280:

- using ALSA hw:0,0 devices work as before
- using ALSA dmix works for the normal audio output, the notification gives me
  the following exception:

     [java] 18:45:25.639 SCHWERWIEGEND: util.UtilActivator.uncaughtException().77 An uncaught exception occurred in thread=Thread[Thread-80,10,main] and message
was: null
     [java] java.lang.NullPointerException
     [java] at net.java.sip.communicator.impl.neomedia.jmfext.media.renderer.audio.PortAudioRenderer.open(PortAudioRenderer.java:334)
     [java] at net.java.sip.communicator.impl.neomedia.notify.PortAudioClipImpl.runOnceInPlayThread(PortAudioClipImpl.java:234)
     [java] at net.java.sip.communicator.impl.neomedia.notify.PortAudioClipImpl.runInPlayThread(PortAudioClipImpl.java:131)
     [java] at net.java.sip.communicator.impl.neomedia.notify.PortAudioClipImpl.access$000(PortAudioClipImpl.java:24)
     [java] at net.java.sip.communicator.impl.neomedia.notify.PortAudioClipImpl$1.run(PortAudioClipImpl.java:70)

thus I cannot hear the "dialing sound".

Another exception was thrown when I used the ALSA hw:0,0 devices, but I'm not sure
if this belongs to your modifications. However, here is the exception (I don't know
why there is a re-invite in the first place - I just use iptel's echo service to do
the basic tests):

     [java] 18:44:22.444 SCHWERWIEGEND: impl.protocol.sip.CallPeerSipImpl.processReInvite().559 Error while trying to send a response

     [java] java.lang.NullPointerException
     [java] at net.java.sip.communicator.impl.protocol.sip.CallPeerMediaHandler.configureAndStartStream(CallPeerMediaHandler.java:906)

     [java] at net.java.sip.communicator.impl.protocol.sip.CallPeerMediaHandler.initStream(CallPeerMediaHandler.java:852)

     [java] at net.java.sip.communicator.impl.protocol.sip.CallPeerMediaHandler.createMediaDescriptionsForAnswer(CallPeerMediaHandler.java:1281)

     [java] at net.java.sip.communicator.impl.protocol.sip.CallPeerMediaHandler.processUpdateOffer(CallPeerMediaHandler.java:1122)

     [java] at net.java.sip.communicator.impl.protocol.sip.CallPeerMediaHandler.processOffer(CallPeerMediaHandler.java:1060)

     [java] at net.java.sip.communicator.impl.protocol.sip.CallPeerSipImpl.processReInvite(CallPeerSipImpl.java:543)

     [java] at net.java.sip.communicator.impl.protocol.sip.OperationSetBasicTelephonySipImpl.processInvite(OperationSetBasicTelephonySipImpl.java:899)

     [java] at net.java.sip.communicator.impl.protocol.sip.OperationSetBasicTelephonySipImpl.processRequest(OperationSetBasicTelephonySipImpl.java:318)

     [java] at net.java.sip.communicator.impl.protocol.sip.ProtocolProviderServiceSipImpl.processRequest(ProtocolProviderServiceSipImpl.java:831)

     [java] at net.java.sip.communicator.impl.protocol.sip.SipStackSharing.processRequest(SipStackSharing.java:609)

     [java] at gov.nist.javax.sip.EventScanner.deliverEvent(EventScanner.java:226)
     [java] at gov.nist.javax.sip.SipProviderImpl.handleEvent(SipProviderImpl.java:193)
     [java] at gov.nist.javax.sip.DialogFilter.processRequest(DialogFilter.java:1275)
     [java] at gov.nist.javax.sip.stack.SIPServerTransaction.processRequest(SIPServerTransaction.java:848)
     [java] at gov.nist.javax.sip.stack.UDPMessageChannel.processMessage(UDPMessageChannel.java:508)
     [java] at gov.nist.javax.sip.stack.UDPMessageChannel.processIncomingDataPacket(UDPMessageChannel.java:468)
     [java] at gov.nist.javax.sip.stack.UDPMessageChannel.run(UDPMessageChannel.java:304)

Regards,
Werner

···

Am 17.06.2010 21:22, schrieb Lubomir Marinov:

If Pa_IsFormatSupported tells me what the device actually supports, I'll try
to get most of it working tonight. Of course, GatherEntropy will double a
good deal of the effort because it's still not JMF generic... but I'll get
there.

On 17 Jun 2010 21:33, "Werner Dittmann" <Werner.Dittmann@t-online.de> wrote:

Hi Lubo,

thanks for taking care of GatherEntropy - I'll do some tests
during the weekend. Javasound was (still is) on my TODO list :slight_smile:

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#11

Hi Lubo,

with your latest fix notifications sounds work again on my ALSA
direkt devices, also on the very sensitive dmix with its fixed
sample rate settings.

Thnaks alot.

Regards,
Werner

···

Am 18.06.2010 18:54, schrieb Werner Dittmann:

Hi Lubo,

thanks for the fast fix (btw, when do you sleep? :slight_smile: ).

Some report about the latest version r7280:

- using ALSA hw:0,0 devices work as before
- using ALSA dmix works for the normal audio output, the notification gives me
  the following exception:

     [java] 18:45:25.639 SCHWERWIEGEND: util.UtilActivator.uncaughtException().77 An uncaught exception occurred in thread=Thread[Thread-80,10,main] and message
was: null
     [java] java.lang.NullPointerException
     [java] at net.java.sip.communicator.impl.neomedia.jmfext.media.renderer.audio.PortAudioRenderer.open(PortAudioRenderer.java:334)
     [java] at net.java.sip.communicator.impl.neomedia.notify.PortAudioClipImpl.runOnceInPlayThread(PortAudioClipImpl.java:234)
     [java] at net.java.sip.communicator.impl.neomedia.notify.PortAudioClipImpl.runInPlayThread(PortAudioClipImpl.java:131)
     [java] at net.java.sip.communicator.impl.neomedia.notify.PortAudioClipImpl.access$000(PortAudioClipImpl.java:24)
     [java] at net.java.sip.communicator.impl.neomedia.notify.PortAudioClipImpl$1.run(PortAudioClipImpl.java:70)

thus I cannot hear the "dialing sound".

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net