[jitsi-dev] Android audio streams


#1

Hi,

I'm trying to implement volume control in Android, but in order to do so I
need some info about our media service and the javax.media.Rednerer.

On Android there is an AudioManager which exposes methods for controlling
the volume. It defines several stream types and each stream has it's own
volume control. For example we have STREAM_MUSIC, STREAM_NOTIFICATION
and STREAM_VOICE_CALL.

Currently we are
using org.jitsi.impl.neomedia.jmfext.media.renderer.audio.AudioTrackRender
to playback sound on Android using native AudioTrack class. The problem is
that we use 0 = STREAM_VOICE_CALL for all audio. It seems to be right for
calls, but we have also notifcation sounds and I suppose they should use
STREAM_NOTIFICATION volume level.

I need to know if it's possible to discover audio use case in the Renderer
or before it is created ? Then I could set the native streams properly.

Thanks,
Pawel


#2

Hi, Pawel!

The AudioTrackRenderer in jitsi-android as it is currently available
in the Subversion repository is instantiated using a constructor with
a boolean argument enableGainControl. A value of false for that
argument signals (in spite of its name) that the Renderer will be used
for a sound notification.

The argument in question has (mostly) been retrofitted/refactored in
libjitsi to signify the Renderer use case (and AbstractAudioRenderer
has a dataFlow property there which reflect that flag after/outside
construction).

Regards,
Lyubomir


#3

Hi Lyubomir,

Thanks for response ! Unfortunately both streams are created with the same
flag by method AudioSystem createRenderer(playback=true). Notification
sound is created from AudioSystemClipImpl and the call sound stream
from AudioMediaDeviceImpl. So I can't use it to set different stream type.

The data flow property is never changed since object creation(by default
playback data flow is used). I can see that the AudioRecordSystem
in doInitialize() registers only caputre devices. Should I try to implement
and register the notifcation device ?

Regards,
Pawel

···

On Tue, May 7, 2013 at 3:14 PM, Lyubomir Marinov <lyubomir.marinov@jitsi.org > wrote:

Hi, Pawel!

The AudioTrackRenderer in jitsi-android as it is currently available
in the Subversion repository is instantiated using a constructor with
a boolean argument enableGainControl. A value of false for that
argument signals (in spite of its name) that the Renderer will be used
for a sound notification.

The argument in question has (mostly) been retrofitted/refactored in
libjitsi to signify the Renderer use case (and AbstractAudioRenderer
has a dataFlow property there which reflect that flag after/outside
construction).

Regards,
Lyubomir


#4

I'm unable to compile jitsi-android because of the following error so
I can only observe the source code:

The method createOutgoingCall(CallJabberImpl, String,
Iterable<PacketExtension>) in the type
OperationSetBasicTelephonyJabberImpl is not applicable for the
arguments (CallJabberImpl, String, String, null)
OperationSetResAwareTelephonyJabberImpl.java
/jitsi-android/src/net/java/sip/communicator/impl/protocol/jabber line
133

By looking at the code, I see the following:

AudioNotifierService.createAudio(String uri)
AudioNotifierService.createAudio(String uri, boolean playback)

AudioNotifierServiceImpl.createAudio(String uri)
    { return createAudio(uri, false); }
AudioNotifierServiceImpl.createAudio(String uri, boolean playback)
    { ... new AudioSystemClipImpl(..., playback); ... }

AudioSystemClipImpl.enterRunInPlayThread()
    { ... audioSystem.createRenderer(playback); ... }

AudioRecordSystem.createRenderer(boolean playback)
    { return new AudioTrackRenderer(playback); }

As far as I understand, the playback value is preserved one and the
same from AudioNotifierService to AudioTrackRenderer.

Could you please provide a stack trace which yields the incorrect behaviour?

···

2013/5/7 Paweł Domas <pawel.domas@jitsi.org>:

Thanks for response ! Unfortunately both streams are created with the same
flag by method AudioSystem createRenderer(playback=true). Notification sound
is created from AudioSystemClipImpl and the call sound stream from
AudioMediaDeviceImpl. So I can't use it to set different stream type.


#5

I've been debugging it an it's because of start() function
of net.java.sip.communicator.impl.notification.SoundNotificationHandlerImpl:

public void start(SoundNotificationAction action, NotificationData data)
    {
        if(isMute())
            return;

        boolean playOnlyOnPlayback = true;

        AudioNotifierService audioNotifService
            = NotificationActivator.getAudioNotifier();

        if(audioNotifService != null)
        {
            playOnlyOnPlayback

···

=
audioNotifService.audioOutAndNotificationsShareSameDevice();
        }

        if(playOnlyOnPlayback)
        {
            if(action.isSoundNotificationEnabled()
                    >> action.isSoundPlaybackEnabled())
            {
                play(action, data, SCAudioClipDevice.PLAYBACK);
            }
        }
        else
        {
            if(action.isSoundNotificationEnabled())
                play(action, data, SCAudioClipDevice.NOTIFICATION);
            if(action.isSoundPlaybackEnabled())
                play(action, data, SCAudioClipDevice.PLAYBACK);
        }

        if(action.isSoundPCSpeakerEnabled())
            play(action, data, SCAudioClipDevice.PC_SPEAKER);
    }

playOnlyOnPlayback is evaluted to true after the call:

if(audioNotifService != null)
        {
            playOnlyOnPlayback
                =
audioNotifService.audioOutAndNotificationsShareSameDevice();
        }

And this function looks like:

public boolean audioOutAndNotificationsShareSameDevice()
    {
        AudioSystem audioSystem = getDeviceConfiguration().getAudioSystem();
        CaptureDeviceInfo notify
            = audioSystem.getSelectedDevice(AudioSystem.DataFlow.NOTIFY);
        CaptureDeviceInfo playback
            = audioSystem.getSelectedDevice(AudioSystem.DataFlow.PLAYBACK);

        return
            (notify == null)
                ? (playback == null)
                : notify.getLocator().equals(playback.getLocator());
    }

I've checked and actually there are no playback nor notification devices
returned and I belive the problem is at this point.

The stacktrace for audio clip:

org.jitsi.impl.neomedia.device.AudioRecordSystem.createRenderer().50 Create
renderer: true
java.lang.Throwable
at
org.jitsi.impl.neomedia.device.AudioRecordSystem.createRenderer(AudioRecordSystem.java:50)
at
org.jitsi.impl.neomedia.notify.AudioSystemClipImpl.enterRunInPlayThread(AudioSystemClipImpl.java:82)
at
org.jitsi.service.audionotifier.AbstractSCAudioClip.runInPlayThread(AbstractSCAudioClip.java:335)
at
org.jitsi.service.audionotifier.AbstractSCAudioClip.access$200(AbstractSCAudioClip.java:20)
at
org.jitsi.service.audionotifier.AbstractSCAudioClip$1.run(AbstractSCAudioClip.java:295)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
at java.lang.Thread.run(Thread.java:856)

The stacktrace for call audio stream:

org.jitsi.impl.neomedia.device.AudioRecordSystem.createRenderer().50 Create
renderer: true
java.lang.Throwable
at
org.jitsi.impl.neomedia.device.AudioRecordSystem.createRenderer(AudioRecordSystem.java:50)
at
org.jitsi.impl.neomedia.device.AudioMediaDeviceImpl.createRenderer(AudioMediaDeviceImpl.java:219)
at
org.jitsi.impl.neomedia.device.MediaDeviceSession.playerConfigureComplete(MediaDeviceSession.java:1229)
at
org.jitsi.impl.neomedia.device.AudioMediaDeviceSession.playerConfigureComplete(AudioMediaDeviceSession.java:117)
at
org.jitsi.impl.neomedia.device.MediaDeviceSession.playerControllerUpdate(MediaDeviceSession.java:1271)
at
org.jitsi.impl.neomedia.device.MediaDeviceSession$1.controllerUpdate(MediaDeviceSession.java:474)
at net.sf.fmj.media.BasicController.dispatchEvent(BasicController.java:405)
at net.sf.fmj.media.SendEventQueue.processEvent(BasicController.java:1183)
at
net.sf.fmj.media.util.ThreadedEventQueue.dispatchEvents(ThreadedEventQueue.java:54)
at net.sf.fmj.media.util.ThreadedEventQueue.run(ThreadedEventQueue.java:91)

About the compilation I temporary remove the parameter in hope files will
get synchronized once we move to github. So the invalid call may look like:

CallPeer callPeer = jabberTelephony
            .createOutgoingCall(call, uri, null);

Regards,
Pawel

On Tue, May 7, 2013 at 8:46 PM, Lyubomir Marinov <lyubomir.marinov@jitsi.org > wrote:

2013/5/7 Paweł Domas <pawel.domas@jitsi.org>:
> Thanks for response ! Unfortunately both streams are created with the
same
> flag by method AudioSystem createRenderer(playback=true). Notification
sound
> is created from AudioSystemClipImpl and the call sound stream from
> AudioMediaDeviceImpl. So I can't use it to set different stream type.

I'm unable to compile jitsi-android because of the following error so
I can only observe the source code:

The method createOutgoingCall(CallJabberImpl, String,
Iterable<PacketExtension>) in the type
OperationSetBasicTelephonyJabberImpl is not applicable for the
arguments (CallJabberImpl, String, String, null)
OperationSetResAwareTelephonyJabberImpl.java
/jitsi-android/src/net/java/sip/communicator/impl/protocol/jabber line
133

By looking at the code, I see the following:

AudioNotifierService.createAudio(String uri)
AudioNotifierService.createAudio(String uri, boolean playback)

AudioNotifierServiceImpl.createAudio(String uri)
    { return createAudio(uri, false); }
AudioNotifierServiceImpl.createAudio(String uri, boolean playback)
    { ... new AudioSystemClipImpl(..., playback); ... }

AudioSystemClipImpl.enterRunInPlayThread()
    { ... audioSystem.createRenderer(playback); ... }

AudioRecordSystem.createRenderer(boolean playback)
    { return new AudioTrackRenderer(playback); }

As far as I understand, the playback value is preserved one and the
same from AudioNotifierService to AudioTrackRenderer.

Could you please provide a stack trace which yields the incorrect
behaviour?


#6

Thank you very much!

Generally, AudioSystem.DataFlow.NOTIFY and PLAYBACK should be used on
AudioSystem instances which support
FEATURE_NOTIFY_AND_PLAYBACK_DEVICES (please refer to the
DeviceSystem.getFeatures() method).

As to the audioOutAndNotificationsShareSameDevice() method, I think it
should presently be safe and easiest to modify it to return false for
AudioSystem instances which do not support
FEATURE_NOTIFY_AND_PLAYBACK_DEVICES. Could you please do it and let
me/us know if that fixes the problem for you?

···

2013/5/7 Paweł Domas <pawel.domas@jitsi.org>:

public boolean audioOutAndNotificationsShareSameDevice()
    {
        AudioSystem audioSystem = getDeviceConfiguration().getAudioSystem();
        CaptureDeviceInfo notify
            = audioSystem.getSelectedDevice(AudioSystem.DataFlow.NOTIFY);
        CaptureDeviceInfo playback
            = audioSystem.getSelectedDevice(AudioSystem.DataFlow.PLAYBACK);

        return
            (notify == null)
                ? (playback == null)
                : notify.getLocator().equals(playback.getLocator());
    }


#7

Thanks for help ! I've changed the method to return false for audio systems
without FEATURE_NOTIFY_AND_PLAYBACK_DEVICES. Now it runs the renderer with
different values for call sound and notification.

However the problem is more complicated, as in case of tablets it will play
notification sound two times on the same device: once for notification and
once for playback. In case of phones it should be played on different
devices(the call and the loud speaker), but I don't have how to check that.
I need to think about it a bit more.

Regards,
Pawel

···

On Tue, May 7, 2013 at 10:21 PM, Lyubomir Marinov < lyubomir.marinov@jitsi.org> wrote:

2013/5/7 Paweł Domas <pawel.domas@jitsi.org>:
> public boolean audioOutAndNotificationsShareSameDevice()
> {
> AudioSystem audioSystem =
getDeviceConfiguration().getAudioSystem();
> CaptureDeviceInfo notify
> = audioSystem.getSelectedDevice(AudioSystem.DataFlow.NOTIFY);
> CaptureDeviceInfo playback
> =
audioSystem.getSelectedDevice(AudioSystem.DataFlow.PLAYBACK);
>
> return
> (notify == null)
> ? (playback == null)
> : notify.getLocator().equals(playback.getLocator());
> }

Thank you very much!

Generally, AudioSystem.DataFlow.NOTIFY and PLAYBACK should be used on
AudioSystem instances which support
FEATURE_NOTIFY_AND_PLAYBACK_DEVICES (please refer to the
DeviceSystem.getFeatures() method).

As to the audioOutAndNotificationsShareSameDevice() method, I think it
should presently be safe and easiest to modify it to return false for
AudioSystem instances which do not support
FEATURE_NOTIFY_AND_PLAYBACK_DEVICES. Could you please do it and let
me/us know if that fixes the problem for you?


#8

Hi,

Thanks for help ! I've changed the method to return false for audio systems without FEATURE_NOTIFY_AND_PLAYBACK_DEVICES. Now it runs the renderer with different values for call sound and notification.

However the problem is more complicated, as in case of tablets it will play notification sound two times on the same device: once for notification and once for playback. In case of phones it should be played on different devices(the call and the loud speaker), but I don't have how to check that. I need to think about it a bit more.

Regards,
  Pawel

public boolean audioOutAndNotificationsShareSameDevice()
     {
         AudioSystem audioSystem = getDeviceConfiguration().getAudioSystem();
         CaptureDeviceInfo notify
             = audioSystem.getSelectedDevice(AudioSystem.DataFlow.NOTIFY);
         CaptureDeviceInfo playback
             = audioSystem.getSelectedDevice(AudioSystem.DataFlow.PLAYBACK);

         return
             (notify == null)
                 ? (playback == null)
                 : notify.getLocator().equals(playback.getLocator());
     }

Thank you very much!

Generally, AudioSystem.DataFlow.NOTIFY and PLAYBACK should be used on
AudioSystem instances which support
FEATURE_NOTIFY_AND_PLAYBACK_DEVICES (please refer to the
DeviceSystem.getFeatures() method).

As to the audioOutAndNotificationsShareSameDevice() method, I think it
should presently be safe and easiest to modify it to return false for
AudioSystem instances which do not support
FEATURE_NOTIFY_AND_PLAYBACK_DEVICES. Could you please do it and let
me/us know if that fixes the problem for you?

Back to the thread....

I can't just make the notifications to be played on both VOICE_CALL
and NOTIFICATION streams, because they are the same device on tablets.

We can endup in the same situation on the desktop. I believe we simply don't play notifications once if we detect that both devices are the same.

Wouldn't this work on anndroid as well?

Cheers,
Emil

···

On 13.06.13, 16:12, Paweł Domas wrote:

2013/5/8 Paweł Domas <pawel.domas@jitsi.org>

On Tue, May 7, 2013 at 10:21 PM, Lyubomir Marinov <lyubomir.marinov@jitsi.org> wrote:

2013/5/7 Paweł Domas <pawel.domas@jitsi.org>:

As a result incoming call sound will be played twice with small delay.
So I thought it may work to implement pseudo devices which will be
bound to these streams. Then the user will be able to configure in
settings which one he wants to use. What do you think ?

If it could work then the question is how do I add such device to JFM
? I saw that during initialization AudioSystem is registering some
devices, but I can't find "auto" device resoultion. When the
notifications are played method createdRenderer is used and it returns
the renderer for playback or notification. Should I at this point
check the configuration and select the device ?

--
Regards,
  Pawel

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
https://jitsi.org


#9

Hi,

Thanks for help ! I've changed the method to return false for audio
systems without FEATURE_NOTIFY_AND_PLAYBACK_DEVICES. Now it runs the
renderer with different values for call sound and notification.

However the problem is more complicated, as in case of tablets it will
play notification sound two times on the same device: once for notification
and once for playback. In case of phones it should be played on different
devices(the call and the loud speaker), but I don't have how to check that.
I need to think about it a bit more.

Regards,
  Pawel

public boolean audioOutAndNotificationsShareSameDevice()
     {
         AudioSystem audioSystem =
getDeviceConfiguration().getAudioSystem();
         CaptureDeviceInfo notify
             =
audioSystem.getSelectedDevice(AudioSystem.DataFlow.NOTIFY);
         CaptureDeviceInfo playback
             =
audioSystem.getSelectedDevice(AudioSystem.DataFlow.PLAYBACK);

         return
             (notify == null)
                 ? (playback == null)
                 : notify.getLocator().equals(playback.getLocator());
     }

Thank you very much!

Generally, AudioSystem.DataFlow.NOTIFY and PLAYBACK should be used on
AudioSystem instances which support
FEATURE_NOTIFY_AND_PLAYBACK_DEVICES (please refer to the
DeviceSystem.getFeatures() method).

As to the audioOutAndNotificationsShareSameDevice() method, I think it
should presently be safe and easiest to modify it to return false for
AudioSystem instances which do not support
FEATURE_NOTIFY_AND_PLAYBACK_DEVICES. Could you please do it and let
me/us know if that fixes the problem for you?

Back to the thread....

I can't just make the notifications to be played on both VOICE_CALL
and NOTIFICATION streams, because they are the same device on tablets.

We can endup in the same situation on the desktop. I believe we simply don't
play notifications once if we detect that both devices are the same.

Wouldn't this work on anndroid as well?

Correct me if I'm wrong. On Android I can't just select particular
device, but stream. AudoManager defines:

STREAM_ALARM - The audio stream for alarms
STREAM_DTMF - The audio stream for DTMF Tones
STREAM_MUSIC - The audio stream for music playback
STREAM_NOTIFICATION - The audio stream for notifications
STREAM_RING - The audio stream for the phone ring
STREAM_SYSTEM - The audio stream for system sounds
STREAM_VOICE_CALL - The audio stream for phone calls

By looking at the API and docs there is no way to find out if
STREAM_VOICE_CALL and STREAM_NOTIFICATION are the same device(and this
is the case on tablet). Maybe it would be enough to always play
notification on the notification stream and the calls on voice call.
The system will do it's best to handle it right regards on what device
it's running ?

Regards,
Pawel

···

On Thu, Jun 13, 2013 at 4:36 PM, Emil Ivov <emcho@jitsi.org> wrote:

On 13.06.13, 16:12, Paweł Domas wrote:

2013/5/8 Paweł Domas <pawel.domas@jitsi.org>

On Tue, May 7, 2013 at 10:21 PM, Lyubomir Marinov >>> <lyubomir.marinov@jitsi.org> wrote:

2013/5/7 Paweł Domas <pawel.domas@jitsi.org>:

Cheers,
Emil

As a result incoming call sound will be played twice with small delay.
So I thought it may work to implement pseudo devices which will be
bound to these streams. Then the user will be able to configure in
settings which one he wants to use. What do you think ?

If it could work then the question is how do I add such device to JFM
? I saw that during initialization AudioSystem is registering some
devices, but I can't find "auto" device resoultion. When the
notifications are played method createdRenderer is used and it returns
the renderer for playback or notification. Should I at this point
check the configuration and select the device ?

--
Regards,
  Pawel

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
https://jitsi.org

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#10

By looking at the API and docs there is no way to find out if
STREAM_VOICE_CALL and STREAM_NOTIFICATION are the same device(and this
is the case on tablet). Maybe it would be enough to always play
notification on the notification stream and the calls on voice call.
The system will do it's best to handle it right regards on what device
it's running ?

Yes, I think this makes sense!

Emil

···

On 13.06.13, 16:57, Paweł Domas wrote:

Regards,
  Pawel

Cheers,
Emil

As a result incoming call sound will be played twice with small delay.
So I thought it may work to implement pseudo devices which will be
bound to these streams. Then the user will be able to configure in
settings which one he wants to use. What do you think ?

If it could work then the question is how do I add such device to JFM
? I saw that during initialization AudioSystem is registering some
devices, but I can't find "auto" device resoultion. When the
notifications are played method createdRenderer is used and it returns
the renderer for playback or notification. Should I at this point
check the configuration and select the device ?

--
Regards,
   Pawel

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
https://jitsi.org

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
https://jitsi.org