[jitsi-dev] Switching devices during a call


#1

Ingo,

First off, thank you for the response - I appreciate it.

The newest code in GitHub is for the 2.9 version (still in development), correct? Would the 2.8 (stable) version contain these same libjitsi and fmj improvements? The move from 2.2 -> 2.4 did help, but the release notes from 2.6/2.8 did not indicate changes in this area. So, I did not upgrade further.

It is true that I am a developer, and I have chased this (countless hours) down the rabbit hole about as far as I can go. I do know that the problems I have with switching audio devices tend to center around changing the devices during an audio call. This is likely due to the fact that Jitsi has to not only switch devices but also tear down/set up the audio streams that are in play. The issues get even worse if the audio switch happens around the initiation or termination (hang-up) of a call - race conditions in these instances. I try to guard against those scenarios by checking the SIP call state before switching audio devices and only changing the devices outside of these scenarios. But, a primary use-case for me is the user hanging-up a call by hanging-up a handset (going on-hook) - that sets in motion a SIP call BYE and a switch from handset back to hands-free.

As far as I can tell, Jitsi seems to be doing what it is supposed to do:

1. Change the capture/playback/notification audio device.
2. Propagate property change notifications throughout the system.
3. Tear down/set up the audio streams, as needed.

However, deep down in the JMF code, something is going awry. Most times, I see the first device (playback) change, and Jitsi "acknowledges" the change by issuing a "Looking for configured audio devices" log. Then, all is quiet. That thread I am changing devices on never ever issues another log until the application is restarted. It seems locked waiting on something, but not deadlocked per JVisualVM. I occasionally see the NPE below, but I am not sure if the SSRCInfo error means it just doesn't like the Asterisk 11 server I use or if this is related to the audio device switch itself. Regardless, it seems to affect stability with the entire SIP/RTP subsystem. The NPE is related to the RTP streams, which are directly related to the audio streams Jitsi creates for the audio devices.

I guess I'll ask a basic question, to maybe point me to a place to focus: is there something inherently wrong with trying to change audio devices on a separate thread (76-pool-7-thread-1 in the NPE example below) from the main/UI thread? I don't want to lock-up the application on a device switch, but I am wondering if this separate thread could be causing some issues? Do the audio device changes *have to* occur on a certain thread to be effected correctly? With 2.2, it seemed like they had to be effected on the main/UI thread to even work.

Thank you again for your help,

Neil S.

···

On 2015-11-19 16:10, dev-request@jitsi.org wrote:

Some of this code is dependent on native resources and thus
unfortunately rather brittle. Switching devices is not something you'd
do regularly inside Jitsi, so these aren't issues that were
encountered there.
Some tips:
- Use thes newest code from Github, there have been some improvementd
in the libjitsi and fmj
- You're a developer, so please try to find e.g. why the NPE you
posted appears and ideally create a PR with a fix

Sorry if the last sentence is a bit harsh, but the device access code
isn't something thats used by Jitsi Meet or the Videobridge and thus
further development depends entirely on the community.

Freundliche Grüsse,
Ingo Bauersachs

-- sent from my mobile

Le 19.11.2015 à 17:31, Neil Sharp <developer@akamaiapps.com> a écrit :

Is there anyone that can point me to some help on this? Do I need to file a bug to get a response? If so, I can.

My co-workers and I have been through the libjitsi/jitsi codebase quite a lot (AudioSystem, MediaStreamImpl, PropertyChangeNotifier, etc.), but we cannot figure out why Jitsi/JMF is so fragile when switching audio devices (especially after call audio has started).

I am battling the result of this issue right now, which is that the audio switching thread just stops after a while; and, the user gets stuck with a _hands-free_ capture device and a _handset_ speaker/notification device. This gives the appearance that everything has stopped working in the application.

Thanks in advance for any help,

Neil S.

On 2015-11-11 16:31, developer@akamaiapps.com wrote:
Long-time lurker, first time poster. I have been experimenting with
Jitsi for a while, and I keep bumping into an issue that I'm hoping
someone here can shed some light on.
I am working on an application that incorporates libjitsi, where the
user can switch audio device sets during an active call - similar to
switching from a handset to a speakerphone (and back). Most times,
switching between these device sets (since there is a capture,
playback and notification device for each switch) happens without
issue. However, over a few days of use, I see a variety of issues:
- NullPointerExceptions happening when switching the capture device
(usually) - see below.
- Thread "deadlocks" when switching devices (could be the capture,
playback or notification device). Note that the deadlocks do not seem
to be true resource deadlocks, at least per JVisualVM - the thread to
set a device just never returns or exits. Thus, I can never set any
device after that, until I restart my app.
- Threads that never get cleaned up inside the JMF/Jitsi code,
resulting in CPU usage spikes over time. This is difficult to show,
but JVisualVM displays a thread count that grows during calls and is
not completely cleaned up afterwards. As far as I can tell, these are
threads related to Jitsi and/or JMF.
I can give an example (when switching the capture device during an active call):
2015-11-11T11:01:01.016 EST [SEVERE] 76-pool-7-thread-1 ::
org.jitsi.util.Logger.error() :: Failed to close send stream
net.sf.fmj.media.rtp.SendSSRCInfo@14b5abc
java.lang.NullPointerException
at net.sf.fmj.media.rtp.RTCPTransmitter.bye(RTCPTransmitter.java:82)
at net.sf.fmj.media.rtp.RTCPReporter.releasessrc(RTCPReporter.java:45)
at net.sf.fmj.media.rtp.RTCPReporter.close(RTCPReporter.java:39)
at net.sf.fmj.media.rtp.RTPSessionMgr.stopParticipating(RTPSessionMgr.java:2743)
at net.sf.fmj.media.rtp.RTPSessionMgr.removeSendStream(RTPSessionMgr.java:1988)
at net.sf.fmj.media.rtp.SendSSRCInfo.close(SendSSRCInfo.java:95)
at
org.jitsi.impl.neomedia.MediaStreamImpl.stopSendStreams(MediaStreamImpl.java:2326)
at
org.jitsi.impl.neomedia.MediaStreamImpl.stopSendStreams(MediaStreamImpl.java:2283)
at
org.jitsi.impl.neomedia.MediaStreamImpl.closeSendStreams(MediaStreamImpl.java:637)
at
org.jitsi.impl.neomedia.MediaStreamImpl.recreateSendStreams(MediaStreamImpl.java:1404)
at
org.jitsi.impl.neomedia.MediaStreamImpl.deviceSessionChanged(MediaStreamImpl.java:770)
at
org.jitsi.impl.neomedia.AudioMediaStreamImpl.deviceSessionChanged(AudioMediaStreamImpl.java:371)
at org.jitsi.impl.neomedia.MediaStreamImpl.setDevice(MediaStreamImpl.java:1707)
at
net.java.sip.communicator.service.protocol.media.CallPeerMediaHandler.callPropertyChange(CallPeerMediaHandler.java:395)
at
net.java.sip.communicator.service.protocol.media.CallPeerMediaHandler.access$000(CallPeerMediaHandler.java:39)
at
net.java.sip.communicator.service.protocol.media.CallPeerMediaHandler$CallPropertyChangeListener.propertyChange(CallPeerMediaHandler.java:1992)
at java.beans.PropertyChangeSupport.fire(Unknown Source)
at java.beans.PropertyChangeSupport.firePropertyChange(Unknown Source)
at java.beans.PropertyChangeSupport.firePropertyChange(Unknown Source)
at
net.java.sip.communicator.service.protocol.AbstractCall.firePropertyChange(AbstractCall.java:187)
at
net.java.sip.communicator.service.protocol.media.MediaAwareCall.firePropertyChange(MediaAwareCall.java:947)
at
net.java.sip.communicator.service.protocol.media.MediaAwareCall.propertyChange(MediaAwareCall.java:856)
at
org.jitsi.util.event.PropertyChangeNotifier.firePropertyChange(PropertyChangeNotifier.java:127)
at
net.java.sip.communicator.service.protocol.media.MediaAwareCallConference.propertyChange(MediaAwareCallConference.java:389)
at
net.java.sip.communicator.service.protocol.media.MediaAwareCallConference.access$000(MediaAwareCallConference.java:27)
at
net.java.sip.communicator.service.protocol.media.MediaAwareCallConference$1.propertyChange(MediaAwareCallConference.java:72)
at
net.java.sip.communicator.service.protocol.media.MediaAwareCallConference$WeakPropertyChangeListener.propertyChange(MediaAwareCallConference.java:574)
at
org.jitsi.util.event.PropertyChangeNotifier.firePropertyChange(PropertyChangeNotifier.java:127)
at
org.jitsi.impl.neomedia.MediaServiceImpl.deviceConfigurationPropertyChange(MediaServiceImpl.java:1468)
at org.jitsi.impl.neomedia.MediaServiceImpl.access$000(MediaServiceImpl.java:49)
at
org.jitsi.impl.neomedia.MediaServiceImpl$1.propertyChange(MediaServiceImpl.java:154)
at
org.jitsi.util.event.PropertyChangeNotifier.firePropertyChange(PropertyChangeNotifier.java:127)
at
org.jitsi.impl.neomedia.device.DeviceConfiguration.propertyChange(DeviceConfiguration.java:947)
at
org.jitsi.util.event.PropertyChangeNotifier.firePropertyChange(PropertyChangeNotifier.java:127)
at
org.jitsi.impl.neomedia.device.AudioSystem.propertyChange(AudioSystem.java:651)
at org.jitsi.impl.neomedia.device.Devices.setDevice(Devices.java:390)
at org.jitsi.impl.neomedia.device.AudioSystem.setDevice(AudioSystem.java:709)
       <-- my code entry which calls AudioSystem.setDevice() -->
My app is a Java-based application, using libjitsi 2.4.4997. It was
using libjitsi 2.2, with similar but more severe issues. Upgrading
helped a bit, but I am still battling these core issues. Any help or
insight into what I could be doing wrong would be appreciated.
I would especially like to know if something else, other than
AudioSystem.setDevice(), should be used to perform the device
switches. As far as I can tell from the Jitsi client source code
(granted the GitHub code is 2.9...), I am switching the devices in a
similar way - just programmatically instead of being driven off of
user clicks on the combo boxes. From the stack trace above, I can see
Jitsi detecting the device change and propagating the property change
to all interested parties. Jitsi seems to manage the stopping and
re-starting of the audio streams as well, such that I should not have
to manually do that myself.
If there is any other information that could aid in getting to the
bottom of this, I can try to provide it.
Thanks in advance,
Neil S.


#2

First off, thank you for the response - I appreciate it.

The newest code in GitHub is for the 2.9 version (still in development),
correct? Would the 2.8 (stable) version contain these same libjitsi and
fmj improvements? The move from 2.2 -> 2.4 did help, but the release
notes from 2.6/2.8 did not indicate changes in this area. So, I did not
upgrade further.

I don't think there were many changes, if any at all. However, if you debug or post stacktraces, it's easier to compare them against current code.

It is true that I am a developer, and I have chased this (countless
hours) down the rabbit hole about as far as I can go. I do know that the
problems I have with switching audio devices tend to center around
changing the devices during an audio call. This is likely due to the
fact that Jitsi has to not only switch devices but also tear down/set up
the audio streams that are in play. The issues get even worse if the
audio switch happens around the initiation or termination (hang-up) of a
call - race conditions in these instances. I try to guard against those
scenarios by checking the SIP call state before switching audio devices
and only changing the devices outside of these scenarios. But, a primary
use-case for me is the user hanging-up a call by hanging-up a handset
(going on-hook) - that sets in motion a SIP call BYE and a switch from
handset back to hands-free.

As far as I can tell, Jitsi seems to be doing what it is supposed to do:

1. Change the capture/playback/notification audio device.
2. Propagate property change notifications throughout the system.
3. Tear down/set up the audio streams, as needed.

However, deep down in the JMF code, something is going awry. Most times,
I see the first device (playback) change, and Jitsi "acknowledges" the
change by issuing a "Looking for configured audio devices" log. Then,
all is quiet. That thread I am changing devices on never ever issues
another log until the application is restarted. It seems locked waiting
on something, but not deadlocked per JVisualVM. I occasionally see the
NPE below, but I am not sure if the SSRCInfo error means it just doesn't
like the Asterisk 11 server I use or if this is related to the audio
device switch itself. Regardless, it seems to affect stability with the
entire SIP/RTP subsystem. The NPE is related to the RTP streams, which
are directly related to the audio streams Jitsi creates for the audio
devices.

I guess I'll ask a basic question, to maybe point me to a place to
focus: is there something inherently wrong with trying to change audio
devices on a separate thread (76-pool-7-thread-1 in the NPE example
below) from the main/UI thread? I don't want to lock-up the application
on a device switch, but I am wondering if this separate thread could be
causing some issues? Do the audio device changes *have to* occur on a
certain thread to be effected correctly? With 2.2, it seemed like they
had to be effected on the main/UI thread to even work.

I have absolutely no idea. You found out more than I know so far and all other developers who know/knew this code are now working on Jitsi Meet. So you're basically on your own here. Sorry.

Thank you again for your help,

Neil S.

Ingo