[jitsi-dev] Switching devices during a call


#1

Long-time lurker, first time poster. I have been experimenting with Jitsi for a while, and I keep bumping into an issue that I'm hoping someone here can shed some light on.

I am working on an application that incorporates libjitsi, where the user can switch audio device sets during an active call - similar to switching from a handset to a speakerphone (and back). Most times, switching between these device sets (since there is a capture, playback and notification device for each switch) happens without issue. However, over a few days of use, I see a variety of issues:

- NullPointerExceptions happening when switching the capture device (usually) - see below.

- Thread "deadlocks" when switching devices (could be the capture, playback or notification device). Note that the deadlocks do not seem to be true resource deadlocks, at least per JVisualVM - the thread to set a device just never returns or exits. Thus, I can never set any device after that, until I restart my app.

- Threads that never get cleaned up inside the JMF/Jitsi code, resulting in CPU usage spikes over time. This is difficult to show, but JVisualVM displays a thread count that grows during calls and is not completely cleaned up afterwards. As far as I can tell, these are threads related to Jitsi and/or JMF.

I can give an example (when switching the capture device during an active call):

2015-11-11T11:01:01.016 EST [SEVERE] 76-pool-7-thread-1 :: org.jitsi.util.Logger.error() :: Failed to close send stream net.sf.fmj.media.rtp.SendSSRCInfo@14b5abc
java.lang.NullPointerException
  at net.sf.fmj.media.rtp.RTCPTransmitter.bye(RTCPTransmitter.java:82)
  at net.sf.fmj.media.rtp.RTCPReporter.releasessrc(RTCPReporter.java:45)
  at net.sf.fmj.media.rtp.RTCPReporter.close(RTCPReporter.java:39)
  at net.sf.fmj.media.rtp.RTPSessionMgr.stopParticipating(RTPSessionMgr.java:2743)
  at net.sf.fmj.media.rtp.RTPSessionMgr.removeSendStream(RTPSessionMgr.java:1988)
  at net.sf.fmj.media.rtp.SendSSRCInfo.close(SendSSRCInfo.java:95)
  at org.jitsi.impl.neomedia.MediaStreamImpl.stopSendStreams(MediaStreamImpl.java:2326)
  at org.jitsi.impl.neomedia.MediaStreamImpl.stopSendStreams(MediaStreamImpl.java:2283)
  at org.jitsi.impl.neomedia.MediaStreamImpl.closeSendStreams(MediaStreamImpl.java:637)
  at org.jitsi.impl.neomedia.MediaStreamImpl.recreateSendStreams(MediaStreamImpl.java:1404)
  at org.jitsi.impl.neomedia.MediaStreamImpl.deviceSessionChanged(MediaStreamImpl.java:770)
  at org.jitsi.impl.neomedia.AudioMediaStreamImpl.deviceSessionChanged(AudioMediaStreamImpl.java:371)
  at org.jitsi.impl.neomedia.MediaStreamImpl.setDevice(MediaStreamImpl.java:1707)
  at net.java.sip.communicator.service.protocol.media.CallPeerMediaHandler.callPropertyChange(CallPeerMediaHandler.java:395)
  at net.java.sip.communicator.service.protocol.media.CallPeerMediaHandler.access$000(CallPeerMediaHandler.java:39)
  at net.java.sip.communicator.service.protocol.media.CallPeerMediaHandler$CallPropertyChangeListener.propertyChange(CallPeerMediaHandler.java:1992)
  at java.beans.PropertyChangeSupport.fire(Unknown Source)
  at java.beans.PropertyChangeSupport.firePropertyChange(Unknown Source)
  at net.java.sip.communicator.service.protocol.AbstractCall.firePropertyChange(AbstractCall.java:187)
  at net.java.sip.communicator.service.protocol.media.MediaAwareCall.firePropertyChange(MediaAwareCall.java:947)
  at net.java.sip.communicator.service.protocol.media.MediaAwareCall.propertyChange(MediaAwareCall.java:856)
  at org.jitsi.util.event.PropertyChangeNotifier.firePropertyChange(PropertyChangeNotifier.java:127)
  at net.java.sip.communicator.service.protocol.media.MediaAwareCallConference.propertyChange(MediaAwareCallConference.java:389)
  at net.java.sip.communicator.service.protocol.media.MediaAwareCallConference.access$000(MediaAwareCallConference.java:27)
  at net.java.sip.communicator.service.protocol.media.MediaAwareCallConference$1.propertyChange(MediaAwareCallConference.java:72)
  at net.java.sip.communicator.service.protocol.media.MediaAwareCallConference$WeakPropertyChangeListener.propertyChange(MediaAwareCallConference.java:574)
  at org.jitsi.util.event.PropertyChangeNotifier.firePropertyChange(PropertyChangeNotifier.java:127)
  at org.jitsi.impl.neomedia.MediaServiceImpl.deviceConfigurationPropertyChange(MediaServiceImpl.java:1468)
  at org.jitsi.impl.neomedia.MediaServiceImpl.access$000(MediaServiceImpl.java:49)
  at org.jitsi.impl.neomedia.MediaServiceImpl$1.propertyChange(MediaServiceImpl.java:154)
  at org.jitsi.util.event.PropertyChangeNotifier.firePropertyChange(PropertyChangeNotifier.java:127)
  at org.jitsi.impl.neomedia.device.DeviceConfiguration.propertyChange(DeviceConfiguration.java:947)
  at org.jitsi.util.event.PropertyChangeNotifier.firePropertyChange(PropertyChangeNotifier.java:127)
  at org.jitsi.impl.neomedia.device.AudioSystem.propertyChange(AudioSystem.java:651)
  at org.jitsi.impl.neomedia.device.Devices.setDevice(Devices.java:390)
  at org.jitsi.impl.neomedia.device.AudioSystem.setDevice(AudioSystem.java:709)
         <-- my code entry which calls AudioSystem.setDevice() -->

My app is a Java-based application, using libjitsi 2.4.4997. It was using libjitsi 2.2, with similar but more severe issues. Upgrading helped a bit, but I am still battling these core issues. Any help or insight into what I could be doing wrong would be appreciated.

I would especially like to know if something else, other than AudioSystem.setDevice(), should be used to perform the device switches. As far as I can tell from the Jitsi client source code (granted the GitHub code is 2.9...), I am switching the devices in a similar way - just programmatically instead of being driven off of user clicks on the combo boxes. From the stack trace above, I can see Jitsi detecting the device change and propagating the property change to all interested parties. Jitsi seems to manage the stopping and re-starting of the audio streams as well, such that I should not have to manually do that myself.

If there is any other information that could aid in getting to the bottom of this, I can try to provide it.

Thanks in advance,

Neil S.


#2

Is there anyone that can point me to some help on this? Do I need to file a bug to get a response? If so, I can.

My co-workers and I have been through the libjitsi/jitsi codebase quite a lot (AudioSystem, MediaStreamImpl, PropertyChangeNotifier, etc.), but we cannot figure out why Jitsi/JMF is so fragile when switching audio devices (especially after call audio has started).

I am battling the result of this issue right now, which is that the audio switching thread just stops after a while; and, the user gets stuck with a _hands-free_ capture device and a _handset_ speaker/notification device. This gives the appearance that everything has stopped working in the application.

Thanks in advance for any help,

Neil S.

···

On 2015-11-11 16:31, developer@akamaiapps.com wrote:

Long-time lurker, first time poster. I have been experimenting with
Jitsi for a while, and I keep bumping into an issue that I'm hoping
someone here can shed some light on.

I am working on an application that incorporates libjitsi, where the
user can switch audio device sets during an active call - similar to
switching from a handset to a speakerphone (and back). Most times,
switching between these device sets (since there is a capture,
playback and notification device for each switch) happens without
issue. However, over a few days of use, I see a variety of issues:

- NullPointerExceptions happening when switching the capture device
(usually) - see below.

- Thread "deadlocks" when switching devices (could be the capture,
playback or notification device). Note that the deadlocks do not seem
to be true resource deadlocks, at least per JVisualVM - the thread to
set a device just never returns or exits. Thus, I can never set any
device after that, until I restart my app.

- Threads that never get cleaned up inside the JMF/Jitsi code,
resulting in CPU usage spikes over time. This is difficult to show,
but JVisualVM displays a thread count that grows during calls and is
not completely cleaned up afterwards. As far as I can tell, these are
threads related to Jitsi and/or JMF.

I can give an example (when switching the capture device during an active call):

2015-11-11T11:01:01.016 EST [SEVERE] 76-pool-7-thread-1 ::
org.jitsi.util.Logger.error() :: Failed to close send stream
net.sf.fmj.media.rtp.SendSSRCInfo@14b5abc
java.lang.NullPointerException
at net.sf.fmj.media.rtp.RTCPTransmitter.bye(RTCPTransmitter.java:82)
at net.sf.fmj.media.rtp.RTCPReporter.releasessrc(RTCPReporter.java:45)
at net.sf.fmj.media.rtp.RTCPReporter.close(RTCPReporter.java:39)
at net.sf.fmj.media.rtp.RTPSessionMgr.stopParticipating(RTPSessionMgr.java:2743)
at net.sf.fmj.media.rtp.RTPSessionMgr.removeSendStream(RTPSessionMgr.java:1988)
at net.sf.fmj.media.rtp.SendSSRCInfo.close(SendSSRCInfo.java:95)
at
org.jitsi.impl.neomedia.MediaStreamImpl.stopSendStreams(MediaStreamImpl.java:2326)
at
org.jitsi.impl.neomedia.MediaStreamImpl.stopSendStreams(MediaStreamImpl.java:2283)
at
org.jitsi.impl.neomedia.MediaStreamImpl.closeSendStreams(MediaStreamImpl.java:637)
at
org.jitsi.impl.neomedia.MediaStreamImpl.recreateSendStreams(MediaStreamImpl.java:1404)
at
org.jitsi.impl.neomedia.MediaStreamImpl.deviceSessionChanged(MediaStreamImpl.java:770)
at
org.jitsi.impl.neomedia.AudioMediaStreamImpl.deviceSessionChanged(AudioMediaStreamImpl.java:371)
at org.jitsi.impl.neomedia.MediaStreamImpl.setDevice(MediaStreamImpl.java:1707)
at
net.java.sip.communicator.service.protocol.media.CallPeerMediaHandler.callPropertyChange(CallPeerMediaHandler.java:395)
at
net.java.sip.communicator.service.protocol.media.CallPeerMediaHandler.access$000(CallPeerMediaHandler.java:39)
at
net.java.sip.communicator.service.protocol.media.CallPeerMediaHandler$CallPropertyChangeListener.propertyChange(CallPeerMediaHandler.java:1992)
at java.beans.PropertyChangeSupport.fire(Unknown Source)
at java.beans.PropertyChangeSupport.firePropertyChange(Unknown Source)
at
net.java.sip.communicator.service.protocol.AbstractCall.firePropertyChange(AbstractCall.java:187)
at
net.java.sip.communicator.service.protocol.media.MediaAwareCall.firePropertyChange(MediaAwareCall.java:947)
at
net.java.sip.communicator.service.protocol.media.MediaAwareCall.propertyChange(MediaAwareCall.java:856)
at
org.jitsi.util.event.PropertyChangeNotifier.firePropertyChange(PropertyChangeNotifier.java:127)
at
net.java.sip.communicator.service.protocol.media.MediaAwareCallConference.propertyChange(MediaAwareCallConference.java:389)
at
net.java.sip.communicator.service.protocol.media.MediaAwareCallConference.access$000(MediaAwareCallConference.java:27)
at
net.java.sip.communicator.service.protocol.media.MediaAwareCallConference$1.propertyChange(MediaAwareCallConference.java:72)
at
net.java.sip.communicator.service.protocol.media.MediaAwareCallConference$WeakPropertyChangeListener.propertyChange(MediaAwareCallConference.java:574)
at
org.jitsi.util.event.PropertyChangeNotifier.firePropertyChange(PropertyChangeNotifier.java:127)
at
org.jitsi.impl.neomedia.MediaServiceImpl.deviceConfigurationPropertyChange(MediaServiceImpl.java:1468)
at org.jitsi.impl.neomedia.MediaServiceImpl.access$000(MediaServiceImpl.java:49)
at
org.jitsi.impl.neomedia.MediaServiceImpl$1.propertyChange(MediaServiceImpl.java:154)
at
org.jitsi.util.event.PropertyChangeNotifier.firePropertyChange(PropertyChangeNotifier.java:127)
at
org.jitsi.impl.neomedia.device.DeviceConfiguration.propertyChange(DeviceConfiguration.java:947)
at
org.jitsi.util.event.PropertyChangeNotifier.firePropertyChange(PropertyChangeNotifier.java:127)
at
org.jitsi.impl.neomedia.device.AudioSystem.propertyChange(AudioSystem.java:651)
at org.jitsi.impl.neomedia.device.Devices.setDevice(Devices.java:390)
at org.jitsi.impl.neomedia.device.AudioSystem.setDevice(AudioSystem.java:709)
        <-- my code entry which calls AudioSystem.setDevice() -->

My app is a Java-based application, using libjitsi 2.4.4997. It was
using libjitsi 2.2, with similar but more severe issues. Upgrading
helped a bit, but I am still battling these core issues. Any help or
insight into what I could be doing wrong would be appreciated.

I would especially like to know if something else, other than
AudioSystem.setDevice(), should be used to perform the device
switches. As far as I can tell from the Jitsi client source code
(granted the GitHub code is 2.9...), I am switching the devices in a
similar way - just programmatically instead of being driven off of
user clicks on the combo boxes. From the stack trace above, I can see
Jitsi detecting the device change and propagating the property change
to all interested parties. Jitsi seems to manage the stopping and
re-starting of the audio streams as well, such that I should not have
to manually do that myself.

If there is any other information that could aid in getting to the
bottom of this, I can try to provide it.

Thanks in advance,

Neil S.


#3

Some of this code is dependent on native resources and thus unfortunately rather brittle. Switching devices is not something you'd do regularly inside Jitsi, so these aren't issues that were encountered there.
Some tips:
- Use thes newest code from Github, there have been some improvementd in the libjitsi and fmj
- You're a developer, so please try to find e.g. why the NPE you posted appears and ideally create a PR with a fix

Sorry if the last sentence is a bit harsh, but the device access code isn't something thats used by Jitsi Meet or the Videobridge and thus further development depends entirely on the community.

Freundliche Grüsse,
Ingo Bauersachs

-- sent from my mobile

···

Le 19.11.2015 à 17:31, Neil Sharp <developer@akamaiapps.com> a écrit :

Is there anyone that can point me to some help on this? Do I need to file a bug to get a response? If so, I can.

My co-workers and I have been through the libjitsi/jitsi codebase quite a lot (AudioSystem, MediaStreamImpl, PropertyChangeNotifier, etc.), but we cannot figure out why Jitsi/JMF is so fragile when switching audio devices (especially after call audio has started).

I am battling the result of this issue right now, which is that the audio switching thread just stops after a while; and, the user gets stuck with a _hands-free_ capture device and a _handset_ speaker/notification device. This gives the appearance that everything has stopped working in the application.

Thanks in advance for any help,

Neil S.

On 2015-11-11 16:31, developer@akamaiapps.com wrote:
Long-time lurker, first time poster. I have been experimenting with
Jitsi for a while, and I keep bumping into an issue that I'm hoping
someone here can shed some light on.
I am working on an application that incorporates libjitsi, where the
user can switch audio device sets during an active call - similar to
switching from a handset to a speakerphone (and back). Most times,
switching between these device sets (since there is a capture,
playback and notification device for each switch) happens without
issue. However, over a few days of use, I see a variety of issues:
- NullPointerExceptions happening when switching the capture device
(usually) - see below.
- Thread "deadlocks" when switching devices (could be the capture,
playback or notification device). Note that the deadlocks do not seem
to be true resource deadlocks, at least per JVisualVM - the thread to
set a device just never returns or exits. Thus, I can never set any
device after that, until I restart my app.
- Threads that never get cleaned up inside the JMF/Jitsi code,
resulting in CPU usage spikes over time. This is difficult to show,
but JVisualVM displays a thread count that grows during calls and is
not completely cleaned up afterwards. As far as I can tell, these are
threads related to Jitsi and/or JMF.
I can give an example (when switching the capture device during an active call):
2015-11-11T11:01:01.016 EST [SEVERE] 76-pool-7-thread-1 ::
org.jitsi.util.Logger.error() :: Failed to close send stream
net.sf.fmj.media.rtp.SendSSRCInfo@14b5abc
java.lang.NullPointerException
at net.sf.fmj.media.rtp.RTCPTransmitter.bye(RTCPTransmitter.java:82)
at net.sf.fmj.media.rtp.RTCPReporter.releasessrc(RTCPReporter.java:45)
at net.sf.fmj.media.rtp.RTCPReporter.close(RTCPReporter.java:39)
at net.sf.fmj.media.rtp.RTPSessionMgr.stopParticipating(RTPSessionMgr.java:2743)
at net.sf.fmj.media.rtp.RTPSessionMgr.removeSendStream(RTPSessionMgr.java:1988)
at net.sf.fmj.media.rtp.SendSSRCInfo.close(SendSSRCInfo.java:95)
at
org.jitsi.impl.neomedia.MediaStreamImpl.stopSendStreams(MediaStreamImpl.java:2326)
at
org.jitsi.impl.neomedia.MediaStreamImpl.stopSendStreams(MediaStreamImpl.java:2283)
at
org.jitsi.impl.neomedia.MediaStreamImpl.closeSendStreams(MediaStreamImpl.java:637)
at
org.jitsi.impl.neomedia.MediaStreamImpl.recreateSendStreams(MediaStreamImpl.java:1404)
at
org.jitsi.impl.neomedia.MediaStreamImpl.deviceSessionChanged(MediaStreamImpl.java:770)
at
org.jitsi.impl.neomedia.AudioMediaStreamImpl.deviceSessionChanged(AudioMediaStreamImpl.java:371)
at org.jitsi.impl.neomedia.MediaStreamImpl.setDevice(MediaStreamImpl.java:1707)
at
net.java.sip.communicator.service.protocol.media.CallPeerMediaHandler.callPropertyChange(CallPeerMediaHandler.java:395)
at
net.java.sip.communicator.service.protocol.media.CallPeerMediaHandler.access$000(CallPeerMediaHandler.java:39)
at
net.java.sip.communicator.service.protocol.media.CallPeerMediaHandler$CallPropertyChangeListener.propertyChange(CallPeerMediaHandler.java:1992)
at java.beans.PropertyChangeSupport.fire(Unknown Source)
at java.beans.PropertyChangeSupport.firePropertyChange(Unknown Source)
at java.beans.PropertyChangeSupport.firePropertyChange(Unknown Source)
at
net.java.sip.communicator.service.protocol.AbstractCall.firePropertyChange(AbstractCall.java:187)
at
net.java.sip.communicator.service.protocol.media.MediaAwareCall.firePropertyChange(MediaAwareCall.java:947)
at
net.java.sip.communicator.service.protocol.media.MediaAwareCall.propertyChange(MediaAwareCall.java:856)
at
org.jitsi.util.event.PropertyChangeNotifier.firePropertyChange(PropertyChangeNotifier.java:127)
at
net.java.sip.communicator.service.protocol.media.MediaAwareCallConference.propertyChange(MediaAwareCallConference.java:389)
at
net.java.sip.communicator.service.protocol.media.MediaAwareCallConference.access$000(MediaAwareCallConference.java:27)
at
net.java.sip.communicator.service.protocol.media.MediaAwareCallConference$1.propertyChange(MediaAwareCallConference.java:72)
at
net.java.sip.communicator.service.protocol.media.MediaAwareCallConference$WeakPropertyChangeListener.propertyChange(MediaAwareCallConference.java:574)
at
org.jitsi.util.event.PropertyChangeNotifier.firePropertyChange(PropertyChangeNotifier.java:127)
at
org.jitsi.impl.neomedia.MediaServiceImpl.deviceConfigurationPropertyChange(MediaServiceImpl.java:1468)
at org.jitsi.impl.neomedia.MediaServiceImpl.access$000(MediaServiceImpl.java:49)
at
org.jitsi.impl.neomedia.MediaServiceImpl$1.propertyChange(MediaServiceImpl.java:154)
at
org.jitsi.util.event.PropertyChangeNotifier.firePropertyChange(PropertyChangeNotifier.java:127)
at
org.jitsi.impl.neomedia.device.DeviceConfiguration.propertyChange(DeviceConfiguration.java:947)
at
org.jitsi.util.event.PropertyChangeNotifier.firePropertyChange(PropertyChangeNotifier.java:127)
at
org.jitsi.impl.neomedia.device.AudioSystem.propertyChange(AudioSystem.java:651)
at org.jitsi.impl.neomedia.device.Devices.setDevice(Devices.java:390)
at org.jitsi.impl.neomedia.device.AudioSystem.setDevice(AudioSystem.java:709)
       <-- my code entry which calls AudioSystem.setDevice() -->
My app is a Java-based application, using libjitsi 2.4.4997. It was
using libjitsi 2.2, with similar but more severe issues. Upgrading
helped a bit, but I am still battling these core issues. Any help or
insight into what I could be doing wrong would be appreciated.
I would especially like to know if something else, other than
AudioSystem.setDevice(), should be used to perform the device
switches. As far as I can tell from the Jitsi client source code
(granted the GitHub code is 2.9...), I am switching the devices in a
similar way - just programmatically instead of being driven off of
user clicks on the combo boxes. From the stack trace above, I can see
Jitsi detecting the device change and propagating the property change
to all interested parties. Jitsi seems to manage the stopping and
re-starting of the audio streams as well, such that I should not have
to manually do that myself.
If there is any other information that could aid in getting to the
bottom of this, I can try to provide it.
Thanks in advance,
Neil S.

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev