[jitsi-dev] JVB deadlock


#1

Hi,

When trying to run a call, I was not getting a session initiate from focus even though both JVB & Jicofo were running ( confirmed after checking both for the process and netstat for the ports).

Since both were running, I took to taking a thread dump for both and found that JVB was having a deadlock.

Found one Java-level deadlock:============================="pool-2-thread-17": waiting to lock monitor 0x00007f9ebfc9ad98 (object 0x0000000701f3d038, a org.jitsi.videobridge.IceUdpTransportManager), which is held by "pool-2-thread-2""pool-2-thread-2": waiting to lock monitor 0x00007fa1ccaa75d8 (object 0x0000000701f3d090, a java.util.LinkedList), which is held by "pool-2-thread-17"
Java stack information for the threads listed above:==================================================="pool-2-thread-17": at org.jitsi.videobridge.IceUdpTransportManager.close(IceUdpTransportManager.java:628) - waiting to lock <0x0000000701f3d038> (a org.jitsi.videobridge.IceUdpTransportManager) at org.jitsi.videobridge.Conference.closeTransportManager(Conference.java:287) - locked <0x0000000701f43170> (a java.util.HashMap) at org.jitsi.videobridge.TransportManager.close(TransportManager.java:144) - locked <0x0000000701f3d090> (a java.util.LinkedList) at org.jitsi.videobridge.IceUdpTransportManager.close(IceUdpTransportManager.java:691) at org.jitsi.videobridge.Channel.expire(Channel.java:360) - locked <0x0000000701f40ac0> (a java.lang.Object) at org.jitsi.videobridge.SctpConnection.expire(SctpConnection.java:332) at org.jitsi.videobridge.Channel.setExpire(Channel.java:663) at org.jitsi.videobridge.Videobridge.handleColibriConferenceIQ(Videobridge.java:988) at org.jitsi.videobridge.Videobridge.handleColibriConferenceIQ(Videobridge.java:569) at org.jitsi.videobridge.xmpp.ComponentImpl.handleColibriConferenceIQ(ComponentImpl.java:271) at org.jitsi.videobridge.xmpp.ComponentImpl.handleIQRequest(ComponentImpl.java:501) at org.jitsi.videobridge.xmpp.ComponentImpl.handleIQ(ComponentImpl.java:432) at org.jitsi.videobridge.xmpp.ComponentImpl.handleIQ(ComponentImpl.java:383) at org.jitsi.videobridge.xmpp.ComponentImpl.handleIQSet(ComponentImpl.java:607) at org.xmpp.component.AbstractComponent.processIQRequest(AbstractComponent.java:515) at org.xmpp.component.AbstractComponent.processIQ(AbstractComponent.java:289) at org.xmpp.component.AbstractComponent.processQueuedPacket(AbstractComponent.java:239) at org.xmpp.component.AbstractComponent.access$100(AbstractComponent.java:81) at org.xmpp.component.AbstractComponent$PacketProcessor.run(AbstractComponent.java:1051) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)"pool-2-thread-2": at org.jitsi.videobridge.TransportManager.getChannels(TransportManager.java:247) - waiting to lock <0x0000000701f3d090> (a java.util.LinkedList) at org.jitsi.videobridge.IceUdpTransportManager.iceAgentStateChange(IceUdpTransportManager.java:1683) at org.jitsi.videobridge.IceUdpTransportManager.access$000(IceUdpTransportManager.java:52) at org.jitsi.videobridge.IceUdpTransportManager$1.propertyChange(IceUdpTransportManager.java:240) at org.ice4j.ice.Agent.fireStateChange(Agent.java:717) at org.ice4j.ice.Agent.setState(Agent.java:742) at org.ice4j.ice.Agent.startConnectivityEstablishment(Agent.java:544) - locked <0x0000000701f3f4e8> (a java.lang.Object) at org.jitsi.videobridge.IceUdpTransportManager.doStartConnectivityEstablishment(IceUdpTransportManager.java:1170) - locked <0x0000000701f3d038> (a org.jitsi.videobridge.IceUdpTransportManager) at org.jitsi.videobridge.IceUdpTransportManager.startConnectivityEstablishment(IceUdpTransportManager.java:1892) at org.jitsi.videobridge.Videobridge.handleColibriConferenceIQ(Videobridge.java:1037) at org.jitsi.videobridge.Videobridge.handleColibriConferenceIQ(Videobridge.java:569) at org.jitsi.videobridge.xmpp.ComponentImpl.handleColibriConferenceIQ(ComponentImpl.java:271) at org.jitsi.videobridge.xmpp.ComponentImpl.handleIQRequest(ComponentImpl.java:501) at org.jitsi.videobridge.xmpp.ComponentImpl.handleIQ(ComponentImpl.java:432) at org.jitsi.videobridge.xmpp.ComponentImpl.handleIQ(ComponentImpl.java:383) at org.jitsi.videobridge.xmpp.ComponentImpl.handleIQSet(ComponentImpl.java:607) at org.xmpp.component.AbstractComponent.processIQRequest(AbstractComponent.java:515) at org.xmpp.component.AbstractComponent.processIQ(AbstractComponent.java:289) at org.xmpp.component.AbstractComponent.processQueuedPacket(AbstractComponent.java:239) at org.xmpp.component.AbstractComponent.access$100(AbstractComponent.java:81) at org.xmpp.component.AbstractComponent$PacketProcessor.run(AbstractComponent.java:1051) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Found 1 deadlock.

Thanks,

Swapnil


#2

Hi,

Hi,

When trying to run a call, I was not getting a session initiate from
focus even though both JVB & Jicofo were running ( confirmed after
checking both for the process and netstat for the ports).

Since both were running, I took to taking a thread dump for both and
found that JVB was having a deadlock.

Found one Java-level deadlock:

"pool-2-thread-17":
   waiting to lock monitor 0x00007f9ebfc9ad98 (object
0x0000000701f3d038, a org.jitsi.videobridge.IceUdpTransportManager),
   which is held by "pool-2-thread-2"
"pool-2-thread-2":
   waiting to lock monitor 0x00007fa1ccaa75d8 (object
0x0000000701f3d090, a java.util.LinkedList),
   which is held by "pool-2-thread-17"

Thank you for the report!

Examining the code, it looks like one of locks involved (on TransportManager#channels) is unnecessary. I remove it here:
https://github.com/jitsi/jitsi-videobridge/pull/119

Lyubo, does that seem right?

Regards,
Boris

···

On 30/12/15 14:46, swapnil bagadia wrote: