[jitsi-dev] Jicofo erroring out for some unknown reason


#1

Hi,

My jicofo is erroring out at times as the task is rejected by ThreadPoolExecutor. May I know what could be the reason for this and how can I resolve it? Is it possibly getting rejected because :
1) the Executor has been shut down, or
2) the Executor uses finite bounds for both maximum threads and work queue capacity, and is saturated.

How do I fix this in the code? Where is the thread executor being instantiated in the code?

An error log snippet is as follows:

[2015-12-21 18:57:06.678] [SEVERE][30] [org.jitsi.jicofo.xmpp.FocusComponent.processPacket()] (serving component 'Jitsi Meet Focus') Unable to process packet! Is the thread pool queue exhausted? Packet dropped in component 'Jitsi Meet Focus'.
java.util.concurrent.RejectedExecutionException: Task org.xmpp.component.AbstractComponent$PacketProcessor@79abaf84 rejected from java.util.concurrent.ThreadPoolExecutor@6c8ecd7a[Running, pool size = 17, active threads = 17, queued tasks = 1000, completed tasks = 462] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) at org.xmpp.component.AbstractComponent.processPacket(AbstractComponent.java:213) at org.jivesoftware.whack.ExternalComponent$1.run(ExternalComponent.java:338) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)


#2

Hi,

···

On 22 Dec 2015, at 12:29, swapnil bagadia <swapnilbagadia@live.com> wrote:

How do I fix this in the code? Where is the thread executor being instantiated in the code?

An error log snippet is as follows:

[2015-12-21 18:57:06.678] [SEVERE][30] [org.jitsi.jicofo.xmpp.FocusComponent.processPacket()] (serving component 'Jitsi Meet Focus') Unable to process packet! Is the thread pool queue exhausted? Packet dropped in component 'Jitsi Meet Focus'.

java.util.concurrent.RejectedExecutionException: Task org.xmpp.component.AbstractComponent$PacketProcessor@79abaf84 rejected from java.util.concurrent.ThreadPoolExecutor@6c8ecd7a[Running, pool size = 17, active threads = 17, queued tasks = 1000, completed tasks = 462]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
        at org.xmpp.component.AbstractComponent.processPacket(AbstractComponent.java:213)
        at org.jivesoftware.whack.ExternalComponent$1.run(ExternalComponent.java:338)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

This particular executor is handled internally by Whack. Try taking a thread dump and check if there are any threads blocking the queue.

Regards,
Pawel


#3

Hi,

As per your suggestion I checked the thread dump and it is giving me a deadlock for one thread.

2015-12-22 15:31:19 Thread dump snippet Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode): "pool-3-thread-4" #32 prio=5 os_prio=0 tid=0x00007f854c008800 nid=0x65a6 waiting for monitor entry [0x00007f8550190000] java.lang.Thread.State: BLOCKED (on object monitor) at org.jitsi.jicofo.JitsiMeetConference.stop(JitsiMeetConference.java:1238) - waiting to lock <0x0000000700e76db8> (a org.jitsi.jicofo.JitsiMeetConference) at org.jitsi.jicofo.FocusManager.createConference(FocusManager.java:307) at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:245) - locked <0x0000000700d18b60> (a org.jitsi.jicofo.FocusManager) at org.jitsi.jicofo.xmpp.FocusComponent.handleConferenceIq(FocusComponent.java:414) at org.jitsi.jicofo.xmpp.FocusComponent.handleIQSet(FocusComponent.java:257) at org.xmpp.component.AbstractComponent.processIQRequest(AbstractComponent.java:515) at org.xmpp.component.AbstractComponent.processIQ(AbstractComponent.java:289) at org.xmpp.component.AbstractComponent.processQueuedPacket(AbstractComponent.java:239) at org.xmpp.component.AbstractComponent.access$100(AbstractComponent.java:81) at org.xmpp.component.AbstractComponent$PacketProcessor.run(AbstractComponent.java:1051) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
"pool-3-thread-3" #30 prio=5 os_prio=0 tid=0x00007f8510045000 nid=0x65a4 waiting for monitor entry [0x00007f8550592000] java.lang.Thread.State: BLOCKED (on object monitor) at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:237) - waiting to lock <0x0000000700d18b60> (a org.jitsi.jicofo.FocusManager) at org.jitsi.jicofo.xmpp.FocusComponent.handleConferenceIq(FocusComponent.java:414) at org.jitsi.jicofo.xmpp.FocusComponent.handleIQSet(FocusComponent.java:257) at org.xmpp.component.AbstractComponent.processIQRequest(AbstractComponent.java:515) at org.xmpp.component.AbstractComponent.processIQ(AbstractComponent.java:289) at org.xmpp.component.AbstractComponent.processQueuedPacket(AbstractComponent.java:239) at org.xmpp.component.AbstractComponent.access$100(AbstractComponent.java:81) at org.xmpp.component.AbstractComponent$PacketProcessor.run(AbstractComponent.java:1051) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
"Smack Listener Processor (0)" #25 daemon prio=5 os_prio=0 tid=0x00007f8524079800 nid=0x659f waiting for monitor entry [0x00007f8550996000] java.lang.Thread.State: BLOCKED (on object monitor) at org.jitsi.jicofo.FocusManager.conferenceEnded(FocusManager.java:337) - waiting to lock <0x0000000700d18b60> (a org.jitsi.jicofo.FocusManager) at org.jitsi.jicofo.JitsiMeetConference.stop(JitsiMeetConference.java:1258) - locked <0x0000000700e76db8> (a org.jitsi.jicofo.JitsiMeetConference) at org.jitsi.jicofo.ChatRoomRoleAndPresence.verifyFocusRole(ChatRoomRoleAndPresence.java:302) at org.jitsi.jicofo.ChatRoomRoleAndPresence.localUserRoleChanged(ChatRoomRoleAndPresence.java:320) at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.fireLocalUserRoleEvent(ChatRoomImpl.java:400) at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.setLocalUserRole(ChatRoomImpl.java:411) at org.jitsi.impl.protocol.xmpp.ChatRoomImpl$ParticipantListener.processOwnPresence(ChatRoomImpl.java:1156) at org.jitsi.impl.protocol.xmpp.ChatRoomImpl$ParticipantListener.processPacket(ChatRoomImpl.java:1113) at org.jivesoftware.smack.Connection$ListenerWrapper.notifyListener(Connection.java:819) at org.jivesoftware.smack.PacketReader$ListenerNotification.run(PacketReader.java:457) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Found one Java-level deadlock: ============================= "pool-3-thread-17": waiting to lock monitor 0x00007f8524032f18 (object 0x0000000700d18b60, a org.jitsi.jicofo.FocusManager), which is held by "pool-3-thread-4" "pool-3-thread-4": waiting to lock monitor 0x00007f856859a6c8 (object 0x0000000700e76db8, a org.jitsi.jicofo.JitsiMeetConference), which is held by "Smack Listener Processor (0)" "Smack Listener Processor (0)": waiting to lock monitor 0x00007f8524032f18 (object 0x0000000700d18b60, a org.jitsi.jicofo.FocusManager), which is held by "pool-3-thread-4" Java stack information for the threads listed above: =================================================== "pool-3-thread-17": at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:237) - waiting to lock <0x0000000700d18b60> (a org.jitsi.jicofo.FocusManager) at org.jitsi.jicofo.xmpp.FocusComponent.handleConferenceIq(FocusComponent.java:414) at org.jitsi.jicofo.xmpp.FocusComponent.handleIQSet(FocusComponent.java:257) at org.xmpp.component.AbstractComponent.processIQRequest(AbstractComponent.java:515) at org.xmpp.component.AbstractComponent.processIQ(AbstractComponent.java:289) at org.xmpp.component.AbstractComponent.processQueuedPacket(AbstractComponent.java:239) at org.xmpp.component.AbstractComponent.access$100(AbstractComponent.java:81) at org.xmpp.component.AbstractComponent$PacketProcessor.run(AbstractComponent.java:1051) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "pool-3-thread-4": at org.jitsi.jicofo.JitsiMeetConference.stop(JitsiMeetConference.java:1238) - waiting to lock <0x0000000700e76db8> (a org.jitsi.jicofo.JitsiMeetConference) at org.jitsi.jicofo.FocusManager.createConference(FocusManager.java:307) at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:245) - locked <0x0000000700d18b60> (a org.jitsi.jicofo.FocusManager) at org.jitsi.jicofo.xmpp.FocusComponent.handleConferenceIq(FocusComponent.java:414) at org.jitsi.jicofo.xmpp.FocusComponent.handleIQSet(FocusComponent.java:257) at org.xmpp.component.AbstractComponent.processIQRequest(AbstractComponent.java:515) at org.xmpp.component.AbstractComponent.processIQ(AbstractComponent.java:289) at org.xmpp.component.AbstractComponent.processQueuedPacket(AbstractComponent.java:239) at org.xmpp.component.AbstractComponent.access$100(AbstractComponent.java:81) at org.xmpp.component.AbstractComponent$PacketProcessor.run(AbstractComponent.java:1051) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "Smack Listener Processor (0)": at org.jitsi.jicofo.FocusManager.conferenceEnded(FocusManager.java:337) - waiting to lock <0x0000000700d18b60> (a org.jitsi.jicofo.FocusManager) at org.jitsi.jicofo.JitsiMeetConference.stop(JitsiMeetConference.java:1258) - locked <0x0000000700e76db8> (a org.jitsi.jicofo.JitsiMeetConference) at org.jitsi.jicofo.ChatRoomRoleAndPresence.verifyFocusRole(ChatRoomRoleAndPresence.java:302) at org.jitsi.jicofo.ChatRoomRoleAndPresence.localUserRoleChanged(ChatRoomRoleAndPresence.java:320) at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.fireLocalUserRoleEvent(ChatRoomImpl.java:400) at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.setLocalUserRole(ChatRoomImpl.java:411) at org.jitsi.impl.protocol.xmpp.ChatRoomImpl$ParticipantListener.processOwnPresence(ChatRoomImpl.java:1156) at org.jitsi.impl.protocol.xmpp.ChatRoomImpl$ParticipantListener.processPacket(ChatRoomImpl.java:1113) at org.jivesoftware.smack.Connection$ListenerWrapper.notifyListener(Connection.java:819) at org.jivesoftware.smack.PacketReader$ListenerNotification.run(PacketReader.java:457) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Found 1 deadlock.

Is this the issue which is causing the Jicofo to crash? If yes, can you please tell what is causing this deadlock ?

Regards,
Swapnil.

···

From: pawel.domas@sip-communicator.org
Date: Tue, 22 Dec 2015 12:43:35 +0100
To: dev@jitsi.org
Subject: Re: [jitsi-dev] Jicofo erroring out for some unknown reason

Hi,

> On 22 Dec 2015, at 12:29, swapnil bagadia <swapnilbagadia@live.com> wrote:
>
> How do I fix this in the code? Where is the thread executor being instantiated in the code?
>
>
> An error log snippet is as follows:
>
>
>
> [2015-12-21 18:57:06.678] [SEVERE][30] [org.jitsi.jicofo.xmpp.FocusComponent.processPacket()] (serving component 'Jitsi Meet Focus') Unable to process packet! Is the thread pool queue exhausted? Packet dropped in component 'Jitsi Meet Focus'.
>
> java.util.concurrent.RejectedExecutionException: Task org.xmpp.component.AbstractComponent$PacketProcessor@79abaf84 rejected from java.util.concurrent.ThreadPoolExecutor@6c8ecd7a[Running, pool size = 17, active threads = 17, queued tasks = 1000, completed tasks = 462]
> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
> at org.xmpp.component.AbstractComponent.processPacket(AbstractComponent.java:213)
> at org.jivesoftware.whack.ExternalComponent$1.run(ExternalComponent.java:338)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>

This particular executor is handled internally by Whack. Try taking a thread dump and check if there are any threads blocking the queue.

Regards,
Pawel

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#4

Hi,

···

On 22 Dec 2015, at 20:44, swapnil bagadia <swapnilbagadia@live.com> wrote:

Hi,

As per your suggestion I checked the thread dump and it is giving me a deadlock for one thread.

Thanks ! Which Jicofo version is this ?

Regards,
Pawel


#5

Hi,

Jitsi/Jicofo release #117.

Regards,

Swapnil

···

Sent from Outlook Mobile

From: Paweł Domas

Sent: Wednesday 23 December 22:50

Subject: Re: [jitsi-dev] Jicofo erroring out for some unknown reason

To: Jitsi Developers

Hi,

On 22 Dec 2015, at 20:44, swapnil bagadia < swapnilbagadia@live.com> wrote:

Hi,

As per your suggestion I checked the thread dump and it is giving me a deadlock for one thread.

Thanks ! Which Jicofo version is this ?

Regards,

Pawel


#6

Hi,

···

On 29 Dec 2015, at 14:41, swapnilbagadia@live.com wrote:

Hi,

Jitsi/Jicofo release #117.

I remember fixing similar deadlock some time ago. Can you try the latest version ?

Regards,
Pawel