After last Jicofo update room no more accessible

Hi, after the last update of Jitsi I found that Jicofo(jicofo/stable,now 1.0-813-1), once a room is destroyed via my module on prosody, it seems to apply a tombstone to it. I tried from code in the module to invoke event.room:destroy(); twice and to set muc_tombstones = false but in both cases the room is no longer accessible untile Jicofo is restarted. Do you have any advice? No errors are tracked in Jicofo or Prosody logs

1 Like

You don’t see this log: jicofo/JitsiMeetConferenceImpl.java at 2b22d862047643fe8f53a6c5af17c06642a12719 · bgrozev/jicofo · GitHub
?
There was a smack update in the latest stable release which may have brought some bugs …
@shawn I know in the reservation module the room is also destroyed in some cases, do you see the same?

@LongJohnLafitte when you destroy the room, can you get a stack trace from jicofo sudo jstack ${PID} and attach it here? Thanks.

Haven’t had a chance yet to test against latest release I’m afraid. I’ll try to get this done Monday and report back.

1 Like

Hi @damencho, I have produced three types of stacktrace and thread dump with this script:

#!/bin/bash

JVM_PID=2703749
COUNT=1
SERVICE_NAME="jicofo"

mkdir ${SERVICE_NAME}
cd ${SERVICE_NAME}

while [ ${COUNT} -lt 10 ]
do
    TTIME=$(date '+%Y-%m-%d-%H-%M-%S')
    ps -Le -o pid,user,s,lwp,pcpu | grep ${JVM_PID} > thread-list-${JVM_PID}-${TTIME}.txt
    jstack ${JVM_PID} > stack-${JVM_PID}-${TTIME}.txt
    ((COUNT++))
    sleep 0.2
done

I’ll explain how they were produced:

  1. jicofo_with_double_destroy_from_prosody.tar.gz: in prosody module the event.room:destroy(); is invoked twice and as I told you it is no longer possible to access the room.
  2. jicofo_with_single_destroy_from_prosody.tar.gz: in this case I invoke only once the event.room:destroy(); and I can’t enter the room as I would expect because of the tombstone.
1 Like
  1. jicofo_with_kick_and_double_destroy_from_prosody.tar.gz: in this case I first set new affiliation for all users (with “member” affiliation) with set_affiliation(true, participant.jid, "outcast"); and then I invoke the event.room:destroy(); twice. So in this case first the room become empty from each participant, then destroyed. Then I can log in again without having to restart Jicofo
2 Likes

This upload seems deleted, at least this is what I see …
But there is no need to produce so many dumps … I want to check one stack trace after the destroy has passed and you have the problem with the room.

Here jstack’s output after the destroy. Thanks!

2021-10-11 16:15:55
Full thread dump OpenJDK 64-Bit Server VM (16.0.1+9-Ubuntu-120.04 mixed mode, sharing):

Threads class SMR info:
_java_thread_list=0x00007f664c0049b0, length=44, elements={
0x00007f66b8011850, 0x00007f66b81412a0, 0x00007f66b8142820, 0x00007f66b8149900,
0x00007f66b814ae50, 0x00007f66b814c370, 0x00007f66b814de50, 0x00007f66b814f4b0,
0x00007f66b8150a50, 0x00007f66b81602a0, 0x00007f66b8165ec0, 0x00007f66b8a33450,
0x00007f66b8a72e60, 0x00007f66b8a73de0, 0x00007f66b8b3fb50, 0x00007f662c00ed20,
0x00007f662c012830, 0x00007f662c0131d0, 0x00007f66b8c55a60, 0x00007f66b8c56690,
0x00007f66b8c577c0, 0x00007f66b8c588f0, 0x00007f66b8c59ae0, 0x00007f66b8c5acd0,
0x00007f66b8c5bed0, 0x00007f66b8c5d0d0, 0x00007f6624007200, 0x00007f6628069460,
0x00007f663803a120, 0x00007f662c021100, 0x00007f663c13b270, 0x00007f662c059790,
0x00007f6640001590, 0x00007f663c109d30, 0x00007f66480d4360, 0x00007f66480d5b50,
0x00007f663c10adb0, 0x00007f662c057fc0, 0x00007f6644008a20, 0x00007f6640002cc0,
0x00007f665000cd80, 0x00007f6678000e50, 0x00007f663c10c010, 0x00007f664c003f20
}

"main" #1 prio=5 os_prio=0 cpu=2980.52ms elapsed=129.46s tid=0x00007f66b8011850 nid=0x297c9c waiting on condition  [0x00007f66bf2c5000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x00000007405839b8> (a java.util.concurrent.CountDownLatch$Sync)
	at java.util.concurrent.locks.LockSupport.park(java.base@16.0.1/LockSupport.java:211)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@16.0.1/AbstractQueuedSynchronizer.java:714)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@16.0.1/AbstractQueuedSynchronizer.java:1046)
	at java.util.concurrent.CountDownLatch.await(java.base@16.0.1/CountDownLatch.java:232)
	at org.jitsi.shutdown.ShutdownServiceImpl.waitForShutdown(ShutdownServiceImpl.kt:35)
	at org.jitsi.jicofo.Main.main(Main.java:73)

"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=1.29ms elapsed=129.45s tid=0x00007f66b81412a0 nid=0x297ca4 waiting on condition  [0x00007f66bce35000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.ref.Reference.waitForReferencePendingList(java.base@16.0.1/Native Method)
	at java.lang.ref.Reference.processPendingReferences(java.base@16.0.1/Reference.java:243)
	at java.lang.ref.Reference$ReferenceHandler.run(java.base@16.0.1/Reference.java:215)

"Finalizer" #3 daemon prio=8 os_prio=0 cpu=0.33ms elapsed=129.44s tid=0x00007f66b8142820 nid=0x297ca5 in Object.wait()  [0x00007f66bcd34000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(java.base@16.0.1/Native Method)
	- waiting on <0x00000007401ab478> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@16.0.1/ReferenceQueue.java:155)
	- locked <0x00000007401ab478> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@16.0.1/ReferenceQueue.java:176)
	at java.lang.ref.Finalizer$FinalizerThread.run(java.base@16.0.1/Finalizer.java:171)

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.47ms elapsed=129.44s tid=0x00007f66b8149900 nid=0x297ca6 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Service Thread" #5 daemon prio=9 os_prio=0 cpu=1.29ms elapsed=129.44s tid=0x00007f66b814ae50 nid=0x297ca7 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Monitor Deflation Thread" #6 daemon prio=9 os_prio=0 cpu=3.30ms elapsed=129.44s tid=0x00007f66b814c370 nid=0x297ca8 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #7 daemon prio=9 os_prio=0 cpu=3245.95ms elapsed=129.44s tid=0x00007f66b814de50 nid=0x297ca9 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C1 CompilerThread0" #9 daemon prio=9 os_prio=0 cpu=1443.09ms elapsed=129.44s tid=0x00007f66b814f4b0 nid=0x297caa waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Sweeper thread" #10 daemon prio=9 os_prio=0 cpu=0.09ms elapsed=129.44s tid=0x00007f66b8150a50 nid=0x297cab runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Notification Thread" #11 daemon prio=9 os_prio=0 cpu=0.16ms elapsed=129.43s tid=0x00007f66b81602a0 nid=0x297cb2 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #12 daemon prio=8 os_prio=0 cpu=1.78ms elapsed=129.43s tid=0x00007f66b8165ec0 nid=0x297cb4 in Object.wait()  [0x00007f66738f6000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(java.base@16.0.1/Native Method)
	- waiting on <0x00000007401a1bf8> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@16.0.1/ReferenceQueue.java:155)
	- locked <0x00000007401a1bf8> (a java.lang.ref.ReferenceQueue$Lock)
	at jdk.internal.ref.CleanerImpl.run(java.base@16.0.1/CleanerImpl.java:140)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)
	at jdk.internal.misc.InnocuousThread.run(java.base@16.0.1/InnocuousThread.java:134)

"FocusExpireThread" #15 prio=5 os_prio=0 cpu=2.35ms elapsed=127.75s tid=0x00007f66b8a33450 nid=0x297cf1 in Object.wait()  [0x00007f66733ee000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(java.base@16.0.1/Native Method)
	- waiting on <0x0000000740400638> (a java.lang.Object)
	at org.jitsi.jicofo.FocusManager$FocusExpireThread.expireLoop(FocusManager.java:601)
	- locked <0x0000000740400638> (a java.lang.Object)
	at org.jitsi.jicofo.FocusManager$FocusExpireThread$$Lambda$61/0x0000000800d9d308.run(Unknown Source)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Smack DefaultReactor Thread #0" #16 daemon prio=5 os_prio=0 cpu=3.65ms elapsed=127.67s tid=0x00007f66b8a72e60 nid=0x297cf2 runnable  [0x00007f66732ed000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@16.0.1/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@16.0.1/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@16.0.1/SelectorImpl.java:129)
	- locked <0x0000000740400c88> (a sun.nio.ch.Util$2)
	- locked <0x00000007404008e0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@16.0.1/SelectorImpl.java:141)
	at org.jivesoftware.smack.SmackReactor$Reactor.handleScheduledActionsOrPerformSelect(SmackReactor.java:257)
	- locked <0x00000007404008e0> (a sun.nio.ch.EPollSelectorImpl)
	at org.jivesoftware.smack.SmackReactor$Reactor.reactorLoop(SmackReactor.java:188)
	at org.jivesoftware.smack.SmackReactor$Reactor.run(SmackReactor.java:173)

"Smack DefaultReactor Thread #1" #17 daemon prio=5 os_prio=0 cpu=4.95ms elapsed=127.66s tid=0x00007f66b8a73de0 nid=0x297cf3 waiting for monitor entry  [0x00007f66731ec000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.jivesoftware.smack.SmackReactor$Reactor.handleScheduledActionsOrPerformSelect(SmackReactor.java:215)
	- waiting to lock <0x00000007404008e0> (a sun.nio.ch.EPollSelectorImpl)
	at org.jivesoftware.smack.SmackReactor$Reactor.reactorLoop(SmackReactor.java:188)
	at org.jivesoftware.smack.SmackReactor$Reactor.run(SmackReactor.java:173)

"Jicofo Global Scheduled Poolpool-3-thread-1" #18 daemon prio=5 os_prio=0 cpu=66.80ms elapsed=127.25s tid=0x00007f66b8b3fb50 nid=0x297cfd waiting on condition  [0x00007f66727fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x0000000740400f68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@16.0.1/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@16.0.1/AbstractQueuedSynchronizer.java:505)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@16.0.1/ForkJoinPool.java:3137)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@16.0.1/AbstractQueuedSynchronizer.java:1614)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@16.0.1/ThreadPoolExecutor.java:1056)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.1/ThreadPoolExecutor.java:1116)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.1/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Smack Cached Executor" #20 daemon prio=5 os_prio=0 cpu=18.88ms elapsed=127.22s tid=0x00007f662c00ed20 nid=0x297cff waiting on condition  [0x00007f66725fc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x0000000740401408> (a java.util.concurrent.SynchronousQueue$TransferStack)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@16.0.1/LockSupport.java:252)
	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@16.0.1/SynchronousQueue.java:462)
	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@16.0.1/SynchronousQueue.java:361)
	at java.util.concurrent.SynchronousQueue.poll(java.base@16.0.1/SynchronousQueue.java:937)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@16.0.1/ThreadPoolExecutor.java:1055)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.1/ThreadPoolExecutor.java:1116)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.1/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Smack Writer (0)" #21 daemon prio=5 os_prio=0 cpu=81.41ms elapsed=127.20s tid=0x00007f662c012830 nid=0x297d00 waiting on condition  [0x00007f66726fd000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x000000074045e220> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@16.0.1/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@16.0.1/AbstractQueuedSynchronizer.java:505)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@16.0.1/ForkJoinPool.java:3137)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@16.0.1/AbstractQueuedSynchronizer.java:1614)
	at org.jivesoftware.smack.util.ArrayBlockingQueueWithShutdown.take(ArrayBlockingQueueWithShutdown.java:415)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketWriter.nextStreamElement(XMPPTCPConnection.java:1292)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketWriter.writePackets(XMPPTCPConnection.java:1307)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketWriter.access$4200(XMPPTCPConnection.java:1169)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketWriter$1.run(XMPPTCPConnection.java:1219)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Smack Reader (0)" #22 daemon prio=5 os_prio=0 cpu=586.19ms elapsed=127.20s tid=0x00007f662c0131d0 nid=0x297d01 runnable  [0x00007f6671bfd000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.SocketDispatcher.read0(java.base@16.0.1/Native Method)
	at sun.nio.ch.SocketDispatcher.read(java.base@16.0.1/SocketDispatcher.java:47)
	at sun.nio.ch.NioSocketImpl.tryRead(java.base@16.0.1/NioSocketImpl.java:261)
	at sun.nio.ch.NioSocketImpl.implRead(java.base@16.0.1/NioSocketImpl.java:312)
	at sun.nio.ch.NioSocketImpl.read(java.base@16.0.1/NioSocketImpl.java:350)
	at sun.nio.ch.NioSocketImpl$1.read(java.base@16.0.1/NioSocketImpl.java:803)
	at java.net.Socket$SocketInputStream.read(java.base@16.0.1/Socket.java:976)
	at sun.security.ssl.SSLSocketInputRecord.read(java.base@16.0.1/SSLSocketInputRecord.java:478)
	at sun.security.ssl.SSLSocketInputRecord.readHeader(java.base@16.0.1/SSLSocketInputRecord.java:472)
	at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(java.base@16.0.1/SSLSocketInputRecord.java:70)
	at sun.security.ssl.SSLSocketImpl.readApplicationRecord(java.base@16.0.1/SSLSocketImpl.java:1444)
	at sun.security.ssl.SSLSocketImpl$AppInputStream.read(java.base@16.0.1/SSLSocketImpl.java:1048)
	at sun.nio.cs.StreamDecoder.readBytes(java.base@16.0.1/StreamDecoder.java:297)
	at sun.nio.cs.StreamDecoder.implRead(java.base@16.0.1/StreamDecoder.java:339)
	at sun.nio.cs.StreamDecoder.read(java.base@16.0.1/StreamDecoder.java:188)
	- locked <0x000000074047b200> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(java.base@16.0.1/InputStreamReader.java:178)
	at java.io.BufferedReader.read1(java.base@16.0.1/BufferedReader.java:210)
	at java.io.BufferedReader.read(java.base@16.0.1/BufferedReader.java:287)
	- locked <0x000000074047b200> (a java.io.InputStreamReader)
	at org.jivesoftware.smack.util.ObservableReader.read(ObservableReader.java:42)
	at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(java.xml@16.0.1/XMLEntityScanner.java:1904)
	at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.peekChar(java.xml@16.0.1/XMLEntityScanner.java:508)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(java.xml@16.0.1/XMLDocumentFragmentScannerImpl.java:2664)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(java.xml@16.0.1/XMLDocumentScannerImpl.java:605)
	at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(java.xml@16.0.1/XMLNSDocumentScannerImpl.java:112)
	at com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next(java.xml@16.0.1/XMLStreamReaderImpl.java:601)
	at org.jivesoftware.smack.xml.stax.StaxXmlPullParser.next(StaxXmlPullParser.java:191)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1140)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$700(XMPPTCPConnection.java:913)
	at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:936)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"qtp1472759652-23" #23 prio=5 os_prio=0 cpu=1.59ms elapsed=126.60s tid=0x00007f66b8c55a60 nid=0x297d0d runnable  [0x00007f6671afd000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@16.0.1/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@16.0.1/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@16.0.1/SelectorImpl.java:129)
	- locked <0x0000000741600168> (a sun.nio.ch.Util$2)
	- locked <0x0000000741600110> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@16.0.1/SelectorImpl.java:146)
	at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:183)
	at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:190)
	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)
	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
	at org.eclipse.jetty.io.ManagedSelector$$Lambda$336/0x0000000800ffd868.run(Unknown Source)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"qtp1472759652-24" #24 prio=5 os_prio=0 cpu=0.35ms elapsed=126.59s tid=0x00007f66b8c56690 nid=0x297d0e runnable  [0x00007f66719fc000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@16.0.1/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@16.0.1/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@16.0.1/SelectorImpl.java:129)
	- locked <0x0000000741600870> (a sun.nio.ch.Util$2)
	- locked <0x0000000741600760> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@16.0.1/SelectorImpl.java:146)
	at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:183)
	at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:190)
	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)
	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
	at org.eclipse.jetty.io.ManagedSelector$$Lambda$336/0x0000000800ffd868.run(Unknown Source)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"qtp1472759652-25-acceptor-0@e031ea5-ServerConnector@5e268ce6{HTTP/1.1, (http/1.1)}{0.0.0.0:8888}" #25 prio=3 os_prio=0 cpu=1.82ms elapsed=126.59s tid=0x00007f66b8c577c0 nid=0x297d0f runnable  [0x00007f66718fb000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.Net.accept(java.base@16.0.1/Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.implAccept(java.base@16.0.1/ServerSocketChannelImpl.java:424)
	at sun.nio.ch.ServerSocketChannelImpl.accept(java.base@16.0.1/ServerSocketChannelImpl.java:390)
	at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388)
	at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:702)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"qtp1472759652-26" #26 prio=5 os_prio=0 cpu=0.37ms elapsed=126.58s tid=0x00007f66b8c588f0 nid=0x297d10 waiting on condition  [0x00007f66717fa000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x00000007409844a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@16.0.1/LockSupport.java:252)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@16.0.1/AbstractQueuedSynchronizer.java:1661)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:973)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1023)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"qtp1472759652-27" #27 prio=5 os_prio=0 cpu=0.47ms elapsed=126.58s tid=0x00007f66b8c59ae0 nid=0x297d11 waiting on condition  [0x00007f66716f9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x00000007409844a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@16.0.1/LockSupport.java:252)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@16.0.1/AbstractQueuedSynchronizer.java:1661)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:973)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1023)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"qtp1472759652-28" #28 prio=5 os_prio=0 cpu=0.31ms elapsed=126.57s tid=0x00007f66b8c5acd0 nid=0x297d12 waiting on condition  [0x00007f66715f8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x00000007409844a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@16.0.1/LockSupport.java:252)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@16.0.1/AbstractQueuedSynchronizer.java:1661)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:973)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1023)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"qtp1472759652-29" #29 prio=5 os_prio=0 cpu=0.58ms elapsed=126.56s tid=0x00007f66b8c5bed0 nid=0x297d13 waiting on condition  [0x00007f66714f7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x00000007409844a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@16.0.1/LockSupport.java:252)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@16.0.1/AbstractQueuedSynchronizer.java:1661)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:973)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1023)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"qtp1472759652-30" #30 prio=5 os_prio=0 cpu=0.23ms elapsed=126.56s tid=0x00007f66b8c5d0d0 nid=0x297d14 waiting on condition  [0x00007f66713f6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x00000007409844a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@16.0.1/LockSupport.java:252)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@16.0.1/AbstractQueuedSynchronizer.java:1661)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:973)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1023)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Smack Cached Executor" #31 daemon prio=5 os_prio=0 cpu=57.47ms elapsed=126.52s tid=0x00007f6624007200 nid=0x297d15 waiting on condition  [0x00007f66712f5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x0000000740401408> (a java.util.concurrent.SynchronousQueue$TransferStack)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@16.0.1/LockSupport.java:252)
	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@16.0.1/SynchronousQueue.java:462)
	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@16.0.1/SynchronousQueue.java:361)
	at java.util.concurrent.SynchronousQueue.poll(java.base@16.0.1/SynchronousQueue.java:937)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@16.0.1/ThreadPoolExecutor.java:1055)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.1/ThreadPoolExecutor.java:1116)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.1/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Smack Cached Executor" #32 daemon prio=5 os_prio=0 cpu=41.21ms elapsed=126.51s tid=0x00007f6628069460 nid=0x297d16 waiting on condition  [0x00007f66711f4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x0000000740401408> (a java.util.concurrent.SynchronousQueue$TransferStack)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@16.0.1/LockSupport.java:252)
	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@16.0.1/SynchronousQueue.java:462)
	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@16.0.1/SynchronousQueue.java:361)
	at java.util.concurrent.SynchronousQueue.poll(java.base@16.0.1/SynchronousQueue.java:937)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@16.0.1/ThreadPoolExecutor.java:1055)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.1/ThreadPoolExecutor.java:1116)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.1/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Jicofo Global Scheduled Poolpool-3-thread-2" #33 daemon prio=5 os_prio=0 cpu=2.36ms elapsed=121.65s tid=0x00007f663803a120 nid=0x297d2f waiting on condition  [0x00007f66734ef000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x0000000740400f68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@16.0.1/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@16.0.1/AbstractQueuedSynchronizer.java:505)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@16.0.1/ForkJoinPool.java:3137)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@16.0.1/AbstractQueuedSynchronizer.java:1614)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@16.0.1/ThreadPoolExecutor.java:1056)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.1/ThreadPoolExecutor.java:1116)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.1/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Jicofo Global Scheduled Poolpool-3-thread-3" #34 daemon prio=5 os_prio=0 cpu=1.92ms elapsed=111.64s tid=0x00007f662c021100 nid=0x297d3c waiting on condition  [0x00007f66706fd000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x0000000740400f68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@16.0.1/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@16.0.1/AbstractQueuedSynchronizer.java:505)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@16.0.1/ForkJoinPool.java:3137)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@16.0.1/AbstractQueuedSynchronizer.java:1614)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@16.0.1/ThreadPoolExecutor.java:1056)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.1/ThreadPoolExecutor.java:1116)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.1/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Jicofo Global Scheduled Poolpool-3-thread-4" #35 daemon prio=5 os_prio=0 cpu=0.94ms elapsed=101.65s tid=0x00007f663c13b270 nid=0x297d3d waiting on condition  [0x00007f66707fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x0000000740400f68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@16.0.1/LockSupport.java:252)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@16.0.1/AbstractQueuedSynchronizer.java:1661)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:1182)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@16.0.1/ThreadPoolExecutor.java:1056)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.1/ThreadPoolExecutor.java:1116)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.1/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Jicofo Global Scheduled Poolpool-3-thread-5" #37 daemon prio=5 os_prio=0 cpu=0.98ms elapsed=91.65s tid=0x00007f662c059790 nid=0x297d4d waiting on condition  [0x00007f66704fb000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x0000000740400f68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@16.0.1/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@16.0.1/AbstractQueuedSynchronizer.java:505)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@16.0.1/ForkJoinPool.java:3137)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@16.0.1/AbstractQueuedSynchronizer.java:1614)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@16.0.1/ThreadPoolExecutor.java:1056)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.1/ThreadPoolExecutor.java:1116)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.1/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Jicofo Global Scheduled Poolpool-3-thread-6" #38 daemon prio=5 os_prio=0 cpu=0.92ms elapsed=81.65s tid=0x00007f6640001590 nid=0x297d4e waiting on condition  [0x00007f66703fa000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x0000000740400f68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@16.0.1/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@16.0.1/AbstractQueuedSynchronizer.java:505)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@16.0.1/ForkJoinPool.java:3137)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@16.0.1/AbstractQueuedSynchronizer.java:1614)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@16.0.1/ThreadPoolExecutor.java:1056)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.1/ThreadPoolExecutor.java:1116)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.1/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Jicofo Global Scheduled Poolpool-3-thread-7" #39 daemon prio=5 os_prio=0 cpu=1.10ms elapsed=71.65s tid=0x00007f663c109d30 nid=0x297d56 waiting on condition  [0x00007f66702f9000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x0000000740400f68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@16.0.1/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@16.0.1/AbstractQueuedSynchronizer.java:505)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@16.0.1/ForkJoinPool.java:3137)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@16.0.1/AbstractQueuedSynchronizer.java:1614)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@16.0.1/ThreadPoolExecutor.java:1056)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.1/ThreadPoolExecutor.java:1116)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.1/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Jicofo Global IO Poolpool-2-thread-2" #40 prio=5 os_prio=0 cpu=48.38ms elapsed=68.53s tid=0x00007f66480d4360 nid=0x297d57 waiting on condition  [0x00007f66701f8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x00000007406701e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@16.0.1/LockSupport.java:252)
	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@16.0.1/SynchronousQueue.java:462)
	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@16.0.1/SynchronousQueue.java:361)
	at java.util.concurrent.SynchronousQueue.poll(java.base@16.0.1/SynchronousQueue.java:937)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@16.0.1/ThreadPoolExecutor.java:1055)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.1/ThreadPoolExecutor.java:1116)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.1/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Jicofo Global IO Poolpool-2-thread-3" #41 prio=5 os_prio=0 cpu=29.88ms elapsed=68.53s tid=0x00007f66480d5b50 nid=0x297d58 waiting on condition  [0x00007f661bffe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x00000007406701e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@16.0.1/LockSupport.java:252)
	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@16.0.1/SynchronousQueue.java:462)
	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@16.0.1/SynchronousQueue.java:361)
	at java.util.concurrent.SynchronousQueue.poll(java.base@16.0.1/SynchronousQueue.java:937)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@16.0.1/ThreadPoolExecutor.java:1055)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.1/ThreadPoolExecutor.java:1116)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.1/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Jicofo Global Scheduled Poolpool-3-thread-8" #42 daemon prio=5 os_prio=0 cpu=0.24ms elapsed=61.65s tid=0x00007f663c10adb0 nid=0x297d84 waiting on condition  [0x00007f661befd000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x0000000740400f68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@16.0.1/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@16.0.1/AbstractQueuedSynchronizer.java:505)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@16.0.1/ForkJoinPool.java:3137)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@16.0.1/AbstractQueuedSynchronizer.java:1614)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@16.0.1/ThreadPoolExecutor.java:1056)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.1/ThreadPoolExecutor.java:1116)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.1/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Jicofo Global Scheduled Poolpool-3-thread-9" #43 daemon prio=5 os_prio=0 cpu=0.36ms elapsed=51.65s tid=0x00007f662c057fc0 nid=0x297d8d waiting on condition  [0x00007f661bdfc000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x0000000740400f68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@16.0.1/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@16.0.1/AbstractQueuedSynchronizer.java:505)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@16.0.1/ForkJoinPool.java:3137)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@16.0.1/AbstractQueuedSynchronizer.java:1614)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@16.0.1/ThreadPoolExecutor.java:1056)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.1/ThreadPoolExecutor.java:1116)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.1/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Jicofo Global Scheduled Poolpool-3-thread-10" #44 daemon prio=5 os_prio=0 cpu=0.20ms elapsed=41.65s tid=0x00007f6644008a20 nid=0x297d95 waiting on condition  [0x00007f661bcfb000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x0000000740400f68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@16.0.1/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@16.0.1/AbstractQueuedSynchronizer.java:505)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@16.0.1/ForkJoinPool.java:3137)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@16.0.1/AbstractQueuedSynchronizer.java:1614)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@16.0.1/ThreadPoolExecutor.java:1056)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.1/ThreadPoolExecutor.java:1116)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.1/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Jicofo Global Scheduled Poolpool-3-thread-11" #45 daemon prio=5 os_prio=0 cpu=0.16ms elapsed=31.65s tid=0x00007f6640002cc0 nid=0x297d9f waiting on condition  [0x00007f661bbfa000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x0000000740400f68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@16.0.1/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@16.0.1/AbstractQueuedSynchronizer.java:505)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@16.0.1/ForkJoinPool.java:3137)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@16.0.1/AbstractQueuedSynchronizer.java:1614)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@16.0.1/ThreadPoolExecutor.java:1056)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.1/ThreadPoolExecutor.java:1116)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.1/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Jicofo Global Scheduled Poolpool-3-thread-12" #46 daemon prio=5 os_prio=0 cpu=0.19ms elapsed=21.65s tid=0x00007f665000cd80 nid=0x297da2 waiting on condition  [0x00007f661baf9000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x0000000740400f68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@16.0.1/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@16.0.1/AbstractQueuedSynchronizer.java:505)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@16.0.1/ForkJoinPool.java:3137)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@16.0.1/AbstractQueuedSynchronizer.java:1614)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@16.0.1/ThreadPoolExecutor.java:1056)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.1/ThreadPoolExecutor.java:1116)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.1/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Attach Listener" #47 daemon prio=9 os_prio=0 cpu=0.73ms elapsed=20.72s tid=0x00007f6678000e50 nid=0x297db7 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Jicofo Global Scheduled Poolpool-3-thread-13" #48 daemon prio=5 os_prio=0 cpu=0.15ms elapsed=11.65s tid=0x00007f663c10c010 nid=0x297dc0 waiting on condition  [0x00007f661b8f7000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x0000000740400f68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@16.0.1/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@16.0.1/AbstractQueuedSynchronizer.java:505)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@16.0.1/ForkJoinPool.java:3137)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@16.0.1/AbstractQueuedSynchronizer.java:1614)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@16.0.1/ThreadPoolExecutor.java:1056)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.1/ThreadPoolExecutor.java:1116)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.1/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"Jicofo Global Scheduled Poolpool-3-thread-14" #49 daemon prio=5 os_prio=0 cpu=0.28ms elapsed=1.65s tid=0x00007f664c003f20 nid=0x297dc1 waiting on condition  [0x00007f66705fc000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@16.0.1/Native Method)
	- parking to wait for  <0x0000000740400f68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@16.0.1/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@16.0.1/AbstractQueuedSynchronizer.java:505)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@16.0.1/ForkJoinPool.java:3137)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@16.0.1/AbstractQueuedSynchronizer.java:1614)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@16.0.1/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@16.0.1/ThreadPoolExecutor.java:1056)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.1/ThreadPoolExecutor.java:1116)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.1/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run(java.base@16.0.1/Thread.java:831)

"VM Thread" os_prio=0 cpu=18.09ms elapsed=129.45s tid=0x00007f66b813ece0 nid=0x297ca3 runnable  

"GC Thread#0" os_prio=0 cpu=36.08ms elapsed=129.46s tid=0x00007f66b80556b0 nid=0x297c9d runnable  

"GC Thread#1" os_prio=0 cpu=44.31ms elapsed=129.16s tid=0x00007f667c004270 nid=0x297cb6 runnable  

"GC Thread#2" os_prio=0 cpu=30.29ms elapsed=129.15s tid=0x00007f667c004e10 nid=0x297cb7 runnable  

"GC Thread#3" os_prio=0 cpu=43.53ms elapsed=129.15s tid=0x00007f667c0059b0 nid=0x297cb8 runnable  

"G1 Main Marker" os_prio=0 cpu=0.37ms elapsed=129.46s tid=0x00007f66b8062e00 nid=0x297c9e runnable  

"G1 Conc#0" os_prio=0 cpu=14.84ms elapsed=129.46s tid=0x00007f66b8063e90 nid=0x297c9f runnable  

"G1 Refine#0" os_prio=0 cpu=1.37ms elapsed=129.46s tid=0x00007f66b8107630 nid=0x297ca1 runnable  

"G1 Service" os_prio=0 cpu=22.42ms elapsed=129.46s tid=0x00007f66b8108650 nid=0x297ca2 runnable  

"VM Periodic Task Thread" os_prio=0 cpu=73.95ms elapsed=129.43s tid=0x00007f66b8161d70 nid=0x297cb3 waiting on condition  

JNI global refs: 15, weak refs: 0

Thanks, yep nothing suspicious … this needs more debugging …
“both cases the room is no longer accessible untile Jicofo is restarted.”
Do you have any authentication configured or not?
Will I repro if I do a vanilla install and make some custom module to destroy the room … and that room will no longer be accessible for clients?

Haven’t really tested this with reservations plugin specifically, but I can indeed repro issue when I manually destroy meeting from admin telnet using:

> hosts["conference.meet.domain.com"].modules.muc.get_room_from_jid("roomname@conference.meet.domain.com"):destroy()

On rejoin to same room, I see a CONFERENCE FAILED: conference.connectionError.notAllowed in console, and the following presence stanza.

<presence to='8aa5cef7-e56e-4f5a-be3c-a40db7d0590f@meet.domain.com/AvPTV6ze' xmlns='jabber:client' type='error' from='roomname@conference.meet.domain.com/8aa5cef7'>
    <error type='cancel'>
        <not-allowed xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'/>
        <text xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'> 
            Room creation is restricted
        </text>
    </error>
</presence>

Nothing interesting in Jicofo or Prosody logs.

As reported, everything works as expected once Jicofo is restarted.

In case it makes a difference, I only use JWT auth.

2 Likes

That will help a lot for debugging it! Thanks @shawn.

We tracked it down to a change in behavior of Smack muc: fix roomDestroyed() callback · igniterealtime/Smack@0f7b7df · GitHub and we are trying to figure it out with Smack and Prosody friends how to fix it.

1 Like

Hi! :grinning:
I can confirm we’re getting the same conference.connectionError.notAllowed after the :destroy(), and we’re using JWT authentication (with RS256) !

4 Likes