Debian Buster JVB2 fatal error detected by JRE happening outside the JVM in native code fresh install

Hello,

We are using Jitsi Meet on Uni server for keeping in touch with students and everything is working neatly, resources are not maxed out, however from time to time with 100+ users (possibly not everyone using Chrome) we get a segmentation fault of the Java VM in [libc.so.6+0xa7164] in libjnisctp.so. Part of the crash log is present here, any suggestion is well appreciated:

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007f2c54b5a164, pid=718, tid=1246

JRE version: OpenJDK Runtime Environment (11.0.7+10) (build 11.0.7+10-post-Debian-3deb10u1)

Java VM: OpenJDK 64-Bit Server VM (11.0.7+10-post-Debian-3deb10u1, mixed mode, sharing, tiered, compressed oops, concurrent mark sweep gc, linux-amd64)

Problematic frame:

C [libc.so.6+0xa7164]

No core dump will be written. Core dumps have been disabled. To enable core dumping, try “ulimit -c unlimited” before starting Java again

If you would like to submit a bug report, please visit:

https://bugs.debian.org/openjdk-11

The crash happened outside the Java Virtual Machine in native code.

See problematic frame for where to report the bug.

--------------- S U M M A R Y ------------

Command Line: -Xmx20480m -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/videobridge/logging.properties org.jitsi.videobridge.Main --host=localhost --domain=jarvis.cs.upt.ro --port=5347 --secret=DHXInet5 --apis=,

Host: Intel® Xeon® CPU E5-2450L 0 @ 1.80GHz, 12 cores, 31G, Debian GNU/Linux 10 (buster)
Time: Tue May 5 12:24:58 2020 EEST elapsed time: 752549 seconds (8d 17h 2m 29s)

--------------- T H R E A D ---------------

Current thread (0x00007f2ae4001000): JavaThread “Thread-4907” daemon [_thread_in_native, id=1246, stack(0x00007f2b73801000,0x00007f2b74000000)]

Stack: [0x00007f2b73801000,0x00007f2b74000000], sp=0x00007f2b73ffe098, free space=8180k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.6+0xa7164]
C [libjnisctp.so+0x8373d] sctp_invoke_recv_callback+0x290
C [libjnisctp.so+0x8431c] sctp_add_to_readq+0x6a6
C [libjnisctp.so+0x7f80d]
C [libjnisctp.so+0x811b6] sctp_ulp_notify+0x3b8
C [libjnisctp.so+0x82251] sctp_report_all_outbound+0xc1e
C [libjnisctp.so+0x825e0] sctp_abort_notification+0xeb
C [libjnisctp.so+0x82843] sctp_abort_an_association+0xfb
C [libjnisctp.so+0x3ce9b]
C [libjnisctp.so+0x3e052] sctp_chunk_output+0x1d0
C [libjnisctp.so+0x7c40a] sctp_timeout_handler+0x4a1
C [libjnisctp.so+0x9bb23]
C [libjnisctp.so+0x9bc52] user_sctp_timer_iterate+0xd8

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x000000005eb130ea
Register to memory mapping:

RAX=0x00007f2ae406aa10 points into unknown readable memory: 00 00 00 00 00 00 00 00
RBX=0x0 is NULL
RCX=0x000000005eb130ea is an unknown value
RDX=0x00000000000da29c is an unknown value
RSP=0x00007f2b73ffe098 is pointing into the stack for thread: 0x00007f2ae4001000
RBP=0x00007f2b73ffe0d0 is pointing into the stack for thread: 0x00007f2ae4001000
RSI=0x000000005eb130ea is an unknown value
RDI=0x00007f2ae406aa10 points into unknown readable memory: 00 00 00 00 00 00 00 00
R8 =0x00007f2ae406a9f0 points into unknown readable memory: 0e 00 01 00 20 00 00 00
R9 =0x0000000000082000 is an unknown value
R10=0x00007f2ae4082000 points into unknown readable memory: 00 00 00 00 00 00 00 00
R11=0x0000000000000206 is an unknown value
R12=0x00007f2c11eeef6e is pointing into the stack for thread: 0x00007f2bd40de800
R13=0x00007f2c11eeef6f is pointing into the stack for thread: 0x00007f2bd40de800
R14=0x00007f2b73ffefc0 is pointing into the stack for thread: 0x00007f2ae4001000
R15=0x00007f2b73fff700 is pointing into the stack for thread: 0x00007f2ae4001000

Registers:
RAX=0x00007f2ae406aa10, RBX=0x0000000000000000, RCX=0x000000005eb130ea, RDX=0x00000000000da29c
RSP=0x00007f2b73ffe098, RBP=0x00007f2b73ffe0d0, RSI=0x000000005eb130ea, RDI=0x00007f2ae406aa10
R8 =0x00007f2ae406a9f0, R9 =0x0000000000082000, R10=0x00007f2ae4082000, R11=0x0000000000000206
R12=0x00007f2c11eeef6e, R13=0x00007f2c11eeef6f, R14=0x00007f2b73ffefc0, R15=0x00007f2b73fff700
RIP=0x00007f2c54b5a164, EFLAGS=0x0000000000010212, CSGSFS=0x002b000000000033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f2b73ffe098)
0x00007f2b73ffe098: 00007f2b7bdc072c 00007f2c11eeef6f
0x00007f2b73ffe0a8: 00007f2ae406aa10 00000000000da29c
0x00007f2b73ffe0b8: 00007f2bb012d470 0000000000000000
0x00007f2b73ffe0c8: 000da29c73ffe190 00007f2b73ffe190

Instructions: (pc=0x00007f2c54b5a164)
0x00007f2c54b5a064: 47 0f 10 06 0f 10 4e 10 0f 10 56 20 0f 10 5e 30
0x00007f2c54b5a074: 0f 10 64 16 f0 0f 10 6c 16 e0 0f 10 74 16 d0 0f
0x00007f2c54b5a084: 10 7c 16 c0 0f 11 07 0f 11 4f 10 0f 11 57 20 0f
0x00007f2c54b5a094: 11 5f 30 0f 11 64 17 f0 0f 11 6c 17 e0 0f 11 74
0x00007f2c54b5a0a4: 17 d0 0f 11 7c 17 c0 c3 0f 10 06 0f 10 4e 10 0f
0x00007f2c54b5a0b4: 10 54 16 f0 0f 10 5c 16 e0 0f 11 07 0f 11 4f 10
0x00007f2c54b5a0c4: 0f 11 54 17 f0 0f 11 5c 17 e0 c3 48 39 f7 0f 87
0x00007f2c54b5a0d4: 8c 00 00 00 0f 84 28 ff ff ff 0f 10 26 0f 10 6c
0x00007f2c54b5a0e4: 16 f0 0f 10 74 16 e0 0f 10 7c 16 d0 44 0f 10 44
0x00007f2c54b5a0f4: 16 c0 49 89 fb 48 8d 4c 17 f0 49 89 f8 49 83 e0
0x00007f2c54b5a104: 0f 49 83 e8 10 4c 29 c6 4c 29 c7 4c 01 c2 48 3b
0x00007f2c54b5a114: 15 07 b1 11 00 0f 87 cc 00 00 00 0f 10 06 0f 10
0x00007f2c54b5a124: 4e 10 0f 10 56 20 0f 10 5e 30 48 83 c6 40 48 83
0x00007f2c54b5a134: ea 40 0f 29 07 0f 29 4f 10 0f 29 57 20 0f 29 5f
0x00007f2c54b5a144: 30 48 83 c7 40 48 83 fa 40 77 d0 0f 11 29 0f 11
0x00007f2c54b5a154: 71 f0 0f 11 79 e0 44 0f 11 41 d0 41 0f 11 23 c3
0x00007f2c54b5a164: 0f 10 26 0f 10 6e 10 0f 10 76 20 0f 10 7e 30 44
0x00007f2c54b5a174: 0f 10 44 16 f0 4c 8d 5c 17 f0 48 8d 4c 16 f0 4d
0x00007f2c54b5a184: 89 d9 4d 89 d8 49 83 e0 0f 4c 29 c1 4d 29 c1 4c
0x00007f2c54b5a194: 29 c2 48 3b 15 83 b0 11 00 0f 87 af 00 00 00 0f
0x00007f2c54b5a1a4: 10 01 0f 10 49 f0 0f 10 51 e0 0f 10 59 d0 48 83
0x00007f2c54b5a1b4: e9 40 48 83 ea 40 41 0f 29 01 41 0f 29 49 f0 41
0x00007f2c54b5a1c4: 0f 29 51 e0 41 0f 29 59 d0 49 83 e9 40 48 83 fa
0x00007f2c54b5a1d4: 40 77 cc 0f 11 27 0f 11 6f 10 0f 11 77 20 0f 11
0x00007f2c54b5a1e4: 7f 30 45 0f 11 03 c3 4c 8d 14 17 4c 39 d6 0f 82
0x00007f2c54b5a1f4: 27 ff ff ff 0f 18 8e 80 00 00 00 0f 18 8e c0 00
0x00007f2c54b5a204: 00 00 0f 10 06 0f 10 4e 10 0f 10 56 20 0f 10 5e
0x00007f2c54b5a214: 30 48 83 c6 40 48 83 ea 40 66 0f e7 07 66 0f e7
0x00007f2c54b5a224: 4f 10 66 0f e7 57 20 66 0f e7 5f 30 48 83 c7 40
0x00007f2c54b5a234: 48 83 fa 40 77 be 0f ae f8 0f 11 29 0f 11 71 f0
0x00007f2c54b5a244: 0f 11 79 e0 44 0f 11 41 d0 41 0f 11 23 c3 4c 8d
0x00007f2c54b5a254: 14 11 4d 39 d1 0f 82 44 ff ff ff 0f 18 49 80 0f

Stack slot to memory mapping:
stack at sp + 0 slots: 0x00007f2b7bdc072c: m_copydata+0x000000000000007a in /tmp/nativeutils10237530610/libjnisctp.so at 0x00007f2b7bd34000
stack at sp + 1 slots: 0x00007f2c11eeef6f is pointing into the stack for thread: 0x00007f2bd40de800
stack at sp + 2 slots: 0x00007f2ae406aa10 points into unknown readable memory: 00 00 00 00 00 00 00 00
stack at sp + 3 slots: 0x00000000000da29c is an unknown value
stack at sp + 4 slots: 0x00007f2bb012d470 points into unknown readable memory: 00 00 00 00 00 00 00 00
stack at sp + 5 slots: 0x0 is NULL
stack at sp + 6 slots: 0x000da29c73ffe190 is an unknown value
stack at sp + 7 slots: 0x00007f2b73ffe190 is pointing into the stack for thread: 0x00007f2ae4001000

--------------- P R O C E S S ---------------

Threads class SMR info:
_java_thread_list=0x00007f2b7402e880, length=471, elements={
0x00007f2c4c017000, 0x00007f2c4c0d7000, 0x00007f2c4c0d9000, 0x00007f2c4c0e7000,
0x00007f2c4c0e9800, 0x00007f2c4c0eb800, 0x00007f2c4c0ed800, 0x00007f2c4c168800,
0x00007f2c4c172800, 0x00007f2bd400a800, 0x00007f2bd4098000, 0x00007f2bd4099000,
0x00007f2bd40db000, 0x00007f2bd40de800, 0x00007f2bc8097000, 0x00007f2bd4241000,
0x00007f2bb001b000, 0x00007f2bc810f000, 0x00007f2bb002e800, 0x00007f2bb0030000,
0x00007f2bc00b0000, 0x00007f2bd40d8000, 0x00007f2bcc008000, 0x00007f2bd423e000,
0x00007f2bd40d5800, 0x00007f2bcc004800, 0x00007f2bcc00e800, 0x00007f2bd4224000,
0x00007f2b2c00a000, 0x00007f2bcc010800, 0x00007f2bd424d800, 0x00007f2bcc7b9000,
0x00007f2bd424e800, 0x00007f2bd424f000, 0x00007f2ae4001000, 0x00007f2ab8003000,
0x00007f2bc82bc000, 0x00007f2bc82a0800, 0x00007f2bc82a1800, 0x00007f2ba800f000,

We don’t know why JRE crashes in libc with the latest OpenJDK deb package if there is any issue in JVB2.

Thanks for an extraordinary piece of software . Any help is appreciated. Please help us find a workaround or identify the root cause of the problem.

Thank you!

Hi Sivraj,
I’m also hosting a jitsi-meet instance and I have also seen three crashes of the Jitsi instance during the last month with an exception that looks like yours.

C [libc.so.6+0xbb6c4]
C [libjnisctp.so+0x8373d] sctp_invoke_recv_callback+0x290

My instance is running Ubuntu 18.04 and Java8 so the versions don’t seem to cause the problem.
Before the crash Sctp has also produced many warnings and errors in the jvb-log

Sctp send error: : Resource temporarily unavailable

This has also been discussed on github https://github.com/jitsi/jitsi-videobridge/issues/314
As Sctp seems to cause this problem I’ll try changing from Sctp to Websockets and see if this solves the problem.
Let’s see if this problem can be solved
Kind Regards
Lennart