Conference crashes since latest update

Install & Config
#1

Hello,
I have upgraded to the latest Jitsi version (on debian), but I experience crashes I have never had with previous versions.
It can happen during a conference with 3, 8 participants or with 30 participants (many of which are usually with video muted), after 10 minutes or after 1 hour.
Only one conference happens at a time.
The conference stops, and restarts after two or three automatic attempts to restart.
Could you please help me understand what could be the problem ?
LOGS :
in /var/log/jitsi/jicofo.log.2.gz

Jicofo 2021-02-22 13:55:09.954 INFOS: [517] org.jitsi.jicofo.AbstractChannelAllocator.log() Using jvbbrewery@internal.auth.visio.mydomain.com/8d67c44c-c373-4827-a139-80a0daf4d4b7 to allocate channels for: Participant[roomname@conference.visio.mydomain.com/ba601082]@1882080772
Jicofo 2021-02-22 13:55:21.710 GRAVE: [511] org.jitsi.jicofo.AbstractChannelAllocator.log() jvbbrewery@internal.auth.visio.mydomain.com/8d67c44c-c373-4827-a139-80a0daf4d4b7 - failed to allocate channels, will consider the bridge faulty: Timed out waiting for a response.
org.jitsi.protocol.xmpp.colibri.exception.TimeoutException: Timed out waiting for a response.
at org.jitsi.impl.protocol.xmpp.colibri.ColibriConferenceImpl.maybeThrowOperationFailed(ColibriConferenceImpl.java:328)
at org.jitsi.impl.protocol.xmpp.colibri.ColibriConferenceImpl.createColibriChannels(ColibriConferenceImpl.java:268)
at org.jitsi.protocol.xmpp.colibri.ColibriConference.createColibriChannels(ColibriConference.java:102)
at org.jitsi.jicofo.ParticipantChannelAllocator.doAllocateChannels(ParticipantChannelAllocator.java:110)
at org.jitsi.jicofo.AbstractChannelAllocator.allocateChannels(AbstractChannelAllocator.java:271)
at org.jitsi.jicofo.AbstractChannelAllocator.doRun(AbstractChannelAllocator.java:190)
at org.jitsi.jicofo.AbstractChannelAllocator.run(AbstractChannelAllocator.java:150)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Jicofo 2021-02-22 13:55:21.714 GRAVE: [511] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() One of our bridges failed: jvbbrewery@internal.auth.visio.mydomain.com/8d67c44c-c373-4827-a139-80a0daf4d4b7
Jicofo 2021-02-22 13:55:21.714 AVERTISSEMENT: [511] org.jitsi.jicofo.AbstractParticipant.log() Canceling ParticipantChannelAllocator[BridgeSession[id=45520_8f48fc, bridge=Bridge[jid=jvbbrewery@internal.auth.visio.mydomain.com/8d67c44c-c373-4827-a139-80a0daf4d4b7, relayId=null, region=null, stress=0,11]]@470681232, Participant[roomname@conference.visio.mydomain.com/26610eb1]@1183602231]@1065474849
Jicofo 2021-02-22 13:55:21.722 INFOS: [511] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=45520: null, null, null, null, null, null, null
Jicofo 2021-02-22 13:55:21.722 AVERTISSEMENT: [511] org.jitsi.jicofo.AbstractParticipant.log() Canceling ParticipantChannelAllocator[BridgeSession[id=45520_8f48fc, bridge=Bridge[jid=jvbbrewery@internal.auth.visio.mydomain.com/8d67c44c-c373-4827-a139-80a0daf4d4b7, relayId=null, region=null, stress=0,11]]@470681232, Participant[roomname@conference.visio.mydomain.com/e24526d5]@1212919958]@1002326025

IN during the same time /var/log/jvb.log.1

2021-01-22 13:55:06.468 INFOS: [26] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-01-22 13:55:16.468 INFOS: [26] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-01-22 13:55:26.467 INFOS: [26] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-01-22 13:55:36.467 INFOS: [26] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-01-22 13:55:36.490 INFOS: [25] VideobridgeExpireThread.expire#140: Running expire()
2021-01-22 13:55:46.467 INFOS: [26] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-01-22 13:55:56.467 INFOS: [26] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-01-22 13:56:06.467 INFOS: [26] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-01-22 13:56:16.467 INFOS: [26] HealthChecker.run#170: Performed a successful health check in PT0S. Sticky failure: false
2021-01-22 13:56:26.467 INFOS: [26] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-01-22 13:56:36.468 INFOS: [26] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false
2021-01-22 13:56:36.490 INFOS: [25] VideobridgeExpireThread.expire#140: Running expire()
2021-01-22 13:56:46.467 INFOS: [26] HealthChecker.run#170: Performed a successful health check in PT0.000001S. Sticky failure: false

Thanks for your help or advice

#2

what are the outputs?

df -h
free -m
lscpu
dpkg -l "jitsi-*"
dpkg -l "prosody*"
#3

miroslav@CT109:~$ df -h
Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
/dev/loop4 7,8G 2,0G 5,4G 27% /
none 492K 0 492K 0% /dev
tmpfs 1,6G 80K 1,6G 1% /run
tmpfs 5,0M 0 5,0M 0% /run/lock
tmpfs 903M 0 903M 0% /dev/shm

miroslav@CT109:~$ free -m
total used free shared buff/cache available
Mem: 4096 576 3464 428 55 3519
Swap: 512 512 0

miroslav@CT109:~$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 36 bits physical, 48 bits virtual
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 58
Model name: Intel® Xeon® CPU E3-1225 V2 @ 3.20GHz
Stepping: 9
CPU MHz: 3522.500
CPU max MHz: 3600,0000
CPU min MHz: 1600,0000
BogoMIPS: 6385.34
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 8192K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm retpoline kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts

Souhait=inconnU/Installé/suppRimé/Purgé/H=à garder
| État=Non/Installé/fichier-Config/dépaqUeté/échec-conFig/H=semi-installé/W=attend-traitement-déclenchements
|/ Err?=(aucune)/besoin Réinstallation (État,Err: majuscule=mauvais)
||/ Nom Version Architecture Description
++±=====================-===================-============-=================================================
ii jitsi-archive-keyring 1.0.1 all The public key for the Jitsi packages repository
ii jitsi-meet 2.0.5390-3 all WebRTC JavaScript video conferences
ii jitsi-meet-prosody 1.0.4628-1 all Prosody configuration for Jitsi Meet
un jitsi-meet-tokens (aucune description n’est disponible)
un jitsi-meet-turnserver (aucune description n’est disponible)
ii jitsi-meet-web 1.0.4628-1 all WebRTC JavaScript video conferences
ii jitsi-meet-web-config 1.0.4628-1 all Configuration for web serving of Jitsi Meet
un jitsi-videobridge (aucune description n’est disponible)
ii jitsi-videobridge2 2.1-416-g2f43d1b4-1 all WebRTC compatible Selective Forwarding Unit (SFU)

miroslav@CT109:~$ dpkg -l “prosody*”
Souhait=inconnU/Installé/suppRimé/Purgé/H=à garder
| État=Non/Installé/fichier-Config/dépaqUeté/échec-conFig/H=semi-installé/W=attend-traitement-déclenchements
|/ Err?=(aucune)/besoin Réinstallation (État,Err: majuscule=mauvais)
||/ Nom Version Architecture Description
++±===============-============-============-=====================================
ii prosody 0.11.2-1 amd64 Lightweight Jabber/XMPP server
un prosody-0.11 (aucune description n’est disponible)
un prosody-modules (aucune description n’est disponible)
un prosody-trunk (aucune description n’est disponible)

#4

The memory seems low. It’s needed at least 8 GB RAM for a default installation.
Search for VIDEOBRIDGE_MAX_MEMORY and JICOFO_MAX_MEMORY

#5

to each his personal gripe :-). 0 swap free is not a good thing. I have never seen a Linux computer behave happily with all swap used. 512 M swap seems low for 4 Gbytes Ram, btw.

#6

Thank you for your help. I will update as soon as I have some feedback to share.

#7

Hello,

I have changed the server settings as recommended, but experienced an other crash. We were only 7 in the room, an not everyone had the camera on. The room crashed after a few minutes, when last members connected to the room. After that romm reconnected and everything went fine for one hour until the end of the meeting.

Memory : 8G
Swap : 1G
htop → see attached file

Capture du 2021-03-08 20-16-23_v1
Capture du 2021-03-08 20-16-23_v11680×945 355 KB

jicofo.log :

2021-03-08 19:16:00.904 INFOS: [72] org.jitsi.jicofo.xmpp.ConferenceIqHandler.log() Focus request for room:
myconfroom@conference.visio.mydomain.com
Jicofo 2021-03-08 19:16:01.213 INFOS: [662] org.jitsi.jicofo.ChatRoomRoleAndPresence.log()
Chat room event ChatRoomMemberPresenceChangeEvent[type=MemberJoined sourceRoom=org.jitsi.impl.protocol.xmpp.ChatRoomImpl@59284a9b
member=ChatMember[myconfroom@conference.visio.mydomain.com/09cba196, jid: null]@79358389]
Jicofo 2021-03-08 19:16:01.213 INFOS: [662]
org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Member myconfroom@conference.visio.mydomain.com/09cba196 joined.
Jicofo 2021-03-08 19:16:01.213 INFOS: [662] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Added participant jid=
myconfroom@conference.visio.mydomain.com/09cba196,
bridge=jvbbrewery@internal.auth.visio.mydomain.com/8d67c44c-c373-4827-a139-80a0daf4d4b7
Jicofo 2021-03-08 19:16:01.213 INFOS: [662] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=32284: [[null, null, null, null, null, null, null, null]]
Jicofo 2021-03-08
19:16:01.213 INFOS: [725] org.jitsi.jicofo.discovery.DiscoveryUtil.log() Doing feature discovery for
myconfroom@conference.visio.mydomain.com/09cba196
Jicofo 2021-03-08 19:16:01.214 INFOS: [725]
org.jitsi.jicofo.discovery.DiscoveryUtil.log() Successfully discovered features for myconfroom@conference.visio.mydomain.com/09cba196
in 1
Jicofo 2021-03-08 19:16:01.214 INFOS: [725] org.jitsi.jicofo.AbstractChannelAllocator.log() Using
jvbbrewery@internal.auth.visio.mydomain.com/8d67c44c-c373-4827-a139-80a0daf4d4b7 to allocate channels for:
Participant[myconfroom@conference.visio.mydomain.com/09cba196]@995368235 J
icofo 2021-03-08 19:16:16.214 GRAVE: [725]
org.jitsi.jicofo.AbstractChannelAllocator.log() jvbbrewery@internal.auth.visio.mydomain.com/8d67c44c-c373-4827-a139-80a0daf4d4b7 - failed to allocate
channels, will consider the bridge faulty: Timed out waiting for a response. org.jitsi.protocol.xmpp.colibri.exception.TimeoutException: Timed out waiting for
a response.
at org.jitsi.impl.protocol.xmpp.colibri.ColibriConferenceImpl.maybeThrowOperationFailed(ColibriConferenceImpl.java:328)
at org.jitsi.impl.protocol.xmpp.colibri.ColibriConferenceImpl.createColibriChannels(ColibriConferenceImpl.java:268)
at org.jitsi.protocol.xmpp.colibri.ColibriConference.createColibriChannels(ColibriConference.java:102)
at org.jitsi.jicofo.ParticipantChannelAllocator.doAllocateChannels(ParticipantChannelAllocator.java:110)
at org.jitsi.jicofo.AbstractChannelAllocator.allocateChannels(AbstractChannelAllocator.java:271)
at org.jitsi.jicofo.AbstractChannelAllocator.doRun(AbstractChannelAllocator.java:190)
at org.jitsi.jicofo.AbstractChannelAllocator.run(AbstractChannelAllocator.java:150)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Jicofo 2021-03-08 19:16:16.307 GRAVE: [725] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() One of our bridges failed: jvbbrewery@internal.auth.visio.mydomain.com$
Jicofo 2021-03-08 19:16:16.314 INFOS: [725] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=32284: [[null, null, null, null, null, null, null]]
Jicofo 2021-03-08 19:16:16.314 INFOS: [725] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=32284: [[null, null, null, null, null, null]]
Jicofo 2021-03-08 19:16:16.314 INFOS: [725] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=32284: [[null, null, null, null, null]]
Jicofo 2021-03-08 19:16:16.314 INFOS: [725] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=32284: [[null, null, null, null]]
Jicofo 2021-03-08 19:16:16.314 INFOS: [725] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=32284: [[null, null, null]]
Jicofo 2021-03-08 19:16:16.314 INFOS: [725] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=32284: [[null, null]]

root@CT109:/var/log/jitsi# df -h
Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
/dev/loop4 7,8G 2,1G 5,4G 28% /
none 492K 0 492K 0% /dev
tmpfs 1,6G 80K 1,6G 1% /run
tmpfs 5,0M 0 5,0M 0% /run/lock
tmpfs 903M 0 903M 0% /dev/shm
root@CT109:/var/log/jitsi#

root@CT109:/var/log/jitsi# free -m
total used free shared buff/cache available
Mem: 8192 660 7497 445 34 7531
Swap: 1024 446 577

root@CT109:/var/log/jitsi# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 36 bits physical, 48 bits virtual
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 58
Model name: Intel® Xeon® CPU E3-1225 V2 @ 3.20GHz
Stepping: 9
CPU MHz: 3115.625
CPU max MHz: 3600,0000
CPU min MHz: 1600,0000
BogoMIPS: 6385.34
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 8192K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm retpoline kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts

root@CT109:/var/log/jitsi# dpkg -l “jitsi-*”
Souhait=inconnU/Installé/suppRimé/Purgé/H=à garder
| État=Non/Installé/fichier-Config/dépaqUeté/échec-conFig/H=semi-installé/W=attend-traitement-déclenchements
|/ Err?=(aucune)/besoin Réinstallation (État,Err: majuscule=mauvais)
||/ Nom Version Architecture Description
++±=====================-===================-============-=================================================
ii jitsi-archive-keyring 1.0.1 all The public key for the Jitsi packages repository
ii jitsi-meet 2.0.5390-3 all WebRTC JavaScript video conferences
ii jitsi-meet-prosody 1.0.4628-1 all Prosody configuration for Jitsi Meet
un jitsi-meet-tokens (aucune description n’est disponible)
un jitsi-meet-turnserver (aucune description n’est disponible)
ii jitsi-meet-web 1.0.4628-1 all WebRTC JavaScript video conferences
ii jitsi-meet-web-config 1.0.4628-1 all Configuration for web serving of Jitsi Meet
un jitsi-videobridge (aucune description n’est disponible)
ii jitsi-videobridge2 2.1-416-g2f43d1b4-1 all WebRTC compatible Selective Forwarding Unit (SFU)

root@CT109:/var/log/jitsi# dpkg -l “prosody*”
Souhait=inconnU/Installé/suppRimé/Purgé/H=à garder
| État=Non/Installé/fichier-Config/dépaqUeté/échec-conFig/H=semi-installé/W=attend-traitement-déclenchements
|/ Err?=(aucune)/besoin Réinstallation (État,Err: majuscule=mauvais)
||/ Nom Version Architecture Description
++±===============-============-============-=====================================
ii prosody 0.11.2-1 amd64 Lightweight Jabber/XMPP server
un prosody-0.11 (aucune description n’est disponible)
un prosody-modules (aucune description n’est disponible)
un prosody-trunk (aucune description n’est disponible)

#8

You have been very descriptive on the hardware and the Jitsi packages, I am curious however on the more general setup of your system; I noticed that you have installed jitsi-archive-keyring, a package I did not know and that seems to be associated to jitsi-desktop; is your computer a local workstation used for local conferences ? is it a virtual machine ? I’m asking because there could be priorities issues with a VM, and Jitsi-meet is a real time system. If that is the case, could you try to set the real time priority on the VM ? Also I wonder if you upgraded from a recent (last stable) or a very old Jitsi-meet version. If the latter, maybe some old config could be harmful.

#9

Hello,
Jitsi is intalled in a LXC Container, on a server (not a local workstation).
First install was done last year (around August/September 2020). Then we have upgraded via Debian repository.

I 'll check the real time priority and will inform you as soon as I have some feedback.
Thanks for your follow up.

#10

normally there should not be priority problems then. There could be resources problem though. No error message in jvb.log that were not there before upgrading ?

there were serious changes since then. Did You check that jvb web sockets are working ?

#11

How can I check this ?

#12

browser console errors talking of websockets