WebSocket connection to Failed followed by CONFERENCE FAILED: conference.connectionError

Greetings,
We have been using jitsi installed on digital ocean since last year for running k12 school.
We have installed jitsi-meet, jicofo and prosody on a server and configured 8 to 16 servers running JVB based on the load.

Recently we have upgraded the created new server to implement JWT by following the emrah-tools/jitsi/installer/school at main · emrahcom/emrah-tools · GitHub.

As needed by the school. The persons with valid JWT tokens are able to join. During the school hours there are around 10 to 20 parallel online classes conducted.

Problem:
At random the a room(class) is getting closed and all the members of the room are disconnected including the moderator. The error messages in the chrome console are

  1. WebSocket connection to ‘XXXX’ failed.
  2. [modules/RTC/BridgeChannel.js] <webSocket.e.onClose>: channel closed: 1006
  3. [conference.js] <ae.onConferenceFailed>: CONFERENCE FAILED: conference.connectionError

When the teacher/moderator tries to login or student tries to login there is a login error and screen shows “Sorry! you are not allowed to be here :(”

All the other rooms run without any problem except this room.
Only after restarting prosody and jicofo the access to this room is restored.

Is there a way to fix this issue?

Thanks in advance!

Kind regards,
Krishna

The current versions on the new installation are

ii coturn 4.5.1.1-1.1ubuntu0.20.04.2 amd64 TURN and STUN server for VoIP
ii jicofo 1.0-798-1 all JItsi Meet COnference FOcus
ii jitsi-meet 2.0.6293-1 all WebRTC JavaScript video conferences
ii jitsi-meet-prosody 1.0.5307-1 all Prosody configuration for Jitsi Meet
ii jitsi-meet-tokens 1.0.5307-1 all Prosody token authentication plugin for Jitsi Meet
ii jitsi-meet-turnserver 1.0.5307-1 all Configures coturn to be used with Jitsi Meet
ii jitsi-meet-web 1.0.5307-1 all WebRTC JavaScript video conferences
ii jitsi-meet-web-config 1.0.5307-1 all Configuration for web serving of Jitsi Meet
ii jitsi-videobridge2 2.1-551-g2ad6eb0b-1 all WebRTC compatible Selective Forwarding Unit (SFU)
ii libnginx-mod-http-image-filter 1.18.0-0ubuntu1.2 amd64 HTTP image filter module for Nginx
ii libnginx-mod-http-xslt-filter 1.18.0-0ubuntu1.2 amd64 XSLT Transformation module for Nginx
ii libnginx-mod-mail 1.18.0-0ubuntu1.2 amd64 Mail module for Nginx
ii libnginx-mod-stream 1.18.0-0ubuntu1.2 amd64 Stream module for Nginx
ii nginx 1.18.0-0ubuntu1.2 all small, powerful, scalable web/proxy server
ii nginx-common 1.18.0-0ubuntu1.2 all small, powerful, scalable web/proxy server - common files
ii nginx-core 1.18.0-0ubuntu1.2 amd64 nginx web/proxy server (standard version)
ii prosody 0.11.4-1 amd64 Lightweight Jabber/XMPP server

The JVB’s are running

=========================
ii jitsi-videobridge2 2.1-551-g2ad6eb0b-1

======================================================================================================
ports open on the jitsi server

80/tcp ALLOW Anywhere
21/tcp ALLOW Anywhere
22 ALLOW Anywhere
22/tcp ALLOW Anywhere
5347/tcp ALLOW Anywhere
5222/tcp ALLOW Anywhere
443/tcp ALLOW Anywhere
5280/tcp ALLOW Anywhere
80/tcp (v6) ALLOW Anywhere (v6)
21/tcp (v6) ALLOW Anywhere (v6)
22 (v6) ALLOW Anywhere (v6)
22/tcp (v6) ALLOW Anywhere (v6)
5347/tcp (v6) ALLOW Anywhere (v6)
5222/tcp (v6) ALLOW Anywhere (v6)
443/tcp (v6) ALLOW Anywhere (v6)
5280/tcp (v6) ALLOW Anywhere (v6)

This may be related with the expired token.

The main problem seems like a resource issue

The token is expiry time is two hours from the creation of the link. The same link works after restarting the jicofo and prosody servers.

The JVB’s are well within the 60% CPU limit. Jicofo and prosody server is below 20% all the time.

Is this an auto-scaling system? How do you add and remove the additional JVBs?

We do it through cron jobs.

How do you decide to shutdown a JVB?
How do you check active conferences?

As we use it for a K12 school there is a specific schedule and based on the schedule we start the number of required servers in the morning and shutdown them in the evening all at the same time.

prosody can use only one core. Check the CPU load per core

We monitored the server regularly and the load is very low on the server. When there is problem these are the values on the server.

   _x86_64_        (4 CPU)

13:44:56 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
13:44:57 all 1.49 0.00 0.50 0.00 0.00 0.25 0.25 0.00 0.00 97.52
13:44:57 0 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 99.00
13:44:57 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
13:44:57 2 1.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.99
13:44:57 3 4.81 0.00 1.92 0.00 0.00 0.96 0.00 0.00 0.00 92.31

Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
Average: all 1.49 0.00 0.50 0.00 0.00 0.25 0.25 0.00 0.00 97.52
Average: 0 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 99.00
Average: 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
Average: 2 1.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.99
Average: 3 4.81 0.00 1.92 0.00 0.00 0.96 0.00 0.00 0.00 92.31

Tasks: 133 total, 2 running, 131 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.8 us, 0.8 sy, 0.0 ni, 96.1 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
MiB Mem : 7961.8 total, 2528.4 free, 773.0 used, 4660.3 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 6873.1 avail Mem

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND

119696 prosody 20 0 138664 132344 6448 S 10.0 1.6 11:49.73 lua5.2
115264 www-data 20 0 5672916 107100 17176 S 3.3 1.3 5:04.49 nginx
115266 www-data 20 0 238868 32908 17176 S 1.7 0.4 2:25.30 nginx
119733 jicofo 20 0 7122844 304744 29304 S 0.7 3.7 1:57.18 java
122710 root 20 0 11004 3856 3348 R 0.3 0.0 0:00.01 top
1 root 20 0 168896 13044 8496 S 0.0 0.2 0:24.57 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.11 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par+
6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
9 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_perc+
10 root 20 0 0 0 0 S 0.0 0.0 0:00.90 ksoftir+
11 root 20 0 0 0 0 R 0.0 0.0 1:21.62 rcu_sch+
12 root rt 0 0 0 0 S 0.0 0.0 0:01.71 migrati+
13 root -51 0 0 0 0 S 0.0 0.0 0:00.00 idle_in+
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/1