Question about server setup for school

Install & Config
#1

Hi,

Do you know when you decide to offer help and end up biting more than you can chew? Well, that’s me. I am at the moment in charge of the 2 servers running jitsi meet for my children’s primary school.

How it came to this is possibly a long story, and irrelevant at the moment. I ended setting up two identical servers from Hetzner (not dedicated): 8 CPUS, 32 Gbs RAM. Servers located in Nuremberg, less than 20kms from us.

Each server was set up using GitHub - udima-university/ansible-jitsi-meet: An ansible playbook to install Jitsi Meet in one or more servers with or without user authentication
The only change to the system is that I removed the blue volume bubbles for participants as I heard they are heavy on the server side, and disabled p2p as I understand that it’s only helpful for 2 person meetings.

If you are wondering how I ended up with two servers, well. I set up one server for 1 class - all components running on this class. Then another class wanted to use it, and then another. The server CPU maxed out at 3 classes. So I got another and the classes split the meetings times so that only 3 classes per server are on. Each class has an average of 17 students, and they mostly have cameras on, with the teacher generally speaking and/or sharing the screen.

I have noticed two things over the last 15 days of homeschooling:

  1. The server performance seems to degrade over time. After one week parents started complaining that their children lose connection, or have bad audio etc. However, nothing on the server looks particularly different resource wise. Even with 3 classes simultaneously, memory is mostly unused. So I decided to reboot the servers overnight at 4 to see if it helps. Yesterday I got no complaints, so it might have done the trick.
  2. Sometimes kids simply get kicked out of class (meeting). Or one of the kids stops hearing the others until they refresh the browser to get in once again.

So I have a few questions… I am not a jitsi expert by far, but I am keen to aid development by providing the necessary logs to solve these issues and/or debug the issue on my end. What could I do to understand what’s going on? Both regarding the system degradation over time and the children being kicked off the meeting. I can imagine that I can possibly look at some debug logs (might have to enable them), or run some analysis/stress tool on the server?

Any comments/help/suggestions would be appreciated.

Thanks in advance for the great free software that’s helping these kids have homeschooling during these crazy times.

User disconnects
#2

Your server specs are fine for the load you’re serving. What’s your bandwidth like though? Jitsi is mostly bandwidth-dependent. CPU and Memory use are very modest, but bandwidth is the big demand.

#3

2021-01-26-205358_1696x1613_scrot
2021-01-26-205358_1696x1613_scrot1696×1613 158 KB

As you can see when the meetings took place this morning from the graphs. Network seems just fine.out peaked between 10 and 15 Mbps and in peaked at just over 5 Mbps. That shouldn’t be the problem as far as I understand it.

#4

What is the prosody version that you use?

#5

That’s XMPP comm server, right?

root@eckental-jitsi-2:~# apt-cache policy prosody
prosody:
  Installed: 0.11.2-1
  Candidate: 0.11.2-1
root@eckental-jitsi-2:~# apt-cache policy jitsi-meet-prosody
jitsi-meet-prosody:
  Installed: 1.0.4628-1
  Candidate: 1.0.4628-1
#6

  • If all video streams go to all clients, this may cause some issues on the client sides. Check ChannelLastN for this

  • The yaml file seems that it installs jibri too. Do you need recording or livestream? If no, you can disable or uninstall jibri

  • Is this Debian 9 Stretch or has it Java8?

#7

I’m also running a Jitsi Server for my school. Hetzner AX51-NVME. It’s used by about 380 participants, whereas only the teacher has video turned on, in order to reduce the traffic on the client sides (not every pupil has good enough connection). It’s running stable.

It’s important to have a dedicated server and not a vServer!
How fresh is your install. I had the problem that the updated installation from May 2019 did contain some wrong or missing settings in the config files, also regarding the websocket configuration. So I either had to do a lot of manual reconfiguration or a fresh install. I did the latter. So you might want to start from scratch, with a new Linux installation and fresh nginx and jitsi installation:

I’m running it on Debian 10.
I tuned a few settings, like

  • disable lots of indicators in the interface.js
  • disabled processing on the server side, disableAudioLevels
  • disabled p2p
  • joining muted and without video.
  • ChannelLastN set to 4
  • I disabled Simulcast to have better constant video quality but more traffic (not sutiable for you)

In a different post I had some nginx limits which caused lots of disconnects with many participants, maybe you also want to check the nginx log files

#8

I have jibri disabled, running java 8 on debian 10.

#9

openjdk-11 may be better. IIRC there are some issues related with using java8 on Debian Buster

#10

Thanks for the tip - I will try first all changes suggested by @frank23 before reinstalling java.

#11

Thanks for the post @frank23. I cannot have ChannelLastN set. The teachers (primary school here in Germany) are keen on seeing all kids while they do some independent work in front of the camera. I haven’t seen the errors you mention but since we are seeing some participants dropping, I will increase the nginx limits anyway. I will report back.

#12

Although I just noticed that the official guide recommends java 8 even on debian 10…

#13
#14

Thank you for the references.