After 2 months with intensive use of Jitsi I came to the conclusion that the main problems seems to be the participants’ hardware. We had meetings with 20 people or so were only 1 person had video turned on and all others were in audio-only mode. As soon as more than 13 odd people turned video on we had significant lag and choppy audio. I read somewhere that the limit for others was 30 people, but I cannot confirm this.
Thus, the problem was never the server. I used a VM with 4 dedicated Ryzen Epic Cores and 16GB of RAM. Bandwith was also sufficient. Looking at the stats during such meetings the CPU-usage was at around 20% and network utilization was at 70 Mbits total (up/down) when 13 people had video turned on. So there was still a lot of headroom left.
To me it seems that many systems cannot cope with the number of participants, the more have video turned on the higher the cpu-load on each system. I guess that one problem is that VP8 doesn’t have any hardware acceleration and the users’ CPUs juts bottleneck.