Performance drop after some time and how to boost performance?

after a couple of days of usage (around 3) we can observe a decline in call quality in our docker hosted jitsi instance.
A restart of the docker services puts Jitsi in a good state again, making a call with 30 persons flawless, where before a call with 5 persons struggled / nearly unusable.

The 16 core CPU has only an usage of around 7.0 in the 5 min average so there is plenty of headroom available. Memory usage has only an usage of 4GB out of available 64GB.

What could be the reason for this problem and how one could one fix this?