RTT Times and overall call stability

Hi, we are running jitsi on AWS autoscaling clusters and for the most part everything is very stable. From time to time, we see a spike in RTT which affects all users- even if they are not participating on the same network. When they spike we’ve seen the RTT range anywhere from 5K-15K ms. The last time it happened we had 20 conferences with 10 people each that ran perfectly for 3 hours and then all of a sudden the connections went poor and RTT spiked. Once a few callers dropped everything stabilized. The only thing I saw on cloudwatch was our JVB memory it 4.5GB and our network in/out traffic dropped but CPU usage stayed consistent at 10%.
Any thoughts as to what can cause this?

noisy neighbor maybe? do you have stats for stolen cpu?

Hi damencho, thanks for addressing this. Here are the stats from cloudwatch. I don’t think its a tenant issue- the issue happened concurrently in two separate data centers- the graphs are practically identical. There is a slight peak and then a drop where the issue occurred, just before 12pm.

This sounds like a network issue. The first graph shows network drops and that is reflected in jvb cpu usage, no packets arrive nothing to process …

@damencho thanks! Do you think these would be user network issues or AWS network issues?

Can be both … we have seen in the past few months reports for problems and at the end it turns out it was Verizon or Comcast issues. If you have some stats (like callstats) you can check participants seeing packet loss and check their networks and see it may come out they are from the same provider …

Thanks! One last question. I did a bit of digging and it turns out that most, but not all users were on the same corporate network. If the majority of the users begin to experience long RTT can that degrade the rest of the users who otherwise would have had good connections? We saw a lot of red!

Well, bad network from a user will not affect the rest of incoming videos.
Of course those watching the video from the user with bad network probably will have problems seeing high quality video with good frame rate…

Thanks!