WARNING: Suspiciously high rtt value

Hi, sometimes we get this error when there are several people connected.

WARNING: Suspiciously high rtt value: 7167.816292 ms, remote processing delay was PT0.608001708S (39846), srSentTime was 2021-11-08T08:05:48.986182Z, received time was 2021-11-08T08:05:56.762Z

Can you explain me what does it means exactly ? Is it RTT between client and JVB ? Do you have any idea what we should monitor to fix this warning.

Thank you

Hello
I’m a bit late to reply to you but it’s for a good reason: when you posted this I did not know. I still don’t but I’m a bit less sure about my ignorance.

I have taken a look because I’m curious but it was a bit difficult to track. I think it’s in jitsi-media-transform, file src/main/kotlin/org/jitsi/nlj/stats/EndpointConnectionStats.kt, line circa 150-170. Jitsi-media-transform is one of those libraries where Jitsi videobridge developers are putting ‘common functions’, its code is included in the bridge.

I think that RTT is related to TCC (Transport-Wide Congestion control: don’t ask me what is the exact meaning of ‘Wide’ I absolutely have no idea).

My understanding of TCC is that periodically the sender (aka JVB) sends a special packet summarizing all the normal data packets that were sent since previous TCC packet. The receiver then searches in his memory what has happened to these packets and sends back a reply giving for each of these packets a concise information along the lines of ‘not received’, ‘received normally’, ‘too late’ (or too early but it’s probably not a common occurrence).

Now this TCC packet is one of the few packets sent that needs a reply and it’s about these packets that Jvb calculate a RTT (Round Trip Time). It is my understanding that this round trip time is used to limit the frequency of TCC packets (since these are ping pong packets their use must be limited else it could actually harm the real data use in case of loaded networks).
You can see in the code that the limit is 7 seconds. What could be the reason a client should take the full enormous time of 7 seconds to reply to a single packet ? IMO one of the principal possibilities is that the client is overwhelmed and has too much to do. In my tests I have seen that even moderate activity on a PC can produce a significant jump in RTT (not that much though; certainly not 7 seconds).

Maybe some complicated network configuration (across the world) could account for one or 2 seconds also.

What could you monitor ? IMO using the logs is awful for that, but maybe you could figure out some way to find if some clients exhibit this behavior more than others: in this case the ‘overwhelmed client’ could be the preferred hypothesis.
If all clients are showing this problem at some point of time, but not normally, it could be a network problem. I hope that you are monitoring your JVB(s) so the possibility of an overwhelmed JVB is not thinkable.
If you are using Octo, the general network problem is much more likely of course.

Well, that’s all I can say and maybe some part is wrong, I don’t belong to the ’ highly qualified and supremely knowledgeable with all things Jitsi’ people posting in this forum.

You can find the RFC here and the code is here, hope that you will gather knowledge from these readings.

1 Like

Thank you very much for your research. Indeed‚ you gave some interesting track. I confirm we have several jvb behind octo and also behind coturn and network firewall. I’m curious if RTT can be impacted when users take times to connect to a conference e.g jicofo/ prosody constraint and in case of high numbers of users connect at the same time process queue increase this RTT.

very high network congestion is not very likely with UDP, you have good or bad links. With TCP (that is, when using coturn) it’s a different story. A single defective network adapter could be a real pain. I don’t know if you have a permanent setup of it’s all dynamically allocated JVBs and coturns from something like AWS, if the latter a bad network adapter is not likely to cause recurrent problems.
Beyond these network pearls of wisdom that are known to everyone I’m afraid that I don’t know more to help you and I believe that it’s not really possible to debug complex networks remotely.