Jvb sudden drop then recover

We are seeing this issue from time to time in our jvb deployments.

When this happened, it seems like jvb’s data sockets suddenly stopped working. However the signaling connection remains as I can still see people joining and leaving the meeting. I saw lots of “timeout for pair” appear in the log when this happened. Both udp and tcp pairs. Then the users started to leave the meeting (because everything will be freeze at the time). And as people dropping off, jvb somehow recovered by itself. We usually saw jvb back to normal after few minutes.

We run jvb in docker on aws ec2. We are on jvb1. Because it only happens on the data sockets, maybe this is related to ice4j?

We set SO_RCVBUF as 33554432. We also confirmed there’s no buffer errors with netstat -su. Our docker’s run in bridge mode, so I installed net-tools and run netstat inside the container. Jvb’s max open files is set as 65000, so that should not be a problem either.

We are really struggled and running out of idea. Any thoughts are appreciated. Thanks!