IceUdpTransportManager attempts to bind ports from a "tracked" starting value, in code that starts here:
The problem is that if multiple threads enter this code in parallel, all can start the search from the same value. Under sustained conference creation load, this ultimately leads to a bind failure, because there is an inner retry loop that fails after 50 attempts (it can be bumped to 100, but that 100 limit is hardcoded in NetworkAddressManagerServiceImpl.createIceStream). If the retry loop is unsuccessful, conference creation fails.
I can get a failure with a conference creation rate of 9/sec over 10 seconds on an EC2 m4.large instance.
Are there any plans to improve this port assignment code? We are experimenting with a synchronized block around the portTracker code mentioned above. That eliminates the errors, but results in a port assignment time that is a order of magnitude slower under load. We may consider using multiple port trackers or some kind of port free list, but we wanted to see if anyone else had run into this.
Reply to this email directly or view it on GitHub: