TURNS transport failure?

turns

#1

In the logs below, I’ve replaced the domains and IPs used with example.org to protect the innocent.

In an environment where previously this was working, users now fail to set-up a working meet.

The environment is configured to use an external coturn server, using TURNS (to circumvent certain network restrictions, by mimicing https-traffic: encrypted TCP data on port 443).

The XMPP server supports external service discovery (when I disable that feature, the Meet javascript console warns me that it’s unavailable, suggesting me to check if things are properly installed - that’s an indication that it’s at the very least being invoked by the meet code).

When I set up a conference between two users, neither user can see the other user.

The first few lines of each session tab in chrome://webrtc-internals/ reports the address of the coturn server that I configured using external service discovery:

https://demo.example.org/ofmeet/testroom, { iceServers: [turns:demo-relay.example.org:443?transport=tcp], iceTransportPolicy: all, bundlePolicy: balanced, rtcpMuxPolicy: require, iceCandidatePoolSize: 0 }, {advanced: [{googHighStartBitrate: {exact: 0}}, {googPayloadPadding: {exact: true}}, {googScreencastMinBitrate: {exact: 400}}, {googCpuOveruseDetection: {exact: true}}, {googCpuOveruseEncodeUsage: {exact: true}}, {googCpuUnderuseThreshold: {exact: 55}}, {googCpuOveruseThreshold: {exact: 85}}, {googEnableVideoSuspendBelowMinBitrate: {exact: true}}]}

The only iceServers entry defines the turns protocol, has the correct hostname (which properly resolves to the intended IP) and defines the tcp transport. That’s what I’d expect.

The same tab reports that these candidates are found:

sdpMid: audio, sdpMLineIndex: 0, candidate: candidate:545250262 1 udp 2122260223 192.168.178.76 35822 typ host generation 0 ufrag daux network-id 1 network-cost 10
sdpMid: audio, sdpMLineIndex: 0, candidate: candidate:1862018854 1 tcp 1518280447 192.168.178.76 9 typ host tcptype active generation 0 ufrag daux network-id 1 network-cost 10
sdpMid: audio, sdpMLineIndex: 0, candidate: candidate:3999088975 1 udp 8331007 198.51.100.1 49248 typ relay raddr 203.0.113.52 rport 52958 generation 0 ufrag daux network-id 1 network-cost 10

Here, 198.51.100.1 is the IP for demo-relay.example.org, and 203.0.113.52 the IP of the end-user.

What strikes me as odd, is that the last candidate defines udp as its transport. I expected a tcp transport to be defined. Also the port number doesn’t match: I expected the port 443 to be used.

Although the server did get some minor changes, I’ve been unsuccessful in restoring functionality by rolling back those changes. I wonder if I missed something, or if something else is going wrong. Did either Meet, Videobridge, or Chrome itself start to behave differently?


#2

The plot thickens…

I hooked on Wireshark, to see what kind of traffic was being sent from the client. To my surprise, I did see TURN related traffic.

This was logged by coturn (again, IP addresses have been replaced. In addition to the addresses above, 192.0.2.104 is used to replace the real IP that is one of the private interfaces on the Amazon EC2 instance that is running both the videobridge as well as coturn).

Jul  9 17:40:44 demo turnserver: 2898: IPv4. tcp or tls connected to: 203.0.113.52:36782
Jul  9 17:40:44 demo turnserver: 2898: session 001000000000000003: realm <example.org> user <>: incoming packet message processed, error 401: Unauthorized
Jul  9 17:40:44 demo turnserver: 2898: IPv4. Local relay addr: 192.0.2.104:52286
Jul  9 17:40:44 demo turnserver: 2898: session 001000000000000003: new, realm=<example.org>, username=<6u1lflde1c%40demo.example.org:1531244435>, lifetime=600, cipher=ECDHE-RSA-AES256-GCM-SHA384, method=TLSv1.2
Jul  9 17:40:44 demo turnserver: 2898: session 001000000000000003: realm <example.org> user <6u1lflde1c%40demo.example.org:1531244435>: incoming packet ALLOCATE processed, success
Jul  9 17:40:45 demo turnserver: 2899: IPv4. tcp or tls connected to: 203.0.113.52:36808
Jul  9 17:40:45 demo turnserver: 2899: session 000000000000000004: realm <example.org> user <>: incoming packet message processed, error 401: Unauthorized
Jul  9 17:40:45 demo turnserver: 2899: IPv4. Local relay addr: 192.0.2.104:58850
Jul  9 17:40:45 demo turnserver: 2899: session 000000000000000004: new, realm=<example.org>, username=<7df3yn92kc%40demo.example.org:1531244444>, lifetime=600, cipher=ECDHE-RSA-AES256-GCM-SHA384, method=TLSv1.2
Jul  9 17:40:45 demo turnserver: 2899: session 000000000000000004: realm <example.org> user <7df3yn92kc%40demo.example.org:1531244444>: incoming packet ALLOCATE processed, success
Jul  9 17:49:44 demo turnserver: 3438: session 001000000000000003: refreshed, realm=<example.org>, username=<6u1lflde1c%40demo.example.org:1531244435>, lifetime=600, cipher=ECDHE-RSA-AES256-GCM-SHA384, method=TLSv1.2
Jul  9 17:49:44 demo turnserver: 3438: session 001000000000000003: realm <example.org> user <6u1lflde1c%40demo.example.org:1531244435>: incoming packet REFRESH processed, success
Jul  9 17:49:45 demo turnserver: 3439: session 000000000000000004: refreshed, realm=<example.org>, username=<7df3yn92kc%40demo.example.org:1531244444>, lifetime=600, cipher=ECDHE-RSA-AES256-GCM-SHA384, method=TLSv1.2
Jul  9 17:49:45 demo turnserver: 3439: session 000000000000000004: realm <example.org> user <7df3yn92kc%40demo.example.org:1531244444>: incoming packet REFRESH processed, success

It’s strange to see that these sessions get established, and ever refreshed, but their users be unable to see/hear each-other.