One way media only (NAT issues) after upgrade


#1

One way media only (NAT issues)

After updating to the latest stable versions I started having issues with a specific client that is under heavy firewall setup (hospital). We didn’t change IPs or ports to justify it stopped working. Having that said, could you help me out figure out what is happening?

The concrete problem is that once we enter the room all is good and after a few seconds (less than 5) the other party just gets disconnected and appear to her that she is alone in the room. On the other hand, for me, I can still hear and see everything like the connection is still happening (see screenshots).

Below I put the log I saw on my console, and I also attach a video that the customer that faced the issue recorded from her browser console (errors in the end of the video).

I was not able to reproduce this issue on meet.jit.si environemnt, which means that either you are not using the latest stable or it’s a config thing.

Any hints?

Thanks in advance,
Jose

LOGS FROM CONSOLE (from computer that was hearing and seeing the whole time)

Logger.js:125 [JitsiMeetJS.js] <Object.getGlobalOnErrorHandler>: UnhandledError: null Script: null Line: null Column: null StackTrace: Error: Jingle error: {“reason”:“timeout”,“session”:“JingleSessionPC[p2p=true,initiator=false,sid=f38afd3b0b20]”}
at JingleSessionPC.js:2154
at s.TimedHandler.handler (strophe.js:3405)
at s.TimedHandler.run (strophe.js:2618)
at s.Connection._onIdle (strophe.js:4455)
at s.Connection. (strophe.js:4473)
n @ Logger.js:125
Logger.js:125 [JitsiMeetJS.js] <Object.getGlobalOnErrorHandler>: UnhandledError: null Script: null Line: null Column: null StackTrace: Error: Jingle error: {“reason”:“timeout”,“session”:“JingleSessionPC[p2p=false,initiator=false,sid=b6urrpa92k1sb]”}
at JingleSessionPC.js:2154
at s.TimedHandler.handler (strophe.js:3405)
at s.TimedHandler.run (strophe.js:2618)
at s.Connection._onIdle (strophe.js:4455)
at s.Connection. (strophe.js:4473)
n @ Logger.js:125
Logger.js:125 [JitsiConference.js] <>: Failed to accept incoming P2P Jingle session Object
n @ Logger.js:125

Logger.js:125 [JitsiMeetJS.js] <Object.getGlobalOnErrorHandler>: UnhandledError: null Script: null Line: null Column: null StackTrace: Object
n @ Logger.js:125
Logger.js:125 [JitsiConference.js] <>: Failed to accept incoming Jingle session Object
n @ Logger.js:125
Logger.js:125 [JitsiMeetJS.js] <Object.getGlobalOnErrorHandler>: UnhandledError: null Script: null Line: null Column: null StackTrace: Error: Jingle error: {“reason”:“timeout”,“session”:“JingleSessionPC[p2p=true,initiator=false,sid=f38afd3b0b20]”}
at JingleSessionPC.js:2154
at s.TimedHandler.handler (strophe.js:3405)
at s.TimedHandler.run (strophe.js:2618)
at s.Connection._onIdle (strophe.js:4455)
at s.Connection. (strophe.js:4473)
n @ Logger.js:125
Logger.js:125 [JitsiMeetJS.js] <Object.getGlobalOnErrorHandler>: UnhandledError: null Script: null Line: null Column: null StackTrace: Error: Jingle error: {“reason”:“timeout”,“session”:“JingleSessionPC[p2p=false,initiator=false,sid=b6urrpa92k1sb]”}
at JingleSessionPC.js:2154
at s.TimedHandler.handler (strophe.js:3405)
at s.TimedHandler.run (strophe.js:2618)
at s.Connection._onIdle (strophe.js:4455)
at s.Connection. (strophe.js:4473)
Logger.js:125 [conference.js] <e.value>: CONFERENCE FAILED: conference.setup_failed Error: ICE fail
at r.peerconnection.oniceconnectionstatechange (JingleSessionPC.js:479)
at RTCPeerConnection.peerconnection.oniceconnectionstatechange (TraceablePeerConnection.js:252)

Video screenshots from the customer with issues


#2

Sorry I don’t see the video. Is the problem seeing ‘user is having connectivity issues’?
From the pasted console logs I see ICE failed which is strange. Can you double check whether this is configured correctly: https://github.com/jitsi/jitsi-meet/blob/master/doc/quick-install.md#advanced-configuration I know it is update you did, but check it, just in case.
On meet.jit.si we also had setup turn servers which are very useful in cases of restricted firewalls.


#3

Not quite… before we upgrade Coturn (see issue User is having connectivity issues) we had that connectivity issue. Now, it seems to the user that he is the only one on the call but the other party can clearly see and hear everything (just one way communication).

I could not update the video so I took screenshots. The error logs on the client side are on the last image attached. The strange thing is that despite the errors on my side (the log posted directly in this post) I was able to see and hear her all the time.

Having that said, any hints?


#4

Had you checked the config, this ice failed is not good, especially if you have and turn.
Do 3 way call work for you?


#5

Here you have the video: https://www.useloom.com/share/ea1fdb8f527b44368d01e7d3d3986f72

As for the 3 way call, it does not work. The person behind the firewall does not see or hear (neither can we) but once one of the parties leave, the call transfer to P2P and the behavior is the same as before.

Here is the config: https://sessions.zenklub.com/config.js

Next steps?


#6

What I see is that when the participant that joins from behind the restricted firewall is not always hearing/seeing? Needs to refresh the page and then it works?

What I saw is that you have a configured turn, but no turns and your turn is listening on the default port. For restricted firewalls I would recommend running the turn server on port 443 and having a turns candidate with a valid certificate. This is the way turn servers are configured on meet.jit.si.

{ type = "stun", host = "turnserver.com", port = 443 },
  { type = "turn", host = "turnserver.com", port = 443, transport = "udp" },
  { type = "turns", host = "turnserver.com", port = 443, transport = "tcp" },

And configure the valid certificate for turnserver.com in coturn config.

You said you are having a problem running a 3way call, do you have the same problem when using meet.jit.si?


Turn Server
#7

@damachenco where exactly are you putting that config? Because I looked at your live config and I can’t find those… Does it work if you use NGINX with proxy? Because I was already using 443. Is there a guide on how to optimize for coturn?


#8

This is prosody config, you should already have some because I saw in webrtc internals that a turn server was advertised in your deployment.
I would say dedicate a separate machine for coturn so you can bind it to port 443, here is meet example config

lt-cred-mech
use-auth-secret
static-auth-secret=SomeSecretSharedBetweenProsodyAnbCoturn
realm=some-realm
listening-port=443
cert=some_valid_cert.crt
pkey=some_valid_cert.key

#10

I was able to setup using por 443 and now we don’t see each other… This particular session occurred using the bridge… seems like it failed the p2p.

Do you use any special config on sip-communicator.properties, jicofo, prosody video-bridge?

Errors:

Logger.js:125 [JitsiConference.js] <p2pJingleSession.terminate.reason>: An error occurred while trying to terminate P2P Jingle session {code: undefined, reason: “item-not-found”, session: “JingleSessionPC[p2p=true,initiator=true,sid=e4287637d3ed]”}

When I hangout the call, I see this message:

Logger.js:125 [modules/connectivity/ParticipantConnectionStatus.js] <e.value>: No participant for id: d535d222

See the logs of the other user behind the firewall:


#11

Here is another trace:


#12

Nope, you are still advertising the old turn:

Did you make sure your turn server uses port 443?
Is it using a valid cert for the advertised hostname (conf.zenklub.com)?
Did you modify prosody config to include turn and turns using port 443?
Did you restart prosody?


#13

To test just open chrome://webrtc-internals and then open two tabs, you should have similar to my screenshot above.


#14

Sorry for not being clear, but I am testing in a replica environment… This time Jitsi Meet is running on conf.zenklub.com and the turn server is on turn.zenklub.com

You can test like this: https://conf.zenklub.com/test1234

Certificates are valid (generated with letsencrypt and seem fine). I was able to use TURN properly on some networks setups. The problem only occurs on certain network conditions of our customer.

Prosody configs:

    turncredentials_secret = "XXX";
    turncredentials_host = "turn.zenklub.com";
    turncredentials_port = 443;

I checked the configs in https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/ and everything seemed fine.

I did change the config in prosody as well as did a restart. Any other hints?


#15

Had you enabled the module turncredentials?
Didn’t you have the version of turncredentials module that supports multiple turn servers, I think you do cause the other server was using multiple turn servers

turncredentials_secret = "XXX";
turncredentials = {
  { type = "stun", host = "turnserver.com", port = 443 },
  { type = "turn", host = "turnserver.com", port = 443, transport = "udp" },
  { type = "turns", host = "turnserver.com", port = 443, transport = "tcp" }
};

#16

Verify the use of turn servers using webrtc-internals page.


#17

Where exactly do I configure this? I was not able to find the right documentation. Should be be directly in /usr/lib/prosody/modules/mod_turncredentials.lua or /etc/prosody/conf.d/conf.zenklub.com.cfg.lua ?

I tried both but with no luck… I am currently using a single Turn server… Here is my /usr/lib/prosody/modules/mod_turncredentials.lua contents:

-- XEP-0215 implementation for time-limited turn credentials
-- Copyright (C) 2012-2013 Philipp Hancke
-- This file is MIT/X11 licensed.


local st = require "util.stanza";
local hmac_sha1 = require "util.hashes".hmac_sha1;
local base64 = require "util.encodings".base64;
local os_time = os.time;
local secret = module:get_option_string("turncredentials_secret");
local host = module:get_option_string("turncredentials_host"); -- use ip addresses here to avoid further dns lookup latency
local port = module:get_option_number("turncredentials_port", 3478);
local ttl = module:get_option_number("turncredentials_ttl", 86400);
if not (secret and host) then
    module:log("error", "turncredentials not configured");
    return;
end

module:add_feature("urn:xmpp:extdisco:1");

module:hook("iq-get/host/urn:xmpp:extdisco:1:services", function(event)
    local origin, stanza = event.origin, event.stanza;
    if origin.type ~= "c2s" then
        return;
    end
    local now = os_time() + ttl;
    local userpart = tostring(now);
    local nonce = base64.encode(hmac_sha1(secret, tostring(userpart), false));
    origin.send(st.reply(stanza):tag("services", {xmlns = "urn:xmpp:extdisco:1"})
        :tag("service", { type = "stun", host = host, port = port }):up()
        :tag("service", { type = "turn", host = host, port = port, username = userpart, password = nonce, ttl = ttl}):up()
    );
    return true;
end);

#18

This is what I see:

What I noticed is that only the first party joining receives that. For instance, if I am in the room https://conf.zenklub.com/teste and someone joins, I can see that log above. If it’s the other way around (i join a room with someone there) the iceServers comes empty. I think you can easily test this for yourself.


#19

So you have enabled useStunTurn: true in the main part of config.js, this will set turn servers to the bridge connection, but only those that are turns will be added there. So you are seeing it empty on the bridge peer connection and you see all of them on the p2p peer connection.

The version of turn credentials we are using should be this:


I cannot find it online, but the one above should be fine, maybe we will commit our version in jitsi-meet repository at some point.
The one you are using is using only one host from turncredentials_host, where the other one is getting multiple hosts: local hosts = module:get_option("turncredentials") or {};


#20

The configuration goes in /etc/prosody/conf.d/conf.zenklub.com.cfg.lua
But needs to in the general part, not under some virtual host.


#21

Apparently with the new config file you sent, things seem to be working. Are the logs files as expected now?

24 32

At least I can see something inside iceServer in both ways.

However, I don’t understand why sometimes it still opts for the bridge and not the TURN server or P2P given that it’s just 2 people. Is there a case where this could / should occur?