[jitsi-dev] How to add remote-candidates to colibri with trickle ICE using REST?


#1

I am trying to add remote candidates using REST. This is not covered in "doc/rest-videobridge.md" [the document should continue after patching the sdp].

No matter, if I send "a=ice-options:trickle" in my initial sdp or not, Chrome/webrtc will generate the sdp answer first, and then the candidates. Sure, I could work around this and wait until "oniceconnectionstatechange == completed", then stick the candidates into the sdp and send everything together, but I guess Jitsi is able to handle trickle ICE (as Emil states in several interviews).

So let's say we have set up a colibri conference, sdp offer and answer has been exchanged, and Chrome has all Jitsi candidates (which I sent with the initial sdp). Chrome is sending candidate by candidate, and I must patch these into the colibri conference.

Problem:

- When I add candidates into "transport.candidates" list, they don't show up in colibri at all. This makes sense, as candidates list seems to contain local Jitsi candidates. (I still see them in the log though, see below.)

- I saw in code [this seems nowhere documented] that there is a "transport.remote-candidate" property. If I stick the incoming candidates into this property, these show up in colibri, but Jitsi sends no streams. Also, "remote-candidate" is a single property, not a list, and on each incoming candidate I overwrite the previous one. I thought Jitsi might keep the most valuable candidate and use it, which would make perfect sense, but it does not. It keeps the last (and typically most invaluable candidate) [at least colibri say that] and still doesn't send streams.

So what's the secret? How do I need to patch remote candidates into an existing colibri session so that Jitsi uses them.

NB: I can see that "TransportManager.startConnectivityEstablishment" is run once for the sdp answer from Chrome, but ICE Agent status stays "WAITING", as there are no candidates yet. I can also see candidates coming in when I patch them into colibri, but the log keeps saying "set use-candidate false for pair..." and "Receive STUN checks before our ICE has started", and still no streams.


#2

It looks to me as if Candidates cannot be trickled at all using REST. Is this the fact?

Looking at your code in JSONDeserializer, I can see that

- Jitsi creates a new transportIQ when the NAMESPACE is provided (which is the case when the original sdp andwer is received),

- otherwise is sets transportIQ as null, which would mean any transport information that is patched into colibri after the initial one *IS LOST*, no matter how I patch it.

Is there a trick or does the REST interface not support ICE trickle?

Object xmlns = transport.get(JSONSerializer.XMLNS);
Object fingerprints = transport.get(JSONSerializer.FINGERPRINTS);
Object candidateList = transport.get(JSONSerializer.CANDIDATE_LIST);
Object remoteCandidate
    = transport.get(RemoteCandidatePacketExtension.ELEMENT_NAME);
Object rtcpMux = transport.get(RtcpmuxPacketExtension.ELEMENT_NAME);

if (IceUdpTransportPacketExtension.NAMESPACE.equals(xmlns))
    transportIQ = new IceUdpTransportPacketExtension();
else if (RawUdpTransportPacketExtension.NAMESPACE.equals(xmlns))
    transportIQ = new RawUdpTransportPacketExtension();
else
    transportIQ = null;


#3

I am able to add candidates with trickle ice and REST.

e.g. candidate json body:

var candidatePatch = {
          'foundation': parsedCandidate.foundation,
          'component': parsedCandidate.component,
          'priority': parsedCandidate.priority,
          'ip': parsedCandidate.ip,
          'port': parsedCandidate.port,
          'generation': parsedCandidate.generation,
          'type': parsedCandidate.type,
          'rel-addr': parsedCandidate.raddr,
          'rel-port': parsedCandidate.rport,
          'protocol': parsedCandidate.protocol,
          'network' : parsedCandidate.network,
          'id' : parsedCandidate.id
        };
        var patch: any = {
          id: msg.streams.publisher,
          'channel-bundles': [{
            'id': msg.streams.channelBundleId,
            transport: {
              'xmlns': 'urn:xmpp:jingle:transports:ice-udp:1',
              'setup': 'passive',
              'remote-candidate': candidatePatch
            }
          }]
        };

Also look at https://github.com/jitsi/lib-jitsi-meet/blob/master/
modules/xmpp/JingleSessionPC.js#L330
and I recommend to replace the answerer offer with "passive"

          // XXX Videobridge is the (SDP) offerer and WebRTC (e.g. Chrome)
is the
          // answerer (as orchestrated by Jicofo). In accord with
          // http://tools.ietf.org/html/rfc5245#section-5.2 and because
both peers
          // are ICE FULL agents, Videobridge will take on the controlling
role and
          // WebRTC will take on the controlled role. In accord with
          // https://tools.ietf.org/html/rfc5763#section-5, Videobridge
will use the
          // setup attribute value of setup:actpass and WebRTC will be
allowed to
          // choose either the setup attribute value of setup:active or
          // setup:passive. Chrome will by default choose setup:active
because it is
          // RECOMMENDED by the respective RFC since setup:passive adds
additional
          // latency. The case of setup:active allows WebRTC to send a DTLS
          // ClientHello as soon as an ICE connectivity check of its
succeeds.
          // Unfortunately, Videobridge will be unable to respond
immediately because
          // may not have WebRTC's answer or may have not completed the ICE
          // connectivity establishment. Even more unfortunate is that in
the
          // described scenario Chrome's DTLS implementation will insist on
          // retransmitting its ClientHello after a second (the time is in
accord
          // with the respective RFC) and will thus cause the whole
connection
          // establishment to exceed at least 1 second. To work around
Chrome's
          // idiosyncracy, don't allow it to send a ClientHello i.e. change
its
          // default choice of setup:active to setup:passive.

···

2016-10-08 2:35 GMT+02:00 Oliver Hausler <oliver@closeup.cc>:

I am trying to add remote candidates using REST. This is not covered in
"doc/rest-videobridge.md" [the document should continue after patching
the sdp].

No matter, if I send "a=ice-options:trickle" in my initial sdp or not,
Chrome/webrtc will generate the sdp answer first, and then the candidates.
Sure, I could work around this and wait until "oniceconnectionstatechange
== completed", then stick the candidates into the sdp and send everything
together, but I guess Jitsi is able to handle trickle ICE (as Emil states
in several interviews).

So let's say we have set up a colibri conference, sdp offer and answer has
been exchanged, and Chrome has all Jitsi candidates (which I sent with the
initial sdp). Chrome is sending candidate by candidate, and I must patch
these into the colibri conference.

Problem:

- When I add candidates into "transport.candidates" list, they
don't show up in colibri at all. This makes sense, as candidates list seems
to contain local Jitsi candidates. (I still see them in the log though, see
below.)

- I saw in code [this seems nowhere documented] that there is a
"transport.remote-candidate" property. If I stick the incoming candidates
into this property, these show up in colibri, but Jitsi sends no streams.
Also, "remote-candidate" is a single property, not a list, and on each
incoming candidate I overwrite the previous one. I thought Jitsi might keep
the most valuable candidate and use it, which would make perfect sense, but
it does not. It keeps the last (and typically most invaluable candidate)
[at least colibri say that] and still doesn't send streams.

So what's the secret? How do I need to patch remote candidates into an
existing colibri session so that Jitsi uses them.

NB: I can see that "TransportManager.startConnectivityEstablishment" is
run once for the sdp answer from Chrome, but ICE Agent status stays
"WAITING", as there are no candidates yet. I can also see candidates coming
in when I patch them into colibri, but the log keeps saying "set
use-candidate false for pair…" and "Receive STUN checks before our ICE has
started", and still no streams.

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#4

// Once again because the ICE Agent does not support adding
// candidates after the connectivity establishment has been started
// and because multiple transport-info JingleIQs may be used to send
// the whole set of transport candidates from the remote peer to the
// local peer, do not really start the connectivity establishment
// until we have at least one remote candidate per ICE Component.

This comment it ICE Agent makes me believe I am right. And Jitsi is sending streams when I send a complete SDP with all candidates.

Can anybody comment what's the status here, please.


#5

Hi Oliver,

I see that in ice4j's Agent class there is a "trickle" flag:

/**
* Determines whether this agent should perform trickling.
*/
private boolean trickle = false;

Maybe you can try to play with this flag (I never tried it though).

Cheers,

Matteo

···

________________________________
From: dev <dev-bounces@jitsi.org> on behalf of Oliver Hausler <oliver@closeup.cc>
Sent: Sunday, October 9, 2016 6:19 PM
To: Jitsi Dev Mailinglist (dev@jitsi.org)
Subject: Re: [jitsi-dev] How to add remote-candidates to colibri with trickle ICE using REST?

// Once again because the ICE Agent does not support adding
// candidates after the connectivity establishment has been started
// and because multiple transport-info JingleIQs may be used to send
// the whole set of transport candidates from the remote peer to the
// local peer, do not really start the connectivity establishment
// until we have at least one remote candidate per ICE Component.

This comment it ICE Agent makes me believe I am right. And Jitsi is sending streams when I send a complete SDP with all candidates.

Can anybody comment what's the status here, please.


#6

I am able to add candidates with trickle ice and REST.

e.g. candidate json body:

var candidatePatch = {
          'foundation': parsedCandidate.foundation,
          'component': parsedCandidate.component,
          'priority': parsedCandidate.priority,
          'ip': parsedCandidate.ip,
          'port': parsedCandidate.port,
          'generation': parsedCandidate.generation,
          'type': parsedCandidate.type,
          'rel-addr': parsedCandidate.raddr,
          'rel-port': parsedCandidate.rport,
          'protocol': parsedCandidate.protocol,
          'network' : parsedCandidate.network,
          'id' : parsedCandidate.id
        };
        var patch: any = {
          id: msg.streams.publisher,
          'channel-bundles': [{
            'id': msg.streams.channelBundleId,
            transport: {
              'xmlns': 'urn:xmpp:jingle:transports:ice-udp:1',
              'setup': 'passive',
              'remote-candidate': candidatePatch
            }
          }]
        };

Also look at
https://github.com/jitsi/lib-jitsi-meet/blob/master/modules/xmpp/JingleSessionPC.js#L330
and I recommend to replace the answerer offer with "passive"

          // XXX Videobridge is the (SDP) offerer and WebRTC (e.g. Chrome)
is the
          // answerer (as orchestrated by Jicofo). In accord with
          // http://tools.ietf.org/html/rfc5245#section-5.2 and because
both peers
          // are ICE FULL agents, Videobridge will take on the controlling
role and
          // WebRTC will take on the controlled role. In accord with
          // https://tools.ietf.org/html/rfc5763#section-5, Videobridge
will use the
          // setup attribute value of setup:actpass and WebRTC will be
allowed to
          // choose either the setup attribute value of setup:active or
          // setup:passive. Chrome will by default choose setup:active
because it is
          // RECOMMENDED by the respective RFC since setup:passive adds
additional
          // latency. The case of setup:active allows WebRTC to send a DTLS
          // ClientHello as soon as an ICE connectivity check of its
succeeds.
          // Unfortunately, Videobridge will be unable to respond
immediately because
          // may not have WebRTC's answer or may have not completed the ICE
          // connectivity establishment. Even more unfortunate is that in
the
          // described scenario Chrome's DTLS implementation will insist on
          // retransmitting its ClientHello after a second (the time is in
accord
          // with the respective RFC) and will thus cause the whole
connection
          // establishment to exceed at least 1 second. To work around
Chrome's
          // idiosyncracy, don't allow it to send a ClientHello i.e. change
its
          // default choice of setup:active to setup:passive.

···

2016-10-09 18:19 GMT+02:00 Oliver Hausler <oliver@closeup.cc>:

*// Once again because the ICE Agent does not support adding // candidates
after the connectivity establishment has been started // and because
multiple transport-info JingleIQs may be used to send // the whole set of
transport candidates from the remote peer to the // local peer, do not
really start the connectivity establishment // until we have at least one
remote candidate per ICE Component.*

This comment it ICE Agent makes me believe I am right. And Jitsi is
sending streams when I send a complete SDP with all candidates.

Can anybody comment what's the status here, please.

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#7

Thank you Philipp, I just noticed you sent a response to this.

Interesting, you patch each candidate into 'remote-candidate' separately. I will try this tomorrow.

···

From: dev [mailto:dev-bounces@jitsi.org] On Behalf Of Philipp Seeser
Sent: Monday, October 10, 2016 03:02
To: Jitsi Developers <dev@jitsi.org>
Subject: Re: [jitsi-dev] How to add remote-candidates to colibri with trickle ICE using REST?

I am able to add candidates with trickle ice and REST.

e.g. candidate json body:

var candidatePatch = {
          'foundation': parsedCandidate.foundation,
          'component': parsedCandidate.component,
          'priority': parsedCandidate.priority,
          'ip': parsedCandidate.ip,
          'port': parsedCandidate.port,
          'generation': parsedCandidate.generation,
          'type': parsedCandidate.type,
          'rel-addr': parsedCandidate.raddr,
          'rel-port': parsedCandidate.rport,
          'protocol': parsedCandidate.protocol,
          'network' : parsedCandidate.network,
          'id' : parsedCandidate.id
        };
        var patch: any = {
          id: msg.streams.publisher,
          'channel-bundles': [{
            'id': msg.streams.channelBundleId,
            transport: {
              'xmlns': 'urn:xmpp:jingle:transports:ice-udp:1',
              'setup': 'passive',
              'remote-candidate': candidatePatch
            }
          }]
        };

Also look at https://github.com/jitsi/lib-jitsi-meet/blob/master/modules/xmpp/JingleSessionPC.js#L330
and I recommend to replace the answerer offer with "passive"

          // XXX Videobridge is the (SDP) offerer and WebRTC (e.g. Chrome) is the
          // answerer (as orchestrated by Jicofo). In accord with
          // http://tools.ietf.org/html/rfc5245#section-5.2 and because both peers
          // are ICE FULL agents, Videobridge will take on the controlling role and
          // WebRTC will take on the controlled role. In accord with
          // https://tools.ietf.org/html/rfc5763#section-5, Videobridge will use the
          // setup attribute value of setup:actpass and WebRTC will be allowed to
          // choose either the setup attribute value of setup:active or
          // setup:passive. Chrome will by default choose setup:active because it is
          // RECOMMENDED by the respective RFC since setup:passive adds additional
          // latency. The case of setup:active allows WebRTC to send a DTLS
          // ClientHello as soon as an ICE connectivity check of its succeeds.
          // Unfortunately, Videobridge will be unable to respond immediately because
          // may not have WebRTC's answer or may have not completed the ICE
          // connectivity establishment. Even more unfortunate is that in the
          // described scenario Chrome's DTLS implementation will insist on
          // retransmitting its ClientHello after a second (the time is in accord
          // with the respective RFC) and will thus cause the whole connection
          // establishment to exceed at least 1 second. To work around Chrome's
          // idiosyncracy, don't allow it to send a ClientHello i.e. change its
          // default choice of setup:active to setup:passive.

2016-10-09 18:19 GMT+02:00 Oliver Hausler <oliver@closeup.cc<mailto:oliver@closeup.cc>>:
// Once again because the ICE Agent does not support adding
// candidates after the connectivity establishment has been started
// and because multiple transport-info JingleIQs may be used to send
// the whole set of transport candidates from the remote peer to the
// local peer, do not really start the connectivity establishment
// until we have at least one remote candidate per ICE Component.

This comment it ICE Agent makes me believe I am right. And Jitsi is sending streams when I send a complete SDP with all candidates.

Can anybody comment what's the status here, please.

_______________________________________________
dev mailing list
dev@jitsi.org<mailto:dev@jitsi.org>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#8

I am sorry I am chiming in so late but, why exactly do you need to send any
additional candidates to the bridge in the first place?

···

On Sunday, 16 October 2016, Oliver Hausler <oliver@closeup.cc> wrote:

Thank you Philipp, I just noticed you sent a response to this.

Interesting, you patch each candidate into 'remote-candidate' separately.
I will try this tomorrow.

*From:* dev [mailto:dev-bounces@jitsi.org
<javascript:_e(%7B%7D,'cvml','dev-bounces@jitsi.org');>] *On Behalf Of *Philipp
Seeser
*Sent:* Monday, October 10, 2016 03:02
*To:* Jitsi Developers <dev@jitsi.org
<javascript:_e(%7B%7D,'cvml','dev@jitsi.org');>>
*Subject:* Re: [jitsi-dev] How to add remote-candidates to colibri with
trickle ICE using REST?

I am able to add candidates with trickle ice and REST.

e.g. candidate json body:

var candidatePatch = {

          'foundation': parsedCandidate.foundation,

          'component': parsedCandidate.component,

          'priority': parsedCandidate.priority,

          'ip': parsedCandidate.ip,

          'port': parsedCandidate.port,

          'generation': parsedCandidate.generation,

          'type': parsedCandidate.type,

          'rel-addr': parsedCandidate.raddr,

          'rel-port': parsedCandidate.rport,

          'protocol': parsedCandidate.protocol,

          'network' : parsedCandidate.network,

          'id' : parsedCandidate.id

        };

        var patch: any = {

          id: msg.streams.publisher,

          'channel-bundles': [{

            'id': msg.streams.channelBundleId,

            transport: {

              'xmlns': 'urn:xmpp:jingle:transports:ice-udp:1',

              'setup': 'passive',

              'remote-candidate': candidatePatch

            }

          }]

        };

Also look at https://github.com/jitsi/lib-jitsi-meet/blob/master/
modules/xmpp/JingleSessionPC.js#L330

and I recommend to replace the answerer offer with "passive"

          // XXX Videobridge is the (SDP) offerer and WebRTC (e.g. Chrome)
is the
          // answerer (as orchestrated by Jicofo). In accord with
          // http://tools.ietf.org/html/rfc5245#section-5.2 and because
both peers
          // are ICE FULL agents, Videobridge will take on the controlling
role and
          // WebRTC will take on the controlled role. In accord with
          // https://tools.ietf.org/html/rfc5763#section-5, Videobridge
will use the
          // setup attribute value of setup:actpass and WebRTC will be
allowed to
          // choose either the setup attribute value of setup:active or
          // setup:passive. Chrome will by default choose setup:active
because it is
          // RECOMMENDED by the respective RFC since setup:passive adds
additional
          // latency. The case of setup:active allows WebRTC to send a DTLS
          // ClientHello as soon as an ICE connectivity check of its
succeeds.
          // Unfortunately, Videobridge will be unable to respond
immediately because
          // may not have WebRTC's answer or may have not completed the ICE
          // connectivity establishment. Even more unfortunate is that in
the
          // described scenario Chrome's DTLS implementation will insist on
          // retransmitting its ClientHello after a second (the time is in
accord
          // with the respective RFC) and will thus cause the whole
connection
          // establishment to exceed at least 1 second. To work around
Chrome's
          // idiosyncracy, don't allow it to send a ClientHello i.e.
change its
          // default choice of setup:active to setup:passive.

2016-10-09 18:19 GMT+02:00 Oliver Hausler <oliver@closeup.cc
<javascript:_e(%7B%7D,'cvml','oliver@closeup.cc');>>:

*// Once again because the ICE Agent does not support adding // candidates
after the connectivity establishment has been started // and because
multiple transport-info JingleIQs may be used to send // the whole set of
transport candidates from the remote peer to the // local peer, do not
really start the connectivity establishment // until we have at least one
remote candidate per ICE Component.*

This comment it ICE Agent makes me believe I am right. And Jitsi is
sending streams when I send a complete SDP with all candidates.

Can anybody comment what's the status here, please.

_______________________________________________
dev mailing list
dev@jitsi.org <javascript:_e(%7B%7D,'cvml','dev@jitsi.org');>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
sent from my mobile


#9

Emil, Philipp, sorry for the late response.

@Philipp: It doesn't work for me with passive. Can you see any issues with my logs below? Might be entirely unrelated as I see DTLS warnings in the Jitsi log. I posted this separately: "Unknown DTLS handshake message and no streams from Jitsi (REST)"

@Emil: I assume we should be using ICE trickle as this is standard in WebRTC, so I tried to do that:

- I am creating an OFFER SDP which contains all Candidates discovered by Jitsi (no ICE trickle, makes no sense here as we have candidates in colibri already).

In Chrome I have 2 options:

1. ICE trickle: I listen to createAnswerOnSuccess(), send the ANSWER SDP to Jitsi, wait for onIceCandidate() and send each Candidate as it is discovered.

2. No ICE trickle: I listen to iceGatheringStateChange() == complete, then send the full ANSWER SDP with all Candidates.

I thought 1. would be the preferred way when using WebRTC/libjingle, so I followed what Philipp Seeser suggested and it looks like Jitsi accepts the "remote-candidate"(s) as it replays it:

"channel-bundles": [{
                                "id": "AKQQeYSlhEwltx69_ZWinO8|||||N-N77FcLReCCe5yoieF_7A",
                                "transport": {
                                                "xmlns": "urn:xmpp:jingle:transports:ice-udp:1",
                                                "rtcp-mux": true,
                                                "remote-candidate": {
                                                                "component": 1,
                                                                "protocol": "udp",
                                                                "priority": 1685987071,
                                                                "ip": "68.96.117.185",
                                                                "port": 56117,
                                                                "type": "srflx",
                                                                "generation": 0,
                                                                "foundation": "1855365925",
                                                                "network": 1
                                                },
                                                "setup": "passive"
                                }
                }]

I assume I must add each Candidate like that, and Jitsi picks the one which it will use. Because the last 2 candidates I am receiving use throw-away port 9 [why is Chrome generating these at all?], if Jitsi would use the last Candidate, it wouldn't work, but I believe it's smart enough and uses the best candidate, even though this isn't reflected in colibri (except for the immediate response, there is no change in colibri thereafter, so I cannot prove this).

Finally, I see a slightly distinct result between 1 and 2, though, when looking at "GET /colibri/conferences/{conferenceId}":

For 1. I see 2 audio channels per endpoint:

  "contents": [
    {
      "channels": [
        {
          "endpoint": "AKQQeYSlhEwltx69_ZWinO8|||||AKPDvx2Egk8dqVFUHTgkE_A",
          "channel-bundle-id": "AKQQeYSlhEwltx69_ZWinO8|||||AKPDvx2Egk8dqVFUHTgkE_A",
          "sources": [
3204522869
          ],
          "rtp-level-relay-type": "mixer",
          "expire": 300,
          "initiator": true,
          "ssrcs": [
            3204522869,
336257926
          ],
          "id": "61baaec9d48429b4",
          "direction": "sendrecv"
        },
        {
          "endpoint": "AKQQeYSlhEwltx69_ZWinO8|||||APkpD2Im3kfzq2hM1o49z3E",
          "channel-bundle-id": "AKQQeYSlhEwltx69_ZWinO8|||||APkpD2Im3kfzq2hM1o49z3E",
          "sources": [
1722705425
          ],
         "rtp-level-relay-type": "mixer",
          "expire": 300,
          "initiator": true,
          "ssrcs": [
            1722705425,
3673416731
          ],
          "id": "b41ab150c9739171",
          "direction": "sendrecv"
        }
      ],
     "name": "audio"
    },

For 2. one of the ssrcs seems to be missing for one of the channels (so Philipps approach with ICE trickle seems to be the better working):

  "contents": [
    {
      "channels": [
        {
          "endpoint": "AKQQeYSlhEwltx69_ZWinO8|||||MEIiSaRqTGep7C2cDR42sQ",
          "channel-bundle-id": "AKQQeYSlhEwltx69_ZWinO8|||||MEIiSaRqTGep7C2cDR42sQ",
          "sources": [
916465418
          ],
          "rtp-level-relay-type": "mixer",
          "expire": 300,
          "initiator": true,
          "ssrcs": [
3268560502
          ],
          "id": "700d2d43eae73a28",
          "direction": "sendrecv"
        },
        {
          "endpoint": "AKQQeYSlhEwltx69_ZWinO8|||||ANZL23OQrkVkqzUZ-XMyyPk",
          "channel-bundle-id": "AKQQeYSlhEwltx69_ZWinO8|||||ANZL23OQrkVkqzUZ-XMyyPk",
          "sources": [
814905801
          ],
          "rtp-level-relay-type": "mixer",
          "expire": 300,
          "initiator": true,
          "ssrcs": [
            814905801,
2637946124
          ],
          "id": "ba07908e22e3b2a2",
          "direction": "sendrecv"
        }
      ],
      "name": "audio"
    },

···

From: dev [mailto:dev-bounces@jitsi.org] On Behalf Of Emil Ivov
Sent: Sunday, October 16, 2016 22:30
To: Jitsi Developers <dev@jitsi.org>
Subject: Re: [jitsi-dev] How to add remote-candidates to colibri with trickle ICE using REST?

I am sorry I am chiming in so late but, why exactly do you need to send any additional candidates to the bridge in the first place?

On Sunday, 16 October 2016, Oliver Hausler <oliver@closeup.cc<mailto:oliver@closeup.cc>> wrote:
Thank you Philipp, I just noticed you sent a response to this.

Interesting, you patch each candidate into 'remote-candidate' separately. I will try this tomorrow.

From: dev [mailto:dev-bounces@jitsi.org<javascript:_e(%7B%7D,'cvml','dev-bounces@jitsi.org');>] On Behalf Of Philipp Seeser
Sent: Monday, October 10, 2016 03:02
To: Jitsi Developers <dev@jitsi.org<javascript:_e(%7B%7D,'cvml','dev@jitsi.org');>>
Subject: Re: [jitsi-dev] How to add remote-candidates to colibri with trickle ICE using REST?

I am able to add candidates with trickle ice and REST.

e.g. candidate json body:

var candidatePatch = {
          'foundation': parsedCandidate.foundation,
          'component': parsedCandidate.component,
          'priority': parsedCandidate.priority,
          'ip': parsedCandidate.ip,
          'port': parsedCandidate.port,
          'generation': parsedCandidate.generation,
          'type': parsedCandidate.type,
          'rel-addr': parsedCandidate.raddr,
          'rel-port': parsedCandidate.rport,
          'protocol': parsedCandidate.protocol,
          'network' : parsedCandidate.network,
          'id' : parsedCandidate.id
        };
        var patch: any = {
          id: msg.streams.publisher,
          'channel-bundles': [{
            'id': msg.streams.channelBundleId,
            transport: {
              'xmlns': 'urn:xmpp:jingle:transports:ice-udp:1',
              'setup': 'passive',
              'remote-candidate': candidatePatch
            }
          }]
        };

Also look at https://github.com/jitsi/lib-jitsi-meet/blob/master/modules/xmpp/JingleSessionPC.js#L330
and I recommend to replace the answerer offer with "passive"

          // XXX Videobridge is the (SDP) offerer and WebRTC (e.g. Chrome) is the
          // answerer (as orchestrated by Jicofo). In accord with
          // http://tools.ietf.org/html/rfc5245#section-5.2 and because both peers
          // are ICE FULL agents, Videobridge will take on the controlling role and
          // WebRTC will take on the controlled role. In accord with
          // https://tools.ietf.org/html/rfc5763#section-5, Videobridge will use the
          // setup attribute value of setup:actpass and WebRTC will be allowed to
          // choose either the setup attribute value of setup:active or
          // setup:passive. Chrome will by default choose setup:active because it is
          // RECOMMENDED by the respective RFC since setup:passive adds additional
          // latency. The case of setup:active allows WebRTC to send a DTLS
          // ClientHello as soon as an ICE connectivity check of its succeeds.
          // Unfortunately, Videobridge will be unable to respond immediately because
          // may not have WebRTC's answer or may have not completed the ICE
          // connectivity establishment. Even more unfortunate is that in the
          // described scenario Chrome's DTLS implementation will insist on
          // retransmitting its ClientHello after a second (the time is in accord
          // with the respective RFC) and will thus cause the whole connection
          // establishment to exceed at least 1 second. To work around Chrome's
          // idiosyncracy, don't allow it to send a ClientHello i.e. change its
          // default choice of setup:active to setup:passive.

2016-10-09 18:19 GMT+02:00 Oliver Hausler <oliver@closeup.cc<javascript:_e(%7B%7D,'cvml','oliver@closeup.cc');>>:
// Once again because the ICE Agent does not support adding
// candidates after the connectivity establishment has been started
// and because multiple transport-info JingleIQs may be used to send
// the whole set of transport candidates from the remote peer to the
// local peer, do not really start the connectivity establishment
// until we have at least one remote candidate per ICE Component.

This comment it ICE Agent makes me believe I am right. And Jitsi is sending streams when I send a complete SDP with all candidates.

Can anybody comment what's the status here, please.

_______________________________________________
dev mailing list
dev@jitsi.org<javascript:_e(%7B%7D,'cvml','dev@jitsi.org');>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
sent from my mobile


#10

Oliver,

From what you have said it doesn't sound like you need to trickle any

candidates. As long as your videobridge is on a public network it will
discover all client candidates as peer reflexive (this is still trickle ICE
but with now actual client-side trickling).

This is what we do on all Jitsi Meet deployments including meet.jit.si

Emil

···

On Saturday, 22 October 2016, Oliver Hausler <oliver@closeup.cc> wrote:

Emil, Philipp, sorry for the late response.

@Philipp: It doesn't work for me with passive. Can you see any issues with
my logs below? Might be entirely unrelated as I see DTLS warnings in the
Jitsi log. I posted this separately: "Unknown DTLS handshake message and no
streams from Jitsi (REST)"

@Emil: I assume we should be using ICE trickle as this is standard in
WebRTC, so I tried to do that:

- I am creating an OFFER SDP which contains all Candidates
discovered by Jitsi (no ICE trickle, makes no sense here as we have
candidates in colibri already).

In Chrome I have 2 options:

1. ICE trickle: I listen to createAnswerOnSuccess(), send the
ANSWER SDP to Jitsi, wait for onIceCandidate() and send each Candidate as
it is discovered.

2. No ICE trickle: I listen to iceGatheringStateChange() ==
complete, then send the full ANSWER SDP with all Candidates.

I thought 1. would be the preferred way when using WebRTC/libjingle, so I
followed what Philipp Seeser suggested and it looks like Jitsi accepts
the "remote-candidate"(s) as it replays it:

"channel-bundles": [{

                                "id": "AKQQeYSlhEwltx69_ZWinO8|||||
N-N77FcLReCCe5yoieF_7A",

                                "transport": {

                                                "xmlns":
"urn:xmpp:jingle:transports:ice-udp:1",

                                                "rtcp-mux": true,

                                                "remote-candidate": {

"component": 1,

"protocol": "udp",

"priority": 1685987071,

                                                                "ip":
"68.96.117.185",

                                                                "port":
56117,

                                                                "type":
"srflx",

"generation": 0,

"foundation": "1855365925",

"network": 1

                                                },

                                                "setup": "passive"

                                }

                }]

I assume I must add each Candidate like that, and Jitsi picks the one
which it will use. Because the last 2 candidates I am receiving use
throw-away port 9 [why is Chrome generating these at all?], if Jitsi would
use the last Candidate, it wouldn't work, but I believe it's smart enough
and uses the best candidate, even though this isn't reflected in colibri
(except for the immediate response, there is no change in colibri
thereafter, so I cannot prove this).

Finally, I see a slightly distinct result between 1 and 2, though, when
looking at "GET /colibri/conferences/{conferenceId}":

For 1. I see 2 audio channels per endpoint:

  "contents": [

    {

      "channels": [

        {

          "endpoint": "AKQQeYSlhEwltx69_ZWinO8|||||
AKPDvx2Egk8dqVFUHTgkE_A",

          "channel-bundle-id": "AKQQeYSlhEwltx69_ZWinO8|||||
AKPDvx2Egk8dqVFUHTgkE_A",

          "sources": [

3204522869

          ],

          "rtp-level-relay-type": "mixer",

          "expire": 300,

          "initiator": true,

          "ssrcs": [

            3204522869,

336257926

          ],

          "id": "61baaec9d48429b4",

          "direction": "sendrecv"

        },

        {

          "endpoint": "AKQQeYSlhEwltx69_ZWinO8|||||
APkpD2Im3kfzq2hM1o49z3E",

          "channel-bundle-id": "AKQQeYSlhEwltx69_ZWinO8|||||
APkpD2Im3kfzq2hM1o49z3E",

          "sources": [

1722705425

          ],

         "rtp-level-relay-type": "mixer",

          "expire": 300,

          "initiator": true,

          "ssrcs": [

            1722705425,

3673416731

          ],

          "id": "b41ab150c9739171",

          "direction": "sendrecv"

        }

      ],

     "name": "audio"

    },

For 2. one of the ssrcs seems to be missing for one of the channels (so
Philipps approach with ICE trickle seems to be the better working):

  "contents": [

    {

      "channels": [

        {

          "endpoint": "AKQQeYSlhEwltx69_ZWinO8|||||
MEIiSaRqTGep7C2cDR42sQ",

          "channel-bundle-id": "AKQQeYSlhEwltx69_ZWinO8|||||
MEIiSaRqTGep7C2cDR42sQ",

          "sources": [

916465418

          ],

          "rtp-level-relay-type": "mixer",

          "expire": 300,

          "initiator": true,

          "ssrcs": [

3268560502

          ],

          "id": "700d2d43eae73a28",

          "direction": "sendrecv"

        },

        {

          "endpoint": "AKQQeYSlhEwltx69_ZWinO8|||||
ANZL23OQrkVkqzUZ-XMyyPk",

          "channel-bundle-id": "AKQQeYSlhEwltx69_ZWinO8|||||
ANZL23OQrkVkqzUZ-XMyyPk",

          "sources": [

814905801

          ],

          "rtp-level-relay-type": "mixer",

          "expire": 300,

          "initiator": true,

          "ssrcs": [

            814905801,

2637946124

          ],

          "id": "ba07908e22e3b2a2",

          "direction": "sendrecv"

        }

      ],

      "name": "audio"

    },

*From:* dev [mailto:dev-bounces@jitsi.org
<javascript:_e(%7B%7D,'cvml','dev-bounces@jitsi.org');>] *On Behalf Of *Emil
Ivov
*Sent:* Sunday, October 16, 2016 22:30
*To:* Jitsi Developers <dev@jitsi.org
<javascript:_e(%7B%7D,'cvml','dev@jitsi.org');>>
*Subject:* Re: [jitsi-dev] How to add remote-candidates to colibri with
trickle ICE using REST?

I am sorry I am chiming in so late but, why exactly do you need to send
any additional candidates to the bridge in the first place?

On Sunday, 16 October 2016, Oliver Hausler <oliver@closeup.cc > <javascript:_e(%7B%7D,'cvml','oliver@closeup.cc');>> wrote:

Thank you Philipp, I just noticed you sent a response to this.

Interesting, you patch each candidate into 'remote-candidate' separately.
I will try this tomorrow.

*From:* dev [mailto:dev-bounces@jitsi.org] *On Behalf Of *Philipp Seeser
*Sent:* Monday, October 10, 2016 03:02
*To:* Jitsi Developers <dev@jitsi.org>
*Subject:* Re: [jitsi-dev] How to add remote-candidates to colibri with
trickle ICE using REST?

I am able to add candidates with trickle ice and REST.

e.g. candidate json body:

var candidatePatch = {

          'foundation': parsedCandidate.foundation,

          'component': parsedCandidate.component,

          'priority': parsedCandidate.priority,

          'ip': parsedCandidate.ip,

          'port': parsedCandidate.port,

          'generation': parsedCandidate.generation,

          'type': parsedCandidate.type,

          'rel-addr': parsedCandidate.raddr,

          'rel-port': parsedCandidate.rport,

          'protocol': parsedCandidate.protocol,

          'network' : parsedCandidate.network,

          'id' : parsedCandidate.id

        };

        var patch: any = {

          id: msg.streams.publisher,

          'channel-bundles': [{

            'id': msg.streams.channelBundleId,

            transport: {

              'xmlns': 'urn:xmpp:jingle:transports:ice-udp:1',

              'setup': 'passive',

              'remote-candidate': candidatePatch

            }

          }]

        };

Also look at https://github.com/jitsi/lib-jitsi-meet/blob/master/
modules/xmpp/JingleSessionPC.js#L330

and I recommend to replace the answerer offer with "passive"

          // XXX Videobridge is the (SDP) offerer and WebRTC (e.g. Chrome)
is the
          // answerer (as orchestrated by Jicofo). In accord with
          // http://tools.ietf.org/html/rfc5245#section-5.2 and because
both peers
          // are ICE FULL agents, Videobridge will take on the controlling
role and
          // WebRTC will take on the controlled role. In accord with
          // https://tools.ietf.org/html/rfc5763#section-5, Videobridge
will use the
          // setup attribute value of setup:actpass and WebRTC will be
allowed to
          // choose either the setup attribute value of setup:active or
          // setup:passive. Chrome will by default choose setup:active
because it is
          // RECOMMENDED by the respective RFC since setup:passive adds
additional
          // latency. The case of setup:active allows WebRTC to send a DTLS
          // ClientHello as soon as an ICE connectivity check of its
succeeds.
          // Unfortunately, Videobridge will be unable to respond
immediately because
          // may not have WebRTC's answer or may have not completed the ICE
          // connectivity establishment. Even more unfortunate is that in
the
          // described scenario Chrome's DTLS implementation will insist on
          // retransmitting its ClientHello after a second (the time is in
accord
          // with the respective RFC) and will thus cause the whole
connection
          // establishment to exceed at least 1 second. To work around
Chrome's
          // idiosyncracy, don't allow it to send a ClientHello i.e.
change its
          // default choice of setup:active to setup:passive.

2016-10-09 18:19 GMT+02:00 Oliver Hausler <oliver@closeup.cc>:

*// Once again because the ICE Agent does not support adding // candidates
after the connectivity establishment has been started // and because
multiple transport-info JingleIQs may be used to send // the whole set of
transport candidates from the remote peer to the // local peer, do not
really start the connectivity establishment // until we have at least one
remote candidate per ICE Component.*

This comment it ICE Agent makes me believe I am right. And Jitsi is
sending streams when I send a complete SDP with all candidates.

Can anybody comment what's the status here, please.

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
sent from my mobile

--
sent from my mobile


#11

Hi Emil,

Are you sure about that?

https://github.com/jitsi/lib-jitsi-meet/blob/master/modules/xmpp/JingleSessionPC.js#L330
This tells the opposite and also my wireshark logs tell the opposite.

That comment means the bridge will send the DTLS ClientHello, but after it
receives the client SDP (including all gathered candidates from the client
after ice gathering is complete).

From my perspective:

1.) If your browser is allowed (setup:active) to send the "DTLS
ClientHello", what Emil described as client-side trickling is true,
assuming the bridge receives the client SDP (with DTLS fingerprint) on
time. (Besides the limitations in Chrome currently mentioned in the
comment, not sure if that's still the case)

2.) If your browser becomes the controlled role (setup:passive) the bridge
will send the "DTLS ClientHello". Then trickle-ice becomes important. It
looks like all Jitsi Meet deployments, use this option but don't
trickle-ice, instead the browser will send the client SDP after gathering
all ice candidates. So this can be optimized.

Philipp

···

2016-10-22 23:24 GMT+02:00 Emil Ivov <eivov@atlassian.com>:

Oliver,

From what you have said it doesn't sound like you need to trickle any
candidates. As long as your videobridge is on a public network it will
discover all client candidates as peer reflexive (this is still trickle ICE
but with now actual client-side trickling).

This is what we do on all Jitsi Meet deployments including meet.jit.si

Emil

On Saturday, 22 October 2016, Oliver Hausler <oliver@closeup.cc> wrote:

Emil, Philipp, sorry for the late response.

@Philipp: It doesn't work for me with passive. Can you see any issues
with my logs below? Might be entirely unrelated as I see DTLS warnings in
the Jitsi log. I posted this separately: "Unknown DTLS handshake message
and no streams from Jitsi (REST)"

@Emil: I assume we should be using ICE trickle as this is standard in
WebRTC, so I tried to do that:

- I am creating an OFFER SDP which contains all Candidates
discovered by Jitsi (no ICE trickle, makes no sense here as we have
candidates in colibri already).

In Chrome I have 2 options:

1. ICE trickle: I listen to createAnswerOnSuccess(), send the
ANSWER SDP to Jitsi, wait for onIceCandidate() and send each Candidate as
it is discovered.

2. No ICE trickle: I listen to iceGatheringStateChange() ==
complete, then send the full ANSWER SDP with all Candidates.

I thought 1. would be the preferred way when using WebRTC/libjingle, so I
followed what Philipp Seeser suggested and it looks like Jitsi accepts
the "remote-candidate"(s) as it replays it:

"channel-bundles": [{

                                "id": "AKQQeYSlhEwltx69_ZWinO8|||||N
-N77FcLReCCe5yoieF_7A",

                                "transport": {

                                                "xmlns":
"urn:xmpp:jingle:transports:ice-udp:1",

                                                "rtcp-mux": true,

                                                "remote-candidate": {

"component": 1,

"protocol": "udp",

"priority": 1685987071,

                                                                "ip":
"68.96.117.185",

                                                                "port":
56117,

                                                                "type":
"srflx",

"generation": 0,

"foundation": "1855365925",

"network": 1

                                                },

                                                "setup": "passive"

                                }

                }]

I assume I must add each Candidate like that, and Jitsi picks the one
which it will use. Because the last 2 candidates I am receiving use
throw-away port 9 [why is Chrome generating these at all?], if Jitsi would
use the last Candidate, it wouldn't work, but I believe it's smart enough
and uses the best candidate, even though this isn't reflected in colibri
(except for the immediate response, there is no change in colibri
thereafter, so I cannot prove this).

Finally, I see a slightly distinct result between 1 and 2, though, when
looking at "GET /colibri/conferences/{conferenceId}":

For 1. I see 2 audio channels per endpoint:

  "contents": [

    {

      "channels": [

        {

          "endpoint": "AKQQeYSlhEwltx69_ZWinO8|||||A
KPDvx2Egk8dqVFUHTgkE_A",

          "channel-bundle-id": "AKQQeYSlhEwltx69_ZWinO8|||||A
KPDvx2Egk8dqVFUHTgkE_A",

          "sources": [

3204522869

          ],

          "rtp-level-relay-type": "mixer",

          "expire": 300,

          "initiator": true,

          "ssrcs": [

            3204522869,

336257926

          ],

          "id": "61baaec9d48429b4",

          "direction": "sendrecv"

        },

        {

          "endpoint": "AKQQeYSlhEwltx69_ZWinO8|||||A
PkpD2Im3kfzq2hM1o49z3E",

          "channel-bundle-id": "AKQQeYSlhEwltx69_ZWinO8|||||A
PkpD2Im3kfzq2hM1o49z3E",

          "sources": [

1722705425

          ],

         "rtp-level-relay-type": "mixer",

          "expire": 300,

          "initiator": true,

          "ssrcs": [

            1722705425,

3673416731

          ],

          "id": "b41ab150c9739171",

          "direction": "sendrecv"

        }

      ],

     "name": "audio"

    },

For 2. one of the ssrcs seems to be missing for one of the channels (so
Philipps approach with ICE trickle seems to be the better working):

  "contents": [

    {

      "channels": [

        {

          "endpoint": "AKQQeYSlhEwltx69_ZWinO8|||||M
EIiSaRqTGep7C2cDR42sQ",

          "channel-bundle-id": "AKQQeYSlhEwltx69_ZWinO8|||||M
EIiSaRqTGep7C2cDR42sQ",

          "sources": [

916465418

          ],

          "rtp-level-relay-type": "mixer",

          "expire": 300,

          "initiator": true,

          "ssrcs": [

3268560502

          ],

          "id": "700d2d43eae73a28",

          "direction": "sendrecv"

        },

        {

          "endpoint": "AKQQeYSlhEwltx69_ZWinO8|||||A
NZL23OQrkVkqzUZ-XMyyPk",

          "channel-bundle-id": "AKQQeYSlhEwltx69_ZWinO8|||||A
NZL23OQrkVkqzUZ-XMyyPk",

          "sources": [

814905801

          ],

          "rtp-level-relay-type": "mixer",

          "expire": 300,

          "initiator": true,

          "ssrcs": [

            814905801,

2637946124

          ],

          "id": "ba07908e22e3b2a2",

          "direction": "sendrecv"

        }

      ],

      "name": "audio"

    },

*From:* dev [mailto:dev-bounces@jitsi.org] *On Behalf Of *Emil Ivov
*Sent:* Sunday, October 16, 2016 22:30
*To:* Jitsi Developers <dev@jitsi.org>
*Subject:* Re: [jitsi-dev] How to add remote-candidates to colibri with
trickle ICE using REST?

I am sorry I am chiming in so late but, why exactly do you need to send
any additional candidates to the bridge in the first place?

On Sunday, 16 October 2016, Oliver Hausler <oliver@closeup.cc> wrote:

Thank you Philipp, I just noticed you sent a response to this.

Interesting, you patch each candidate into 'remote-candidate' separately.
I will try this tomorrow.

*From:* dev [mailto:dev-bounces@jitsi.org] *On Behalf Of *Philipp Seeser
*Sent:* Monday, October 10, 2016 03:02
*To:* Jitsi Developers <dev@jitsi.org>
*Subject:* Re: [jitsi-dev] How to add remote-candidates to colibri with
trickle ICE using REST?

I am able to add candidates with trickle ice and REST.

e.g. candidate json body:

var candidatePatch = {

          'foundation': parsedCandidate.foundation,

          'component': parsedCandidate.component,

          'priority': parsedCandidate.priority,

          'ip': parsedCandidate.ip,

          'port': parsedCandidate.port,

          'generation': parsedCandidate.generation,

          'type': parsedCandidate.type,

          'rel-addr': parsedCandidate.raddr,

          'rel-port': parsedCandidate.rport,

          'protocol': parsedCandidate.protocol,

          'network' : parsedCandidate.network,

          'id' : parsedCandidate.id

        };

        var patch: any = {

          id: msg.streams.publisher,

          'channel-bundles': [{

            'id': msg.streams.channelBundleId,

            transport: {

              'xmlns': 'urn:xmpp:jingle:transports:ice-udp:1',

              'setup': 'passive',

              'remote-candidate': candidatePatch

            }

          }]

        };

Also look at https://github.com/jitsi/lib-jitsi-meet/blob/master/modul
es/xmpp/JingleSessionPC.js#L330

and I recommend to replace the answerer offer with "passive"

          // XXX Videobridge is the (SDP) offerer and WebRTC (e.g.
Chrome) is the
          // answerer (as orchestrated by Jicofo). In accord with
          // http://tools.ietf.org/html/rfc5245#section-5.2 and because
both peers
          // are ICE FULL agents, Videobridge will take on the
controlling role and
          // WebRTC will take on the controlled role. In accord with
          // https://tools.ietf.org/html/rfc5763#section-5, Videobridge
will use the
          // setup attribute value of setup:actpass and WebRTC will be
allowed to
          // choose either the setup attribute value of setup:active or
          // setup:passive. Chrome will by default choose setup:active
because it is
          // RECOMMENDED by the respective RFC since setup:passive adds
additional
          // latency. The case of setup:active allows WebRTC to send a
DTLS
          // ClientHello as soon as an ICE connectivity check of its
succeeds.
          // Unfortunately, Videobridge will be unable to respond
immediately because
          // may not have WebRTC's answer or may have not completed the
ICE
          // connectivity establishment. Even more unfortunate is that in
the
          // described scenario Chrome's DTLS implementation will insist
on
          // retransmitting its ClientHello after a second (the time is
in accord
          // with the respective RFC) and will thus cause the whole
connection
          // establishment to exceed at least 1 second. To work around
Chrome's
          // idiosyncracy, don't allow it to send a ClientHello i.e.
change its
          // default choice of setup:active to setup:passive.

2016-10-09 18:19 GMT+02:00 Oliver Hausler <oliver@closeup.cc>:

*// Once again because the ICE Agent does not support adding //
candidates after the connectivity establishment has been started // and
because multiple transport-info JingleIQs may be used to send // the whole
set of transport candidates from the remote peer to the // local peer, do
not really start the connectivity establishment // until we have at least
one remote candidate per ICE Component.*

This comment it ICE Agent makes me believe I am right. And Jitsi is
sending streams when I send a complete SDP with all candidates.

Can anybody comment what's the status here, please.

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
sent from my mobile

--
sent from my mobile

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev