[jitsi-dev] libjitsi on smack to implement jingle


#1

Hey all,

I am interested in using libjitsi to add jingle features (specifically A/V chat) to my app that's already using smack. Ultimately, I'd like to support ICE, but that's not the first thing I need to do.

I can't really see how to get there from the sample code, which, if I'm reading it correctly, just provides an example of data transmission given the IP addresses and ports of the other host.

What I'd like to do as a first step is create some sample code I can donate back to libjitsi that does the following:

1st app:

a1) open a connection to an xmpp server (U/P given at the commandline) using the smack library.
b1) listen for jingle connections
c1) when a connection is requested, accept is and negotiate the connection
d1) share a mike or video connection (specified at the commandline) via the negotiated connection

2nd app:

a2) open a connection to an xmpp server (U/P given at the commandline) using the smack library.
b2) make a request for a jingle connection to the user of the 1st app.
c2) when the connection is accepted, negotiate the connection
d2) share a mike or video connection (specified at the commandline) via the negotiated connection

a) is pretty trivial, and d) is more or less what's shown in the sample code. I've looked a bit through the jitsi sample code, to see how it manages b) and c), but it's a vast codebase and I'm not finding it yet. Can anyone give me hints on this?

Judging by posts on the ignite and stack overflow, I know others would benefit from this in a big way.

I'd really appreciate any help.

Thanks,

  bjorn

···

-----------------------------
Bjorn Roche
http://www.xonami.com
Audio Collaboration
http://blog.bjornroche.com
@xonamiaudio


#2

Hey Bjorn,

Hey all,

I am interested in using libjitsi to add jingle features (specifically A/V chat) to my app that's already using smack. Ultimately, I'd like to support ICE, but that's not the first thing I need to do.

Great, that's also how we are using it.

I can't really see how to get there from the sample code, which, if I'm reading it correctly, just provides an example of data transmission given the IP addresses and ports of the other host.

Correct.

What I'd like to do as a first step is create some sample code I can donate back to libjitsi that does the following:

1st app:

a1) open a connection to an xmpp server (U/P given at the commandline) using the smack library.
b1) listen for jingle connections
c1) when a connection is requested, accept is and negotiate the connection
d1) share a mike or video connection (specified at the commandline) via the negotiated connection

2nd app:

a2) open a connection to an xmpp server (U/P given at the commandline) using the smack library.
b2) make a request for a jingle connection to the user of the 1st app.
c2) when the connection is accepted, negotiate the connection
d2) share a mike or video connection (specified at the commandline) via the negotiated connection

Both of the above would be welcome.

a) is pretty trivial, and d) is more or less what's shown in the sample code. I've looked a bit through the jitsi sample code, to see how it manages b) and c), but it's a vast codebase and I'm not finding it yet. Can anyone give me hints on this?

Judging by posts on the ignite and stack overflow, I know others would benefit from this in a big way.

Everything happens in

net.java.sip.communicator.impl.protocol.jabber

The entry point for most of the calling features is:

OperationSetBasicTelephonyJabberImpl

createOutoingCall() in all its variants and all they do is what you'd
need for the 1st app.
processJingleIQ() and everything it does would be the body of your 2nd app.

Both of these methods would create Call-s and CallPeer-s, which would
have MediaHandler-s and TransportManager-s. The former deal with the
libjitsi initialisation while the latter are where the ice4j methods are
being called.

Hope this helps.

Cheers,
Emil

···

On 29.04.13, 19:59, Bjorn Roche wrote:

I'd really appreciate any help.

Thanks,

  bjorn

-----------------------------
Bjorn Roche
http://www.xonami.com
Audio Collaboration
http://blog.bjornroche.com
@xonamiaudio

--
https://jitsi.org


#3

Hey Bjorn,

Hey all,

I am interested in using libjitsi to add jingle features (specifically A/V chat) to my app that's already using smack. Ultimately, I'd like to support ICE, but that's not the first thing I need to do.

Great, that's also how we are using it.

I can't really see how to get there from the sample code, which, if I'm reading it correctly, just provides an example of data transmission given the IP addresses and ports of the other host.

Correct.

Great, thanks.

What I'd like to do as a first step is create some sample code I can donate back to libjitsi that does the following:

1st app:

a1) open a connection to an xmpp server (U/P given at the commandline) using the smack library.
b1) listen for jingle connections
c1) when a connection is requested, accept is and negotiate the connection
d1) share a mike or video connection (specified at the commandline) via the negotiated connection

2nd app:

a2) open a connection to an xmpp server (U/P given at the commandline) using the smack library.
b2) make a request for a jingle connection to the user of the 1st app.
c2) when the connection is accepted, negotiate the connection
d2) share a mike or video connection (specified at the commandline) via the negotiated connection

Both of the above would be welcome.

a) is pretty trivial, and d) is more or less what's shown in the sample code. I've looked a bit through the jitsi sample code, to see how it manages b) and c), but it's a vast codebase and I'm not finding it yet. Can anyone give me hints on this?

Judging by posts on the ignite and stack overflow, I know others would benefit from this in a big way.

Everything happens in

net.java.sip.communicator.impl.protocol.jabber
The entry point for most of the calling features is:

OperationSetBasicTelephonyJabberImpl

Okay, I was looking in that package, but did not see that class, so thanks for the tip.

I've looked through the code some more and I have many thoughts buzzing in my head -- it's always challenging looking at a new codebase and trying to figure out what the possibilities are. I guess I'll just see where I can get and ask more questions as I need to.

···

On Apr 30, 2013, at 9:46 AM, Emil Ivov wrote:

On 29.04.13, 19:59, Bjorn Roche wrote:

createOutoingCall() in all its variants and all they do is what you'd
need for the 1st app.
processJingleIQ() and everything it does would be the body of your 2nd app.

Both of these methods would create Call-s and CallPeer-s, which would
have MediaHandler-s and TransportManager-s. The former deal with the
libjitsi initialisation while the latter are where the ice4j methods are
being called.

Hope this helps.

Cheers,
Emil

I'd really appreciate any help.

Thanks,

  bjorn

-----------------------------
Bjorn Roche
http://www.xonami.com
Audio Collaboration
http://blog.bjornroche.com
@xonamiaudio

--
https://jitsi.org

-----------------------------
Bjorn Roche
http://www.xonami.com
Audio Collaboration
http://blog.bjornroche.com
@xonamiaudio


#4

So I've made some basic progress on this. I'm initiating a jingle session and exchanging the first few jingle iq packets and I have a very basic structure in place.

I am having a bit of trouble with the negotiation step. Specifically, in the call to

JinglePacketFactory.createSessionAccept(....);

I'm not sure how to build my content list. I can figure out how to build it manually, but obviously that's not the right way to do it. It seems that the right way to do it is to:

- generate content descriptions from libjitsi. libjistsi will want to know if we are making a video connection, audio connection, which device we are using, and so on.
- generate the transport candidates from ice4j. ice4j will need to make some calls to stun server, first, presumably. I believe I can get stun server info from the XMPP (openfire) server, but I'm still researching how to do that.

I am trying to see how jitsi does this, but again, I'm getting lost. Any tips? Thanks,

  bjorn

--- no more comments

···

On Apr 30, 2013, at 12:31 PM, Bjorn Roche wrote:

On Apr 30, 2013, at 9:46 AM, Emil Ivov wrote:

Hey Bjorn,

On 29.04.13, 19:59, Bjorn Roche wrote:

Hey all,

I am interested in using libjitsi to add jingle features (specifically A/V chat) to my app that's already using smack. Ultimately, I'd like to support ICE, but that's not the first thing I need to do.

Great, that's also how we are using it.

I can't really see how to get there from the sample code, which, if I'm reading it correctly, just provides an example of data transmission given the IP addresses and ports of the other host.

Correct.

Great, thanks.

What I'd like to do as a first step is create some sample code I can donate back to libjitsi that does the following:

1st app:

a1) open a connection to an xmpp server (U/P given at the commandline) using the smack library.
b1) listen for jingle connections
c1) when a connection is requested, accept is and negotiate the connection
d1) share a mike or video connection (specified at the commandline) via the negotiated connection

2nd app:

a2) open a connection to an xmpp server (U/P given at the commandline) using the smack library.
b2) make a request for a jingle connection to the user of the 1st app.
c2) when the connection is accepted, negotiate the connection
d2) share a mike or video connection (specified at the commandline) via the negotiated connection

Both of the above would be welcome.

a) is pretty trivial, and d) is more or less what's shown in the sample code. I've looked a bit through the jitsi sample code, to see how it manages b) and c), but it's a vast codebase and I'm not finding it yet. Can anyone give me hints on this?

Judging by posts on the ignite and stack overflow, I know others would benefit from this in a big way.

Everything happens in

net.java.sip.communicator.impl.protocol.jabber
The entry point for most of the calling features is:

OperationSetBasicTelephonyJabberImpl

Okay, I was looking in that package, but did not see that class, so thanks for the tip.

I've looked through the code some more and I have many thoughts buzzing in my head -- it's always challenging looking at a new codebase and trying to figure out what the possibilities are. I guess I'll just see where I can get and ask more questions as I need to.

createOutoingCall() in all its variants and all they do is what you'd
need for the 1st app.
processJingleIQ() and everything it does would be the body of your 2nd app.

Both of these methods would create Call-s and CallPeer-s, which would
have MediaHandler-s and TransportManager-s. The former deal with the
libjitsi initialisation while the latter are where the ice4j methods are
being called.

Hope this helps.

Cheers,
Emil

-----------------------------
Bjorn Roche
http://www.xonami.com
Audio Collaboration
http://blog.bjornroche.com
@xonamiaudio


#5

Hey Bjorn,

So I've made some basic progress on this. I'm initiating a jingle session and exchanging the first few jingle iq packets and I have a very basic structure in place.

I am having a bit of trouble with the negotiation step. Specifically, in the call to

JinglePacketFactory.createSessionAccept(....);

I'm not sure how to build my content list. I can figure out how to build it manually, but obviously that's not the right way to do it. It seems that the right way to do it is to:

- generate content descriptions from libjitsi. libjistsi will want to know if we are making a video connection, audio connection, which device we are using, and so on.

almost but not quite. libjitsi will not generate the actual description
for your because it is protocol agnostic (remember that we are also
using it with SIP where the O/A negotiation protocol is SDP).

So basically what you need to do is ask libjitsi what devices it has and
what formats it supports and then convert that information into XML.
This part happens in the createOffer() method in the media handler (note
that the actual code is split between CallPeerMediaHandler,
AbstractCallPeerMediaHandler and CallPeerMediaHandlerJabberImpl) as well
as in net.java.sip.communicator.impl.protocol.jabber.jinglesdp.JingleUtils.

- generate the transport candidates from ice4j.

Yes.

ice4j will need to make some calls to stun server,

Correct.

first, presumably. I believe I can get stun server info from the XMPP (openfire) server, but I'm still researching how to do that.

I wouldn't rely on this. In Jitsi we first check through DNS whether
there's a STUN server for the domain we are using. If not then we fall
back to a default Jitsi stun server (stun.jitsi.net). You may want to
have a similar setup.

I am trying to see how jitsi does this, but again, I'm getting lost. Any tips? Thanks,

This is the part in the transport managers. Gathering the addresses
happens through the ice4j harvesting.

Hope this helps,
Emil

···

On 01.05.13, 20:13, Bjorn Roche wrote:

  bjorn

--- no more comments

On Apr 30, 2013, at 12:31 PM, Bjorn Roche wrote:

On Apr 30, 2013, at 9:46 AM, Emil Ivov wrote:

Hey Bjorn,

On 29.04.13, 19:59, Bjorn Roche wrote:

Hey all,

I am interested in using libjitsi to add jingle features (specifically A/V chat) to my app that's already using smack. Ultimately, I'd like to support ICE, but that's not the first thing I need to do.

Great, that's also how we are using it.

I can't really see how to get there from the sample code, which, if I'm reading it correctly, just provides an example of data transmission given the IP addresses and ports of the other host.

Correct.

Great, thanks.

What I'd like to do as a first step is create some sample code I can donate back to libjitsi that does the following:

1st app:

a1) open a connection to an xmpp server (U/P given at the commandline) using the smack library.
b1) listen for jingle connections
c1) when a connection is requested, accept is and negotiate the connection
d1) share a mike or video connection (specified at the commandline) via the negotiated connection

2nd app:

a2) open a connection to an xmpp server (U/P given at the commandline) using the smack library.
b2) make a request for a jingle connection to the user of the 1st app.
c2) when the connection is accepted, negotiate the connection
d2) share a mike or video connection (specified at the commandline) via the negotiated connection

Both of the above would be welcome.

a) is pretty trivial, and d) is more or less what's shown in the sample code. I've looked a bit through the jitsi sample code, to see how it manages b) and c), but it's a vast codebase and I'm not finding it yet. Can anyone give me hints on this?

Judging by posts on the ignite and stack overflow, I know others would benefit from this in a big way.

Everything happens in

net.java.sip.communicator.impl.protocol.jabber
The entry point for most of the calling features is:

OperationSetBasicTelephonyJabberImpl

Okay, I was looking in that package, but did not see that class, so thanks for the tip.

I've looked through the code some more and I have many thoughts buzzing in my head -- it's always challenging looking at a new codebase and trying to figure out what the possibilities are. I guess I'll just see where I can get and ask more questions as I need to.

createOutoingCall() in all its variants and all they do is what you'd
need for the 1st app.
processJingleIQ() and everything it does would be the body of your 2nd app.

Both of these methods would create Call-s and CallPeer-s, which would
have MediaHandler-s and TransportManager-s. The former deal with the
libjitsi initialisation while the latter are where the ice4j methods are
being called.

Hope this helps.

Cheers,
Emil

-----------------------------
Bjorn Roche
http://www.xonami.com
Audio Collaboration
http://blog.bjornroche.com
@xonamiaudio

--
https://jitsi.org


#6

Hey Bjorn,

So I've made some basic progress on this. I'm initiating a jingle session and exchanging the first few jingle iq packets and I have a very basic structure in place.

I am having a bit of trouble with the negotiation step. Specifically, in the call to

JinglePacketFactory.createSessionAccept(....);

I'm not sure how to build my content list. I can figure out how to build it manually, but obviously that's not the right way to do it. It seems that the right way to do it is to:

- generate content descriptions from libjitsi. libjistsi will want to know if we are making a video connection, audio connection, which device we are using, and so on.

almost but not quite. libjitsi will not generate the actual description
for your because it is protocol agnostic (remember that we are also
using it with SIP where the O/A negotiation protocol is SDP).

I am hoping it will at least generate the information used to populate the description.

So basically what you need to do is ask libjitsi what devices it has and
what formats it supports and then convert that information into XML.
This part happens in the createOffer() method in the media handler (note
that the actual code is split between CallPeerMediaHandler,
AbstractCallPeerMediaHandler and CallPeerMediaHandlerJabberImpl) as well
as in net.java.sip.communicator.impl.protocol.jabber.jinglesdp.JingleUtils.

- generate the transport candidates from ice4j.

Yes.

ice4j will need to make some calls to stun server,

Correct.

first, presumably. I believe I can get stun server info from the XMPP (openfire) server, but I'm still researching how to do that.

I wouldn't rely on this. In Jitsi we first check through DNS whether
there's a STUN server for the domain we are using. If not then we fall
back to a default Jitsi stun server (stun.jitsi.net). You may want to
have a similar setup.

Okay, I see external service discovery XEP is "deferred" and doesn't seem to be working on my server anyway. This makes things rather tricky....

In the case of my server, I have an SRV record on xmpp.xonami.com pointing to openfire.xonami.com. I can't find any docs on what I need to do to my DNS to get that working (do I add dns records to xmpp. or openfilre. ? which DNS records? etc)

In the meantime, I'll use yours if it will work.

I am trying to see how jitsi does this, but again, I'm getting lost. Any tips? Thanks,

This is the part in the transport managers. Gathering the addresses
happens through the ice4j harvesting.

Thanks,

bjorn

···

On May 1, 2013, at 2:57 PM, Emil Ivov wrote:

On 01.05.13, 20:13, Bjorn Roche wrote:

-----------------------------
Bjorn Roche
http://www.xonami.com
Audio Collaboration
http://blog.bjornroche.com
@xonamiaudio


#7

I wouldn't rely on this. In Jitsi we first check through DNS whether
there's a STUN server for the domain we are using. If not then we fall
back to a default Jitsi stun server (stun.jitsi.net). You may want to
have a similar setup.

Okay, I see external service discovery XEP is "deferred" and doesn't seem to be working on my server anyway. This makes things rather tricky....

Personally I'd say it makes them much simpler but that's only me :slight_smile:

In the case of my server, I have an SRV record on xmpp.xonami.com pointing to openfire.xonami.com. I can't find any docs on what I need to do to my DNS to get that working

STUN Discovery:
http://tools.ietf.org/html/rfc5389#section-9

Example:
$ host -t SRV _stun._udp.jitsi.org
_stun._udp.jitsi.org has SRV record 1 10 3478 stun.jitsi.net.

TURN Discovery:
http://tools.ietf.org/html/rfc5766#section-6.1

$ host -t SRV _turn._udp.jitsi.org
_turn._udp.jitsi.org has SRV record 1 10 3478 stun.jitsi.net.

(do I add dns records to xmpp. or openfilre. ?

From your description it sounds like it will have to be on

xmpp.xonami.com since that's the domain you register with.

Note however that this is entirely up to you since you will be the one
performing the DNS queries, so you get to choose the domains you are
going to make them for.

which DNS records? etc)

see above.

In the meantime, I'll use yours if it will work.

Sure. Always better to have one you control yourself though :slight_smile:

Cheers,
Emil

···

On 01.05.13, 21:45, Bjorn Roche wrote:

--
https://jitsi.org


#8

Okay, here's where I am. I've tried to include the most important calls I use along the way, especially the ones I'm not sure about.

0. Tell smack library that we are handling jingle packets, so it doesn't automatically reply with "cancel" when someone sends a jingle packet. We also tell it how to translate Jingle IQ packets into JingleIQ objects.
1. Both clients (one "Caller" and one "Receiver") connect to the XMPP host using the smack library.
2. Caller finds the stun and turn servers by making a DNS query. (right now I'm falling back on jitsi.org's servers)
3. Caller waits for Receiver's presence.
4. Caller gets a list of video devices/codecs from libjitsi
    + get the default device from mediaService.getDefaultDevice(mediaType, MediaUseCase.CALL);
    + we populate RtpDescriptionPacketExtension object with information from dev.getSupportedFormats()
5. Caller gets a list of "transport candidates" from ice4j
    + a new agent is created
    + agent.setControling(true) is called
    + agent.addCandidateHarvester is called for each turn and stun server
    + a media stream and two components are created
6. Caller uses devices/codecs list and transport candidates to build a jingle session-initiate stanza and send it to the Receiver.
7. Receiver acknowledges session-initiate.
8. If the Receiver recognizes the "from" address of the session-initiate as the Caller, it accepts and continues. Otherwise, it rejects:
      a. Receiver sends Caller a session-terminate.
      b. Caller and Receiver both exit.
9. Receiver gets it's own list of video device codecs and transport candidates (same method as Caller). It sends this info in a jingle session-accept to the caller.
10. The Caller sends back an acknowledgement.
11. The Caller looks at the proposed transport candidates from the Receiver and adds them to it's local Ice Agent as "remote candidates", as well as the password and ufrag.
12. The Caller calls startConnectivityEstablishment() on its agent.

At this point, I can call getCheckList() on the agent's streams and get something, but I'm not sure what it is. Have I completed negotiation? Can the clients start talking? In the Jitsi code I see lots of calls to things that do "Harvest"'ing, but I'm not clear what that's all about.

Thanks again,

  bjorn

···

On May 1, 2013, at 5:37 PM, Emil Ivov wrote:

On 01.05.13, 21:45, Bjorn Roche wrote:

I wouldn't rely on this. In Jitsi we first check through DNS whether
there's a STUN server for the domain we are using. If not then we fall
back to a default Jitsi stun server (stun.jitsi.net). You may want to
have a similar setup.

Okay, I see external service discovery XEP is "deferred" and doesn't seem to be working on my server anyway. This makes things rather tricky....

Personally I'd say it makes them much simpler but that's only me :slight_smile:

In the case of my server, I have an SRV record on xmpp.xonami.com pointing to openfire.xonami.com. I can't find any docs on what I need to do to my DNS to get that working

STUN Discovery:
http://tools.ietf.org/html/rfc5389#section-9

Example:
$ host -t SRV _stun._udp.jitsi.org
_stun._udp.jitsi.org has SRV record 1 10 3478 stun.jitsi.net.

TURN Discovery:
http://tools.ietf.org/html/rfc5766#section-6.1

$ host -t SRV _turn._udp.jitsi.org
_turn._udp.jitsi.org has SRV record 1 10 3478 stun.jitsi.net.

(do I add dns records to xmpp. or openfilre. ?

From your description it sounds like it will have to be on
xmpp.xonami.com since that's the domain you register with.

Note however that this is entirely up to you since you will be the one
performing the DNS queries, so you get to choose the domains you are
going to make them for.

which DNS records? etc)

see above.

In the meantime, I'll use yours if it will work.

Sure. Always better to have one you control yourself though :slight_smile:

Cheers,
Emil

--
https://jitsi.org

-----------------------------
Bjorn Roche
http://www.xonami.com
Audio Collaboration
http://blog.bjornroche.com
@xonamiaudio