[jitsi-users] Writing a jitsi-meet client for raspberry pi


#1

Hi everyone,

I'm currently working on writing a simple C++ client on a raspberry pi that
can connect to jitsi-meet conferences. I'm using Gloox
<https://camaya.net/gloox/> for an xmpp library and OpenWebRTC
<https://www.openwebrtc.org/> for handling the content streams. I've made
some progress, but I'm struggling because there isn't much mid-level
documentation of jicofo and the videobridge, especially in terms of what
exactly they expect from clients (both signaling and actual streams). I've
found this jres article
<https://2013.jres.org/archives/169/paper169_article.pdf> and this
super-helpful gliffy diagram <https://www.gliffy.com/go/publish/7649541/> but
I haven't been able to come up with anything more specific than that short
of the libjitsi api docs. If you could point me to anything already written
about the "api" for interacting with jicofo and jvb on either a signalling
or an RTP level, I'd really appreciate it.

Some more specific questions:
What ssrc information does jvb require from a client? What does it expect a
client to do with ssrc information it sends?
Is the xmlns https://jitsi.org/protocol/focus documented anywhere? Are the
jitsi extensions to jingle (e.g. source-add, source-remove) documented
anywhere?
How is the client expected to handle source-add requests?

One specific problem I'm having right now has to do with ICE candidates and
streams. OpenWebRTC finds and advertises separate candidates for each media
stream (audio, video, and data). In the files I've attached, you can see
that my client is advertising in its session-accept jingle a UDP port 46132
for audio and a UDP port 36943 for video. In the videobridge logs, you can
see that port advertised for audio is validated, but the port advertised
for video has a 401 (unauthorized) error. Despite this, it logs that
transport is connected for both audio and video content. Does this mean
it's sending both audio and video streams to one port? If so, is there any
way to force it to use separate ports?

videobridge | JVB 2016-12-13 17:24:10.880 INFO: [573]
org.ice4j.ice.ConnectivityCheckClient.processSuccessResponse() Pair
succeeded: 172.16.38.32:10000/udp/host -> 172.16.43.25:46132/udp/prflx
(stream.RTP)
videobridge | JVB 2016-12-13 17:24:10.881 INFO: [573]
org.ice4j.ice.ConnectivityCheckClient.processSuccessResponse() Pair
validated: 172.16.38.32:10000/udp/host -> 172.16.43.25:46132/udp/prflx
(stream.RTP)
videobridge | JVB 2016-12-13 17:24:10.881 INFO: [573]
org.ice4j.ice.DefaultNominator.strategyNominateFirstValid() Nominate (first
valid): 172.16.38.32:10000/udp/host -> 172.16.43.25:46132/udp/prflx
(stream.RTP)
videobridge | JVB 2016-12-13 17:24:10.881 INFO: [573]
org.ice4j.ice.Agent.nominate() verify if nominated pair answer again
videobridge | JVB 2016-12-13 17:24:10.881 INFO: [573]
org.ice4j.ice.ConnectivityCheckClient.processSuccessResponse()
IsControlling: true USE-CANDIDATE:false
videobridge | JVB 2016-12-13 17:24:10.881 INFO: [573]
org.ice4j.ice.ConnectivityCheckClient.processErrorResponse() Error response
for pair: 172.16.38.32:10000/udp/host -> 172.16.43.25:36943/udp/prflx
(stream.RTP), failing. Code = 401(class=4; number=1)
videobridge | JVB 2016-12-13 17:24:10.901 INFO: [572]
org.ice4j.ice.ConnectivityCheckClient.processSuccessResponse() Pair
succeeded: 172.16.38.32:10000/udp/host -> 172.16.43.25:46132/udp/prflx
(stream.RTP)
videobridge | JVB 2016-12-13 17:24:10.901 INFO: [572]
org.ice4j.ice.ConnectivityCheckClient.processSuccessResponse()
IsControlling: true USE-CANDIDATE:true
videobridge | JVB 2016-12-13 17:24:10.901 INFO: [572]
org.ice4j.ice.ConnectivityCheckClient.processSuccessResponse() Nomination
confirmed for pair: 172.16.38.32:10000/udp/host ->
172.16.43.25:46132/udp/prflx (stream.RTP)
videobridge | JVB 2016-12-13 17:24:10.902 INFO: [572]
org.ice4j.ice.CheckList.handleNominationConfirmed() Selected pair for
stream stream.RTP: 172.16.38.32:10000/udp/host ->
172.16.43.25:46132/udp/prflx (stream.RTP)
videobridge | JVB 2016-12-13 17:24:10.902 INFO: [572]
org.ice4j.ice.Agent.checkListStatesUpdated() CheckList of stream stream is
COMPLETED
videobridge | JVB 2016-12-13 17:24:10.902 INFO: [572]
org.ice4j.ice.Agent.setState() ICE state changed from Running to Completed
videobridge | JVB 2016-12-13 17:24:10.902 INFO: [572]
org.jitsi.videobridge.IceUdpTransportManager.info() ICE processing state of
IceUdpTransportManager #342e9803 (for channels e908513700f4a17b
1bf58e089a98d982) of conference c35b430063593985 changed from Running to
Completed.
videobridge | JVB 2016-12-13 17:24:10.902 INFO: [577]
org.jitsi.videobridge.Channel.info() Transport connected for channel
e908513700f4a17b of content audio of conference c35b430063593985
videobridge | JVB 2016-12-13 17:24:10.902 INFO: [572]
org.ice4j.ice.Agent.logCandTypes() Harvester used for selected pair for
stream.RTP: host
videobridge | JVB 2016-12-13 17:24:10.908 INFO: [577]
org.jitsi.videobridge.Channel.info() Transport connected for channel
1bf58e089a98d982 of content video of conference

(for reference, my machine's ip ends in 38.32, the raspberry pi's ip ends
in 43.25)

Any advice on this specific problem, or writing a client for jitsi-meet in
general, or where to find documentation would be appreciated!

Thanks,
Zoe

xmpp_messages (11.1 KB)

sample.log (96.4 KB)


#2

Hi,

Hi everyone,

I'm currently working on writing a simple C++ client on a raspberry pi
that can connect to jitsi-meet conferences. I'm using Gloox
<https://camaya.net/gloox/> for an xmpp library and OpenWebRTC
<https://www.openwebrtc.org/> for handling the content streams.

Sounds like an interesting project!

I've
made some progress, but I'm struggling because there isn't much
mid-level documentation of jicofo and the videobridge, especially in
terms of what exactly they expect from clients (both signaling and
actual streams). I've found this jres article
<https://2013.jres.org/archives/169/paper169_article.pdf> and this
super-helpful gliffy diagram
<https://www.gliffy.com/go/publish/7649541/> but I haven't been able to
come up with anything more specific than that short of the libjitsi api
docs. If you could point me to anything already written about the "api"
for interacting with jicofo and jvb on either a signalling or an RTP
level, I'd really appreciate it.

I'm afraid there isn't much documentation available.

Some more specific questions:
What ssrc information does jvb require from a client?

It depends on the use-case. The SSRCs themselves are needed if rtcp-mux + bundle is used, or if simulcast is used. Other than that, I don't think the bridge itself requires them.

What does it
expect a client to do with ssrc information it sends?

The bridge will use the SSRC it generates to send RTCP messages (e.g. keyframe requests). The clients need to handle RTCP messages coming from this SSRC. In practice this means adding it as a source (a=ssrc line) in the remote description, just like the SSRC of other clients.

Is the xmlns https://jitsi.org/protocol/focus documented anywhere? Are
the jitsi extensions to jingle (e.g. source-add, source-remove)
documented anywhere?

I don't think they are.

How is the client expected to handle source-add requests?

It should accept RTP/RTCP packets for this source. In practice, it again adds a=ssrc lines to its remote description.

One specific problem I'm having right now has to do with ICE candidates
and streams. OpenWebRTC finds and advertises separate candidates for
each media stream (audio, video, and data). In the files I've attached,
you can see that my client is advertising in its session-accept jingle a
UDP port 46132 for audio and a UDP port 36943 for video. In the
videobridge logs, you can see that port advertised for audio is
validated, but the port advertised for video has a 401 (unauthorized)
error. Despite this, it logs that transport is connected for both audio
and video content. Does this mean it's sending both audio and video
streams to one port? If so, is there any way to force it to use separate
ports?

This is because of rtcp-mux and bundle. Some docs here:
https://github.com/jitsi/jitsi-videobridge/blob/master/doc/bundle.md

By default jicofo assumes that clients support these, and uses them in the offer (and while setting up the channels on jitsi-videobridge). You can check whether jicofo assumes the default features by looking for one of these two strings in jicofo's logs:
"Service discovery not supported" or
"Failed to discover features"
The code is here: https://github.com/jitsi/jicofo/blob/master/src/main/java/org/jitsi/jicofo/discovery/DiscoveryUtil.java#L123

Make sure your XMPP client supports disco#info, and that you are NOT advertising support for urn:ietf:rfc:5761 (rtcp-mux) and urn:ietf:rfc:5888 (bundle). Or if you are running your own jicofo, you can change the defaults.

Hope this helps,
Boris

···

On 13/12/2016 12:12, V User wrote:


#3

Thanks Boris! That was really helpful.

···

On Tue, Dec 13, 2016 at 5:59 PM, Boris Grozev <boris@jitsi.org> wrote:

Hi,

On 13/12/2016 12:12, V User wrote:

Hi everyone,

I'm currently working on writing a simple C++ client on a raspberry pi
that can connect to jitsi-meet conferences. I'm using Gloox
<https://camaya.net/gloox/> for an xmpp library and OpenWebRTC
<https://www.openwebrtc.org/> for handling the content streams.

Sounds like an interesting project!

I've

made some progress, but I'm struggling because there isn't much
mid-level documentation of jicofo and the videobridge, especially in
terms of what exactly they expect from clients (both signaling and
actual streams). I've found this jres article
<https://2013.jres.org/archives/169/paper169_article.pdf> and this
super-helpful gliffy diagram
<https://www.gliffy.com/go/publish/7649541/> but I haven't been able to
come up with anything more specific than that short of the libjitsi api
docs. If you could point me to anything already written about the "api"
for interacting with jicofo and jvb on either a signalling or an RTP
level, I'd really appreciate it.

I'm afraid there isn't much documentation available.

Some more specific questions:
What ssrc information does jvb require from a client?

It depends on the use-case. The SSRCs themselves are needed if rtcp-mux +
bundle is used, or if simulcast is used. Other than that, I don't think the
bridge itself requires them.

What does it

expect a client to do with ssrc information it sends?

The bridge will use the SSRC it generates to send RTCP messages (e.g.
keyframe requests). The clients need to handle RTCP messages coming from
this SSRC. In practice this means adding it as a source (a=ssrc line) in
the remote description, just like the SSRC of other clients.

Is the xmlns https://jitsi.org/protocol/focus documented anywhere? Are

the jitsi extensions to jingle (e.g. source-add, source-remove)
documented anywhere?

I don't think they are.

How is the client expected to handle source-add requests?

It should accept RTP/RTCP packets for this source. In practice, it again
adds a=ssrc lines to its remote description.

One specific problem I'm having right now has to do with ICE candidates
and streams. OpenWebRTC finds and advertises separate candidates for
each media stream (audio, video, and data). In the files I've attached,
you can see that my client is advertising in its session-accept jingle a
UDP port 46132 for audio and a UDP port 36943 for video. In the
videobridge logs, you can see that port advertised for audio is
validated, but the port advertised for video has a 401 (unauthorized)
error. Despite this, it logs that transport is connected for both audio
and video content. Does this mean it's sending both audio and video
streams to one port? If so, is there any way to force it to use separate
ports?

This is because of rtcp-mux and bundle. Some docs here:
https://github.com/jitsi/jitsi-videobridge/blob/master/doc/bundle.md

By default jicofo assumes that clients support these, and uses them in the
offer (and while setting up the channels on jitsi-videobridge). You can
check whether jicofo assumes the default features by looking for one of
these two strings in jicofo's logs:
"Service discovery not supported" or
"Failed to discover features"
The code is here: https://github.com/jitsi/jicof
o/blob/master/src/main/java/org/jitsi/jicofo/discovery/
DiscoveryUtil.java#L123

Make sure your XMPP client supports disco#info, and that you are NOT
advertising support for urn:ietf:rfc:5761 (rtcp-mux) and urn:ietf:rfc:5888
(bundle). Or if you are running your own jicofo, you can change the
defaults.

Hope this helps,
Boris

_______________________________________________
users mailing list
users@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/users