[jitsi-dev] C/C++ client on Raspberry Pi that can connect to Jisti Meet


#1

Hi everyone,

I'm working on doing videoconferencing from a raspberry pi. I'd like to use
Jisti Meet, but the pi browser isn't performing well enough, and I'm
currently planning on writing a Jitsi Meet-compatible XMPP client in C++
(using openWebRTC for the peer-to-peer connection and gloox for the
XMPP/Jingle signalling). I'm a little nervous about this after finding
Jitsi-meet issue #96 (https://github.com/jitsi/jitsi-meet/issues/96) saying
that you can't call into a Jitsi meet room from Jitsi's XMPP client, even
though the Jitsi client does support audio and video calls. Is there any
detailed documentation of the XMPP transaction between the Jitsi Meet
webclient and the videobridge? One comment on the thread for issue #96 says
"the focus initiates a Jingle call to anyone who joins the MUC", is that
accurate? I've read the JRES paper (
https://2013.jres.org/archives/169/paper169_article.pdf) a few times, but
I'm looking for something more detailed about the specific XMPP/Jingle
messages being exchanged and what the focus user needs to do to set up the
conference before others can join. Who acts as the focus user with Meet
conferences? Where/how does Jitsi Meet use Colibri?

Any direction or resources you could point me at would be greatly
appreciated. My next steps currently are to set up a videobridge locally
and watch the prosody debug logs to see what messages are being exchanged
when a Jitsi Meet client calls in and to poke around in the lib-jitsi-meet
source code, but it'd be great to have some direction and know what to look
for.

Thanks,
Zoe


#2

Hi,

To see the xmpp messages exchanged between clients and the xmpp server
you can open two tabs in chrome using meet.jit.si, then on one of them
open the javascript console and execute: APP.conference.saveLogs()
This will create a file in which you can see all those xmpp messages.
Colibri is used by the focus, which is jicofo to talk to
jitsi-videobride, if you increase your log level in jicofo you can see
those messages in the log file.
To connect Jitsi desktop client and use it in a jitsi-meet conference,
the problem is not the xmpp part, this can be easily added. It is the
media functionality which needs update. Video vp8 is missing stuff
like handling and requesting NACKs. Generally it will work, but
receiving and sending quality will be very low.

Regards
damencho

···

On Fri, Nov 18, 2016 at 8:40 AM, V User <vuser128@gmail.com> wrote:

Hi everyone,

I'm working on doing videoconferencing from a raspberry pi. I'd like to use
Jisti Meet, but the pi browser isn't performing well enough, and I'm
currently planning on writing a Jitsi Meet-compatible XMPP client in C++
(using openWebRTC for the peer-to-peer connection and gloox for the
XMPP/Jingle signalling). I'm a little nervous about this after finding
Jitsi-meet issue #96 (https://github.com/jitsi/jitsi-meet/issues/96) saying
that you can't call into a Jitsi meet room from Jitsi's XMPP client, even
though the Jitsi client does support audio and video calls. Is there any
detailed documentation of the XMPP transaction between the Jitsi Meet
webclient and the videobridge? One comment on the thread for issue #96 says
"the focus initiates a Jingle call to anyone who joins the MUC", is that
accurate? I've read the JRES paper
(https://2013.jres.org/archives/169/paper169_article.pdf) a few times, but
I'm looking for something more detailed about the specific XMPP/Jingle
messages being exchanged and what the focus user needs to do to set up the
conference before others can join. Who acts as the focus user with Meet
conferences? Where/how does Jitsi Meet use Colibri?

Any direction or resources you could point me at would be greatly
appreciated. My next steps currently are to set up a videobridge locally and
watch the prosody debug logs to see what messages are being exchanged when a
Jitsi Meet client calls in and to poke around in the lib-jitsi-meet source
code, but it'd be great to have some direction and know what to look for.

Thanks,
Zoe

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#3

Hi again,

Sorry if this is not an appropriate request for this list. If there's any
detailed technical documentation on jicofo/jvb that you could point me to
or any better audience for technical help, let me know and I can look
there.

I've made a lot of progress on my C++ client but it still hasn't totally
come together. I eventually stumbled on this gliffy
<https://www.gliffy.com/go/publish/7649541/> diagram which answered a lot
of my questions. I now have a client set up on my raspberry pi which is
exchanging jingle messages with a jicofo instance on my machine, and it
seems like I'm receiving an audio stream but not a video stream. The client
joins the MUC for my conference, and sends a jitsi conference iq to the
focus user on my machine. The client gets a session-initiate request from
the focus, replies (using dtls and candidate information gathered by
OpenWebRTC), and the videobridge starts sending a stream of UDP packets to
the port the client offered for audio (so, 39781 in the example below). But
for some reason, the videobridge doesn't appear to be sending video data.
I'm not sure if this is related, but the videobridge is also failing to
make a SCTP connection with my client:
videobridge | JVB 2016-12-08 19:37:43.944 INFO: [251]
org.jitsi.videobridge.SctpConnection.info() SCTP socket accepted for
endpoint fc746b16
videobridge | JVB 2016-12-08 19:37:44.418 WARNING: [250]
org.jitsi.videobridge.Endpoint.warn() SCTP connection with bot not ready
yet.

I'm including relevant xmpp below and I've attached a segment of logs from
jicofo and jvb from a sample run of the program. Any guidance would be
really appreciated!

Here's the relevant xmpp message that got exchanged:
Conference request from the pi:

<iq xmlns="jabber:client" to="focus@auth.localhost" type="set"
from="zs@auth.localhost/bb518f07-7079-4a56-872e-ae08e3eb7456">
  <conference xmlns="https://jitsi.org/protocol/focus"
room="mathematical2@conference.localhost" machine-uid="abcdefg">
    <property name="channelLastN" value="-1"/>
  </conference></iq>

Jingle session-initiate from the focus (sent as soon as I open a browser
window in the same conference):

<iq xmlns=“jabber:client” type=“set”
to=“zs@auth.localhost/bb518f07-7079-4a56-872e-ae08e3eb7456”
from=“mathematical2@conference.localhost/focus”
id=“enNAYXV0aC5sb2NhbGhvc3QvYmI1MThmMDctNzA3OS00YTU2LTg3MmUtYWUwOGUzZWI3NDU2AFM4SkRZLTE0NDYA77yaX7RNZsYt4uQ/uIbrOA==”>
  <jingle xmlns=“urn:xmpp:jingle:1”
initiator=“focus@auth.localhost/focus94873832579447”
sid=“anud1t8gr40tl” action=“session-initiate”>
    <content name=“audio” creator=“initiator” senders=“both”>
      <description xmlns=“urn:xmpp:jingle:apps:rtp:1” maxptime=“60”
media=“audio”>
        <payload-type name=“opus” clockrate=“48000” id=“111” channels=“2”>
          <parameter value=“10” name=“minptime”/>
          <parameter value=“1” name=“useinbandfec”/>
        </payload-type>
        <payload-type id=“103” clockrate=“16000” name=“ISAC”/>
        <payload-type id=“104” clockrate=“32000” name=“ISAC”/>
        <payload-type id=“126” clockrate=“8000” name=“telephone-event”/>
        <rtp-hdrext xmlns=“urn:xmpp:jingle:apps:rtp:rtp-hdrext:0”
id=“1” uri=“urn:ietf:params:rtp-hdrext:ssrc-audio-level”/>
        <rtp-hdrext xmlns=“urn:xmpp:jingle:apps:rtp:rtp-hdrext:0”
id=“3” uri=“http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time”/>
        <rtcp-mux/>
        <source xmlns=“urn:xmpp:jingle:apps:rtp:ssma:0” ssrc=“2586250134”>
          <parameter value=“mixed” name=“cname”/>
          <parameter value=“mixedlabelaudio0” name=“label”/>
          <parameter value=“mixedmslabel mixedlabelaudio0” name=“msid”/>
          <parameter value=“mixedmslabel” name=“mslabel”/>
          <ssrc-info xmlns=“http://jitsi.org/jitmeet” owner=“jvb”/>
        </source>
      </description>
      <transport xmlns=“urn:xmpp:jingle:transports:ice-udp:1”
ufrag=“dtqi11b3i673pg” pwd=“2sqj5t0o8p4qpm4kjj5e4j4pi0”>
        <rtcp-mux/>
        <fingerprint xmlns=“urn:xmpp:jingle:apps:dtls:0” hash=“sha-1”
setup=“actpass”
required=“false”>D6:E1:A3:E4:8B:62:31:3E:EF:64:32:0E:40:4E:05:7D:7B:88:40:3C</fingerprint>
        <candidate type=“host” network=“0”
id=“3d981bc435787d0f3d982a022c13e95d0fffffffffb4e26c1”
ip=“172.16.38.32” component=“1” port=“4443” foundation=“1”
generation=“0” protocol=“ssltcp” priority=“2130706431”
tcptype=“passive”/>
        <candidate type=“host” protocol=“udp”
id=“3d981bc435787d0f3d982a022c13e95d0fffffffffb4e3c76”
ip=“172.16.38.32” component=“1” port=“10000” foundation=“2”
generation=“0” priority=“2130706431” network=“0”/>
      </transport>
      <bundle xmlns=“http://estos.de/ns/bundle”/>
    </content>
    <content name=“video” creator=“initiator” senders=“both”>
      <description xmlns=“urn:xmpp:jingle:apps:rtp:1” media=“video”>
        <payload-type id=“100” clockrate=“90000” name=“VP8”>
          <rtcp-fb xmlns=“urn:xmpp:jingle:apps:rtp:rtcp-fb:0”
type=“ccm” subtype=“fir”/>
          <rtcp-fb xmlns=“urn:xmpp:jingle:apps:rtp:rtcp-fb:0” type=“nack”/>
          <rtcp-fb xmlns=“urn:xmpp:jingle:apps:rtp:rtcp-fb:0”
type=“nack” subtype=“pli”/>
          <rtcp-fb xmlns=“urn:xmpp:jingle:apps:rtp:rtcp-fb:0” type=“goog-remb”/>
          <parameter value=“800” name=“x-google-start-bitrate”/>
        </payload-type>
        <rtp-hdrext xmlns=“urn:xmpp:jingle:apps:rtp:rtp-hdrext:0”
id=“3” uri=“http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time”/>
        <rtcp-mux/>
        <source xmlns=“urn:xmpp:jingle:apps:rtp:ssma:0” ssrc=“3938367210”>
          <parameter value=“mixed” name=“cname”/>
          <parameter value=“mixedlabelvideo0” name=“label”/>
          <parameter value=“mixedmslabel mixedlabelvideo0” name=“msid”/>
          <parameter value=“mixedmslabel” name=“mslabel”/>
          <ssrc-info xmlns=“http://jitsi.org/jitmeet” owner=“jvb”/>
        </source>
      </description>
      <transport xmlns=“urn:xmpp:jingle:transports:ice-udp:1”
ufrag=“dtqi11b3i673pg” pwd=“2sqj5t0o8p4qpm4kjj5e4j4pi0”>
        <rtcp-mux/>
        <fingerprint xmlns=“urn:xmpp:jingle:apps:dtls:0” hash=“sha-1”
setup=“actpass”
required=“false”>D6:E1:A3:E4:8B:62:31:3E:EF:64:32:0E:40:4E:05:7D:7B:88:40:3C</fingerprint>
        <candidate type=“host” network=“0”
id=“3d981bc435787d0f3d982a022c13e95d0fffffffffb4e26c1”
ip=“172.16.38.32” component=“1” port=“4443” foundation=“1”
generation=“0” protocol=“ssltcp” priority=“2130706431”
tcptype=“passive”/>
        <candidate type=“host” protocol=“udp”
id=“3d981bc435787d0f3d982a022c13e95d0fffffffffb4e3c76”
ip=“172.16.38.32” component=“1” port=“10000” foundation=“2”
generation=“0” priority=“2130706431” network=“0”/>
      </transport>
      <bundle xmlns=“http://estos.de/ns/bundle”/>
    </content>
    <content name=“data” creator=“initiator” senders=“both”>
      <description xmlns=“urn:xmpp:jingle:apps:rtp:1” media=“application”>
        <rtcp-mux/>
      </description>
      <transport xmlns=“urn:xmpp:jingle:transports:ice-udp:1”
ufrag=“dtqi11b3i673pg” pwd=“2sqj5t0o8p4qpm4kjj5e4j4pi0”>
        <rtcp-mux/>
        <fingerprint xmlns=“urn:xmpp:jingle:apps:dtls:0” hash=“sha-1”
setup=“actpass”
required=“false”>D6:E1:A3:E4:8B:62:31:3E:EF:64:32:0E:40:4E:05:7D:7B:88:40:3C</fingerprint>
        <sctpmap xmlns=“urn:xmpp:jingle:transports:dtls-sctp:1”
number=“5000” streams=“1024” protocol=“webrtc-datachannel”/>
        <candidate type=“host” network=“0”
id=“3d981bc435787d0f3d982a022c13e95d0fffffffffb4e26c1”
ip=“172.16.38.32” component=“1” port=“4443” foundation=“1”
generation=“0” protocol=“ssltcp” priority=“2130706431”
tcptype=“passive”/>
        <candidate type=“host” protocol=“udp”
id=“3d981bc435787d0f3d982a022c13e95d0fffffffffb4e3c76”
ip=“172.16.38.32” component=“1” port=“10000” foundation=“2”
generation=“0” priority=“2130706431” network=“0”/>
      </transport>
      <bundle xmlns=“http://estos.de/ns/bundle”/>
    </content>
    <group xmlns=“urn:xmpp:jingle:apps:grouping:0” semantics=“BUNDLE”>
      <content name=“audio”/>
      <content name=“video”/>
      <content name=“data”/>
    </group>
  </jingle></iq>

Jingle session-accept reply from the pi (constructed using
candidates/fingerprint found by OpenWebRTC):

<iq xmlns="jabber:client"
to="mathematical2@conference.localhost/focus"
id="uid-d9f31a58-00000005" type="set"
from="zs@auth.localhost/bb518f07-7079-4a56-872e-ae08e3eb7456">
  <jingle xmlns="urn:xmpp:jingle:1" action="session-accept"
responder="zs@auth.localhost/bb518f07-7079-4a56-872e-ae08e3eb7456"
sid="anud1t8gr40tl">
    <content creator="initiator" disposition="session" name="audio"
senders="both">
      <description xmlns="urn:xmpp:jingle:apps:rtp:1" media="audio">
        <rtcp-mux/>
        <payload-type name="opus" id="111" clockrate="48000" channels="2"/>
      </description>
      <transport xmlns="urn:xmpp:jingle:transports:ice-udp:1"
pwd="fjbjTRyzYb2kVaIQtq5JiT" ufrag="hsAm">
        <candidate foundation="1" ip="172.16.43.25" port="39781"
priority="2013266431" protocol="udp" rel-port="2022" type="host"/>
        <candidate foundation="2" ip="172.16.43.25" port="0"
priority="1019216383" protocol="tcp" rel-port="0" type="host"/>
        <candidate foundation="3" ip="172.16.43.25" port="37819"
priority="1015022079" protocol="tcp" rel-port="1701076846"
type="host"/>
        <candidate foundation="13" ip="38.97.68.126" port="39781"
priority="1677722111" protocol="udp" rel-addr="172.16.43.25"
rel-port="39781" type="srflx"/>
        <candidate foundation="14" ip="38.97.68.126" port="0"
priority="847249919" protocol="tcp" rel-addr="172.16.43.25"
rel-port="0" type="srflx"/>
        <candidate foundation="15" ip="38.97.68.126" port="37819"
priority="843055615" protocol="tcp" rel-addr="172.16.43.25"
rel-port="37819" type="srflx"/>
        <fingerprint xmlns="urn:xmpp:jingle:apps:dtls:0"
hash="sha-256" setup="active">DE:0F:5D:97:60:02:B7:16:B5:FD:2D:DC:4C:BB:4D:67:DB:91:D7:07:A3:BC:AD:0B:59:46:3F:03:99:D6:0A:3A</fingerprint>
      </transport>
    </content>
    <content creator="initiator" disposition="session" name="video"
senders="both">
      <description xmlns="urn:xmpp:jingle:apps:rtp:1" media="video">
        <rtcp-mux/>
        <payload-type name="VP8" id="100" clockrate="90000"
channels="1952539753">
          <rtcp-fb type="ccm" subtype="fir"/>
          <rtcp-fb type="nack" subtype="pli"/>
        </payload-type>
      </description>
      <transport xmlns="urn:xmpp:jingle:transports:ice-udp:1"
pwd="DxhYJu4ZAaqE/Ap/3J1e+S" ufrag="sz9X">
        <candidate foundation="4" ip="172.16.43.25" port="59104"
priority="20132 f66431" protocol="udp" rel-port="1848125216"
type="host"/>
        <candidate foundation="5" ip="172.16.43.25" port="0"
priority="1019216383" protocol="tcp" rel-port="1952539753"
type="host"/>
        <candidate foundation="6" ip="172.16.43.25" port="37122"
priority="1015022079" protocol="tcp" rel-port="657026156"
type="host"/>
        <candidate foundation="10" ip="38.97.68.126" port="59104"
priority="1677722111" protocol="udp" rel-addr="172.16.43.25"
rel-port="59104" type="srflx"/>
        <candidate foundation="11" ip="38.97.68.126" port="0"
priority="847249919" protocol="tcp" rel-addr="172.16.43.25"
rel-port="0" type="srflx"/>
        <candidate foundation="12" ip="38.97.68.126" port="37122"
priority="843055615" protocol="tcp" rel-addr="172.16.43.25"
rel-port="37122" type="srflx"/>
        <fingerprint xmlns="urn:xmpp:jingle:apps:dtls:0"
hash="sha-256" setup="active">DE:0F:5D:97:60:02:B7:16:B5:FD:2D:DC:4C:BB:4D:67:DB:91:D7:07:A3:BC:AD:0B:59:46:3F:03:99:D6:0A:3A</fingerprint>
      </transport>
    </content>
    <content creator="initiator" disposition="session"
name="application" senders="both">
      <description xmlns="urn:xmpp:jingle:apps:rtp:1" media="application"/>
      <transport xmlns="urn:xmpp:jingle:transports:ice-udp:1"
pwd="bcuvJEqGyCrnm4JG7zY9tu" ufrag="kVbi">
        <candidate foundation="7" ip="172.16.43.25" port="34953"
priority="2013266431" protocol="udp" rel-port="1684955491"
type="host"/>
        <candidate foundation="8" ip="172.16.43.25" port="0"
priority="1019216383" protocol="tcp" rel-port="28990928" type="host"/>
        <candidate foundation="9" ip="172.16.43.25" port="55155"
priority="1015022079" protocol="tcp" rel-port="859126068"
type="host"/>
        <fingerprint xmlns="urn:xmpp:jingle:apps:dtls:0"
hash="sha-256" setup="active">DE:0F:5D:97:60:02:B7:16:B5:FD:2D:DC:4C:BB:4D:67:DB:91:D7:07:A3:BC:AD:0B:59:46:3F:03:99:D6:0A:3A</fingerprint>
      </transport>
    </content>
  </jingle></iq>

Source-add from the focus:

<iq xmlns="jabber:client" type="set"
to="zs@auth.localhost/bb518f07-7079-4a56-872e-ae08e3eb7456"
from="mathematical2@conference.localhost/focus"
id="enNAYXV0aC5sb2NhbGhvc3QvYmI1MThmMDctNzA3OS00YTU2LTg3MmUtYWUwOGUzZWI3NDU2AFM4SkRZLTE0NTMA77yaX7RNZsYt4uQ/uIbrOA==">
  <jingle xmlns="urn:xmpp:jingle:1" sid="anud1t8gr40tl" action="source-add">
    <content name="audio">
      <description xmlns="urn:xmpp:jingle:apps:rtp:1" media="audio">
        <source xmlns="urn:xmpp:jingle:apps:rtp:ssma:0" ssrc="1025821269">
          <parameter value="Be2cdxlsOyu9eT7A" name="cname"/>
          <parameter value="d7fd83a3-af03-4faa-89ce-fde52b04418b
3b66bb00-88d9-4fa2-b89e-131ad84bda5d" name="msid"/>
          <parameter value="d7fd83a3-af03-4faa-89ce-fde52b04418b"
name="mslabel"/>
          <parameter value="3b66bb00-88d9-4fa2-b89e-131ad84bda5d" name="label"/>
          <ssrc-info xmlns="http://jitsi.org/jitmeet"
owner="mathematical2@conference.localhost/ec2d9c16"/>
        </source>
      </description>
    </content>
    <content name="video">
      <description xmlns="urn:xmpp:jingle:apps:rtp:1" media="video">
        <source xmlns="urn:xmpp:jingle:apps:rtp:ssma:0" ssrc="1954458134">
          <parameter value="2524caf9-0eef-4bd4-96b9-3c2a582d013d
dd6d8d06-ff61-4b95-a9db-401e3ad61b02" name="msid"/>
          <parameter value="Be2cdxlsOyu9eT7A" name="cname"/>
          <ssrc-info xmlns="http://jitsi.org/jitmeet"
owner="mathematical2@conference.localhost/ec2d9c16"/>
        </source>
        <source xmlns="urn:xmpp:jingle:apps:rtp:ssma:0" ssrc="4009611579">
          <parameter value="2524caf9-0eef-4bd4-96b9-3c2a582d013d
dd6d8d06-ff61-4b95-a9db-401e3ad61b02" name="msid"/>
          <parameter value="Be2cdxlsOyu9eT7A" name="cname"/>
          <ssrc-info xmlns="http://jitsi.org/jitmeet"
owner="mathematical2@conference.localhost/ec2d9c16"/>
        </source>
        <source xmlns="urn:xmpp:jingle:apps:rtp:ssma:0" ssrc="2707358922">
          <parameter value="2524caf9-0eef-4bd4-96b9-3c2a582d013d
dd6d8d06-ff61-4b95-a9db-401e3ad61b02" name="msid"/>
          <parameter value="Be2cdxlsOyu9eT7A" name="cname"/>
          <ssrc-info xmlns="http://jitsi.org/jitmeet"
owner="mathematical2@conference.localhost/ec2d9c16"/>
        </source>
        <ssrc-group xmlns="urn:xmpp:jingle:apps:rtp:ssma:0" semantics="SIM">
          <source ssrc="1954458134"/>
          <source ssrc="4009611579"/>
          <source ssrc="2707358922"/>
        </ssrc-group>
      </description>
    </content>
  </jingle></iq>

jvb_jicofo_excerpt.log (72.2 KB)

···

On Fri, Nov 18, 2016 at 9:40 AM, V User <vuser128@gmail.com> wrote:

Hi everyone,

I'm working on doing videoconferencing from a raspberry pi. I'd like to
use Jisti Meet, but the pi browser isn't performing well enough, and I'm
currently planning on writing a Jitsi Meet-compatible XMPP client in C++
(using openWebRTC for the peer-to-peer connection and gloox for the
XMPP/Jingle signalling). I'm a little nervous about this after finding
Jitsi-meet issue #96 (https://github.com/jitsi/jitsi-meet/issues/96)
saying that you can't call into a Jitsi meet room from Jitsi's XMPP client,
even though the Jitsi client does support audio and video calls. Is there
any detailed documentation of the XMPP transaction between the Jitsi Meet
webclient and the videobridge? One comment on the thread for issue #96 says
"the focus initiates a Jingle call to anyone who joins the MUC", is that
accurate? I've read the JRES paper (https://2013.jres.org/
archives/169/paper169_article.pdf) a few times, but I'm looking for
something more detailed about the specific XMPP/Jingle messages being
exchanged and what the focus user needs to do to set up the conference
before others can join. Who acts as the focus user with Meet conferences?
Where/how does Jitsi Meet use Colibri?

Any direction or resources you could point me at would be greatly
appreciated. My next steps currently are to set up a videobridge locally
and watch the prosody debug logs to see what messages are being exchanged
when a Jitsi Meet client calls in and to poke around in the lib-jitsi-meet
source code, but it'd be great to have some direction and know what to look
for.

Thanks,
Zoe