[jitsi-dev] On Bitrate Adaptation for GSoC


#1

Hi Everyone,
Am preparing to work on work on the bitrate adaptation for Google Summer of Code and I have a few questions.
   - First, Jitsi uses the RTP and RTCP protocol.

   - RTP/RTCP provides control mechanisms for media streaming and runs on top the UDP protocol.

   - and Video data(decoded/encoded bytes) is passed from RTP to Client program or vice versa.

   - In simple terms and using the OSI model, am expected to add a mechanism to control the rate at which data (bits) is sent just above layer 4 but below layer 7 in a way such that the mechanism is network aware (bandwidth aware).

   - Can anyone direct me to a folder were I can locate the already existing implementation of the google congestion control algorithm. In colloquial terms or intution, this is layer 4.X - 5 of the OSI were RTP and RTCP terminates (I intend extending the file to incorporate some libraries other clients can use in 2.)

   - I also want were I can find the folder were the client java file resides. I intend to be able to import a library from 1). above to vary the value of the bitrate in real time. In Colloquial terms, I infer this is in layer 6.X - 7 of the OSI model.

Please correct me if am wrong.
A pictorial map of the repo will also be nice. Probably a feature the the repo may want to adopt as the project grows.
Thanks,Regards,Julian.


#2

Hi Julian,

Hi Everyone,

Am preparing to work on work on the bitrate adaptation for Google
Summer of Code and I have a few questions.

Welcome! We're glad and exited that you are interested in this project. See below for some answers.

1. First, Jitsi uses the RTP and RTCP protocol.
2. RTP/RTCP provides control mechanisms for media streaming and runs on
    top the UDP protocol.

Correct!

3. and Video data(decoded/encoded bytes) is passed from RTP to Client
    program or vice versa.

Yes, but note that we implement our own RTP stack, so the distinction between RTP stack and the client may be a bit vague.

4. In simple terms and using the OSI model, am expected to add a
    mechanism to control the rate at which data (bits) is sent just
    above layer 4 but below layer 7 in a way such that the mechanism is
    network aware (bandwidth aware).

You are correct that the mechanism for bandwidth estimation works on top of RTP/RTCP. See below for more on the scope of the project.

  * Can anyone direct me to a folder were I can locate the already
    existing implementation of the google congestion control algorithm.
    In colloquial terms or intution, this is layer 4.X - 5 of the OSI
    were RTP and RTCP terminates (I intend extending the file to
    incorporate some libraries other clients can use in 2.)

The implementation is in our library, libjitsi[0]. You can look for "bandwidth estimation".

  * I also want were I can find the folder were the client java file
    resides. I intend to be able to import a library from 1). above to
    vary the value of the bitrate in real time. In Colloquial terms, I
    infer this is in layer 6.X - 7 of the OSI model.

I'm afraid our projects aren't organized that closely around the OSI model. But you can take a look at the MediaAwareCallPeer and CallPeerMediaHandler classes in the jitsi repo[1].

Please correct me if am wrong.

A pictorial map of the repo will also be nice. Probably a feature the
the repo may want to adopt as the project grows.

About the scope of the project: we now have an implementation of the bandwidth estimation algorithm (GCC) in the library, and we are actively working on improvements in this part. Because of this, we would like to broaden the scope of the project, and improve the VP8 video support in Jitsi in other ways as well. We have the following tasks:
1. Update to the latest libvpx version
2. Fixes and improvements to the VP8 encoder in libjitsi, including support to dynamically change the target bitrate
3. Enable the use of bandwidth estimation and set the encoder's target bitrate accordingly
4. Enable the use of RTCP NACK (negative acknowledgements) to request packets lost over the network.
5. Re-transmit packets requested via NACKs.

These last two are also features already supported in the library, but not in the client.

I think that all of the above could potentially be done in one GSoC project, but they are not all necessary. A successful GSoC project can do 3-4 of them for example.

We will update the project page with the info above shortly.

Regards,
Boris

[0] https://github.com/jitsi/libjitsi
[1] https://github.com/jitsi/jitsi

···

On 03/03/2017 15:01, jcchuks@ymail.com wrote:

Thanks,
Regards,
Julian.

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#3

blockquote, div.yahoo_quoted { margin-left: 0 !important; border-left:1px #715FFA solid !important; padding-left:1ex !important; background-color:white !important; } Thanks Boris for the detailed response, the task list and the links. The task list looks feasible and interesting. I might ask more later.
Thanks,Regards,Julian

Hi Julian,

Hi Everyone,

Am preparing to work on work on the bitrate adaptation for Google
Summer of Code and I have a few questions.

Welcome! We're glad and exited that you are interested in this project.
See below for some answers.

1. First, Jitsi uses the RTP and RTCP protocol.
2. RTP/RTCP provides control mechanisms for media streaming and runs on
top the UDP protocol.

Correct!

3. and Video data(decoded/encoded bytes) is passed from RTP to Client
program or vice versa.

Yes, but note that we implement our own RTP stack, so the distinction
between RTP stack and the client may be a bit vague.

4. In simple terms and using the OSI model, am expected to add a
mechanism to control the rate at which data (bits) is sent just
above layer 4 but below layer 7 in a way such that the mechanism is
network aware (bandwidth aware).

You are correct that the mechanism for bandwidth estimation works on top
of RTP/RTCP. See below for more on the scope of the project.

* Can anyone direct me to a folder were I can locate the already
existing implementation of the google congestion control algorithm.
In colloquial terms or intution, this is layer 4.X - 5 of the OSI
were RTP and RTCP terminates (I intend extending the file to
incorporate some libraries other clients can use in 2.)

The implementation is in our library, libjitsi[0]. You can look for
"bandwidth estimation".

* I also want were I can find the folder were the client java file
resides. I intend to be able to import a library from 1). above to
vary the value of the bitrate in real time. In Colloquial terms, I
infer this is in layer 6.X - 7 of the OSI model.

I'm afraid our projects aren't organized that closely around the OSI
model. But you can take a look at the MediaAwareCallPeer and
CallPeerMediaHandler classes in the jitsi repo[1].

Please correct me if am wrong.

A pictorial map of the repo will also be nice. Probably a feature the
the repo may want to adopt as the project grows.

About the scope of the project: we now have an implementation of the
bandwidth estimation algorithm (GCC) in the library, and we are actively
working on improvements in this part. Because of this, we would like to
broaden the scope of the project, and improve the VP8 video support in
Jitsi in other ways as well. We have the following tasks:
1. Update to the latest libvpx version
2. Fixes and improvements to the VP8 encoder in libjitsi, including
support to dynamically change the target bitrate
3. Enable the use of bandwidth estimation and set the encoder's target
bitrate accordingly
4. Enable the use of RTCP NACK (negative acknowledgements) to request
packets lost over the network.
5. Re-transmit packets requested via NACKs.

These last two are also features already supported in the library, but
not in the client.

I think that all of the above could potentially be done in one GSoC
project, but they are not all necessary. A successful GSoC project can
do 3-4 of them for example.

We will update the project page with the info above shortly.

Regards,
Boris

[0] https://github.com/jitsi/libjitsi
[1] https://github.com/jitsi/jitsi

···

Sent from Yahoo Mail for iPhone
On Monday, March 6, 2017, 6:51 PM, Boris Grozev <boris@jitsi.org> wrote:
On 03/03/2017 15:01, jcchuks@ymail.com wrote:

Thanks,
Regards,
Julian.

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#4

Hello Boris,
I compiled and ran(build, made, run) jitsi just to see the jitsi app working. It worked was able to hook it up to google hangout.Then, I tried to make the libjitsi (to libjitsi.jar) library and replace the libjitsi-1.0-20170207.194207-242.jar in the jitsi/lib/installer-exclude folder.
I built and made libjitsi library in to libjitsi.jar.
I got the following warnings on "ant build -lib /path/to/ant"maven-artifact:dependencies] [WARNING] POM for 'org.osgi:org.osgi.core:pom:4.3.1:compile' is invalid.
[maven-artifact:dependencies]
[maven-artifact:dependencies] Its dependencies (if any) will NOT be available to the current build.
[maven-artifact:dependencies] [WARNING] POM for 'org.osgi:org.osgi.core:pom:4.3.1:compile' is invalid.
[maven-artifact:dependencies]
[maven-artifact:dependencies] Its dependencies (if any) will NOT be available to the current build.

also when I did "ant make",I got
[maven-artifact:dependencies] [WARNING] POM for 'org.osgi:org.osgi.core:pom:4.3.1:compile' is invalid.
[maven-artifact:dependencies]
[maven-artifact:dependencies] Its dependencies (if any) will NOT be available to the current build.
[maven-artifact:dependencies] [WARNING] POM for 'org.osgi:org.osgi.core:pom:4.3.1:compile' is invalid.
[maven-artifact:dependencies]
[maven-artifact:dependencies] Its dependencies (if any) will NOT be available to the current build.

compile:
[javac] /home/user/IdeaProjects/libjitsi/build.xml:86: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable buildsjar:
make:
BUILD SUCCESSFUL
Total time: 3 seconds

Both build and make completed successfully.
Surprisingly, the jar file turned out 2mb. I looked at the libjitsi-1.0-20170207.194207-242 and it was 9mb. Not withstanding, I copied the libjitsi.jar to the same folder as the libjitsi-1.0-20170207.194207-242, and edited the buildfile in jitsi (build.xml) at line 981 top point to libjitsi.jar. It compiled successfully but when I ran it, I got

[java] ERROR: Bundle net.java.sip.communicator.service.util [96] Error starting reference:file:sc-bundles/util.jar (org.osgi.framework.BundleException: Unable to resolve net.java.sip.communicator.service.util [96](R 96.0): missing requirement [net.java.sip.communicator.service.util [96](R 96.0)] osgi.wiring.package; (osgi.wiring.package=net.java.sip.communicator.service.gui.call) [caused by: Unable to resolve net.java.sip.communicator.service.gui [58](R 58.0): missing requirement [net.java.sip.communicator.service.gui [58](R 58.0)] osgi.wiring.package; (osgi.wiring.package=org.jitsi.service.resources)] Unresolved requirements: [[net.java.sip.communicator.service.util [96](R 96.0)] osgi.wiring.package; (osgi.wiring.package=net.java.sip.communicator.service.gui.call)])
[java] ERROR: Bundle net.java.sip.communicator.fileaccess [99] Error starting reference:file:sc-bundles/fileaccess.jar (org.osgi.framework.BundleException: Unable to resolve net.java.sip.communicator.fileaccess [99](R 99.0): missing requirement [net.java.sip.communicator.fileaccess [99](R 99.0)] osgi.wiring.package; (osgi.wiring.package=org.jitsi.service.fileaccess) Unresolved requirements: [[net.java.sip.communicator.fileaccess [99](R 99.0)] osgi.wiring.package; (osgi.wiring.package=org.jitsi.service.fileaccess)])
.........and many more
I see packages were missing. Is there something I need to take note of when compiling libjitsi library from source.

Thanks.
Regards,Julian

#yiv0172610620 blockquote, #yiv0172610620 div.yiv0172610620yahoo_quoted {margin-left:0 !important;border-left:1px #715FFA solid !important;padding-left:1ex !important;background-color:white;}Thanks Boris for the detailed response, the task list and the links. The task list looks feasible and interesting. I might ask more later.
Thanks,Regards,Julian

Hi Julian,

Hi Everyone,

Am preparing to work on work on the bitrate adaptation for Google
Summer of Code and I have a few questions.

Welcome! We're glad and exited that you are interested in this project.
See below for some answers.

1. First, Jitsi uses the RTP and RTCP protocol.
2. RTP/RTCP provides control mechanisms for media streaming and runs on
top the UDP protocol.

Correct!

3. and Video data(decoded/encoded bytes) is passed from RTP to Client
program or vice versa.

Yes, but note that we implement our own RTP stack, so the distinction
between RTP stack and the client may be a bit vague.

4. In simple terms and using the OSI model, am expected to add a
mechanism to control the rate at which data (bits) is sent just
above layer 4 but below layer 7 in a way such that the mechanism is
network aware (bandwidth aware).

You are correct that the mechanism for bandwidth estimation works on top
of RTP/RTCP. See below for more on the scope of the project.

* Can anyone direct me to a folder were I can locate the already
existing implementation of the google congestion control algorithm.
In colloquial terms or intution, this is layer 4.X - 5 of the OSI
were RTP and RTCP terminates (I intend extending the file to
incorporate some libraries other clients can use in 2.)

The implementation is in our library, libjitsi[0]. You can look for
"bandwidth estimation".

* I also want were I can find the folder were the client java file
resides. I intend to be able to import a library from 1). above to
vary the value of the bitrate in real time. In Colloquial terms, I
infer this is in layer 6.X - 7 of the OSI model.

I'm afraid our projects aren't organized that closely around the OSI
model. But you can take a look at the MediaAwareCallPeer and
CallPeerMediaHandler classes in the jitsi repo[1].

Please correct me if am wrong.

A pictorial map of the repo will also be nice. Probably a feature the
the repo may want to adopt as the project grows.

About the scope of the project: we now have an implementation of the
bandwidth estimation algorithm (GCC) in the library, and we are actively
working on improvements in this part. Because of this, we would like to
broaden the scope of the project, and improve the VP8 video support in
Jitsi in other ways as well. We have the following tasks:
1. Update to the latest libvpx version
2. Fixes and improvements to the VP8 encoder in libjitsi, including
support to dynamically change the target bitrate
3. Enable the use of bandwidth estimation and set the encoder's target
bitrate accordingly
4. Enable the use of RTCP NACK (negative acknowledgements) to request
packets lost over the network.
5. Re-transmit packets requested via NACKs.

These last two are also features already supported in the library, but
not in the client.

I think that all of the above could potentially be done in one GSoC
project, but they are not all necessary. A successful GSoC project can
do 3-4 of them for example.

We will update the project page with the info above shortly.

Regards,
Boris

[0] https://github.com/jitsi/libjitsi
[1] https://github.com/jitsi/jitsi

···

On Tuesday, March 7, 2017 10:18 PM, CHIMAOBI JULIAN <jcchuks@ymail.com> wrote:
Sent from Yahoo Mail for iPhone
On Monday, March 6, 2017, 6:51 PM, Boris Grozev <boris@jitsi.org> wrote:
On 03/03/2017 15:01, jcchuks@ymail.com wrote:

Thanks,
Regards,
Julian.

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#5

Hi Julian,

···

On 16/03/2017 20:07, CHIMAOBI JULIAN wrote:

Hello Boris,

I compiled and ran(build, made, run) jitsi just to see the jitsi app
working. It worked was able to hook it up to google hangout.
Then, I tried to make the libjitsi (to /libjitsi/./jar/) library and
replace the /libjitsi-1.0-20170207.194207-242.jar/ in the
/jitsi/lib/installer-exclude/ folder.
I built and made libjitsi library in to/libjitsi.jar. /

I think you need to build libjitsi with maven (mvn install) and then copy the file from the "target" directory. Is this what you have been doing?

Regards,
Boris


#6

Hi Boris,
Thanks, got it. mvn install worked.
For this task;
   
   - 3. Enable the use of bandwidth estimation and set the encoder's target
bitrate accordingly
I think I figured out were changes need to be made - jitsi/src/net/java/sip/communicatior/impl/neomedia/MediaConfigurationImpl.java in createVideoAdvancedSettings() method

1st if I implement the interface from the libjitsi/src/org/jitsi/service/neomedia/rtp/bandwidthEstimator.java in the jitsi project, I can be able to register a listener through the BandwidthEstimator's "addListener(Listener listener)" which retrieves the current estimated bandwidth using the BandwidthEstimator's "getLatestEstimate()";,

Then I can use some "bitrate estimation equation/formular" which am currently looking in to, to set the new bitrate in the JSpinner videoBitrate. Already, I see a changeListener has been added to the JSpinner to pick up any bit rate changes.

Q1. Please is the above reasoning sound?
One other thing, I wanted to test the video call with another contact on google talk (hangout) using the jitsi client. But the video call button wasn't enabled (Both the libjitsi library I built and the old one worked but the video wasn't enabled). Using jitsi client for chatting via works.
my webcam works when viewed under devices.(ie. from jitsi chat console > Tools >options>device)
Q2) A quick question, What happens when all encoding options are checked (> Tools >options> encoding)?Q3) How do I enable the video button ("Call contact")

Thanks,Julian.

Hi Julian,

···

On Friday, March 17, 2017 1:13 AM, Boris Grozev <boris@jitsi.org> wrote:

On 16/03/2017 20:07, CHIMAOBI JULIAN wrote:

Hello Boris,

I compiled and ran(build, made, run) jitsi just to see the jitsi app
working. It worked was able to hook it up to google hangout.
Then, I tried to make the libjitsi (to /libjitsi/./jar/) library and
replace the /libjitsi-1.0-20170207.194207-242.jar/ in the
/jitsi/lib/installer-exclude/ folder.
I built and made libjitsi library in to/libjitsi.jar. /

I think you need to build libjitsi with maven (mvn install) and then
copy the file from the "target" directory. Is this what you have been doing?

Regards,
Boris


#7

Hi Julian,

Hi Boris,
Thanks, got it. mvn install worked.

For this task;

  * 3. Enable the use of bandwidth estimation and set the encoder's target

bitrate accordingly

I think I figured out were changes need to be made -
jitsi/src/net/java/sip/communicatior/impl/neomedia/MediaConfigurationImpl.java
in createVideoAdvancedSettings() method

1st if I implement the interface from the
libjitsi/src/org/jitsi/service/neomedia/rtp/bandwidthEstimator.java in
the jitsi project, I can be able to register a listener through the
BandwidthEstimator's "addListener(Listener listener)" which retrieves
the current estimated bandwidth using the BandwidthEstimator's
"getLatestEstimate()";,

Then I can use some "bitrate estimation equation/formular" which am
currently looking in to, to set the new bitrate in the JSpinner
videoBitrate. Already, I see a changeListener has been added to the
JSpinner to pick up any bit rate changes.

Q1. Please is the above reasoning sound?

Good job researching the problem! I think what you describe could work (I'm a bit rusty with my knowledge of the jitsi client itself). However, I would like to do this this in a different way. The Jitsi client should just tell libjitsi (i.e. the MediaStream instance) to use adaptive rate control, and the library should handle it from there.

I think that right now we don't support setting a target bitrate on the encoder dynamically, so that would need to be implemented.

One other thing, I wanted to test the video call with another contact on
google talk (hangout) using the jitsi client. But the video call button
wasn't enabled (Both the libjitsi library I built and the old one worked
but the video wasn't enabled). Using jitsi client for chatting via works.
my webcam works when viewed under devices.(ie. from jitsi chat console >
Tools >options>device)

Q2) A quick question, What happens when all encoding options are checked
(> Tools >options> encoding)?

I'm not sure I understand the question. Is it is about which encoding ends up being used? When a call is initialized there is an offer-answer procedure, in which the two sides exchange their lists of supported formats. They then use the highest priority format which is supported by both.

Q3) How do I enable the video button ("Call contact")

I think this might have to do with hangouts. Can you try with some regular XMPP accounts? There is a list of public XMPP servers here:
https://xmpp.net/directory.php

Regards,
Boris

···

On 17/03/2017 14:55, CHIMAOBI JULIAN wrote:


#8

Hi Boris,
Thanks for your response.I used regular XMPP accounts and it worked, video calls and desktop sharing became enabled although if I try to initiate a call(voice/video/desktop sharing) from any of the two ends (On windows - downloaded jitsi msi and Ubuntu running compiled jitsi version)
The caller displays:
Call ended by remote side. Reason: failed-application. Error: Could not establish connection (ICE failed and no relay found)
and callee displays - Error: Could not establish connection (ICE failed and no relay found).
As for the google talks, voice calls work. I don't know what might cause the error though I don't feel its the jitsi client. I will still re-try outside school network.

With regards this,
   
   - "The Jitsi client should just tell libjitsi (i.e. the MediaStream instance) to use adaptive rate
control, and the library should handle it from there."

This makes sense. I will look more into this. Thanks.

Thanks,Regards,Julian.

Hi Julian,

Hi Boris,
Thanks, got it. mvn install worked.

For this task;

* 3. Enable the use of bandwidth estimation and set the encoder's target

bitrate accordingly

I think I figured out were changes need to be made -
jitsi/src/net/java/sip/communicatior/impl/neomedia/MediaConfigurationImpl.java
in createVideoAdvancedSettings() method

1st if I implement the interface from the
libjitsi/src/org/jitsi/service/neomedia/rtp/bandwidthEstimator.java in
the jitsi project, I can be able to register a listener through the
BandwidthEstimator's "addListener(Listener listener)" which retrieves
the current estimated bandwidth using the BandwidthEstimator's
"getLatestEstimate()";,

Then I can use some "bitrate estimation equation/formular" which am
currently looking in to, to set the new bitrate in the JSpinner
videoBitrate. Already, I see a changeListener has been added to the
JSpinner to pick up any bit rate changes.

Q1. Please is the above reasoning sound?

Good job researching the problem! I think what you describe could work
(I'm a bit rusty with my knowledge of the jitsi client itself). However,
I would like to do this this in a different way. The Jitsi client should
just tell libjitsi (i.e. the MediaStream instance) to use adaptive rate
control, and the library should handle it from there.

I think that right now we don't support setting a target bitrate on the
encoder dynamically, so that would need to be implemented.

One other thing, I wanted to test the video call with another contact on
google talk (hangout) using the jitsi client. But the video call button
wasn't enabled (Both the libjitsi library I built and the old one worked
but the video wasn't enabled). Using jitsi client for chatting via works.
my webcam works when viewed under devices.(ie. from jitsi chat console >
Tools >options>device)

Q2) A quick question, What happens when all encoding options are checked
(> Tools >options> encoding)?

I'm not sure I understand the question. Is it is about which encoding
ends up being used? When a call is initialized there is an offer-answer
procedure, in which the two sides exchange their lists of supported
formats. They then use the highest priority format which is supported by
both.

Q3) How do I enable the video button ("Call contact")

I think this might have to do with hangouts. Can you try with some
regular XMPP accounts? There is a list of public XMPP servers here:
https://xmpp.net/directory.php

Regards,
Boris

···

On Friday, March 17, 2017 5:00 PM, Boris Grozev <boris@jitsi.org> wrote:
On 17/03/2017 14:55, CHIMAOBI JULIAN wrote: