[jitsi-users] videobridge and audio mixing


#1

Hi,

I am using jitsi videobridge with webrtc (Note: audio only),.

the webpage says that audio mixing is "supported"
https://jitsi.org/Projects/JitsiVideobridge

Can you please clarify if this (audio mixing) is the default behavior? or
if it somehow needs to be enabled?

e.g. if I have (say) 4 people on a bridged call are 4 audio streams getting
mixed together into one stream beteween each client and the bridge? or
does the bridge just relay all streams to all peers?

if the latter then how can I get the mixed behavior? my goal is to get a
stable call experience with minimal bandwidth usage.

currently I am getting choppy / laggy experience with > 4 people on the
call (all on the same wifi network) which led to me wondering if the
bandwidth usage per client was more than I was expecting.

also: with webrtc: has anyone any experience of setting "maxaveragebitrate"
(in the a=fmtp: section of the SDP). I was wondering if that may help the
above.

Thanks,


#2

Mixing is not the default, it needs to be enabled explicitly. You can do that by adding rtp-level-relay-type='mixer' to "channel" elements in COLIBRI. See XEP-0340.
http://xmpp.org/extensions/xep-0340.html

Regards,
Boris

···

On 20/04/16 17:14, Raoul Duke wrote:

Hi,

I am using jitsi videobridge with webrtc (Note: audio only),.

the webpage says that audio mixing is "supported"
https://jitsi.org/Projects/JitsiVideobridge

Can you please clarify if this (audio mixing) is the default behavior?
  or if it somehow needs to be enabled?


#3

Thanks Boris. I'm a little unclear where this fits in the overall picture
I am using jicofo which as I understand to be responsible for the COLIBRI
interactions. do I need to add this to jicofo somehow? or is there maybe
a configuration parameter?

Thanks,

···

On Thu, Apr 21, 2016 at 10:00 AM, Boris Grozev <boris@jitsi.org> wrote:

On 20/04/16 17:14, Raoul Duke wrote:

Hi,

I am using jitsi videobridge with webrtc (Note: audio only),.

the webpage says that audio mixing is "supported"
https://jitsi.org/Projects/JitsiVideobridge

Can you please clarify if this (audio mixing) is the default behavior?
  or if it somehow needs to be enabled?

Mixing is not the default, it needs to be enabled explicitly. You can do
that by adding rtp-level-relay-type='mixer' to "channel" elements in
COLIBRI. See XEP-0340.
http://xmpp.org/extensions/xep-0340.html


#4

Jicofo doesn't support it, but it should be easy to add (see ColibriBuilder.java).

Boris

···

On 21/04/16 09:23, Raoul Duke wrote:

On Thu, Apr 21, 2016 at 10:00 AM, Boris Grozev <boris@jitsi.org > <mailto:boris@jitsi.org>> wrote:

    On 20/04/16 17:14, Raoul Duke wrote:

        Hi,

        I am using jitsi videobridge with webrtc (Note: audio only),.

        the webpage says that audio mixing is "supported"
        https://jitsi.org/Projects/JitsiVideobridge

        Can you please clarify if this (audio mixing) is the default
        behavior?
           or if it somehow needs to be enabled?

    Mixing is not the default, it needs to be enabled explicitly. You
    can do that by adding rtp-level-relay-type='mixer' to "channel"
    elements in COLIBRI. See XEP-0340.
    http://xmpp.org/extensions/xep-0340.html

Thanks Boris. I'm a little unclear where this fits in the overall
picture I am using jicofo which as I understand to be responsible for
the COLIBRI interactions. do I need to add this to jicofo somehow? or
is there maybe a configuration parameter?


#5

Thanks Boris. I think I have figured out the relevant chanegs to jicofo to
try this out. I will try it tomorrow.

While I am on the subject: would you be able to please clarify for me what
the default behavior is of jitsi-videobridge in terms of audio stream
bandwidth usage. does relayed audio simply mean that each client is
getting N individual streams realyed to them from the bridge?

so if (for example) I had a conference with 8 users each using approx
50kbit of audio bandwidth then would that result in each client having a
requirement of approx 400kbit of bandwidth usage to participate in that
conference? or is it not that simple?

I'm just trying to understand how normal users (particularly in mobile
environments) do not run into bandwidth problems with the default behavior.

I read elsewhere about the bridge dropping muted channels. is this an
initiiative to solve the bandwidth issue? how is this normally handled at
the application layer? i.e. assuming 8 people join a call and they don't
explicitly mute then do you have something whereby they are auto-muted on
inactivity? can they then somehow be auto-unmuted on the clientside? I
guess that would be an application problem.

also, I read elsewhere (
http://lists.jitsi.org/pipermail/dev/2015-August/024957.html) about a "last-n
setting" for audio such that the only the least N recently active streams
are actively carrying data (I guess). is there any plans to pursure this?
or is the "maximum-allowed-non-muted" a more likely initiative.

Thanks for your support,

RD

···

On Thu, Apr 21, 2016 at 3:57 PM, Boris Grozev <boris@jitsi.org> wrote:

On 21/04/16 09:23, Raoul Duke wrote:

On Thu, Apr 21, 2016 at 10:00 AM, Boris Grozev <boris@jitsi.org >> <mailto:boris@jitsi.org>> wrote:

    On 20/04/16 17:14, Raoul Duke wrote:

        Hi,

        I am using jitsi videobridge with webrtc (Note: audio only),.

        the webpage says that audio mixing is "supported"
        https://jitsi.org/Projects/JitsiVideobridge

        Can you please clarify if this (audio mixing) is the default
        behavior?
           or if it somehow needs to be enabled?

    Mixing is not the default, it needs to be enabled explicitly. You
    can do that by adding rtp-level-relay-type='mixer' to "channel"
    elements in COLIBRI. See XEP-0340.
    http://xmpp.org/extensions/xep-0340.html

Thanks Boris. I'm a little unclear where this fits in the overall
picture I am using jicofo which as I understand to be responsible for
the COLIBRI interactions. do I need to add this to jicofo somehow? or
is there maybe a configuration parameter?

Jicofo doesn't support it, but it should be easy to add (see
ColibriBuilder.java).


#6

Hi,

                 Hi,

                 I am using jitsi videobridge with webrtc (Note: audio
        only),.

                 the webpage says that audio mixing is "supported"
        https://jitsi.org/Projects/JitsiVideobridge

                 Can you please clarify if this (audio mixing) is the
        default
                 behavior?
                    or if it somehow needs to be enabled?

             Mixing is not the default, it needs to be enabled
        explicitly. You
             can do that by adding rtp-level-relay-type='mixer' to "channel"
             elements in COLIBRI. See XEP-0340.
        http://xmpp.org/extensions/xep-0340.html

        Thanks Boris. I'm a little unclear where this fits in the overall
        picture I am using jicofo which as I understand to be
        responsible for
        the COLIBRI interactions. do I need to add this to jicofo
        somehow? or
        is there maybe a configuration parameter?

    Jicofo doesn't support it, but it should be easy to add (see
    ColibriBuilder.java).

Thanks Boris. I think I have figured out the relevant chanegs to jicofo
to try this out. I will try it tomorrow.

While I am on the subject: would you be able to please clarify for me
what the default behavior is of jitsi-videobridge in terms of audio
stream bandwidth usage. does relayed audio simply mean that each client
is getting N individual streams realyed to them from the bridge?

so if (for example) I had a conference with 8 users each using approx
50kbit of audio bandwidth then would that result in each client having a
requirement of approx 400kbit of bandwidth usage to participate in that
conference? or is it not that simple?

Yes that's it (except that the bridge may drop some silence packets, but I don't think this is significant). Compared to the bandwidth requirements for video, audio is cheap, so it hasn't seen much work.

I'm just trying to understand how normal users (particularly in mobile
environments) do not run into bandwidth problems with the default behavior.

I read elsewhere about the bridge dropping muted channels. is this an
initiiative to solve the bandwidth issue? how is this normally handled
at the application layer? i.e. assuming 8 people join a call and they
don't explicitly mute then do you have something whereby they are
auto-muted on inactivity? can they then somehow be auto-unmuted on the
clientside? I guess that would be an application problem.

also, I read elsewhere
(http://lists.jitsi.org/pipermail/dev/2015-August/024957.html) about a
"last-n setting" for audio such that the only the least N recently
active streams are actively carrying data (I guess). is there any plans
to pursure this? or is the "maximum-allowed-non-muted" a more likely
initiative.

I don't think we have progress on any of these.

Regards,
Boris

···

On 21/04/16 19:46, Raoul Duke wrote:

On Thu, Apr 21, 2016 at 3:57 PM, Boris Grozev <boris@jitsi.org > <mailto:boris@jitsi.org>> wrote:
    On 21/04/16 09:23, Raoul Duke wrote:
        On Thu, Apr 21, 2016 at 10:00 AM, Boris Grozev <boris@jitsi.org > <mailto:boris@jitsi.org> > <mailto:boris@jitsi.org <mailto:boris@jitsi.org>>> wrote:
             On 20/04/16 17:14, Raoul Duke wrote:


#7

that would be my expectation too but I'm seeing quite bad lag / crackle
with > 4 users to the point where it doesn't seem usable. I'm not entirely
sure where to start diagnosising the problem. i.e. users can make
statements like "it is laggy and crackly" but it is hard to turn that into
a diagnosis or pointer to where the problem may lie i.e. could it be the
server, the network bandwidth, webrtc library, the Android device etc. or
some combination of those.

I would have thought that if I were to limit the per-user audio bitrate to
(say) 20 kbit then even worst case that is 200kbit for 10 users which still
doesn't seem in any way an unreasonable bandwidth expectation. would you
agree?

so maybe it isn't bandwidth at all? maybe it is somehow CPU based with
remixing the streams (on the Android devices)? if that is the case then
mixing should help. but it wasnt noticeably better unless maybe I have
configured it incorrectly.

can you suggest any other factors or areas of investigation I can dig into
to try to get a better handle on this? Does jitsi itself have any metrics
or diagnostics that would be helpful here? I was thinking I can rule out
bandwidth from the picture entirely by running the entire system on a LAN.
I will try that next. if I still have problems I know I can look somewhere
else other than bandwidth.

Currently I am using the OPUS codec. would you recommend any other
codecs/settings worth trying? particularly for mobile users.

Thanks again for your feedback.
RD

···

On Fri, Apr 22, 2016 at 3:15 AM, Boris Grozev <boris@jitsi.org> wrote:

Hi,

On 21/04/16 19:46, Raoul Duke wrote:

On Thu, Apr 21, 2016 at 3:57 PM, Boris Grozev <boris@jitsi.org >> <mailto:boris@jitsi.org>> wrote:

    On 21/04/16 09:23, Raoul Duke wrote:

        On Thu, Apr 21, 2016 at 10:00 AM, Boris Grozev <boris@jitsi.org >> <mailto:boris@jitsi.org> >> <mailto:boris@jitsi.org <mailto:boris@jitsi.org>>> wrote:

             On 20/04/16 17:14, Raoul Duke wrote:

                 Hi,

                 I am using jitsi videobridge with webrtc (Note: audio
        only),.

                 the webpage says that audio mixing is "supported"
        https://jitsi.org/Projects/JitsiVideobridge

                 Can you please clarify if this (audio mixing) is the
        default
                 behavior?
                    or if it somehow needs to be enabled?

             Mixing is not the default, it needs to be enabled
        explicitly. You
             can do that by adding rtp-level-relay-type='mixer' to
"channel"
             elements in COLIBRI. See XEP-0340.
        http://xmpp.org/extensions/xep-0340.html

        Thanks Boris. I'm a little unclear where this fits in the overall
        picture I am using jicofo which as I understand to be
        responsible for
        the COLIBRI interactions. do I need to add this to jicofo
        somehow? or
        is there maybe a configuration parameter?

    Jicofo doesn't support it, but it should be easy to add (see
    ColibriBuilder.java).

Thanks Boris. I think I have figured out the relevant chanegs to jicofo
to try this out. I will try it tomorrow.

While I am on the subject: would you be able to please clarify for me
what the default behavior is of jitsi-videobridge in terms of audio
stream bandwidth usage. does relayed audio simply mean that each client
is getting N individual streams realyed to them from the bridge?

so if (for example) I had a conference with 8 users each using approx
50kbit of audio bandwidth then would that result in each client having a
requirement of approx 400kbit of bandwidth usage to participate in that
conference? or is it not that simple?

Yes that's it (except that the bridge may drop some silence packets, but I
don't think this is significant). Compared to the bandwidth requirements
for video, audio is cheap, so it hasn't seen much work.


#8

Hi Raoul,

    Hi,

                          Hi,

                          I am using jitsi videobridge with webrtc
        (Note: audio
                 only),.

                          the webpage says that audio mixing is "supported"
        https://jitsi.org/Projects/JitsiVideobridge

                          Can you please clarify if this (audio mixing)
        is the
                 default
                          behavior?
                             or if it somehow needs to be enabled?

                      Mixing is not the default, it needs to be enabled
                 explicitly. You
                      can do that by adding rtp-level-relay-type='mixer'
        to "channel"
                      elements in COLIBRI. See XEP-0340.
        http://xmpp.org/extensions/xep-0340.html

                 Thanks Boris. I'm a little unclear where this fits in
        the overall
                 picture I am using jicofo which as I understand to be
                 responsible for
                 the COLIBRI interactions. do I need to add this to jicofo
                 somehow? or
                 is there maybe a configuration parameter?

             Jicofo doesn't support it, but it should be easy to add (see
             ColibriBuilder.java).

        Thanks Boris. I think I have figured out the relevant chanegs
        to jicofo
        to try this out. I will try it tomorrow.

        While I am on the subject: would you be able to please clarify
        for me
        what the default behavior is of jitsi-videobridge in terms of audio
        stream bandwidth usage. does relayed audio simply mean that
        each client
        is getting N individual streams realyed to them from the bridge?

        so if (for example) I had a conference with 8 users each using
        approx
        50kbit of audio bandwidth then would that result in each client
        having a
        requirement of approx 400kbit of bandwidth usage to participate
        in that
        conference? or is it not that simple?

    Yes that's it (except that the bridge may drop some silence packets,
    but I don't think this is significant). Compared to the bandwidth
    requirements for video, audio is cheap, so it hasn't seen much work.

that would be my expectation too but I'm seeing quite bad lag / crackle
with > 4 users to the point where it doesn't seem usable. I'm not
entirely sure where to start diagnosising the problem. i.e. users can
make statements like "it is laggy and crackly" but it is hard to turn
that into a diagnosis or pointer to where the problem may lie i.e.
could it be the server, the network bandwidth, webrtc library, the
Android device etc. or some combination of those.

I would have thought that if I were to limit the per-user audio bitrate
to (say) 20 kbit then even worst case that is 200kbit for 10 users which
still doesn't seem in any way an unreasonable bandwidth expectation.
  would you agree?

so maybe it isn't bandwidth at all? maybe it is somehow CPU based with
remixing the streams (on the Android devices)? if that is the case then
mixing should help. but it wasnt noticeably better unless maybe I have
configured it incorrectly.

can you suggest any other factors or areas of investigation I can dig
into to try to get a better handle on this? Does jitsi itself have any
metrics or diagnostics that would be helpful here? I was thinking I can
rule out bandwidth from the picture entirely by running the entire
system on a LAN. I will try that next. if I still have problems I know
I can look somewhere else other than bandwidth.

The chrome://webrtc-internals page should help you with debugging, although it may not be easy to use on android. You can start by looking for packet loss on the audio stream(s).

Currently I am using the OPUS codec. would you recommend any other
codecs/settings worth trying? particularly for mobile users.

I wouldn't know, really. I suppose you can try (since you're recompiling jicofo already this will be easy, just remove opus from JingleOfferFactory.java) so you can to rule out any opus-specific bugs. Modern android devices should be powerful enough to handle a couple of opus streams, so if the problem is not enough cycles for audio decoding, I would suspect the cause is somewhere else.

Regards,
Boris

···

On 22/04/16 15:02, Raoul Duke wrote:

On Fri, Apr 22, 2016 at 3:15 AM, Boris Grozev <boris@jitsi.org > <mailto:boris@jitsi.org>> wrote:
    On 21/04/16 19:46, Raoul Duke wrote:
        On Thu, Apr 21, 2016 at 3:57 PM, Boris Grozev <boris@jitsi.org > <mailto:boris@jitsi.org> > <mailto:boris@jitsi.org <mailto:boris@jitsi.org>>> wrote:
             On 21/04/16 09:23, Raoul Duke wrote:
                 On Thu, Apr 21, 2016 at 10:00 AM, Boris Grozev > <boris@jitsi.org <mailto:boris@jitsi.org> > <mailto:boris@jitsi.org <mailto:boris@jitsi.org>> > <mailto:boris@jitsi.org <mailto:boris@jitsi.org> > <mailto:boris@jitsi.org <mailto:boris@jitsi.org>>>> wrote:
                      On 20/04/16 17:14, Raoul Duke wrote: