[jitsi-dev] A few questions regarding audio conference on mobile


#1

I am having some problems scaling audio-conferences to more than 6 participants, using the video-bridge.

The conferences use "translator" mode, so there is more and more downstream data for each participant who joins.

In this case, the clients are iphones using the stock WebRTC library and a custom-written signaling server (using grpc internally, and REST to jvb). The audio codec is Opus.

Normally we deploy the system for development in AWS on a t2.medium instance. Today I deployed jvb on a m5d.xlarge instance just to rule out throttling on the server as a cause. The problem persisted. Measuring the load on one ios device showed that each participant that joined the conference, added about 5k incoming data-traffic. From my perspective that's insignificant on a wi-fi network.

So, my questions are simply:

   - Are there any known problems on mobile devices, receiving more than 5 - 6 incoming audio-streams?

   - Can jvb be configured to only forward the audio-payload from active streams or the /n/ most recent active streams (I think I have seen such a feature for video-streams).

   - If we switch from translator to mixer mode, will Opus still work? (I tried that today, but it failed with an UnsupportedFormatException:

JVB 2018-06-05 16:13:22.512 INFO: [21] org.jitsi.videobridge.Content.log() CAT=stat create_channel,conf_id=26e4fa36af9b51a8,content=audio,ch_id=596fd25aa5174ff4,endp_id=null,stream=1517118374 transport=default,bun
dle=EdH9hDN6wmXasBYxXn7Ti,initiator=true,media_type=audio,relay_type=mixer
JVB 2018-06-05 16:13:22.570 SEVERE: [21] org.jitsi.impl.neomedia.MediaStreamImpl.log() Failed to create send stream for data source net.sf.fmj.media.multiplexer.RawBufferMux$RawBufferDataSource@1898fa20 and stream
  index 0 because of failed format opus/rtp, 48000.0 Hz, Stereo
javax.media.format.UnsupportedFormatException: Format of Stream not supported in RTP Session Manager
         at net.sf.fmj.media.rtp.RTPSessionMgr.createSendStream(RTPSessionMgr.java:842)
         at net.sf.fmj.media.rtp.RTPSessionMgr.createSendStream(RTPSessionMgr.java:985)
         at org.jitsi.impl.neomedia.rtp.StreamRTPManager.createSendStream(StreamRTPManager.java:121)
         at org.jitsi.impl.neomedia.MediaStreamImpl.createSendStreams(MediaStreamImpl.java:1018)
         at org.jitsi.impl.neomedia.MediaStreamImpl.startSendStreams(MediaStreamImpl.java:3172)
         at org.jitsi.impl.neomedia.MediaStreamImpl.start(MediaStreamImpl.java:3061)
         at org.jitsi.impl.neomedia.MediaStreamImpl.setDirection(MediaStreamImpl.java:2853)
         at org.jitsi.videobridge.RtpChannel.setDirection(RtpChannel.java:1348)
         at org.jitsi.videobridge.Videobridge.handleColibriConferenceIQ(Videobridge.java:942)
         at org.jitsi.videobridge.rest.HandlerImpl.doPatchConferenceJSON(HandlerImpl.java:632)
         at org.jitsi.videobridge.rest.HandlerImpl.handleColibriJSON(HandlerImpl.java:991)
         at org.jitsi.videobridge.rest.HandlerImpl.handleJSON(HandlerImpl.java:1074)
         at org.jitsi.rest.AbstractJSONHandler.handle(AbstractJSONHandler.java:340)
         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
         at org.eclipse.jetty.server.Server.handle(Server.java:497)
         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
         at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
         at java.lang.Thread.run(Thread.java:748)

I hope somebody have an idea about what's wrong - or how to configure jvb around this issue.

All the best,

Jarle

···

--
Jarle Aase

C++ Freelancer: https://cpp-freelancer.com/
Professional blog: https://lastviking.eu/
Open Source projects: https://github.com/jgaa

<<< More than 1 million lines of code written, and counting... >>>


#2

Hi Jarle,

Do you reproduce this if you use jitsi-meet mobile app and
meet.jit.si? I don't think there are any known problems on mobile. In
audio-only mode work just fine even with many people in the call.

The mixing has not been used lately and indeed there can be issues
with it. If anybody in the community is using it can help you with
that.
On meet.jit.si everything is in translator mode and we haven't had
reports as yours till now.

Regards
damencho

···

On Tue, Jun 5, 2018 at 12:22 PM Jarle Aase <jarle@lastviking.eu> wrote:

I am having some problems scaling audio-conferences to more than 6
participants, using the video-bridge.

The conferences use "translator" mode, so there is more and more
downstream data for each participant who joins.

In this case, the clients are iphones using the stock WebRTC library and
a custom-written signaling server (using grpc internally, and REST to
jvb). The audio codec is Opus.

Normally we deploy the system for development in AWS on a t2.medium
instance. Today I deployed jvb on a m5d.xlarge instance just to rule out
throttling on the server as a cause. The problem persisted. Measuring
the load on one ios device showed that each participant that joined the
conference, added about 5k incoming data-traffic. From my perspective
that's insignificant on a wi-fi network.

So, my questions are simply:

   - Are there any known problems on mobile devices, receiving more than
5 - 6 incoming audio-streams?

   - Can jvb be configured to only forward the audio-payload from active
streams or the /n/ most recent active streams (I think I have seen such
a feature for video-streams).

   - If we switch from translator to mixer mode, will Opus still work?
(I tried that today, but it failed with an UnsupportedFormatException:

JVB 2018-06-05 16:13:22.512 INFO: [21]
org.jitsi.videobridge.Content.log() CAT=stat
create_channel,conf_id=26e4fa36af9b51a8,content=audio,ch_id=596fd25aa5174ff4,endp_id=null,stream=1517118374
transport=default,bun
dle=EdH9hDN6wmXasBYxXn7Ti,initiator=true,media_type=audio,relay_type=mixer
JVB 2018-06-05 16:13:22.570 SEVERE: [21]
org.jitsi.impl.neomedia.MediaStreamImpl.log() Failed to create send
stream for data source
net.sf.fmj.media.multiplexer.RawBufferMux$RawBufferDataSource@1898fa20
and stream
  index 0 because of failed format opus/rtp, 48000.0 Hz, Stereo
javax.media.format.UnsupportedFormatException: Format of Stream not
supported in RTP Session Manager
         at
net.sf.fmj.media.rtp.RTPSessionMgr.createSendStream(RTPSessionMgr.java:842)
         at
net.sf.fmj.media.rtp.RTPSessionMgr.createSendStream(RTPSessionMgr.java:985)
         at
org.jitsi.impl.neomedia.rtp.StreamRTPManager.createSendStream(StreamRTPManager.java:121)
         at
org.jitsi.impl.neomedia.MediaStreamImpl.createSendStreams(MediaStreamImpl.java:1018)
         at
org.jitsi.impl.neomedia.MediaStreamImpl.startSendStreams(MediaStreamImpl.java:3172)
         at
org.jitsi.impl.neomedia.MediaStreamImpl.start(MediaStreamImpl.java:3061)
         at
org.jitsi.impl.neomedia.MediaStreamImpl.setDirection(MediaStreamImpl.java:2853)
         at
org.jitsi.videobridge.RtpChannel.setDirection(RtpChannel.java:1348)
         at
org.jitsi.videobridge.Videobridge.handleColibriConferenceIQ(Videobridge.java:942)
         at
org.jitsi.videobridge.rest.HandlerImpl.doPatchConferenceJSON(HandlerImpl.java:632)
         at
org.jitsi.videobridge.rest.HandlerImpl.handleColibriJSON(HandlerImpl.java:991)
         at
org.jitsi.videobridge.rest.HandlerImpl.handleJSON(HandlerImpl.java:1074)
         at
org.jitsi.rest.AbstractJSONHandler.handle(AbstractJSONHandler.java:340)
         at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
         at org.eclipse.jetty.server.Server.handle(Server.java:497)
         at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
         at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
         at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
         at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
         at java.lang.Thread.run(Thread.java:748)

I hope somebody have an idea about what's wrong - or how to configure
jvb around this issue.

All the best,

Jarle

--
Jarle Aase

C++ Freelancer: https://cpp-freelancer.com/
Professional blog: https://lastviking.eu/
Open Source projects: https://github.com/jgaa

<<< More than 1 million lines of code written, and counting... >>>

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#3

Hi Damian,

Thank you for your suggestions.

We tried to reproduce with Jitsi Meet on Android and ios with 8 participants, and we were unable to reproduce the problem. (We did experience alot of disconnects though). So it's probably something I've overlooked.

All the best,

Jarle

···

Den 06. juni 2018 06:59, skrev Damian Minkov:

Hi Jarle,

Do you reproduce this if you use jitsi-meet mobile app and
meet.jit.si? I don't think there are any known problems on mobile. In
audio-only mode work just fine even with many people in the call.

The mixing has not been used lately and indeed there can be issues
with it. If anybody in the community is using it can help you with
that.
On meet.jit.si everything is in translator mode and we haven't had
reports as yours till now.

Regards
damencho

On Tue, Jun 5, 2018 at 12:22 PM Jarle Aase <jarle@lastviking.eu> wrote:

I am having some problems scaling audio-conferences to more than 6
participants, using the video-bridge.

The conferences use "translator" mode, so there is more and more
downstream data for each participant who joins.

In this case, the clients are iphones using the stock WebRTC library and
a custom-written signaling server (using grpc internally, and REST to
jvb). The audio codec is Opus.

Normally we deploy the system for development in AWS on a t2.medium
instance. Today I deployed jvb on a m5d.xlarge instance just to rule out
throttling on the server as a cause. The problem persisted. Measuring
the load on one ios device showed that each participant that joined the
conference, added about 5k incoming data-traffic. From my perspective
that's insignificant on a wi-fi network.

So, my questions are simply:

    - Are there any known problems on mobile devices, receiving more than
5 - 6 incoming audio-streams?

    - Can jvb be configured to only forward the audio-payload from active
streams or the /n/ most recent active streams (I think I have seen such
a feature for video-streams).

    - If we switch from translator to mixer mode, will Opus still work?
(I tried that today, but it failed with an UnsupportedFormatException:

JVB 2018-06-05 16:13:22.512 INFO: [21]
org.jitsi.videobridge.Content.log() CAT=stat
create_channel,conf_id=26e4fa36af9b51a8,content=audio,ch_id=596fd25aa5174ff4,endp_id=null,stream=1517118374
transport=default,bun
dle=EdH9hDN6wmXasBYxXn7Ti,initiator=true,media_type=audio,relay_type=mixer
JVB 2018-06-05 16:13:22.570 SEVERE: [21]
org.jitsi.impl.neomedia.MediaStreamImpl.log() Failed to create send
stream for data source
net.sf.fmj.media.multiplexer.RawBufferMux$RawBufferDataSource@1898fa20
and stream
   index 0 because of failed format opus/rtp, 48000.0 Hz, Stereo
javax.media.format.UnsupportedFormatException: Format of Stream not
supported in RTP Session Manager
          at
net.sf.fmj.media.rtp.RTPSessionMgr.createSendStream(RTPSessionMgr.java:842)
          at
net.sf.fmj.media.rtp.RTPSessionMgr.createSendStream(RTPSessionMgr.java:985)
          at
org.jitsi.impl.neomedia.rtp.StreamRTPManager.createSendStream(StreamRTPManager.java:121)
          at
org.jitsi.impl.neomedia.MediaStreamImpl.createSendStreams(MediaStreamImpl.java:1018)
          at
org.jitsi.impl.neomedia.MediaStreamImpl.startSendStreams(MediaStreamImpl.java:3172)
          at
org.jitsi.impl.neomedia.MediaStreamImpl.start(MediaStreamImpl.java:3061)
          at
org.jitsi.impl.neomedia.MediaStreamImpl.setDirection(MediaStreamImpl.java:2853)
          at
org.jitsi.videobridge.RtpChannel.setDirection(RtpChannel.java:1348)
          at
org.jitsi.videobridge.Videobridge.handleColibriConferenceIQ(Videobridge.java:942)
          at
org.jitsi.videobridge.rest.HandlerImpl.doPatchConferenceJSON(HandlerImpl.java:632)
          at
org.jitsi.videobridge.rest.HandlerImpl.handleColibriJSON(HandlerImpl.java:991)
          at
org.jitsi.videobridge.rest.HandlerImpl.handleJSON(HandlerImpl.java:1074)
          at
org.jitsi.rest.AbstractJSONHandler.handle(AbstractJSONHandler.java:340)
          at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
          at org.eclipse.jetty.server.Server.handle(Server.java:497)
          at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
          at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
          at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
          at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
          at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
          at java.lang.Thread.run(Thread.java:748)

I hope somebody have an idea about what's wrong - or how to configure
jvb around this issue.

All the best,

Jarle

--
Jarle Aase

C++ Freelancer: https://cpp-freelancer.com/
Professional blog: https://lastviking.eu/
Open Source projects: https://github.com/jgaa

<<< More than 1 million lines of code written, and counting... >>>

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
Jarle Aase
mobile: +359 8999 35657 skype: jaase.fb

C++ Freelancer: https://cpp-freelancer.com/
Professional blog: https://lastviking.eu/
Open Source projects: https://github.com/jgaa

<<< More than 1 million lines of code written, and counting... >>>