Need help for scaling and high E2E values

Hi,

we need someone, who looks at our jitsi installation, since it will not scale. It works bad, if there are many users in one room. Users using firefox, chrome and apps. We see high E2E values in these sessions. Hopefully its just an configuration issue.

How many users? What is the size of the conferences? What is the available bandwidth to the machine? Have you consider deploying multiple bridges to allow more bandwidth to the system?
Firefox can bring a lot of bandwidth to bigger conferences.

Sorry for missing that:

120 persons divided into 12-15 conferences. The conference that I had problems had 17 people.

Hardware:
• Jicofo, Prosody, etc. (Xeon 12 Cores, 128 GB RAM), 1 Gbps
• Videobridge 1 (Xeon 16 Cores, 64 GB RAM), 1 Gbps
• Videobrdige 2 (Xeon 24 Cores, 196 GB RAM), 10 Gbps

using Firefox -> bandwith for all people is high?

Yes! It does not use simulcast so we cannot drop hd and send a thumbnail, so whatever is sent is multiplied to other users, even hd is shown in the thumb. It becomes worst if desktop with high resolution is shared.

Also I suppose you are using latest from stable. I think it is worth switching to using jvb2 for the machines with videobridge.
We will soon move it to stable, just need a little more time to manage with all fires.

We used the latest version form unstable, since it is the same version as on jitsi-meet public. Also changed to prosody 0.11.x, since the chrome bug with ghot users was annoying.
We will switch all jvb to jvb2, maybe it will help…

Some metrics from grafana… also some more logs after switching to jvb2:

2020-03-20 10:02:01.371 INFO: [48] org.ice4j.ice.harvest.MappingCandidateHarvesters.initialize: Initialized mapping harvesters (delay=467ms).  stunDiscoveryFailed=false
2020-03-20 10:02:01.373 INFO: [45] org.ice4j.ice.harvest.AbstractUdpListener.<init>: Initialized AbstractUdpListener with address 82.165.203.102:10000/udp. Receive buffer size 2097152 (asked for 10485760)
2020-03-20 10:02:01.374 INFO: [45] org.ice4j.ice.harvest.SinglePortUdpHarvester.<init>: Initialized SinglePortUdpHarvester with address 82.165.203.102:10000/udp


2020-03-20 10:50:53.030 INFO: [59] org.ice4j.ice.harvest.AbstractUdpListener$MySocket.addBuffer: Dropping a packet because the queue is full.
2020-03-20 10:50:53.030 INFO: [59] org.ice4j.ice.harvest.AbstractUdpListener$MySocket.addBuffer: Dropping a packet because the queue is full.
2020-03-20 10:50:53.031 INFO: [59] org.ice4j.ice.harvest.AbstractUdpListener$MySocket.addBuffer: Dropping a packet because the queue is full.
2020-03-20 10:50:53.032 INFO: [59] org.ice4j.ice.harvest.AbstractUdpListener$MySocket.addBuffer: Dropping a packet because the queue is full.
2020-03-20 10:50:53.034 INFO: [59] org.ice4j.ice.harvest.AbstractUdpListener$MySocket.addBuffer: Dropping a packet because the queue is full.
2020-03-20 10:50:53.035 INFO: [59] org.ice4j.ice.harvest.AbstractUdpListener$MySocket.addBuffer: Dropping a packet because the queue is full.

In the jvb1.0 (similar?) logs:

JVB 2020-03-20 09:00:10.559 INFO: [72358] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Times reset() called: 1
JVB 2020-03-20 09:00:10.559 INFO: [72358] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Times grow() called: 3
JVB 2020-03-20 09:00:10.559 INFO: [72358] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Packets dropped because full: 486
JVB 2020-03-20 09:00:10.559 INFO: [72358] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Packets dropped while shrinking: 1
JVB 2020-03-20 09:00:10.559 INFO: [72358] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Packets dropped because they were late: 0
JVB 2020-03-20 09:00:10.559 INFO: [72358] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Packets dropped because they were late by more than MAX_SIZE: 0
JVB 2020-03-20 09:00:10.559 INFO: [72358] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Packets dropped in reset(): 0
JVB 2020-03-20 09:00:10.560 INFO: [72358] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Max size reached: 16
JVB 2020-03-20 09:00:10.560 INFO: [72358] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Adaptive jitter buffer mode was enabled
JVB 2020-03-20 09:00:10.560 INFO: [72358] net.sf.fmj.media.Log.info() Stopping RTPSourceStream.
--
JVB 2020-03-20 09:00:13.401 INFO: [72369] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Total packets added: 1
JVB 2020-03-20 09:00:13.401 INFO: [72369] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Times reset() called: 2
JVB 2020-03-20 09:00:13.401 INFO: [72369] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Times grow() called: 0
JVB 2020-03-20 09:00:13.402 INFO: [72369] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Packets dropped because full: 0
JVB 2020-03-20 09:00:13.402 INFO: [72369] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Packets dropped while shrinking: 0
JVB 2020-03-20 09:00:13.402 INFO: [72369] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Packets dropped because they were late: 0
JVB 2020-03-20 09:00:13.402 INFO: [72369] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Packets dropped because they were late by more than MAX_SIZE: 0
JVB 2020-03-20 09:00:13.402 INFO: [72369] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Packets dropped in reset(): 1
JVB 2020-03-20 09:00:13.402 INFO: [72369] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Max size reached: 4
JVB 2020-03-20 09:00:13.402 INFO: [72369] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Adaptive jitter buffer mode was enabled
JVB 2020-03-20 09:00:13.402 INFO: [72369] net.sf.fmj.media.Log.info() Stopping RTPSourceStream.
--
JVB 2020-03-20 09:00:13.869 INFO: [150864] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Total packets added: 1
JVB 2020-03-20 09:00:13.869 INFO: [150864] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Times reset() called: 1
JVB 2020-03-20 09:00:13.869 INFO: [150864] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Times grow() called: 0
JVB 2020-03-20 09:00:13.869 INFO: [150864] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Packets dropped because full: 0
JVB 2020-03-20 09:00:13.869 INFO: [150864] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Packets dropped while shrinking: 0
JVB 2020-03-20 09:00:13.869 INFO: [150864] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Packets dropped because they were late: 0
JVB 2020-03-20 09:00:13.869 INFO: [150864] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Packets dropped because they were late by more than MAX_SIZE: 0
JVB 2020-03-20 09:00:13.869 INFO: [150864] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Packets dropped in reset(): 0
JVB 2020-03-20 09:00:13.870 INFO: [150864] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Max size reached: 4
JVB 2020-03-20 09:00:13.870 INFO: [150864] net.sf.fmj.media.Log.info() net.sf.fmj.media.rtp.RTPSourceStream Adaptive jitter buffer mode was enabled
JVB 2020-03-20 09:00:13.870 INFO: [150864] net.sf.fmj.media.Log.info() Stopping RTPSourceStream.

JVB 2020-03-19 23:04:53.633 INFO: [41] org.ice4j.ice.harvest.AbstractUdpListener.<init>() Initialized AbstractUdpListener with address 85.214.192.64:10000/udp. Receive buffer size 2097152 (asked for 10485760)

Yep, this is the buffer I said you should increase:

net.core.rmem_max=10485760
net.core.netdev_max_backlog=100000

Do you have stat for “packet receive errors” from executing netstat -anus? This is the way to detect those udp drops, the kernel is dropping them and they do not reach jvb.

Hey Benjamin,

A few quick comments:

  1. From the conferences and participants graph you can see load balancing is not working as expected. How many jicofo instances do you have (and what version)? Do you have graphs for bitrate or packet rate? Or CPU usage and system load?
  2. The loss_rate, rtt, and jitter stats are not consistent. You can ignore them.
  3. How many “AbstractUdpListener$MySocket.addBuffer: Dropping a packet” do you see? Which exact version of jitsi-videobridge are you using? There was a change a few weeks ago that was supposed to reduce/fix some of these.
  4. The jitter buffer logs from the old bridge are not relevant

Boris

1 Like

Hey Boris,

thanks your comments:

  1. We sometimes changed the configuration and stopped some jvbs and restartet them. I thought the loadbalancing would work, since all servers get roughly the same amount on conferences.

meeting2
Jicofo Server + JVB 1.0 (the jvb was disabled today in the morning) (latest unstable version, since we would like to come over the ghot users problem in chrome -> using the same version as it is on the public version) + Prosody 0.11.4 ( 0.10.X did not work well for us) (Xeon 12 Cores, 128 GB RAM, 1 Gbits)
started at 09:30 the jvb service to take another server down

meeting3
JVB 1.0 (Xeon 16 Cores, 64 GB RAM, 1Gbits) (latest unstable version via apt-get, installed two days ago)
stopped today at 09:30, because we thought another jvb would work better

meeting4
JVB 1.0 before 10:00, then stopped and restarted with jvb2.0 (Xeon 24 Cores, 196 GB RAM, 10 Gbits)

  1. Okay, was just an idea to find the cause of the problems…
  2. Installed today with apt-get from unstable

grep Dropping jvb.log | wc -l
meeting2 -> 4085
meeting3 -> 274
meeting4 -> 999 today, system was used for 5 hours in this time

Full logs and also configuration can be found here: https://1drv.ms/u/s!ArJEGI8DTKRMi8ojqgAotdyLcGI0Ww?e=WddZuE

Thanks for any idea, how to solve the problem… love jitsi so far

Benjamin

No, not really, only a small value:

meeting2 (jicofo + jvb1.0 since 09:30)

IcmpMsg:
    InType0: 1
    InType3: 513856
    InType8: 1061
    InType11: 128
    OutType0: 1061
    OutType3: 4102
Udp:
    342708563 packets received
    121963 packets to unknown port received
    11675 packet receive errors
    1027564765 packets sent
    11659 receive buffer errors
    0 send buffer errors
    InCsumErrors: 15
    IgnoredMulti: 478381
UdpLite:
IpExt:
    InBcastPkts: 478381
    InOctets: 162653271906
    OutOctets: 505752314959
    InBcastOctets: 171364470
    InNoECTPkts: 411702753
    InECT1Pkts: 4
    InECT0Pkts: 221112
    InCEPkts: 8

meeting3 (jvb1.0, stopped at 9:30)

IcmpMsg:
    InType3: 559508
    InType8: 1035
    InType11: 98
    OutType0: 1035
    OutType3: 2439
Udp:
    125734247 packets received
    63598 packets to unknown port received
    3829 packet receive errors
    396458589 packets sent
    3688 receive buffer errors
    0 send buffer errors
    InCsumErrors: 137
    IgnoredMulti: 4230
UdpLite:
IpExt:
    InBcastPkts: 4230
    InOctets: 44634498620
    OutOctets: 165571654032
    InBcastOctets: 1511594
    InNoECTPkts: 132105931
    InECT1Pkts: 1
    InECT0Pkts: 290
    InCEPkts: 2

meeting4 (jvb 1.0 before 09:30, then jvb 2.0 after 10:00)

IcmpMsg:
    InType3: 16919
    InType11: 930
    OutType3: 1399
Udp:
    53413449 packets received
    74679 packets to unknown port received
    0 packet receive errors
    137963688 packets sent
    0 receive buffer errors
    0 send buffer errors
UdpLite:
IpExt:
    InBcastPkts: 1334
    InOctets: 42353299613
    OutOctets: 52102153105
    InBcastOctets: 437552
    InNoECTPkts: 70734083
    InECT0Pkts: 1

Can you try if setting this property (for the new bridge) has any effect?
org.jitsi.videobridge.USE_COMPONENT_SOCKET=false

Boris

1 Like

Sure,

my plan is to update all bridges to jvb2.0, set

net.core.rmem_max=10485760
net.core.netdev_max_backlog=100000

(Is a higher value possible? Since I think, we have good hardware - also all dedicated servers)

And use the

org.jitsi.videobridge.USE_COMPONENT_SOCKET=false

Also switching to use desktop and mobile app rather than use firefox, chrome and mobile apps.

Any ideas on testing the performance without the users? Since jitsi-hammer is not compatible with the version.

Wish you the best & thank you for your time first of all!

Benjamin

It’s possible, but likely unnecessary. Monitor dropped packets in the kernel queue (netstat) and increase further if you see an increase.

Switching from firefox to clients that use simulcast should have a huge impact.

You can use the malleus tool in jitsi-meet-torture for load testing. You can run it with a selenium grid:

Boris

Okay, I will monitor that.

The startup log seems to be fine:

2020-03-20 22:33:02.363 WARNING: [1] CmdLine.parse#115: Skipped empty cmd line argument: domain=
2020-03-20 22:33:02.365 WARNING: [1] CmdLine.parse#115: Skipped empty cmd line argument: port=
2020-03-20 22:33:02.366 WARNING: [1] CmdLine.parse#115: Skipped empty cmd line argument: secret=
2020-03-20 22:33:02.605 INFO: [1] LegacyConfigFileLoader$Companion.load#40: Attempting to load legacy config file at path /etc/jitsi, videobridge, sip-communicator.properties
2020-03-20 22:33:02.608 INFO: [1] LegacyConfigFileLoader$Companion.load#40: Attempting to load legacy config file at path /etc/jitsi, videobridge, sip-communicator.properties
2020-03-20 22:33:02.619 INFO: [1] LegacyConfigFileLoader$Companion.load#40: Attempting to load legacy config file at path /etc/jitsi, videobridge, sip-communicator.properties
2020-03-20 22:33:02.620 INFO: [1] LegacyConfigFileLoader$Companion.load#40: Attempting to load legacy config file at path /etc/jitsi, videobridge, sip-communicator.properties
2020-03-20 22:33:02.652 INFO: [21] ConfigurationActivator.start#45: Registered the LegacyConfigurationServiceShim in OSGi.
2020-03-20 22:33:02.655 INFO: [21] AbstractVersionActivator.start#91: VersionService registered: JVB 2.1.146-g33c380ee
2020-03-20 22:33:02.675 INFO: [21] org.eclipse.jetty.util.log.Log.initialized: Logging initialized @476ms to org.eclipse.jetty.util.log.JavaUtilLog
2020-03-20 22:33:02.811 INFO: [21] org.eclipse.jetty.server.Server.doStart: jetty-9.4.15.v20190215; built: 2019-02-15T16:53:49.381Z; git: eb70b240169fcf1abbd86af36482d1c49826fa0b; jvm 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08
2020-03-20 22:33:03.418 INFO: [21] org.eclipse.jetty.server.handler.ContextHandler.doStart: Started o.e.j.s.ServletContextHandler@99e1801{/,null,AVAILABLE}
2020-03-20 22:33:03.446 INFO: [21] org.eclipse.jetty.server.AbstractConnector.doStart: Started MuxServerConnector@7dbffe1{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2020-03-20 22:33:03.446 INFO: [21] org.eclipse.jetty.server.Server.doStart: Started @1247ms
2020-03-20 22:33:03.450 INFO: [21] AbstractJettyBundleActivator.start#613: Not starting the Jetty service for org.jitsi.videobridge.rest.PublicRESTBundleActivator(port=-1)
2020-03-20 22:33:03.451 INFO: [21] AbstractJettyBundleActivator.start#613: Not starting the Jetty service for org.jitsi.videobridge.rest.PublicClearPortRedirectBundleActivator(port=8080)
2020-03-20 22:33:03.500 INFO: [21] UlimitCheck.printUlimits#115: Running with open files limit 65000 (hard 65000), thread limit 65000 (hard 65000).
2020-03-20 22:33:03.503 INFO: [21] VideobridgeExpireThread.start#92: Starting with 60 second interval.
2020-03-20 22:33:03.507 WARNING: [21] Videobridge.start#918: No authorized source regexp configured. Will accept requests from any source.
2020-03-20 22:33:03.533 INFO: [45] Videobridge.createConference#326: create_conf, id=9e14f622a56e6b7 gid=null logging=false
2020-03-20 22:33:03.547 INFO: [45] TaskPools.<clinit>#81: TaskPools detected 24 processors, creating the CPU pool with that many threads
2020-03-20 22:33:03.654 INFO: [21] LegacyConfigFileLoader$Companion.load#40: Attempting to load legacy config file at path /etc/jitsi, videobridge, sip-communicator.properties
2020-03-20 22:33:03.655 INFO: [21] LegacyConfigFileLoader$Companion.load#40: Attempting to load legacy config file at path /etc/jitsi, videobridge, sip-communicator.properties
2020-03-20 22:33:03.687 INFO: [21] OctoRelayService.start#62: Octo relay is disabled.
2020-03-20 22:33:03.698 WARNING: [49] MucClient.createXMPPTCPConnectionConfiguration#113: Disabling certificate verification!
2020-03-20 22:33:03.775 INFO: [47] org.ice4j.ice.harvest.MappingCandidateHarvesters.initialize: Initialized mapping harvesters (delay=113ms).  stunDiscoveryFailed=false
2020-03-20 22:33:03.782 INFO: [45] org.ice4j.ice.harvest.AbstractUdpListener.<init>: Initialized AbstractUdpListener with address 82.165.203.102:10000/udp. Receive buffer size 10485760 (asked for 10485760)
2020-03-20 22:33:03.783 INFO: [45] org.ice4j.ice.harvest.SinglePortUdpHarvester.<init>: Initialized SinglePortUdpHarvester with address 82.165.203.102:10000/udp
2020-03-20 22:33:04.111 INFO: [49] [hostname=meeting2.schule-plus.com id=shard] MucClient$1.connected#230: Connected.
2020-03-20 22:33:04.315 INFO: [49] [hostname=meeting2.schule-plus.com id=shard] MucClient$MucWrapper.join#650: Joined MUC: jvbbrewery@internal.auth.meeting2.schule-plus.com
SCTP JNI load: Linux OS detected
SCTP lib loaded
=====>: org_jitsi_modified_sctp4j_SctpJni.c calling init
=====>: org_jitsi_modified_sctp4j_SctpJni.c about to set SCTP_DEBUG_ALL
2020-03-20 22:33:04.409 INFO: [45] Health.doRun#294: Performed a successful health check in 902ms. Sticky failure: false
2020-03-20 22:33:14.410 INFO: [45] Videobridge.createConference#326: create_conf, id=1d39e7f3309dbc1a gid=null logging=false
2020-03-20 22:33:14.418 INFO: [45] Health.doRun#294: Performed a successful health check in 8ms. Sticky failure: false
2020-03-20 22:33:24.419 INFO: [45] Videobridge.createConference#326: create_conf, id=78053d0d5beed642 gid=null logging=false

And the jicofo.log:

Jicofo 2020-03-20 23:04:36.215 INFO: [12] org.jitsi.service.libjitsi.LibJitsi.log() Successfully started LibJitsi using as implementation: org.jitsi.impl.libjitsi.LibJitsiOSGiImpl
Jicofo 2020-03-20 23:04:36.226 INFO: [12] impl.configuration.ConfigurationActivator.log() Using properties file configuration store.
Jicofo 2020-03-20 23:04:36.229 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() org.jitsi.jicofo.HOSTNAME=localhost
Jicofo 2020-03-20 23:04:36.229 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.runtime.name=OpenJDK Runtime Environment
Jicofo 2020-03-20 23:04:36.229 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() sun.boot.library.path=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64
Jicofo 2020-03-20 23:04:36.229 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.vm.version=25.242-b08
Jicofo 2020-03-20 23:04:36.230 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.vm.vendor=Private Build
Jicofo 2020-03-20 23:04:36.230 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.vendor.url=http://java.oracle.com/
Jicofo 2020-03-20 23:04:36.230 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() path.separator=:
Jicofo 2020-03-20 23:04:36.230 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.util.logging.config.file=/etc/jitsi/jicofo/logging.properties
Jicofo 2020-03-20 23:04:36.230 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.vm.name=OpenJDK 64-Bit Server VM
Jicofo 2020-03-20 23:04:36.230 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() file.encoding.pkg=sun.io
Jicofo 2020-03-20 23:04:36.230 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() user.country=GB
Jicofo 2020-03-20 23:04:36.230 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() sun.java.launcher=SUN_STANDARD
Jicofo 2020-03-20 23:04:36.231 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() sun.os.patch.level=unknown
Jicofo 2020-03-20 23:04:36.231 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() net.java.sip.communicator.service.media.DISABLE_AUDIO_SUPPORT=true
Jicofo 2020-03-20 23:04:36.231 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.vm.specification.name=Java Virtual Machine Specification
Jicofo 2020-03-20 23:04:36.231 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() user.dir=/usr/share/jicofo
Jicofo 2020-03-20 23:04:36.231 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() org.jitsi.jicofo.FOCUS_USER_PASSWORD=KtsLXMUZ
Jicofo 2020-03-20 23:04:36.231 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.runtime.version=1.8.0_242-8u242-b08-0ubuntu3~18.04-b08
Jicofo 2020-03-20 23:04:36.231 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.awt.graphicsenv=sun.awt.X11GraphicsEnvironment
Jicofo 2020-03-20 23:04:36.231 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.endorsed.dirs=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/endorsed
Jicofo 2020-03-20 23:04:36.232 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() os.arch=amd64
Jicofo 2020-03-20 23:04:36.232 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.io.tmpdir=/tmp
Jicofo 2020-03-20 23:04:36.232 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() line.separator=

Jicofo 2020-03-20 23:04:36.232 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.vm.specification.vendor=Oracle Corporation
Jicofo 2020-03-20 23:04:36.232 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() os.name=Linux
Jicofo 2020-03-20 23:04:36.232 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() sun.jnu.encoding=UTF-8
Jicofo 2020-03-20 23:04:36.232 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() net.java.sip.communicator.CONFIGURATION_FILE_IS_READ_ONLY=true
Jicofo 2020-03-20 23:04:36.232 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() org.jitsi.jicofo.FOCUS_USER_DOMAIN=auth.meeting2.schule-plus.com
Jicofo 2020-03-20 23:04:36.233 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
Jicofo 2020-03-20 23:04:36.233 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.specification.name=Java Platform API Specification
Jicofo 2020-03-20 23:04:36.233 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.class.version=52.0
Jicofo 2020-03-20 23:04:36.233 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() org.jitsi.service.audionotifier.AudioNotifierService=org.jitsi.impl.neomedia.notify.AudioNotifierServiceImpl
Jicofo 2020-03-20 23:04:36.233 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() sun.management.compiler=HotSpot 64-Bit Tiered Compilers
Jicofo 2020-03-20 23:04:36.233 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() os.version=4.15.0-76-generic
Jicofo 2020-03-20 23:04:36.233 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() user.home=/usr/share/jicofo
Jicofo 2020-03-20 23:04:36.233 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() user.timezone=Europe/Berlin
Jicofo 2020-03-20 23:04:36.233 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.awt.printerjob=sun.print.PSPrinterJob
Jicofo 2020-03-20 23:04:36.234 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() org.jitsi.jicofo.FOCUS_USER_NAME=focus
Jicofo 2020-03-20 23:04:36.234 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() file.encoding=UTF-8
Jicofo 2020-03-20 23:04:36.234 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.specification.version=1.8
Jicofo 2020-03-20 23:04:36.234 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() net.java.sip.communicator.service.media.DISABLE_VIDEO_SUPPORT=true
Jicofo 2020-03-20 23:04:36.234 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() org.jitsi.jicofo.XMPP_DOMAIN=meeting2.schule-plus.com
Jicofo 2020-03-20 23:04:36.234 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.class.path=/usr/share/jicofo/jicofo.jar:/usr/share/jicofo/lib/agafua-syslog-0.4.jar:/usr/share/jicofo/lib/annotations-15.0.jar:/usr/share/jicofo/lib/aopalliance-repackaged-2.5.0.jar:/usr/share/jicofo/lib/bccontrib-1.0.jar:/usr/share/jicofo/lib/bcpkix-jdk15on-1.54.jar:/usr/share/jicofo/lib/bcprov-jdk15on-1.54.jar:/usr/share/jicofo/lib/cglib-nodep-2.2.jar:/usr/share/jicofo/lib/commons-codec-1.6.jar:/usr/share/jicofo/lib/commons-lang3-3.1.jar:/usr/share/jicofo/lib/commons-logging-1.2.jar:/usr/share/jicofo/lib/concurrentlinkedhashmap-lru-1.0_jdk5.jar:/usr/share/jicofo/lib/core-2.0.1.jar:/usr/share/jicofo/lib/dnsjava-2.1.7.jar:/usr/share/jicofo/lib/dom4j-1.6.1.jar:/usr/share/jicofo/lib/fmj-1.0-SNAPSHOT.jar:/usr/share/jicofo/lib/guava-15.0.jar:/usr/share/jicofo/lib/hk2-api-2.5.0.jar:/usr/share/jicofo/lib/hk2-locator-2.5.0.jar:/usr/share/jicofo/lib/hk2-utils-2.5.0.jar:/usr/share/jicofo/lib/httpclient-4.4.1.jar:/usr/share/jicofo/lib/httpcore-4.4.1.jar:/usr/share/jicofo/lib/ice4j-3.0-1-ge854e4e.jar:/usr/share/jicofo/lib/jackson-annotations-2.9.9.jar:/usr/share/jicofo/lib/jackson-core-2.9.9.jar:/usr/share/jicofo/lib/jackson-databind-2.9.9.jar:/usr/share/jicofo/lib/jackson-module-jaxb-annotations-2.9.9.jar:/usr/share/jicofo/lib/jain-sip-ri-ossonly-1.2.98c7f8c-jitsi-oss1.jar:/usr/share/jicofo/lib/jakarta.annotation-api-1.3.4.jar:/usr/share/jicofo/lib/jakarta.inject-2.5.0.jar:/usr/share/jicofo/lib/jakarta.ws.rs-api-2.1.5.jar:/usr/share/jicofo/lib/java-dogstatsd-client-2.5.jar:/usr/share/jicofo/lib/java-sdp-nist-bridge-1.1.jar:/usr/share/jicofo/lib/javassist-3.22.0-CR2.jar:/usr/share/jicofo/lib/javax.servlet-api-3.1.0.jar:/usr/share/jicofo/lib/jbosh-0.9.2.jar:/usr/share/jicofo/lib/jcip-annotations-1.0.jar:/usr/share/jicofo/lib/jcl-core-2.8.jar:/usr/share/jicofo/lib/jersey-client-2.29.jar:/usr/share/jicofo/lib/jersey-common-2.29.jar:/usr/share/jicofo/lib/jersey-container-jetty-http-2.29.jar:/usr/share/jicofo/lib/jersey-container-servlet-2.29.jar:/usr/share/jicofo/lib/jersey-container-servlet-core-2.29.jar:/usr/share/jicofo/lib/jersey-entity-filtering-2.29.jar:/usr/share/jicofo/lib/jersey-hk2-2.29.jar:/usr/share/jicofo/lib/jersey-media-jaxb-2.29.jar:/usr/share/jicofo/lib/jersey-media-json-jackson-2.29.jar:/usr/share/jicofo/lib/jersey-server-2.29.jar:/usr/share/jicofo/lib/jetty-client-9.4.15.v20190215.jar:/usr/share/jicofo/lib/jetty-continuation-9.4.12.v20180830.jar:/usr/share/jicofo/lib/jetty-http-9.4.15.v20190215.jar:/usr/share/jicofo/lib/jetty-io-9.4.15.v20190215.jar:/usr/share/jicofo/lib/jetty-proxy-9.4.15.v20190215.jar:/usr/share/jicofo/lib/jetty-security-9.4.15.v20190215.jar:/usr/share/jicofo/lib/jetty-server-9.4.15.v20190215.jar:/usr/share/jicofo/lib/jetty-servlet-9.4.15.v20190215.jar:/usr/share/jicofo/lib/jetty-util-9.4.15.v20190215.jar:/usr/share/jicofo/lib/jetty-webapp-7.0.1.v20091125.jar:/usr/share/jicofo/lib/jetty-xml-7.0.1.v20091125.jar:/usr/share/jicofo/lib/jicoco-1.1-7-g05338ab.jar:/usr/share/jicofo/lib/jitsi-android-osgi-1.0-SNAPSHOT.jar:/usr/share/jicofo/lib/jitsi-configuration-2.13.cb5485e.jar:/usr/share/jicofo/lib/jitsi-credentialsstorage-2.13.cb5485e.jar:/usr/share/jicofo/lib/jitsi-dnsservice-2.13.cb5485e.jar:/usr/share/jicofo/lib/jitsi-netaddr-2.13.cb5485e.jar:/usr/share/jicofo/lib/jitsi-protocol-2.13.cb5485e.jar:/usr/share/jicofo/lib/jitsi-protocol-jabber-2.13.cb5485e.jar:/usr/share/jicofo/lib/jitsi-protocol-media-2.13.cb5485e.jar:/usr/share/jicofo/lib/jitsi-util-2.13.cb5485e.jar:/usr/share/jicofo/lib/jitsi-utils-1.0-33-g2ed4090.jar:/usr/share/jicofo/lib/jitsi-xmpp-extensions-1.0-6-g009420d.jar:/usr/share/jicofo/lib/jna-4.1.0.jar:/usr/share/jicofo/lib/jnsapi-0.0.3-jitsi-smack4.2-3.jar:/usr/share/jicofo/lib/json-simple-1.1.1.jar:/usr/share/jicofo/lib/jxmpp-core-0.6.2.jar:/usr/share/jicofo/lib/jxmpp-jid-0.6.2.jar:/usr/share/jicofo/lib/jxmpp-util-cache-0.6.2.jar:/usr/share/jicofo/lib/libidn-1.15.jar:/usr/share/jicofo/lib/libjitsi-1.0-0-gb3296cf.jar:/usr/share/jicofo/lib/object-cloner-0.1.jar:/usr/share/jicofo/lib/objenesis-2.6.jar:/usr/share/jicofo/lib/orange-extensions-1.3.0.jar:/usr/share/jicofo/lib/org.apache.felix.framework-4.4.0.jar:/usr/share/jicofo/lib/org.apache.felix.main-4.4.0.jar:/usr/share/jicofo/lib/org.osgi.core-4.3.1.jar:/usr/share/jicofo/lib/osgi-resource-locator-1.0.3.jar:/usr/share/jicofo/lib/reflections-0.9.11.jar:/usr/share/jicofo/lib/sdes4j-1.1.3.jar:/usr/share/jicofo/lib/sdp-api-1.0.jar:/usr/share/jicofo/lib/slf4j-api-1.7.26.jar:/usr/share/jicofo/lib/slf4j-jdk14-1.7.26.jar:/usr/share/jicofo/lib/smack-bosh-4.2.4-47d17fc.jar:/usr/share/jicofo/lib/smack-core-4.2.4-47d17fc.jar:/usr/share/jicofo/lib/smack-debug-4.2.4-47d17fc.jar:/usr/share/jicofo/lib/smack-experimental-4.2.4-47d17fc.jar:/usr/share/jicofo/lib/smack-extensions-4.2.4-47d17fc.jar:/usr/share/jicofo/lib/smack-im-4.2.4-47d17fc.jar:/usr/share/jicofo/lib/smack-java7-4.2.4-47d17fc.jar:/usr/share/jicofo/lib/smack-legacy-4.2.4-47d17fc.jar:/usr/share/jicofo/lib/smack-resolver-javax-4.2.4-47d17fc.jar:/usr/share/jicofo/lib/smack-sasl-javax-4.2.4-47d17fc.jar:/usr/share/jicofo/lib/smack-tcp-4.2.4-47d17fc.jar:/usr/share/jicofo/lib/tinder-1.3.0.jar:/usr/share/jicofo/lib/validation-api-2.0.1.Final.jar:/usr/share/jicofo/lib/weupnp-0.1.4.jar:/usr/share/jicofo/lib/xml-apis-1.0.b2.jar:/usr/share/jicofo/lib/xmlpull-1.1.3.4a.jar:/usr/share/jicofo/lib/xpp3-1.1.4c.jar:/usr/share/jicofo/lib/zrtp4j-light-4.1.0-jitsi-1-SNAPSHOT.jar
Jicofo 2020-03-20 23:04:36.235 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() user.name=jicofo
Jicofo 2020-03-20 23:04:36.235 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.vm.specification.version=1.8
Jicofo 2020-03-20 23:04:36.235 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() sun.java.command=org.jitsi.jicofo.Main --host=localhost --domain=meeting2.schule-plus.com --port=5347 --secret=eogsvZss --user_name=focus --user_domain=auth.meeting2.schule-plus.com --user_password=KtsLXMUZ
Jicofo 2020-03-20 23:04:36.235 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
Jicofo 2020-03-20 23:04:36.235 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() sun.arch.data.model=64
Jicofo 2020-03-20 23:04:36.235 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() user.language=en
Jicofo 2020-03-20 23:04:36.235 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.specification.vendor=Oracle Corporation
Jicofo 2020-03-20 23:04:36.235 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() awt.toolkit=sun.awt.X11.XToolkit
Jicofo 2020-03-20 23:04:36.236 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.vm.info=mixed mode
Jicofo 2020-03-20 23:04:36.236 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.version=1.8.0_242
Jicofo 2020-03-20 23:04:36.236 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.ext.dirs=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext:/usr/java/packages/lib/ext
Jicofo 2020-03-20 23:04:36.236 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() sun.boot.class.path=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/classes
Jicofo 2020-03-20 23:04:36.236 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() net.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi
Jicofo 2020-03-20 23:04:36.236 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.vendor=Private Build
Jicofo 2020-03-20 23:04:36.236 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() file.separator=/
Jicofo 2020-03-20 23:04:36.236 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() net.java.sip.communicator.impl.configuration.USE_PROPFILE_CONFIG=true
Jicofo 2020-03-20 23:04:36.236 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() net.java.sip.communicator.SC_HOME_DIR_NAME=jicofo
Jicofo 2020-03-20 23:04:36.237 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() java.vendor.url.bug=http://bugreport.sun.com/bugreport/
Jicofo 2020-03-20 23:04:36.237 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() sun.io.unicode.encoding=UnicodeLittle
Jicofo 2020-03-20 23:04:36.237 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() sun.cpu.endian=little
Jicofo 2020-03-20 23:04:36.237 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() net.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi
Jicofo 2020-03-20 23:04:36.237 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() sun.cpu.isalist=
Jicofo 2020-03-20 23:04:36.237 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() Normal classloader
Jicofo 2020-03-20 23:04:36.238 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() failed to find jitsi-defaults.properties with class loader, will continue without it.
Jicofo 2020-03-20 23:04:36.238 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() Normal classloader
Jicofo 2020-03-20 23:04:36.238 INFO: [12] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() failed to find jitsi-default-overrides.properties with class loader, will continue without it.
Jicofo 2020-03-20 23:04:36.252 INFO: [12] util.NetworkUtils.<clinit>().117 java.net.preferIPv6Addresses=null
Jicofo 2020-03-20 23:04:36.252 INFO: [12] util.NetworkUtils.<clinit>().118 java.net.preferIPv4Stack=null
Jicofo 2020-03-20 23:04:36.256 INFO: [12] impl.netaddr.NetworkAddressManagerServiceImpl.start().92 Network Address Manager         ...[  STARTED ]
Jicofo 2020-03-20 23:04:36.256 INFO: [12] impl.netaddr.NetworkAddressManagerServiceImpl.start().98 Network Address Manager Service ...[REGISTERED]
Jicofo 2020-03-20 23:04:36.259 INFO: [12] org.jitsi.version.AbstractVersionActivator.log() VersionService registered: JiCoFo 1.0.530
Jicofo 2020-03-20 23:04:36.412 INFO: [12] org.jitsi.jicofo.JitsiMeetGlobalConfig.init().223 Automatically grant 'owner' role: true
Jicofo 2020-03-20 23:04:36.412 INFO: [12] org.jitsi.jicofo.JitsiMeetGlobalConfig.init().242 Jibri requests in PENDING state will be timed out after: 90 seconds
Jicofo 2020-03-20 23:04:36.412 INFO: [12] org.jitsi.jicofo.JitsiMeetGlobalConfig.init().255 Will attempt a maximum of 5 Jibri retries after failure
Jicofo 2020-03-20 23:04:36.412 INFO: [12] org.jitsi.jicofo.JitsiMeetGlobalConfig.init().269 Lonely participants will be "terminated" after 20000 milliseconds
Jicofo 2020-03-20 23:04:36.430 WARNING: [12] org.jitsi.jicofo.FocusManager.log() No dedicated JVB MUC XMPP connection configured - falling back to the default XMPP connection
Jicofo 2020-03-20 23:04:36.432 INFO: [12] org.jitsi.jicofo.bridge.BridgeSelector.log() Using org.jitsi.jicofo.bridge.SingleBridgeSelectionStrategy
Jicofo 2020-03-20 23:04:36.432 INFO: [12] org.jitsi.jicofo.bridge.BridgeSelector.log() Bridge failure reset threshold: 300000
Jicofo 2020-03-20 23:04:36.432 INFO: [12] org.jitsi.jicofo.bridge.BridgeSelector.log() Local region: null
Jicofo 2020-03-20 23:04:36.433 INFO: [12] org.jitsi.jicofo.JitsiMeetServices.log() Using a Bridge MUC detector with MUC: JvbBrewery@internal.auth.meeting2.schule-plus.com
Jicofo 2020-03-20 23:04:36.451 INFO: [12] org.eclipse.jetty.util.log.initialized() Logging initialized @399ms to org.eclipse.jetty.util.log.Slf4jLog
Jicofo 2020-03-20 23:04:36.572 INFO: [12] org.eclipse.jetty.server.Server.doStart() jetty-9.4.15.v20190215; built: 2019-02-15T16:53:49.381Z; git: eb70b240169fcf1abbd86af36482d1c49826fa0b; jvm 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08
Jicofo 2020-03-20 23:04:36.804 INFO: [18] org.jitsi.jicofo.ProtocolProviderHandler.registrationStateChanged().144 XmppProtocolProvider(focus@auth.meeting2.schule-plus.com/focus687828810071 (Jabber)): RegistrationStateChangeEvent[ oldState=Unregistered; newState=RegistrationState=Registered; reasonCode=-1; reason=null]
Jicofo 2020-03-20 23:04:36.826 INFO: [18] org.jitsi.jicofo.xmpp.BaseBrewery.start().186 Joined brewery room: JvbBrewery@internal.auth.meeting2.schule-plus.com
Jicofo 2020-03-20 23:04:36.829 INFO: [18] org.jitsi.jicofo.ComponentsDiscovery.log() New component discovered: conference.meeting2.schule-plus.com, null
Jicofo 2020-03-20 23:04:36.829 INFO: [18] org.jitsi.jicofo.JitsiMeetServices.log() MUC component discovered: conference.meeting2.schule-plus.com
Jicofo 2020-03-20 23:04:36.870 INFO: [18] org.jitsi.jicofo.ComponentsDiscovery.log() New component discovered: auth.meeting2.schule-plus.com, Prosody(0.11.4,Linux)
Jicofo 2020-03-20 23:04:36.871 INFO: [18] org.jitsi.jicofo.JitsiMeetServices.log() Detected XMPP server version: Prosody(0.11.4,Linux)
Jicofo 2020-03-20 23:04:36.872 INFO: [18] org.jitsi.jicofo.ComponentsDiscovery.log() Services re-discovery interval: 30000
Jicofo 2020-03-20 23:04:36.873 INFO: [18] org.jitsi.jicofo.FocusManager.log() XMPP provider reg state: RegistrationState=Registered
Jicofo 2020-03-20 23:04:36.878 INFO: [18] org.jitsi.impl.protocol.xmpp.XmppProtocolProvider.doConnect().299 XMPP provider Jabber:focus@auth.meeting2.schule-plus.com/focus687828810071@localhost connected (JID: focus@auth.meeting2.schule-plus.com/focus687828810071)
Jicofo 2020-03-20 23:04:37.017 INFO: [12] org.eclipse.jetty.server.handler.ContextHandler.doStart() Started o.e.j.s.ServletContextHandler@7edfd553{/,null,AVAILABLE}
Jicofo 2020-03-20 23:04:37.033 INFO: [12] org.eclipse.jetty.server.AbstractConnector.doStart() Started MuxServerConnector@55dec6fa{HTTP/1.1,[http/1.1]}{0.0.0.0:8888}
Jicofo 2020-03-20 23:04:37.033 INFO: [12] org.eclipse.jetty.server.Server.doStart() Started @983ms
Jicofo 2020-03-20 23:04:37.034 INFO: [1] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() org.jitsi.jicofo.BRIDGE_MUC=JvbBrewery@internal.auth.meeting2.schule-plus.com
Jicofo 2020-03-20 23:04:37.034 INFO: [1] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() org.jitsi.jicofo.MAX_SSRC_PER_USER=1000
Jicofo 2020-03-20 23:04:37.034 INFO: [1] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() net.java.sip.communicator.impl.protocol.jabber.extensions.caps.EntityCapsManager.CAPS.http://prosody.im#wBlvgkM1+mhpz81eJoge4hYoI0Q==<query xmlns='http://jabber.org/protocol/disco#info' node='http://prosody.im#wBlvgkM1+mhpz81eJoge4hYoI0Q='><identity category='server' name='Prosody' type='im'/><identity category='pubsub' name='Prosody' type='pep'/><feature var='http://jabber.org/protocol/commands'/><feature var='jabber:iq:version'/><feature var='jabber:iq:private'/><feature var='urn:xmpp:carbons:2'/><feature var='jabber:iq:roster'/><feature var='urn:xmpp:ping'/><feature var='http://jabber.org/protocol/disco#info'/><feature var='http://jabber.org/protocol/disco#items'/><feature var='urn:xmpp:time'/><feature var='jabber:iq:time'/><feature var='vcard-temp'/><feature var='http://jabber.org/protocol/pubsub#publish'/><feature var='msgoffline'/><feature var='jabber:iq:register'/><feature var='jabber:iq:last'/><feature var='urn:xmpp:blocking'/></query>
Jicofo 2020-03-20 23:04:37.037 INFO: [1] org.jitsi.xmpp.component.ComponentBase.log() Component org.jitsi.jicofo. config:
Jicofo 2020-03-20 23:04:37.037 INFO: [1] org.jitsi.xmpp.component.ComponentBase.log()   ping interval: 10000 ms
Jicofo 2020-03-20 23:04:37.037 INFO: [1] org.jitsi.xmpp.component.ComponentBase.log()   ping timeout: 5000 ms
Jicofo 2020-03-20 23:04:37.037 INFO: [1] org.jitsi.xmpp.component.ComponentBase.log()   ping threshold: 3
Jicofo 2020-03-20 23:05:03.161 INFO: [29] org.jitsi.jicofo.xmpp.BaseBrewery.processInstanceStatusChanged().329 Added brewery instance: jvbbrewery@internal.auth.meeting2.schule-plus.com/59488033-7a43-4966-a411-7687300f0e14
Jicofo 2020-03-20 23:05:03.161 INFO: [29] org.jitsi.jicofo.bridge.BridgeSelector.log() Added videobridge: jvbbrewery@internal.auth.meeting2.schule-plus.com/59488033-7a43-4966-a411-7687300f0e14 v: null
Jicofo 2020-03-20 23:05:03.161 WARNING: [29] org.jitsi.jicofo.bridge.BridgeSelector.log() No pub-sub node mapped for jvbbrewery@internal.auth.meeting2.schule-plus.com/59488033-7a43-4966-a411-7687300f0e14
Jicofo 2020-03-20 23:05:03.169 INFO: [52] org.jitsi.jicofo.bridge.JvbDoctor.log() Scheduled health-check task for: jvbbrewery@internal.auth.meeting2.schule-plus.com/59488033-7a43-4966-a411-7687300f0e14
Jicofo 2020-03-20 23:05:06.889 INFO: [32] org.jitsi.jicofo.ComponentsDiscovery.log() New component discovered: focus.meeting2.schule-plus.com, null
Jicofo 2020-03-20 23:05:41.123 INFO: [29] org.jitsi.jicofo.xmpp.BaseBrewery.processInstanceStatusChanged().329 Added brewery instance: jvbbrewery@internal.auth.meeting2.schule-plus.com/bridge3
Jicofo 2020-03-20 23:05:41.123 INFO: [29] org.jitsi.jicofo.bridge.BridgeSelector.log() Added videobridge: jvbbrewery@internal.auth.meeting2.schule-plus.com/bridge3 v: null
Jicofo 2020-03-20 23:05:41.123 WARNING: [29] org.jitsi.jicofo.bridge.BridgeSelector.log() No pub-sub node mapped for jvbbrewery@internal.auth.meeting2.schule-plus.com/bridge3
Jicofo 2020-03-20 23:05:41.124 INFO: [52] org.jitsi.jicofo.bridge.JvbDoctor.log() Scheduled health-check task for: jvbbrewery@internal.auth.meeting2.schule-plus.com/bridge3
Jicofo 2020-03-20 23:06:16.105 INFO: [29] org.jitsi.jicofo.xmpp.BaseBrewery.processInstanceStatusChanged().329 Added brewery instance: jvbbrewery@internal.auth.meeting2.schule-plus.com/bridge4
Jicofo 2020-03-20 23:06:16.105 INFO: [29] org.jitsi.jicofo.bridge.BridgeSelector.log() Added videobridge: jvbbrewery@internal.auth.meeting2.schule-plus.com/bridge4 v: null
Jicofo 2020-03-20 23:06:16.105 WARNING: [29] org.jitsi.jicofo.bridge.BridgeSelector.log() No pub-sub node mapped for jvbbrewery@internal.auth.meeting2.schule-plus.com/bridge4
Jicofo 2020-03-20 23:06:16.106 INFO: [52] org.jitsi.jicofo.bridge.JvbDoctor.log() Scheduled health-check task for: jvbbrewery@internal.auth.meeting2.schule-plus.com/bridge4

Maybe nothing related, but we had to set in jicofo´s sip properties

org.jitsi.jicofo.MAX_SSRC_PER_USER=1000

Those values should be fine, I think these are the values on meet.jit.si. But bigger doesn’t make sense.

We are using selenium grid with big machines and every one of them is running one chrome. With this test https://github.com/jitsi/jitsi-meet-torture/blob/master/src/test/java/org/jitsi/meet/test/MalleusJitsificus.java

Setting this value will stop the bridge from working -> Browser logs bridge channel is not opened

By the way: Now using the test framework. Love it so far!

Hi,

today the workload is higher. No packet are dropped, however the E2E values in the clients are high and the audio quality is bad. All bridges using now jvb2.0

Any further ideas…?

Edit: Sorry! The video and audio is now fine! (sorry!) The problem is the initial loading / bosh connection under high load of the system with Jicofo and prosody:

Nginx Logs
2020/03/23 11:13:04 [error] 8312#8312: *167342 connect() to [::1]:5280 failed (101: Network is unreachable) while connecting to upstream, client: 90.186.82.71, server: meeting2.schule-plus.com, request: “POST /http-bind?room=az255 HTTP/2.0”, upstream: “http://[::1]:5280/http-bind?room=az255”, host: “meeting2.schule-plus.com”

2020/03/23 11:13:04 [warn] 8312#8312: *167342 upstream server temporarily disabled while connecting to upstream, client: 90.186.82.71, server: meeting2.schule-plus.com, request: "POST /http-bind?room=az255 HTTP/2.0", upstream: "http://[::1]:5280/http-bind?room=az255", host: "meeting2.schule-plus.com"

2020/03/23 11:13:16 [error] 8312#8312: *149163 connect() to [::1]:5280 failed (101: Network is unreachable) while connecting to upstream, client: 176.199.209.125, server: meeting2.schule-plus.com, request: "POST /http-bind?room=az243 HTTP/2.0", upstream: "http://[::1]:5280/http-bind?room=az243", host: "meeting2.schule-plus.com"

2020/03/23 11:13:16 [warn] 8312#8312: *149163 upstream server temporarily disabled while connecting to upstream, client: 176.199.209.125, server: meeting2.schule-plus.com, request: "POST /http-bind?room=az243 HTTP/2.0", upstream: "http://[::1]:5280/http-bind?room=az243", host: "meeting2.schule-plus.com"

2020/03/23 11:13:27 [error] 8312#8312: *173640 connect() to [::1]:5280 failed (101: Network is unreachable) while connecting to upstream, client: 93.229.21.63, server: meeting2.schule-plus.com, request: "POST /http-bind?room=az258 HTTP/2.0", upstream: "http://[::1]:5280/http-bind?room=az258", host: "meeting2.schule-plus.com"

no prosody oder Jicofo error logs

Make sure you run prosody 0.11 with epoll network backend enabled, this will improve its performance a lot.
Make sure to setup high limits for open file descriptors for nginx and prosody.