Poor Videoquality with JVB2 on dedicated host

eazyadm · November 26, 2020, 3:57pm

Hi,

we are faceing a problem with our Videobridge Cluster regarding the videoquality, maybe since our last update to jitsi-videobridge2 2.1-376-g9f12bfe2-1.

We are running multiple frontendservers hosting jicofo, prosody, nginx so everything which comes if I install jitsi-meet on ubuntu 18.04 with the offical repoistory (deb https://download.jitsi.org stable/) except jitsi-videobridge2 which is disabled on the frontendservers.

We are also running a whole bunch of physical servers which are only running the jitsi-videobridge2 service. Those machine are running all with the same config and holding a connection to all the frontendservers.

Everything works fine, we can handle large conferences and thousends of users at once. But the videoquality is poor and has nothing to do with the connected users to the whole cluster or a single room.

In the moment we are leaving the p2p connection while the 3th user is connected to a room the quality drops. No packetlost, no errors and warnings in the jvb.log just bad video quality. On the JVB2 port 10000 UDP and 443 tcp ist available and i can see traffic.

If I now disable the connection from the JVB2 Cluster and enable the local JVB2 on the Frontendserver the quality is perfect. Same internet connection, same OS, same software version, the only change is the local jvb instead of the external one.

The running hardware setup:
36 frontend servers running as KVM VM with 2 Cores, 4 GB RAM on Intel Xeon Gold 6244 @ 3,6 GHz with 10 GBE Uplink direct connected to the internet without NAT.
15 JVB2 Hosts are running physical on AMD Ryzen 9 3950X 16-Core with 32 GB RAM, SWAP disabled connected over 2x10GBE LACP direct connected to the internet, no NAT.
Frontendservers and JVB2 Hosts are connected to the same switch.

Installed Versions running on Ubuntu 18.04 including all updates:
jitsi-meet 2.0.5142-1
jitsi-meet-prosody 1.0.4466-1
jitsi-meet-turnserver 1.0.4466-1
jitsi-meet-web 1.0.4466-1
jitsi-meet-web-config 1.0.4466-1
jitsi-videobridge2 2.1-376-g9f12bfe2-1

running config for /etc/jitsi/meet/meet.foo.bar-config.js

var config = {
hosts: {
domain: ‘meet.foo.bar’,
muc: ‘conference.meet.foo.bar’
},

testing: {
    p2pTestMode: false
},

disableAudioLevels: true,
enableNoAudioDetection: true,
enableNoisyMicDetection: true,
maxFullResolutionParticipants: -1,
channelLastN: -1,
openBridgeChannel: 'websocket',
enableWelcomePage: true,
defaultLanguage: 'de',
enableUserRolesBasedOnToken: false,
disableThirdPartyRequests: true,

p2p: {
    enabled: true,

    stunServers: [

        { urls: 'stun:turnrelay.foo.bar:443' }
    ],

    disableH264: true,
},

analytics: {
},
deploymentInfo: {
},
makeJsonParserHappy: 'even if last key had a trailing comma'

};

Is this a know problem? are there any ideas ?

Thanks
bye
ralf

eazyadm · November 27, 2020, 10:27am

I will answer to my own because I think it is easier for others to understand the process and progress of what I’m doing trying to debug the situation.

Yesterday I striped out two Hosts of the production environment for better debugging and I have done a complete new installtion with ubuntu 18.04 server.

Videobridge Server:

export FQDN=meet.foo.bar

# Installing required Systemtools
apt-get install -y wget gnupg

# Jitsi Sources
wget -qO - https://download.jitsi.org/jitsi-key.gpg.key | apt-key add -
echo deb https://download.jitsi.org stable/ > /etc/apt/sources.list.d/jitsi-stable.list

apt update

# Installing
echo "jitsi-videobridge2 jitsi-videobridge/jvb-hostname string $FQDN" | debconf-set-selections

apt install -y jitsi-videobridge2

Frontendserver:

export FQDN=meet.foo.bar

# Installing required Systemtools
apt-get install -y wget gnupg

# Adding Prosody and Jitsi Sources
wget -qO- https://prosody.im/files/prosody-debian-packages.key | apt-key add -
echo deb http://packages.prosody.im/debian $(lsb_release -sc) main > /etc/apt/sources.list.d/prosody.list

wget -qO - https://download.jitsi.org/jitsi-key.gpg.key | apt-key add -
echo deb https://download.jitsi.org stable/ > /etc/apt/sources.list.d/jitsi-stable.list

apt update

# Installing
echo "jitsi-videobridge2 jitsi-videobridge/jvb-hostname string $FQDN" | debconf-set-selections
echo "jitsi-meet-web-config jitsi-meet/cert-choice  select  Generate a new self-signed certificate (You will later get a chance to obtain Let's encrypt certificate)" | debconf-set-selections

apt install -y jitsi-meet

# Going on with installation
chown prosody:prosody /etc/prosody/certs/*

/usr/share/jitsi-meet/scripts/install-letsencrypt-cert.sh << 'EOF'
ssl@foo.bar
EOF

# disable local Videobridge we don't need it. we're using the shared bridges
systemctl disable jitsi-videobridge2
/etc/init.d/jitsi-videobridge stop

# Send Videobridge data to shared bridges
SHARD=`echo $FQDN | sed "s/\./_/g"`
grep "org.jitsi.videobridge.xmpp.user.shard." /etc/jitsi/videobridge/sip-communicator.properties | sed "s/.shard./.shard-$SHARD./g" | sed "s/localhost/$FQDN/g" > /tmp/sip-communicator.properties
echo "org.jitsi.videobridge.xmpp.user.shard.DISABLE_CERTIFICATE_VERIFICATION=true" | sed "s/.shard./.shard-$SHARD./g" >> /tmp/sip-communicator.properties

cat /tmp/sip-communicator.properties | ssh videobridge "cat - >> /etc/jitsi/videobridge/sip-communicator.properties"

Starting a conference with 3 participants all have a green dot with good connection quality, but the resolution is only 320x180 also if I focus on one participant. If I stop the videobridge service on the videobridge host and start it direct on the frontend server, quaility is good.

So now I switched back to videobridge on videobridge host and stopped the local videobridge on the frontend server.
If I set disableSimulcast: true, the video quality is quite perfect but i get a maximum of about 20 participants in a room which is clear because of the bandwith issue at the client.

So I have exaclity the same problem as in our production environment without load and only 3 Users on a unmodified installation.

Than I tried all that on meet.jit.si everything looks good but wenn I focus to a participant it looks like the video is reconnecting and switching to a higher resolution, that is not happening in my environment.
Is there an option which has to be enabled?

Thanks

Th3R3al · November 27, 2020, 11:44am

Hey eazyadm,

the “upscaling” is not happening thing, could be connected to a problem there the client could not establish a websocket connection to the jvb. Is the Callstats and active speaker marking also not working? Thats another indicator for that.

Greets
Th3R3al

b00nsai · November 27, 2020, 12:07pm

Hi,

i have the same problem. With simucast active I can’t get over 320p. If I check the dev-console in chrome while in the videoconference I see a lot of errors about websockets.

BridgeChannel.js:88 WebSocket connection to 'wss://myURL.tld/colibri-ws/default-id/65389b9cf7fee9c2/ebb640b5?pwd=40ukhtmt543efucmcdu2j5g30b' failed: Error during WebSocket handshake: Unexpected response code: 404
_initWebSocket @ BridgeChannel.js:88
t @ BridgeChannel.js:107
Logger.js:154 2020-11-27T12:02:29.587Z [modules/RTC/BridgeChannel.js] <WebSocket.e.onclose>:  Channel closed by server
Logger.js:154 2020-11-27T12:02:29.587Z [modules/RTC/BridgeChannel.js] <WebSocket.e.onclose>:  Channel closed: 1006 
o @ Logger.js:154
e.onclose @ BridgeChannel.js:376
Logger.js:154 2020-11-27T12:02:30.021Z [modules/RTC/BridgeChannel.js] <l._send>:  Bridge Channel send: no opened channel.
o @ Logger.js:154
_send @ BridgeChannel.js:400
sendMessage @ BridgeChannel.js:192
sendChannelMessage @ RTC.js:932
re.sendEndpointMessage @ JitsiConference.js:2512
re.sendMessage @ JitsiConference.js:2556
(anonymous) @ JitsiConference.js:349
sendRequest @ e2eping.js:93
Logger.js:154 2020-11-27T12:02:30.021Z [JitsiConference.js] <u.sendMessage>:  Failed to send E2E ping request or response. undefined

I use the following verisons:

dpkg -l | grep jitsi
ii  jitsi-meet                           2.0.5142-1                        all          WebRTC JavaScript video conferences
ii  jitsi-meet-prosody                   1.0.4466-1                        all          Prosody configuration for Jitsi Meet
ii  jitsi-meet-web                       1.0.4466-1                        all          WebRTC JavaScript video conferences
ii  jitsi-meet-web-config                1.0.4466-1                        all          Configuration for web serving of Jitsi Meet
ii  jitsi-videobridge2                   2.1-376-g9f12bfe2-1               all          WebRTC compatible Selective Forwarding Unit (SFU)

I have a VPS with a static ip connected to the internet. If I disable simucast i still get the errors, but I get the HD videostream. Are there any pointers where to search?

Cheers

Th3R3al · November 27, 2020, 12:10pm

Hey boonsai,

that sounds like the websocket cant connect to the jvb. Configure that according to the documentation:

github.com

jitsi/jitsi-videobridge/blob/master/doc/web-sockets.md

# Introduction
WebSockets can be used instead of WebRTC Data Channels for transport of Colibri
client-to-bridge messages. This needs support from the bridge as well as the
client.

When this is enabled, a bridge will advertise a Colibri WebSocket URL together with
its ICE candidates. The URL will be specific to an endpoint (in fact the ICE username
fragment is reused, encoded as a URL parameter, for authentication), and connections
to it will be routed to the Endpoint representation in the bridge.

The URL has the following format:
```
wss://example.com/colibri-ws/server-id/conf-id/endpoint-id?pwd=123
```

# Bridge configuration
To enable WebSockets on the bridge, both the publicly-accessible HTTP server and WebSockets
themselves must be enabled.

The following block in jvb.conf configures the publicly-accessible HTTP server:

This file has been truncated. show original

And it will work

Greets
Th3r3al

b00nsai · November 27, 2020, 12:42pm

Hi Th3r3al (and sorry OP to hijack your thread),

I have the letsencrypt SSL certificate from the provided script. My current jvb.conf looks like this:

videobridge {
    http-servers {
        public {
            port = 9090
        }
    }
    websockets {
        enabled = true
        domain = "meet.myUrl.tld:443"
        tls = true
    }
}

That probably means i should configure the videobridge to also accept tls, right?
Sadly, I’m a real beginner, when it comes to web-stuff. So I don’t know where my TLS keystore should be or what my keystore password is. Do i have to set that on my own somewhere?
I’m only running one instance of jitsi-meet, and all my clients will be browsers or the mobile app. So I guess I’m not running a HTTP-Proxy. (I really just use a fresh Ubuntu 20.04LTS and apt packages.)

Cheers

Lichtinger_Bernhard · November 27, 2020, 2:01pm

You should definitely also set a uniqe server-id for each videobridge in the websockets section.

Then you need in your webserver config for each videobridge a corresponding location definition for proxy_pass of the websocket connections to each videobridge.

The example in the documentation runs two videobridges on localhost, but if you have videobridges on different hosts, you must use the correct IP addresses and not 127.0.0.1 in the proxy_pass line.

eazyadm · November 27, 2020, 2:40pm

Many thanks to @Th3R3al, you saved my weekend!

I tried it and it is working with an nginx proxy between. Now I’ll deploy those settings to als JVBs and everything will be fine.

I’ve no idea why I missed the requirement of the websockets.

Hopefully others with the same problem will find that thread.

Have a nice weekend and give me a hint to which adress I have to send some beer

Th3R3al · November 27, 2020, 7:05pm

Hey,

great that things are now working on your side .

If you have multiple jvbs, please take attention to what @Lichtinger_Bernhard said (unique config + id per JVB) so the clients connect to the right one via websocket.

@b00nsai If you jvbs and the nginx have a trusted network layer in between, you probably doesnt need https, no you can speak plain http between nginx and the jvb, but https to your endpoint.

We are doing things here with a wireguard overlay network, using it as a trusted network layer.

Greets
Th3R3al

PS: @eazyadm I drop you a pm

eazyadm · November 27, 2020, 9:17pm

Hi,

thanks for the hint, I saw it in the documentation. 13 Videobridges are already running and doing now a good job

Don’t forget the PM.

Cheers

b00nsai · November 28, 2020, 9:41pm

Ok it seems the problem is apache2. I replaced it with nginx and the ws config is done properly by default. But still i cant get a connection.

BridgeChannel.js:88 WebSocket connection to 'wss://meet.MyURl.tld/colibri-ws/jvb1/a5761d13aa7f5b09/e6603714?pwd=secret;)' failed: Error during WebSocket handshake: Unexpected response code: 200

And

2020-11-28T20:17:09.223Z [modules/RTC/BridgeChannel.js] <WebSocket.e.onclose>: Channel closed by server

What i really find weird is the “Unexpected response code: 200”
200 usually means everything is alright, or am i mistaken?

EDIT:

I’m stupid. I still had “default-id” in the line

location ~ ^/colibri-ws/default-id/(.*)

instead of

location ~ ^/colibri-ws/jvb1/(.*)

Thanks for your help, works like a charm now