Platform Comparison - Zoom vs Jitsi (VP8, VP9); A comprehensive Test

Install & Config
#1

Ever since I discovered Jitsi and installed my own server, I’ve been on an unending quest to get the best quality possible from it. Having used other video conferencing platforms before Jitsi, I had a baseline expectation and a set of reference points for comparison. After much tinkering, I decided to do a comprehensive test that should hopefully objectively show the difference between the current industry leader (Zoom) and Jitsi. I’m sharing my findings.

SET UP
I tested using 3 instances:

  1. Zoom
  2. Public instance of Jitsi at meet.jit.si - VP8
  3. My own Jitsi installation (MyJitsi) - VP8, VP9, Jitsi Meet version 2.0.5286-1(unstable)

ENVIRONMENT

Server (MyJitsi)
Bare metal server
CPU - 8 cores 2.33GHz
RAM - 16GB
HDD - 300GB RAID
Available bandwidth (ethernet) - 300Mbps up/down

Clients
Client1 – PC Laptop, Windows 10 Pro, Duo Core, Intel i3 - 2.53GHz, 4GB RAM (Jitsi Electron App)
Client2 – Android Phone (Jitsi Android App)
Client3 – Mac, OSx 10.13.6, Quad Core, Inter i7 – 2.5GHz, 16GB RAM (Jitsi Electron App)
Client4 – Mac, OSx 10.13.6, Quad Core, Inter i7 – 2.5GHz, 16GB RAM (Chromium)
Client5 – Mac, OSx 10.13.6, Quad Core, Inter i7 – 2.5GHz, 16GB RAM (Brave)

Additional Information

  • I purposely took all measurements on the low-end PC as it’s the most likely machine I have around to be stressed. I figured it would provide the least performance, so should serve as a good baseline for stress measurement
  • I connected on WiFi on all clients, except Client2 (mobile data)
  • I took readings from the performance tab in the basic windows Task Manager
  • I ran the initial tests (3 clients) on the various apps (rather than browsers) to have a closer comparison to the Zoom test done with the Zoom desktop app
  • I forced VP9 on Jitsi through entries in Jicofo’s sip-communicator-properties
  • I disabled simulcast in some of the tests to get the best resolutions possible
  • I disabled layer suspension in some instances to maintain the attained resolution
  • I ran Jitsi clients in 2 view sizes - the default app size and a maximized full-screen view
  • I recorded visual quality based on how the meeting looked to me (this of course could be subjective)
  • The max resolution possible on Client1’s cam is 640x480; Client2 was also set to 640x40; Client3 (Client4, Client 5) maxes at 1280x720.
  • I set my resolution constraints as shown below:
    Screen Shot 2020-12-02 at 4.30.07 AM

FINDINGS

Screen Shot 2020-12-01 at 10.32.41 AM
Screen Shot 2020-12-01 at 10.32.41 AM1352×115 14.7 KB

Screen Shot 2020-12-01 at 10.38.05 AM
Screen Shot 2020-12-01 at 10.38.05 AM1368×436 52.5 KB

Screen Shot 2020-12-01 at 10.39.06 AM
Screen Shot 2020-12-01 at 10.39.06 AM1355×555 82.3 KB

OBSERVATIONS

  • Jitsi hits client CPUs a lot harder than Zoom, in every single instance
  • Bandwidth usage on default Jitsi settings (meet.jit.si) was much lower than Zoom (Test #1)
  • Jitsi is capable of much better quality, but that currently can only come at significant costs (client CPU, bandwidth)
  • Jitsi’s bandwidth estimation and concurrent resolution adjustment is probably its biggest achilles’ heel
  • VP9 with SVC would be a total game-changer for Jitsi!
  • CPU usage in Jitsi seemed to rise with the additional clients
  • On the server side, my server (MyJitsi) didn’t do any work at all - 2% CPU, 0.9% RAM

CONCLUSION
I think you can draw your own conclusions from these results. One thing that’s certain - Jitsi is a very strong player in this field, with great potential to dominate once the right efficiencies are built into the product. On a side note, Jibri on the other hand is currently its least efficient solution (because of its resource-hungry demands), but the convenience it provides is well worth the investment, nonetheless.

Happy Jitsi-ing! :smiley:

13 Likes
VP9 [with bridge support] soon cometh! Do a happy dance! 🕺🏾
CPU performance, Jitsi Meet (browser) vs Jitsi Desktop
VP9 [with bridge support] soon cometh! Do a happy dance! 🕺🏾
#2

Thanks for sharing this! :heart:

1 Like
#3

The guys at Jitsi got curious – why not put it to the test ? … Full adaptation to limiting the bandwidth took WebRTC 20 seconds. … This starts to sound like the VP8 vs H.264 quality comparisons of the past (I never could tell the … rtcEvent(‘Stop limit’, ‘global’ ); } client // 2 more minutes unlimited .pause(60*sec) .

#4

Your point is?

#5

@mistycarpenter87 yeah, I’m curious too: what exactly is the groundbreaking point you’re trying to make here?

#7

Hi, I loved your test and how much is the possibility that with same clients (lets say 5-15) conference in https://meet.jit.se and in your jitsi deployment scenario has same impact on client side load?
I am facing issues in our deployment which isn’t occuring in meet.jit.si, how are you changing video codecs? by changing versions? and what’s the best config set (codecs, settings, configs) from your point of view for own server deployment to host around min 20-25 persons with video (360p-720p) on and where every end user has bandwidth limit of 2Mb/s up and 2Mb/s down…? if it’s not possible then atleast how much bandwidth a client shoud have? (I wanna put more concentration to reduce client side bandwidth as much possible)

#8

If you’re able to get VP9 to work for you, that’s definitely the superior option. With VP9 enabled, you can set the resolution to as low as 180 (200kbps upload) and still get remarkable video quality. VP9 however still has some restrictions (older versions of Safari and iOS in general, don’t yet support it).

#9

It’s really great to know…! how can I enable/disable vp8/vp9 and check? can you give me a high level overview for tweaking those configurations ?

#10

I meant yeah I could just set preferable codec as VP8 or VP9 but how can I be sure that this is working… or how can I impose that only vp8 should be used from lib-jitsi-meet?

#11

Freddie,

How do I enable VP9 ? (and get it to be used by Edge, Chrome or Chromium)

The jitsi server I am running is self hosted on a Debian 10 VM, jitsi install in Feb 2021. Using Apache2, php 7.3, behind a NAT firewall and configured appropriately as per the instructions for NAT. Jitsi is working well, just no VP9.

I have tried these below changes, but still the participants are connecting using “Codecs (A/V): opus, VP8”

/etc/jitsi/jicofo/sip-communicator.properties
org.jitsi.jicofo.ENABLE_VP9=true

/etc/jitsi/meet/video.mydomain.com-config.js

// Specify the settings for video quality optimizations on the client.
videoQuality: {
disabledCodec: ‘H264’,
preferredCodec: ‘VP9’,
maxBitratesVideo: {
low: 200000,
standard: 500000,
high: 1500000
},
minHeightForQualityLvl: {
360: ‘standard’,
720: ‘high’
},
},

=========================================================

apt list apache2

Listing… Done
apache2/stable,stable,now 2.4.38-3+deb10u4 amd64 [installed,automatic]

apt list php

Listing… Done
php/stable 2:7.3+69 all

apt list php7.3 -a

Listing… Done
php7.3/stable 7.3.27-1~deb10u1 all
php7.3/stable 7.3.19-1~deb10u1 all

dpkg -l | grep jitsi

ii jitsi-meet 2.0.5390-3 all WebRTC JavaScript video conferences
ii jitsi-meet-prosody 1.0.4628-1 all Prosody configuration for Jitsi Meet
ii jitsi-meet-web 1.0.4628-1 all WebRTC JavaScript video conferences
ii jitsi-meet-web-config 1.0.4628-1 all Configuration for web serving of Jitsi Meet
ii jitsi-videobridge2 2.1-416-g2f43d1b4-1 all WebRTC compatible Selective Forwarding Unit (SFU)

apt list prosody

Listing… Done
prosody/stable,now 0.11.2-1 amd64 [installed,automatic]

#12

VP9 has not been released in stable yet. It’s more than just configuration; you have to actually compile and build the bridge to test it out in beta.

#13

Thanks Freddie. Now I know why I could not get it to work : )

I will wait patiently for it to reach production. It sounds like it is an advancement that will be worthwhile.

#14

you should also set ENABLE_VP8 to false here. To try out VP9 you don’t need to compile code, it’s only necessary to do so to get the full experience and use it optimally.

#15

Totally agree with @gpatel-fr

Incase you are thinking of compiling code then

No need to disable VP8 at jicofo level. If it is disabled then it won’t work with safari. Instead change preferredCodec=VP9 in config.js.
so for
chrome client1 <—> chrome client2==> VP9
safari client1 <—> chrome client2 ==> VP8

when safari client left it will fallback to VP9 and when it joins back then it will switch back to VP8.

#16

that’s quite possible I never used my VP9 test setup with anything Mac related

unfortunately I tested this and it did not work for me, video stayed at VP8 - I just upgraded to latest unstable for the occasion - maybe you are using this on an older version ?. And the GP included his config and it contains this instruction.

#17

Last I tested it on 22nd Jan(using unstable only) and It was working at that time. Need to add preferredCodec=VP9 under videoQuality settings and enable VP8,VP9 at jicofo level too.

#18

Do you need to use unstable, to compile and run VP9 ?

If all I need to do is compile some jitsi components, then I would like to give that a try, if there are some easy to follow instructions on what need to be done, e.g. configuration changes, downloading of new code for VP9, how to compile it and how to integrate the newly complied components into an existing Jitsi server.

Otherwise, I guess I just wait for VP9 to be introduced into production, then upgrade.

Actually I read something about how to upgrade a Jitsi server (apt update && apt dis-upgrade, I think), but do you then need to reapply your specific config changes, like domain, ports, your personal enable/disable of various features, security certificates ? If there is an official “How to upgrade Jitsi” page, please post a link in this thread, so I cannot miss it.

#19

this is development stuff, not ready to run software packaged as source. This is no ‘all you need is to run ./configure; make; make install’. Instructions have been posted on this forum. They are complex and may still be usable, or not yet work on your version if it’s old, or not work anymore if it’s more recent.

1 Like