Platform Comparison - Zoom vs Jitsi (VP8, VP9); A comprehensive Test

Ever since I discovered Jitsi and installed my own server, I’ve been on an unending quest to get the best quality possible from it. Having used other video conferencing platforms before Jitsi, I had a baseline expectation and a set of reference points for comparison. After much tinkering, I decided to do a comprehensive test that should hopefully objectively show the difference between the current industry leader (Zoom) and Jitsi. I’m sharing my findings.

SET UP
I tested using 3 instances:

  1. Zoom
  2. Public instance of Jitsi at meet.jit.si - VP8
  3. My own Jitsi installation (MyJitsi) - VP8, VP9, Jitsi Meet version 2.0.5286-1(unstable)

ENVIRONMENT

Server (MyJitsi)
Bare metal server
CPU - 8 cores 2.33GHz
RAM - 16GB
HDD - 300GB RAID
Available bandwidth (ethernet) - 300Mbps up/down

Clients
Client1 – PC Laptop, Windows 10 Pro, Duo Core, Intel i3 - 2.53GHz, 4GB RAM (Jitsi Electron App)
Client2 – Android Phone (Jitsi Android App)
Client3 – Mac, OSx 10.13.6, Quad Core, Inter i7 – 2.5GHz, 16GB RAM (Jitsi Electron App)
Client4 – Mac, OSx 10.13.6, Quad Core, Inter i7 – 2.5GHz, 16GB RAM (Chromium)
Client5 – Mac, OSx 10.13.6, Quad Core, Inter i7 – 2.5GHz, 16GB RAM (Brave)

Additional Information

  • I purposely took all measurements on the low-end PC as it’s the most likely machine I have around to be stressed. I figured it would provide the least performance, so should serve as a good baseline for stress measurement
  • I connected on WiFi on all clients, except Client2 (mobile data)
  • I took readings from the performance tab in the basic windows Task Manager
  • I ran the initial tests (3 clients) on the various apps (rather than browsers) to have a closer comparison to the Zoom test done with the Zoom desktop app
  • I forced VP9 on Jitsi through entries in Jicofo’s sip-communicator-properties
  • I disabled simulcast in some of the tests to get the best resolutions possible
  • I disabled layer suspension in some instances to maintain the attained resolution
  • I ran Jitsi clients in 2 view sizes - the default app size and a maximized full-screen view
  • I recorded visual quality based on how the meeting looked to me (this of course could be subjective)
  • The max resolution possible on Client1’s cam is 640x480; Client2 was also set to 640x40; Client3 (Client4, Client 5) maxes at 1280x720.
  • I set my resolution constraints as shown below:
    Screen Shot 2020-12-02 at 4.30.07 AM

FINDINGS

OBSERVATIONS

  • Jitsi hits client CPUs a lot harder than Zoom, in every single instance
  • Bandwidth usage on default Jitsi settings (meet.jit.si) was much lower than Zoom (Test #1)
  • Jitsi is capable of much better quality, but that currently can only come at significant costs (client CPU, bandwidth)
  • Jitsi’s bandwidth estimation and concurrent resolution adjustment is probably its biggest achilles’ heel
  • VP9 with SVC would be a total game-changer for Jitsi!
  • CPU usage in Jitsi seemed to rise with the additional clients
  • On the server side, my server (MyJitsi) didn’t do any work at all - 2% CPU, 0.9% RAM

CONCLUSION
I think you can draw your own conclusions from these results. One thing that’s certain - Jitsi is a very strong player in this field, with great potential to dominate once the right efficiencies are built into the product. On a side note, Jibri on the other hand is currently its least efficient solution (because of its resource-hungry demands), but the convenience it provides is well worth the investment, nonetheless.

Happy Jitsi-ing! :smiley:

9 Likes

Thanks for sharing this! :heart:

1 Like

The guys at Jitsi got curious – why not put it to the test ? … Full adaptation to limiting the bandwidth took WebRTC 20 seconds. … This starts to sound like the VP8 vs H.264 quality comparisons of the past (I never could tell the … rtcEvent(‘Stop limit’, ‘global’ ); } client // 2 more minutes unlimited .pause(60*sec) .

Your point is?

@mistycarpenter87 yeah, I’m curious too: what exactly is the groundbreaking point you’re trying to make here?