Tutorial: Loadtesting Jitsi with MalleusJitsificus on a Selenium Grid

Loadtesting Jitsi Meet

I needed to do some load testing on my jitsi meet instance to get a feeling for how many participants, audio and videostreams my JVB could handle. You can see my results (on a pretty outdated machine) here.

Today (as requested by a few others on the forum as well) I’m going to describe my Setup and walk through the most important steps.
While this would be possible on Windows as well, I’m going to focus on Linux as that is what I daily-drive and I think it is more convenient as well :wink:

A quick warning: Selenium Grids require insane amounts of resources for larger scale tests.

Topic Overview:

  • Abstract
  • Setting up a Selenium Grid
  • Installing the Maven environment
  • Running tests (in this case MalleusJitsificus. I love that name, thank you to whoever named it that way)

Abstract

There are two tools to do testing on JVBs: Jitsi Hammer and Jitsi Meet Torture. Jitsi Hammer is outdated and doesn’t work anymore, if you want to test up-to-date installations you need to use Jitsi Meet Torture (which also includes a lot of other useful tests as well and could be used in your CI/CD process)

Jitsi Meet Torture requires a Selenium Grid.
A short quote from their Documentation on what Selenium Grid is:

Selenium Grid is a smart proxy server that allows Selenium tests to route commands to remote web browser instances. Its aim is to provide an easy way to run tests in parallel on multiple machines.

With Selenium Grid, one server acts as the hub that routes JSON formatted test commands to one or more registered Grid nodes. Tests contact the hub to obtain access to remote browser instances. The hub has a list of registered servers that it provides access to, and allows control of these instances.

Selenium Grid allows us to run tests in parallel on multiple machines, and to manage different browser versions and browser configurations centrally (instead of in each individual test).

Selenium Grid is not a silver bullet. It solves a subset of common delegation and distribution problems, but will for example not manage your infrastructure, and might not suit your specific needs.

Tl;dr: It launches “real” browser Sessions on Chrome or Firefox and can control them with code you have written. In our case it can open a Webpage of your Jitsi Meet and simulate audio and video. This is as close as you can get to real-life examples without actually having people open browsers and clicking things.

Lets get to it now!

Setting up a Selenium Grid

Requirements:

  • A Java installation
  • Multiple VMs to run your tests. Cloud instances on GCloud or AWS are maybe not a bad option if you are savy enough to do some automation around them and shut them down when no longer needed.

The grid consists of multiple components:

  • The hub
  • The nodes

The hub is the brain of your Selenium Grid. It manages all nodes which run your chrome browsers.
When running tests you always connect to your hub which then launches the appropriate amount of chrome browsers on the nodes.
The hub does not require a lot of resources and can be installed on one of your node VMs as well. I’d still recommend running it on its own VM though.

Setting up the Hub

Grab your self the Selenium Server jar: https://www.selenium.dev/downloads/ I will use v.3.141.59 in this example.

Install Java on your VM (I’m using Java 8) and then run java -jar selenium-server-standalone.jar -role hub you can also use the flag -host xxx.xxx.xxx.xxx to specify which IP it should bind to.
When you navigate to http://theiporhostname:4444/grid/console You should see an almost empty page with a banner “Grid Console vx.xxx.xx”

Setting up nodes

Nodes are where the magic happens. Jitsi Meet Torture tests mostly request chrome instaces by default, this is why we will only set up chrome instances in this tutorial. You can run firefox as well, the setup is pretty much the same.
In an ideal setup you would launch 1 chrome instance per node (node with 2 CPU, 2GB RAM) but as I didn’t want to spend money on cloud servers I set up 4 VMs with 4 cores, 4gb of ram on my local computer and ran 3 chrome instances on them. I couldn’t run tests on the scale as I wanted to as I soon hit 100% CPU usage on my 8c/16t Ryzen CPU.
This works as well but is kinda janky, if you want to do “real” “professional” tests, I’d suggest going with a lot of small cloud servers with one chrome instance each.

We need the following packages (I’m going off a Debian Buster, you can adapt that to whatever distro you are running):

sudo apt-get update
sudo apt-get install -y gnupg curl wget unzip xvfb libxi6 libgconf-2-4 default-jdk
sudo curl -sS -o - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
sudo echo "deb [arch=amd64]  http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list
sudo apt-get -y update
sudo apt-get -y install google-chrome-stable

Make sure you installed your OS with a Desktop Environment, chrome browser needs some kind of X server. Use GNOME, KDE, whatever floats your boat.
This gives us our “base” install with chrome. To control chrome, we now need to get the ChromeDriver which fits your chrome version. Check the official download page for the matching ChromeDriver and download the ZIP from there, unpack it and make it executable. This is just a single binary blob.

The last part needed is Selenium which connects to your Hub and sends the commands to ChromeDriver. Download the same jar you downloaded for your hub but this time run it like this:

java -Dwebdriver.chrome.driver=path/to/chromedriver -jar selenium-server-standalone.jar -role node -maxSession 1 -hub http://yourip:4444/grid/register -browser browserName=chrome,version=80,platform=Linux,maxInstances=1

You can choose a port with -port, otherwise it will use 4444 if avaible. -host again allows you to only bind to a specific IP.

If you want to run tests with video on them, you will need to copy a sample video to each node as well, refer to “Running Tests” for this.

When reloading http://theiporhostname:4444/grid/console you should now see your nodes if they successfully registered.

This might be a lot and kinda overwhelming for new users but don’t worry, we are half way there now.

Installing the Maven environment

Do this on your normal PC or Laptop, you will start and compile your tests here and then send them off to your Selenium Grid. You need a Java JDK installed (I used openjdk-11-jdk-headless)

Jitsi Meet Torture is a collection of different tests written in Java, to compile them you need a correctly configured maven environment.
This is as simple as sudo apt install maven
mvn -version should show you something like this:

Maven home: /usr/share/maven
Java version: 11.0.6, vendor: Debian, runtime: /usr/lib/jvm/java-11-openjdk-amd64
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.19.0-6-amd64", arch: "amd64", family: "unix"

Make sure echo $JAVA_HOME returns your path, if not you need to set that Environment variable with JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:/bin/javac::")

Clone the jitsi-meet-torture repository:

git clone https://github.com/jitsi/jitsi-meet-torture
cd jitsi-meet-torture

Running Tests

To run the MalleusJitsificus test (which can do load testing) there is a nice wrapper script located in scripts/malleus.sh
Just put a --help on there to see all avaible flags, I’m going to give some examples:

./scripts/malleus.sh --conferences=2 --participants=4 --senders=1 --audio-senders=2 --duration=120 --room-name-prefix=hamertesting --hub-url=http://yourSeleniumHub:4444/wd/hub --instance-url=https://theJitsiInstallationYouWantToTest.net 

This will connect to the Selenium hub set with --hub-url and launch chrome browsers pointing to https://theJitsiInstallationYouWantToTest.net/hammertesting0 and https://theJitsiInstallationYouWantToTest.net/hammertesting1
with 4 users in each room, one of them sending video, two of them sending audio, one just watching.

Which video will it play back?

I’m glad you asked because right now it won’t play back any :smiley: we need to provide the nodes with sample files to play back.
Open jitsi-meet-torture/src/test/java/org/jitsi/meet/test/MalleusJitsificus.javain your favourite text editor and check the line private static final String INPUT_VIDEO_FILE
This is by default set to resources/FourPeople_1280x720_30.y4m which doesn’t exist on the nodes.You can download a video you like from here: https://media.xiph.org/video/derf/y4m/ and place it in /usr/share/jitsi-meet-torture/resources/ and change the file name in the .java accordingly. I suggest you just copy the entire resources folder from the repository on to the nodes, it also includes the audio files.

I hope I didn’t miss any important steps, if yes, I will add them. Should I put this as a PR on Github?

8 Likes

Thank you for putting this together @Cookiefamily

Very helpful!

thank you very much for the tutorial!

I’m somehow stucked at step “Running Tests”, I see chrome opening on just one Node and prompting “Chrome is being controlled by automated test software” but there are no conferences running on my jitsi server (I’m monitoring my jitsi-server via colibri-stats)

The other nodes seem to “subscribe” to the hub but chrome won’t start on these nodes so I’ve somehow an error in my setup. I will try a new approach tomorrow by new-installing the other nodes from scratch

Is there a good way to debug the setup?

update: got it working on my VMs, seems as it was a problem with chromedriver on my side, re-installing chromedriver and giving the adequate permissions solved my issue.

next step: stack of small cloud-server instead of local running VMs

thanks again for the tutorial!

1 Like

I am stuck with it as well for few days, getting the following exception :

Driver info: driver.version: unknown
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

Which Chrome driver you guys are using ?

Best Regards,
Mohamed Abada

Sorry I couldn’t help you in time but glad you got it resolved!

@Mohamed_Abada I was using Chrome 80.0.3987.163 With chrome driver 80.0.3987.160. havent updated yet if there is a newer version avaible

1 Like

Thanks @Cookiefamily for your support.
I am using latest 81.0.4044.69 ,will try downgrading now to see if I can get rid of this exception.

One more question, which path you’re using here ?, just to be sure :slight_smile:
-Dwebdriver.chrome.driver=path/to/chromedriver

Best Regards,
Mohamed Abada

It „should“ work with newer versions as well, let’s see how much luck you have with downgrading. In my case the path would just be
-Dwebdriver.chrome.driver=./chromedriver As I have it placed in the same folder from which I run the script.

You can test if your path is correct by just typing it in the terminal and adding a -v which should show you the version

I’m using the latest version ChromeDriver 81.0.4044.69 which works for me

This is not working for me, I have Grid Hub and Node running, installing everything from scratch on a Buster EC2 instance, SG is open to the world, and still I am unable to access this URL.

Are you implementing this on local VMs ?
I tried for the third time over a fresh Ubuntu 18.4 EC2 and still getting the same exception :

Driver info: driver.version: unknown
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.openqa.selenium.remote.W3CHandshakeResponse.lambda$errorHandler$0(W3CHandshakeResponse.java:62)
at org.openqa.selenium.remote.HandshakeResponse.lambda$getResponseFunction$0(HandshakeResponse.java:30)
at org.openqa.selenium.remote.ProtocolHandshake.lambda$createSession$0(ProtocolHandshake.java:126)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958)
at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:531)
at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:128)
at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:74)
at org.openqa.selenium.grid.session.remote.RemoteSession$Factory.performHandshake(RemoteSession.java:147)
at org.openqa.selenium.grid.session.remote.ServicedSession$Factory.apply(ServicedSession.java:161)
at org.openqa.selenium.remote.server.ActiveSessionFactory.lambda$apply$12(ActiveSessionFactory.java:180)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:440)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958)
at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:531)
at org.openqa.selenium.remote.server.ActiveSessionFactory.apply(ActiveSessionFactory.java:183)
at org.openqa.selenium.remote.server.NewSessionPipeline.lambda$null$2(NewSessionPipeline.java:66)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.Collections$2.tryAdvance(Collections.java:4719)
at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:531)
at org.openqa.selenium.remote.server.NewSessionPipeline.lambda$createNewSession$3(NewSessionPipeline.java:69)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.DistinctOps$1$2.accept(DistinctOps.java:175)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.stream.Streams$StreamBuilderImpl.tryAdvance(Streams.java:405)
at java.util.stream.Streams$ConcatSpliterator.tryAdvance(Streams.java:728)
at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:531)
at org.openqa.selenium.remote.server.NewSessionPipeline.createNewSession(NewSessionPipeline.java:72)
at org.openqa.selenium.remote.server.commandhandler.BeginSession.execute(BeginSession.java:65)
at org.openqa.selenium.remote.server.WebDriverServlet.lambda$handle$0(WebDriverServlet.java:235)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

Chrome driver is working with no issue as you can see below :

$ chromedriver
Starting ChromeDriver 81.0.4044.69 (6813546031a4bc83f717a2ef7cd4ac6ec1199132-refs/branch-heads/4044@{#776}) on port 9515
Only local connections are allowed.
Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.

Hence I am not sure what is missing really.

I ran into comparable or maybe the same issues in my setup: I use VPS as my selensium nodes which run chrome and chrome runner. The two changes that got my setup running (with ChromeDriver 81.0.4044.69):

  1. google-chrome does not like to be started as root and will not start as root. I created a second user and start my selenium node (an thereby the chrome-driver) within this user context
  2. I use Xvfb and had to add +extension RANDR to the startup

Maybe this helps you, too

Regards
Markus

1 Like

Thanks a lot @Markus_Schemp for your support.

I am actually using Xvfb for two days now, and I am able to see the user connected to the room However, there’s no Audio and Video for some reason, despite the fact that I tried with different video files.

Any idea what might be the reason behind that ?

Best Regards,
Mohamed Abada

Audio and Video works in my setup:

  1. I created a /usr/share/jitsi-meet-torture/resources/ folder on my selenium nodes
  2. And uploaded a video-input file as mentioned above (thx @Cookiefamily) and the sample audio from https://github.com/jitsi/jitsi-meet-torture/tree/master/resources to my created resource folder on the nodes
  3. I changed the paths in MalieusJitisficus.java (line 36 Input Video File) and /web/WebParticipantOptions.java (line 141 PROP_FAKE_AUDIO) and set them to absolute paths.

MalieusJitsificus.java:

...
private static final String INPUT_VIDEO_FILE = "/usr/share/jitsi-meet-torture/resources/FourPeople_1280x720_60.y4m"; 
...

WebParticipantOptions.java:


defaults.setProperty(PROP_FAKE_AUDIO, "/usr/share/jitsi-meet-torture/resources/fakeAudioStream.wav");

Best Regards,
Markus

Hi,

Are you using a desktop to run these tests?

Thanks
Rajeev

Hi,

How do we install chrome as non-root?

Thanks
Rajeev

Hi :slight_smile:

What exactly do you mean by that? Using a Desktop Computer? It can be done, but you are quite limited by the resources you have. I recommend spreading the load over multiple VPS/Cloud instances or whatever other servers you have avaible.

If you are talking about linux having a graphical interface with a desktop and such: Yes, you need a X Server running as they are actually opening up browser tabs and interacting with video/audio

You can install it as root, but run your tests on a different user (not root)

Hi,

Thank you for the quick reply…
I am installing node with chrome 81 and chrome driver 81, on ubuntu 18.04 server, however, i am just unable to run the test with the following error …

org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: exited abnormally.
(unknown error: DevToolsActivePort file doesn’t exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

Is there anything i am missing?

Thanks
Rajeev

Hi Rajeev,

I think of two things that could be the case in your setup:

  1. no xserver
  2. selenium driver running as root and trying to start chrome as root (will not work out of the box)

For nr 1: On a vps/server I would go with something like Xvfb (X virtual frame buffer). I have this in my setup. You can of course use a “full desktop” GNOME, KDE, … if you want to.

For nr 2: Chrome won’t start as root (for security reasons). If you start your selenium node within an interactive session and are logged in as root you can simply create a new user and switch to it before you start your selenium node:

adduser runner
usermod -aG sudo runner
su runner

The adduser command will create a user called “runner”.
usermod -aG sudo will grant it sudo rights (I think you won’t need it for the selenium nodes but I do it out of habit because I know I will need it someday…).
su runner switches your interactive session to the newly created user.

If you start your selenium node from within this (switched) session it will start the chrome instance in the user context of the newly created “runner” and not root. Therfore the switch user (su) command has to be done before this part of the tutorial above: