Jitsi Load Testing - javascript error: APP is not defined

We are attempting to do load testing for self hosted Jitsi meet using Jitsi meet torture.

Currently, setup selenium hub and selenium node on the same machine

Selenium Grid - Version 3.141.59
Chrome browser - Version 84.0.4147.135 (Official Build) (64-bit)
Chrome Driver (32 bit) - Version 84.0.4147.30

Here are the commands used to setup:

java -jar selenium-server-standalone-3.141.59.jar -role hub

java -Dwebdriver.chrome.driver=chromedriver -jar selenium-server-standalone-3.141.59.jar -role node -maxSession 1 -hub http://localhost:4444/grid/register -browser browserName=chrome,version=84,platform=WIN10,maxInstances=1

mvn test -Dorg.jitsi.malleus.conferences=3 -Dorg.jitsi.malleus.participants=3 -Dorg.jitsi.malleus.senders=1 -Dorg.jitsi.malleus.audio_senders=2 -Dorg.jitsi.malleus.duration=300 -Dorg.jitsi.malleus.room_name_prefix=hamertesting -Djitsi-meet.tests.toRun=MalleusJitsificus -Djitsi-meet.instance.url=“https://jitsiserver03.com

I can see that it attempts to starts 3 conferences each with 3 participants. However, some of the participants fail to join the conference. Here is the error I see in logs:

Exception in thread “Thread-7” org.openqa.selenium.JavascriptException: javascript error: APP is not defined
(Session info: chrome=84.0.4147.135)

I see above error for 4 participants only. Other 5 participants ( out of 9) are able to join conference successfully.

For the 4 participants which are not able to join due to javascript error: APP is not defined, I can see that Chrome continues to wait to load the jitsi meet page.

For all the 9 participants (including the 5 which are able to join conference), I see following warning:

org.openqa.selenium.TimeoutException: timeout: Timed out receiving message from renderer: 18.527

Looking at Jitsi meet torture source code, it seems TimeoutException is due to Chrome bug.

It looks like APP is not defined error comes for the participants for which Chrome is not loading the Jitsi meet page.

As stated above, error comes only for some of the participants but not for the others.
Any advise on what could be the problem? Do we need to test with other version of Chrome etc?

Do you have enough resources for every chrome instance?

@damencho It could be due to resources. I’ll re-test.

I was actually able to launch up to 10 conferences when running Chrome in headless mode but it quickly uses 100% CPU. How many selenium nodes would you suggest (ball park) to test 200-300 conferences?

Thanks!

In general we are using one node per chrome instance, per participant. When cpu is loaded nodes quickly throttle bandwidth and resolution and you are not getting real bridge performance being tested because participants will be sending lower resolution.
Good idea is to monitor all nodes bandwidth, that throttle is easily seen on the graphics.
For 200 participants I would say 200 nodes to run 200 chromes.

1 Like

@damencho Just to confirm - Are you recommending separate VM to setup each selenium node? So, basically 200 VM to run 200 chromes for 200 participants?

Yep, this is what we do. But if you have a very powerful vm that can handle several chromes go with smaller number. Its that encoding and decoding is cpu intensive and several chromes on one machine and they quickly will throttle encoding/resolution

Hello @K100, it was really a resource issue?
I’m trying to do a loadtest using selenium-nodes with docker, but for only 2 participants I’m getting this error of “APP is not defined”

The VM I’m running have a lot of resource, so I don’t think this is my problem.

Thanks.

EDIT:
My jitsi server is using a auto-signed certificate, I was reading about it in another topic. It can be the cause of the problem?

I manage to make it working by add the argument “ignore-certificate-errors” in MalleusJitsiFicus.java, it was really a SSL error.

Cool. In my case, I was getting APP is not defined error only for some of the participants. I noticed high CPU usage the moment I try to launch multiple conference on a machine.

If you were able to complete your load testing, would you mind sharing the results? How are you sizing it based on the participants?

I didn’t complete yet, because I had some problems in sharing the video file in all nodes, but a 110GB RAM VM with 24vCPU reach 100% when I used 70 nodes (1 node is 1 docker container, with each one running 1 chrome instance).
Even with 100% of CPU, although some rendered erros, the participants didn’t disconnect. But I will only have the real performance when I get all conferences running with audio and video. Now it only connects the nodes to the referenced conference rooms.