Browser Hanging using Jitsi Torture

Hello Everyone,

I am trying to run Jitsi torture script of 1 conference for 120 participants with 120 videos and 120 audios for 20 mins. The process goes fine and I get success at the end of the test. However during the test, If I try to open the conference on browser I don’t see the exact number of participants all the time. Sometimes it shows correct number of participants and at that point of time videos or Audios are not working for all participants. At times it shows lesser number of participants (like 35) and then count start increasing slowly but then it stops increasing at a point and the conference is disconnected and it automatically re joins after 15 seconds. If I review the servers resources all are good and not maxed out.

However whatever happens the results of the test comes as success. So I was just wondering if someone can help me understand if this is the right method to test.

  1. Eventually I will be hosting a conference in which there will be 100+ number of users with audio and video who will be joining the conference. I need to prepare the servers to do that. How can I check if a user will be able to open the conference in browser and see all the participants.
  2. I see there is a folder surefire-reports in torture, which gets updated after every result. Does these reports provide some insights of tests completed? How can I see these reports? Can someone points me to some tutorial?

Thanks In Advance.

Are you using a selenium grid with powerful nodes that can handle the chromes? The problem with this testing is that is expensive and needs many and powerful nodes.

Yes Absolutely. I am using 6 selenium node of C5.9xlarge each to carry out the testing. I can confirm that the selenium node are enough to test for 120 users with Video and Audio. I have run multiple tests and then reached to this configuration.

Did you monitor the prosody process while doing the test?

Yes. I have one server hosting jicofo and prosody and 2 JVB servers of c5.2xlarge. All looks good. CPU consumption is less than 50% during test.

btw Is it the right way to check feasibility of 100+ participant conference by checking in to the conference during test. Can you share your thoughts on point 1 and 2 in my first message?

That does not help, you need to monitor prosody process. This machine is with 8 cores where prosody is single-threaded and if it hits the ceiling you will not see it as it is just 1/8 of that percentage.

That is fine, I had been testing entering conferences with 500 participants… as we work on big conferences.

Enter the conference from your browser. In such conferences you should enable lastN so you don’t forward everything.
In the future there will be optimizations in the UI so only what is shown will be forwarded, but we are working on that, check the latest community call.

There is no tutorial and that folder holds reports for failed tests, but that is for the functional testing. The malleus testing with many participant is more to create load and monitor the bridges and the signaling nodes for problems. There is no report of any kind from the malleus side of things.

Thanks for the detailed description. I have monitored the prosody process as well and it is not maxing out. It goes up and down which at times goes up to 75% but mostly stays lesser than 10%. There are no errors in log as well. I assume if prosody stops responding in between it will through an error.

What else can I check. Please help.

@damencho Any thoughts on this plz?

Are you using the full UI or the malleus script with the load test client?

I am using following command to run the script:

./scripts/malleus.sh --conferences=1 --participants=100 --senders=100 --audio-senders=100 --duration=300 --room-name-prefix=testconf --hub-url=http://x.x.x.x:4444/wd/hub --instance-url=https://jitsi.testdomain.in/

Pass --use-load-test and make sure you have this uncommented on your deployment: jitsi-meet/jitsi-meet.example at 98658f573cd6679372d5803f4b14727f5329d9ba · jitsi/jitsi-meet · GitHub

You will use a lighter version of the client so the chromes don’t struggle loading it …

After making the changes in nginx conf, when I run the script I am getting following error:

[ERROR] Tests run: 333, Failures: 1, Errors: 0, Skipped: 332, Time elapsed: 7.741 s <<< FAILURE! - in TestSuite
[ERROR] testMain(org.jitsi.meet.test.MalleusJitsificus) Time elapsed: 1.33 s <<< FAILURE!
java.lang.Exception: Failed with multiple errors. Throws the primary.
at org.jitsi.meet.test.MalleusJitsificus.testMain(MalleusJitsificus.java:273)
Caused by: org.openqa.selenium.WebDriverException:
unknown error: config is not defined
(Session info: chrome=89.0.4389.90)
(Driver info: chromedriver=2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 5.4.0-1045-aws x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 0 milliseconds

What am I missing here?

Have you tried to load the URL:
https://yourdomain.com/_load-test/loadtest, does it work?
Can you see in the js console the config object?

It simply didn’t run. I do see it tries to open the chrome session like this:

[35] web.participant17 is opening URL: https://jitsi.mohaliwalk.in/_load-test/redmoon0#config.testing.testMode=true&config.disableNS=true&config.testing.noAutoPlayVideo=true&config.disableAEC=true&config.analytics.disabled=true&config.startWithAudioMuted=true&config.prejoinPageEnabled=false&interfaceConfig.SHOW_CHROME_EXTENSION_BANNER=false&config.disable1On1Mode=false&config.alwaysVisibleToolbar=true&config.p2p.useStunTurn=true&config.p2p.enabled=true&config.requireDisplayName=false&config.gatherStats=true&config.pcStatsInterval=10000&config.debug=true&config.enableTalkWhileMuted=false&config.startWithVideoMuted=true&config.callStatsID=false&interfaceConfig.DISABLE_FOCUS_INDICATOR=true

But it simply not able to start. It opens up all the sessions and then throws the error: Attached are the complete logs. JitsiTortureLogs.txt (34.9 KB) Please help.

I tried with just 1 conference and 10 participants.

Your link returns 404 when you try to open it, probably your nginx config is wrong and don’t have the needed setting … See nginx logs for errors and fix it

I have added following code in nginx site config file:

location ~ ^/_load-test/([^/?&:’"]+) { rewrite ^/_load-test/(.*) /load-test/index.html break;
}
location ~ ^/_load-test/libs/(.)$ {
add_header ‘Access-Control-Allow-Origin’ '
’;
alias /usr/share/jitsi-meet/load-test/libs/$1;
}

as suggested above. Please correct me if I am wrong here.

Nginx is indeed returning 404 error
1047 open() “/usr/share/jitsi-meet/load-test/index.html” failed (2: No such file or directory)

I don’t see load-test folder in the Jitsi-meet folder. Not sure what I am missing here. Please help.

Which version of jitsi-meet is this?
This should be in /usr/share/jitsi-meet/load-test

I am using following versions:
root@jitsi:/usr/share/jitsi-meet# dpkg -l | grep jitsi
jitsi-meet 2.0.5390-3
jitsi-meet-prosody 1.0.4628-1
jitsi-meet-turnserver 1.0.4628-1
jitsi-meet-web 1.0.4628-1
jitsi-meet-web-config 1.0.4628-1
jitsi-videobridge2 2.1-416-g2f43d1b4-1

There is no load-test folder in my setup. I tried copying load-test folder from resources folder from git repo but that gives the error for libs folder under load-test:

2021/04/26 13:07:07 [error] 3071#3071: *1136 open() “/usr/share/jitsi-meet/load-test/libs/load-test-participant.min.js” failed (2: No such file or directory), client: 110.225.248.141, server: jitsi.mohaliwalk.in, request: “GET /_load-test/libs/load-test-participant.min.js HTTP/1.1”, host: “jitsi.mohaliwalk.in”, referrer: “https://jitsi.mohaliwalk.in/_load-test/redmoon0

You need to update to the latest stable.

Thanks for the guidance. After updating Jitsi Meet I see load-test folder and load-test/libs folder in the location /usr/share/jitsi-meet. I am now able to run the tests by passing --use-load-test.

However the Videos and Audios have stopped working now (whether I run it with --use-load-test or without --use-load-test) For a small conference like upto 5 users I see no video playing or audio playing. If I run conference for around 10 users. Sometimes I get 1 audio or 1 video running(out of 10 video and 10 audio). If I run it for say 50 users I see 1 or 2 videos and 1-2 audio only.

I have made no changes in jitsi torture or selenium nodes and have only updated the main JItsi-meet. Does the latest stable version needs any additional configuration to make ?