Jitsi-hammer stress test

Hi,
I want to stress my environment with jitsi-hammer.
I’m seeing that the jitsi-hammer client is very slow to send the video stream. I mean that it starts after minutes to send the video.
How can I solve this issue?
Thank you very much.
Angelo

1 Like

Sorry for the late reply. We haven’t used jitsi hammer for long time and it is hard to keep it up-to-date and working, so we are not investing time in it.
For measurements and stress we use large selenium grids with multiple browser clients joining. We use jitsi-meet-torture like tests where we can pass a custom 720p video as a source.

1 Like

I read for first time “selenium grids”
do you have a guide to use jitsi torture with selenium grids?
is it possible to use it with a head less chrome?
Thank you very much

Angelo

Yes, selenium can be used with chrome headless. The idea of the grid is where you have a hub and multiple nodes, this way jitsi-meet-torture can start multiple participants at the same time entering a conference.
For the moment we use this for load testing: https://github.com/jitsi/jitsi-meet-torture/tree/malleus we will soon merge this in the master. It is simple as:

mvn \
-Dthreadcount=1 \
-Dorg.jitsi.malleus.conferences=2 \
-Dorg.jitsi.malleus.participants=2 \
-Dorg.jitsi.malleus.senders=2 \
-Dorg.jitsi.malleus.duration=300 \
-Dorg.jitsi.malleus.room_name_prefix="mytestroom" \
-Dremote.address="http://your_grid_hub:4444/wd/hub" \
-Djitsi-meet.tests.toRun=MalleusJitsificus \
-Dremote.resource.path=/usr/share/jitsi-meet-torture \
-Djitsi-meet.instance.url=https://your.jitsi.meet.example.com \
-Djitsi-meet.isRemote=true \
test

this is for joining two conferences and every conference to have 2 participants for 5 minutes.

2 Likes

thank you very much for your help

Angelo

Hi,
I’m trying your script with malleus branch of jitsi-torture.
I launched, inside jitsi-meet-torture, the following command.

mvn -Dthreadcount=1 -Dorg.jitsi.malleus.conferences=2 -Dorg.jitsi.malleus.participants=2 -Dorg.jitsi.malleus.senders=2 -Dorg.jitsi.malleus.duration=300 -Dorg.jitsi.malleus.room_name_prefix=“mytestroom” -Dremote.address=“http://192.168.252.230:4444/wd/hub” -Djitsi-meet.tests.toRun=MalleusJitsificus -Dremote.resource.path=~/jitsi-meet-torture -Djitsi-meet.instance.url=https://jitsitest.myserver.com -Djitsi-meet.isRemote=true test > output.log 2>&1

selenium grid has been launched in the same server.

I have a lot of exceptions (see attached file).
where am I doing wrong?
output.log (31.9 KB)

Thank you very much
Angelo

Hi,
i found that the problem was the chrome driver.
I want to know if I can launch chrome without interface and in “no sandbox” mode and, if it is possible, I want to launch jitsi with a custom video/audio.
Thank you for your help

Yep, there are many options https://github.com/jitsi/jitsi-meet-torture/blob/master/src/test/java/org/jitsi/meet/test/web/WebParticipantFactory.java#L232
https://github.com/jitsi/jitsi-meet-torture/blob/master/src/test/java/org/jitsi/meet/test/web/WebParticipantOptions.java

Hi,
my fundamental problem (for now) is that I set no-sandbox parameter, but chrome does not show me the message “no sandbox is not supported”.

Hi,
I’m trying to launch in a server without user interface.

I have the following error:
> Build info: version: ‘3.13.0’, revision: ‘2f0d292’, time: ‘2018-06-25T15:24:21.231Z’

System info: host: 'jitsi-original-hammer', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0-98-generic', java.version: '1.8.0_181'
Driver info: org.openqa.selenium.remote.RemoteWebDriver
Capabilities {acceptInsecureCerts: false, acceptSslCerts: false, applicationCacheEnabled: false, browserConnectionEnabled: false, browserName: chrome, chrome: {chromedriverVersion: 2.38.552522 (437e6fbedfa876..., userDataDir: /tmp/.org.chromium.Chromium...}, cssSelectorsEnabled: true, databaseEnabled: false, handlesAlerts: true, hasTouchScreen: false, javascriptEnabled: true, locationContextEnabled: true, mobileEmulationEnabled: false, nativeEvents: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, rotatable: false, setWindowRect: true, takesHeapSnapshot: true, takesScreenshot: true, unexpectedAlertBehaviour: , unhandledPromptBehavior: , version: 65.0.3325.210, webStorageEnabled: true, webdriver.remote.sessionid: 50e428f6e27e7c65ab55e65bcff...}
Session ID: 50e428f6e27e7c65ab55e65bcff165d9
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:214)
	at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:166)
	at org.openqa.selenium.remote.http.JsonHttpResponseCodec.reconstructValue(JsonHttpResponseCodec.java:40)
	at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:80)
	at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:44)
	at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:548)
	at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:485)
	at org.jitsi.meet.test.web.WebParticipant.executeScript(WebParticipant.java:193)
	at org.jitsi.meet.test.web.WebParticipant.doJoinConference(WebParticipant.java:132)
	at org.jitsi.meet.test.base.Participant.joinConference(Participant.java:152)
	at org.jitsi.meet.test.MalleusJitsificus.lambda$runAsync$0(MalleusJitsificus.java:154)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-1" org.openqa.selenium.WebDriverException: unknown error: APP is not defined
  (Session info: headless chrome=65.0.3325.210)
  (Driver info: chromedriver=2.38.552522 (437e6fbedfa8762dec75e2c5b3ddb86763dc9dcb),platform=Linux 4.4.0-98-generic x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 0 milliseconds

I don’t understand where is the error.

I launch the script with the following command

reset && mvn -Dthreadcount=1 -Dorg.jitsi.malleus.conferences=1
-Dorg.jitsi.malleus.participants=4 -Dorg.jitsi.malleus.senders=4
-Dorg.jitsi.malleus.duration=300
-Dorg.jitsi.malleus.room_name_prefix=“mytestroom”
-Dremote.address=“http://localhost:4444/wd/hub
-Djitsi-meet.tests.toRun=MalleusJitsificus
-Dremote.resource.path=~/jitsi-meet-torture
-Djitsi-meet.instance.url=https://jitsivb.myserver.com
-Djitsi-meet.isRemote=true test

thank you very much for your help

Have you created a selenium grid? Looking at the parameters you pass: http://localhost:4444/wd/hub your selenium hub is on the same machine, is this correct?
Normally we see “APP is not defined” error when the page(https://jitsivb.myserver.com) did not load for some reason.

thank you very much
it was a problem about the ssl certificate.

I have another question:
I’m trying to launch 50 clients in headless mode.
the infrastructure is very big, 72 core and 256gb of ram.
with 20 clients I have a connection issue, I’m using a normal client to see the events.

could you tell me how you scale a lot in your tests?
Thank you very much
Angelo

So what we are doing is having an auto-scale group for the nodes. Nodes are small instances for aws, cannot recall the size, but can check later. When we need a test like this we just auto-scale to have lets say 50 or even 100 nodes, do the test and down-scale. Every node is one chrome instance.
Running a lot of chrome instances on one machine is tricky, I guess. Resources, cpu and bandwidth … I suppose you hit the bandwidth part…

Hi, thank you for your response.
the bandwidth is not a problem and I’m going to move the clients to a set of servers.

I have a question: in the hammer project it is possible to set how many clients have to be instantiated per second.

is it possible with jitsi-meet-torture?

Thank you very much

Angelo

By this you mean jitsi-hammer? I don’t know.

Stuff like that can be easily coded for torture, it will not be very punctual cause spinning a chrome instance takes time, and can vary, but overall the code base I gave you is spinning the remote selenium drivers in new threads I think, and those threads can be grouped, by conference and/or by time periods, it is a matter of requirements … it can be extended to be generic and those behaviors to be parametrized, everything can be done :slight_smile:

thank you for all your answers.

I have a question about the following parameters

-Dthreadcount=1
-Dorg.jitsi.malleus.conferences=1
-Dorg.jitsi.malleus.participants=20
-Dorg.jitsi.malleus.senders=2

I want to launch 20 clients with only one conference room.
I don’t understand thread count and senders.
if I set senders=20 I have a very high value of in-bandwidth in the videobridge VM.
I’ve modified chromium and the video bandwidth is always 600kbps per every client.
so, what I’m expecting is (about the VB server):
input (600+audio bandwidth)20 kbps (more or less 13mbps)
output 3
(600+audio bandwidth)*20 kbps (ore or less 39mbps)
my lastN value is 4, so I receive, from the VB, 3 flows of other clients. 1 flow is the loopback video streaming.

what I have is different,
in input I have 44mbps
in output I have 40mbps (this value is ok).

so the problem is: what is the sender parameter?
thank you very much.

This means that out of the 20 participants, just 2 will have video, the others will start video muted:

            if (muteVideo)
            {
                _url.appendConfig("config.startWithVideoMuted=true");
            }

In my case I have the following results
input videobridge 43903.40 kbps
output videobridge 44193.78 kbps

but with lastN=4 the value is not true.
the input has to be more or less 12mbps.

Could you tell me about the threadcount parameter?
thank you for your help

ok, the problem was that the simulcast was set to true.
I don’t need simulcast.

thanks for your help

Hi,
I’m trying to launch 50 clients using one instance of selenium grid.
After 20 users I have in selenium log

[1541164179.254][SEVERE]: Timed out receiving message from renderer: -0.004

and the jitsi-meet-torture shows

System info: host: ‘jitsi-original-hammer’, ip: ‘127.0.0.1’, os.name: ‘Linux’, os.arch: ‘amd64’, os.version: ‘4.4.0-98-generic’, java.version: ‘9-internal’
Driver info: org.openqa.selenium.remote.RemoteWebDriver
Capabilities {acceptInsecureCerts: false, acceptSslCerts: false, applicationCacheEnabled: false, browserConnectionEnabled: false, browserName: chrome, chrome: {chromedriverVersion: 2.38.552522 (437e6fbedfa876…, userDataDir: /tmp/.org.chromium.Chromium…}, cssSelectorsEnabled: true, databaseEnabled: false, handlesAlerts: true, hasTouchScreen: false, javascriptEnabled: true, locationContextEnabled: true, mobileEmulationEnabled: false, nativeEvents: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, rotatable: false, setWindowRect: true, takesHeapSnapshot: true, takesScreenshot: true, unexpectedAlertBehaviour: , unhandledPromptBehavior: , version: 65.0.3325.210, webStorageEnabled: true, webdriver.remote.sessionid: 4e00a9be9d552ffd95d6ef661a2…}
Session ID: 4e00a9be9d552ffd95d6ef661a2e6e7c
at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(java.base@9-internal/DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(java.base@9-internal/Constructor.java:453)
at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:214)
at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:166)
at org.openqa.selenium.remote.http.JsonHttpResponseCodec.reconstructValue(JsonHttpResponseCodec.java:40)
at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:80)
at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:44)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:548)
at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:485)
at org.jitsi.meet.test.util.MeetUtils.lambda$waitForPageToLoad$0(MeetUtils.java:244)
at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:248)
at org.jitsi.meet.test.util.MeetUtils.waitForPageToLoad(MeetUtils.java:249)
… 4 more

After this list of messages, the client stops.

do you have a solution or a hint about this error?
Thank you very much

Angelo