Jitsi-hammer stress test


#1

Hi,
I want to stress my environment with jitsi-hammer.
I’m seeing that the jitsi-hammer client is very slow to send the video stream. I mean that it starts after minutes to send the video.
How can I solve this issue?
Thank you very much.
Angelo


#2

Sorry for the late reply. We haven’t used jitsi hammer for long time and it is hard to keep it up-to-date and working, so we are not investing time in it.
For measurements and stress we use large selenium grids with multiple browser clients joining. We use jitsi-meet-torture like tests where we can pass a custom 720p video as a source.


#3

I read for first time “selenium grids”
do you have a guide to use jitsi torture with selenium grids?
is it possible to use it with a head less chrome?
Thank you very much

Angelo


#4

Yes, selenium can be used with chrome headless. The idea of the grid is where you have a hub and multiple nodes, this way jitsi-meet-torture can start multiple participants at the same time entering a conference.
For the moment we use this for load testing: https://github.com/jitsi/jitsi-meet-torture/tree/malleus we will soon merge this in the master. It is simple as:

mvn \
-Dthreadcount=1 \
-Dorg.jitsi.malleus.conferences=2 \
-Dorg.jitsi.malleus.participants=2 \
-Dorg.jitsi.malleus.senders=2 \
-Dorg.jitsi.malleus.duration=300 \
-Dorg.jitsi.malleus.room_name_prefix="mytestroom" \
-Dremote.address="http://your_grid_hub:4444/wd/hub" \
-Djitsi-meet.tests.toRun=MalleusJitsificus \
-Dremote.resource.path=/usr/share/jitsi-meet-torture \
-Djitsi-meet.instance.url=https://your.jitsi.meet.example.com \
-Djitsi-meet.isRemote=true \
test

this is for joining two conferences and every conference to have 2 participants for 5 minutes.


#5

thank you very much for your help

Angelo


#6

Hi,
I’m trying your script with malleus branch of jitsi-torture.
I launched, inside jitsi-meet-torture, the following command.

mvn -Dthreadcount=1 -Dorg.jitsi.malleus.conferences=2 -Dorg.jitsi.malleus.participants=2 -Dorg.jitsi.malleus.senders=2 -Dorg.jitsi.malleus.duration=300 -Dorg.jitsi.malleus.room_name_prefix=“mytestroom” -Dremote.address=“http://192.168.252.230:4444/wd/hub” -Djitsi-meet.tests.toRun=MalleusJitsificus -Dremote.resource.path=~/jitsi-meet-torture -Djitsi-meet.instance.url=https://jitsitest.myserver.com -Djitsi-meet.isRemote=true test > output.log 2>&1

selenium grid has been launched in the same server.

I have a lot of exceptions (see attached file).
where am I doing wrong?
output.log (31.9 KB)

Thank you very much
Angelo


#7

Hi,
i found that the problem was the chrome driver.
I want to know if I can launch chrome without interface and in “no sandbox” mode and, if it is possible, I want to launch jitsi with a custom video/audio.
Thank you for your help


#8

Yep, there are many options https://github.com/jitsi/jitsi-meet-torture/blob/master/src/test/java/org/jitsi/meet/test/web/WebParticipantFactory.java#L232
https://github.com/jitsi/jitsi-meet-torture/blob/master/src/test/java/org/jitsi/meet/test/web/WebParticipantOptions.java


#9

Hi,
my fundamental problem (for now) is that I set no-sandbox parameter, but chrome does not show me the message “no sandbox is not supported”.


#10

Hi,
I’m trying to launch in a server without user interface.

I have the following error:
> Build info: version: ‘3.13.0’, revision: ‘2f0d292’, time: ‘2018-06-25T15:24:21.231Z’

System info: host: 'jitsi-original-hammer', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0-98-generic', java.version: '1.8.0_181'
Driver info: org.openqa.selenium.remote.RemoteWebDriver
Capabilities {acceptInsecureCerts: false, acceptSslCerts: false, applicationCacheEnabled: false, browserConnectionEnabled: false, browserName: chrome, chrome: {chromedriverVersion: 2.38.552522 (437e6fbedfa876..., userDataDir: /tmp/.org.chromium.Chromium...}, cssSelectorsEnabled: true, databaseEnabled: false, handlesAlerts: true, hasTouchScreen: false, javascriptEnabled: true, locationContextEnabled: true, mobileEmulationEnabled: false, nativeEvents: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, rotatable: false, setWindowRect: true, takesHeapSnapshot: true, takesScreenshot: true, unexpectedAlertBehaviour: , unhandledPromptBehavior: , version: 65.0.3325.210, webStorageEnabled: true, webdriver.remote.sessionid: 50e428f6e27e7c65ab55e65bcff...}
Session ID: 50e428f6e27e7c65ab55e65bcff165d9
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:214)
	at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:166)
	at org.openqa.selenium.remote.http.JsonHttpResponseCodec.reconstructValue(JsonHttpResponseCodec.java:40)
	at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:80)
	at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:44)
	at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:548)
	at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:485)
	at org.jitsi.meet.test.web.WebParticipant.executeScript(WebParticipant.java:193)
	at org.jitsi.meet.test.web.WebParticipant.doJoinConference(WebParticipant.java:132)
	at org.jitsi.meet.test.base.Participant.joinConference(Participant.java:152)
	at org.jitsi.meet.test.MalleusJitsificus.lambda$runAsync$0(MalleusJitsificus.java:154)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-1" org.openqa.selenium.WebDriverException: unknown error: APP is not defined
  (Session info: headless chrome=65.0.3325.210)
  (Driver info: chromedriver=2.38.552522 (437e6fbedfa8762dec75e2c5b3ddb86763dc9dcb),platform=Linux 4.4.0-98-generic x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 0 milliseconds

I don’t understand where is the error.

I launch the script with the following command

reset && mvn -Dthreadcount=1 -Dorg.jitsi.malleus.conferences=1
-Dorg.jitsi.malleus.participants=4 -Dorg.jitsi.malleus.senders=4
-Dorg.jitsi.malleus.duration=300
-Dorg.jitsi.malleus.room_name_prefix=“mytestroom”
-Dremote.address=“http://localhost:4444/wd/hub
-Djitsi-meet.tests.toRun=MalleusJitsificus
-Dremote.resource.path=~/jitsi-meet-torture
-Djitsi-meet.instance.url=https://jitsivb.myserver.com
-Djitsi-meet.isRemote=true test

thank you very much for your help


#11

Have you created a selenium grid? Looking at the parameters you pass: http://localhost:4444/wd/hub your selenium hub is on the same machine, is this correct?
Normally we see “APP is not defined” error when the page(https://jitsivb.myserver.com) did not load for some reason.


#12

thank you very much
it was a problem about the ssl certificate.

I have another question:
I’m trying to launch 50 clients in headless mode.
the infrastructure is very big, 72 core and 256gb of ram.
with 20 clients I have a connection issue, I’m using a normal client to see the events.

could you tell me how you scale a lot in your tests?
Thank you very much
Angelo


#13

So what we are doing is having an auto-scale group for the nodes. Nodes are small instances for aws, cannot recall the size, but can check later. When we need a test like this we just auto-scale to have lets say 50 or even 100 nodes, do the test and down-scale. Every node is one chrome instance.
Running a lot of chrome instances on one machine is tricky, I guess. Resources, cpu and bandwidth … I suppose you hit the bandwidth part…


#14

Hi, thank you for your response.
the bandwidth is not a problem and I’m going to move the clients to a set of servers.

I have a question: in the hammer project it is possible to set how many clients have to be instantiated per second.

is it possible with jitsi-meet-torture?

Thank you very much

Angelo


#15

By this you mean jitsi-hammer? I don’t know.

Stuff like that can be easily coded for torture, it will not be very punctual cause spinning a chrome instance takes time, and can vary, but overall the code base I gave you is spinning the remote selenium drivers in new threads I think, and those threads can be grouped, by conference and/or by time periods, it is a matter of requirements … it can be extended to be generic and those behaviors to be parametrized, everything can be done :slight_smile:


#16

thank you for all your answers.

I have a question about the following parameters

-Dthreadcount=1
-Dorg.jitsi.malleus.conferences=1
-Dorg.jitsi.malleus.participants=20
-Dorg.jitsi.malleus.senders=2

I want to launch 20 clients with only one conference room.
I don’t understand thread count and senders.
if I set senders=20 I have a very high value of in-bandwidth in the videobridge VM.
I’ve modified chromium and the video bandwidth is always 600kbps per every client.
so, what I’m expecting is (about the VB server):
input (600+audio bandwidth)20 kbps (more or less 13mbps)
output 3
(600+audio bandwidth)*20 kbps (ore or less 39mbps)
my lastN value is 4, so I receive, from the VB, 3 flows of other clients. 1 flow is the loopback video streaming.

what I have is different,
in input I have 44mbps
in output I have 40mbps (this value is ok).

so the problem is: what is the sender parameter?
thank you very much.


#17

This means that out of the 20 participants, just 2 will have video, the others will start video muted:

            if (muteVideo)
            {
                _url.appendConfig("config.startWithVideoMuted=true");
            }

#18

In my case I have the following results
input videobridge 43903.40 kbps
output videobridge 44193.78 kbps

but with lastN=4 the value is not true.
the input has to be more or less 12mbps.

Could you tell me about the threadcount parameter?
thank you for your help


#19

ok, the problem was that the simulcast was set to true.
I don’t need simulcast.

thanks for your help


#20

Hi,
I’m trying to launch 50 clients using one instance of selenium grid.
After 20 users I have in selenium log

[1541164179.254][SEVERE]: Timed out receiving message from renderer: -0.004

and the jitsi-meet-torture shows

System info: host: ‘jitsi-original-hammer’, ip: ‘127.0.0.1’, os.name: ‘Linux’, os.arch: ‘amd64’, os.version: ‘4.4.0-98-generic’, java.version: ‘9-internal’
Driver info: org.openqa.selenium.remote.RemoteWebDriver
Capabilities {acceptInsecureCerts: false, acceptSslCerts: false, applicationCacheEnabled: false, browserConnectionEnabled: false, browserName: chrome, chrome: {chromedriverVersion: 2.38.552522 (437e6fbedfa876…, userDataDir: /tmp/.org.chromium.Chromium…}, cssSelectorsEnabled: true, databaseEnabled: false, handlesAlerts: true, hasTouchScreen: false, javascriptEnabled: true, locationContextEnabled: true, mobileEmulationEnabled: false, nativeEvents: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, rotatable: false, setWindowRect: true, takesHeapSnapshot: true, takesScreenshot: true, unexpectedAlertBehaviour: , unhandledPromptBehavior: , version: 65.0.3325.210, webStorageEnabled: true, webdriver.remote.sessionid: 4e00a9be9d552ffd95d6ef661a2…}
Session ID: 4e00a9be9d552ffd95d6ef661a2e6e7c
at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(java.base@9-internal/DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(java.base@9-internal/Constructor.java:453)
at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:214)
at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:166)
at org.openqa.selenium.remote.http.JsonHttpResponseCodec.reconstructValue(JsonHttpResponseCodec.java:40)
at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:80)
at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:44)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:548)
at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:485)
at org.jitsi.meet.test.util.MeetUtils.lambda$waitForPageToLoad$0(MeetUtils.java:244)
at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:248)
at org.jitsi.meet.test.util.MeetUtils.waitForPageToLoad(MeetUtils.java:249)
… 4 more

After this list of messages, the client stops.

do you have a solution or a hint about this error?
Thank you very much

Angelo