Jibri - Chrome fails to start / PJSUA issues

amcgowan · 2018-11-19 19:47:26 UTC

When a recording is initialized, the chrome driver fails to start. This is the result in the Jibri log:

2018-11-19 19:12:51.323 INFO: [39] org.jitsi.jibri.JibriManager.startFileRecording() Starting a file recording with params: FileRecordingRequestParams(callParams=CallParams(callUrlInfo=CallUrlInfo(baseUrl=https://dev-sip-jitsi.synzi.com, callName=jibritest, urlParams=[])), sessionId=artokmnpgeoaobde, callLoginParams=XmppCredentials(domain=recorder.dev-sip-jitsi.synzi.com, username=recorder, password=jibrirecorderpass)) finalize script path: /finalize_recording.sh and recordings directory: /recordings
Starting ChromeDriver 2.43.600233 (523efee95e3d68b8719b3a1c83051aa63aa6b10d) on port 14909
Only local connections are allowed.
2018-11-19 19:12:52.163 SEVERE: [39] org.jitsi.jibri.api.xmpp.XmppApi.handleStartService() Error starting service: org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: exited abnormally
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
  (Driver info: chromedriver=2.43.600233 (523efee95e3d68b8719b3a1c83051aa63aa6b10d),platform=Linux 3.16.0-5-amd64 x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 357 milliseconds
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: '9e6e5df4d651', ip: '172.17.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '3.16.0-5-amd64', java.version: '1.8.0_191'
Driver info: driver.version: ChromeDriver with stack: 
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:214)
org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:166)
org.openqa.selenium.remote.JsonWireProtocolResponse.lambda$new$0(JsonWireProtocolResponse.java:53)
org.openqa.selenium.remote.JsonWireProtocolResponse.lambda$getResponseFunction$2(JsonWireProtocolResponse.java:91)
org.openqa.selenium.remote.ProtocolHandshake.lambda$createSession$0(ProtocolHandshake.java:123)
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958)
java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498)
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:464)
org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:126)
org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:73)
org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:142)
org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:83)
org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:601)
org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:219)
org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:142)
org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:181)
org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:168)
org.jitsi.jibri.selenium.JibriSelenium.<init>(JibriSelenium.kt:142)
org.jitsi.jibri.selenium.JibriSelenium.<init>(JibriSelenium.kt:102)
org.jitsi.jibri.service.impl.FileRecordingJibriService.<init>(FileRecordingJibriService.kt:109)
org.jitsi.jibri.JibriManager.startFileRecording(JibriManager.kt:124)
org.jitsi.jibri.api.xmpp.XmppApi.handleStartService(XmppApi.kt:293)
org.jitsi.jibri.api.xmpp.XmppApi.access$handleStartService(XmppApi.kt:72)
org.jitsi.jibri.api.xmpp.XmppApi$handleStartJibriIq$1.run(XmppApi.kt:206)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

2018-11-19 19:12:52.164 FINE: [39] org.jitsi.jibri.statsd.JibriStatsDClient.incrementCounter() Incrementing statsd counter: stop:
2018-11-19 19:12:52.166 INFO: [39] org.jitsi.jibri.JibriManager.stopService() Stopping the current service
2018-11-19 19:12:52.167 INFO: [39] org.jitsi.jibri.status.JibriStatusManager.log() Received component health update: XMPPAPI has status UNHEALTHY (detail: org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: exited abnormally
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
  (Driver info: chromedriver=2.43.600233 (523efee95e3d68b8719b3a1c83051aa63aa6b10d),platform=Linux 3.16.0-5-amd64 x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 357 milliseconds
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: '9e6e5df4d651', ip: '172.17.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '3.16.0-5-amd64', java.version: '1.8.0_191'
Driver info: driver.version: ChromeDriver)
2018-11-19 19:12:52.170 INFO: [39] org.jitsi.jibri.status.JibriStatusManager.log() Health status has changed: HEALTHY -> UNHEALTHY
2018-11-19 19:12:52.177 INFO: [39] org.jitsi.jibri.api.xmpp.XmppApi.invoke() Jibri reports its status is now JibriStatus(busyStatus=IDLE, health=OverallHealth(healthStatus=UNHEALTHY, details={XMPPAPI=ComponentHealthDetails(healthStatus=UNHEALTHY, detail=org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: exited abnormally
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
  (Driver info: chromedriver=2.43.600233 (523efee95e3d68b8719b3a1c83051aa63aa6b10d),platform=Linux 3.16.0-5-amd64 x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 357 milliseconds
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: '9e6e5df4d651', ip: '172.17.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '3.16.0-5-amd64', java.version: '1.8.0_191'
Driver info: driver.version: ChromeDriver)})), publishing presence to connection dev-sip-jitsi.synzi.com

If I try to run chrome directly with the following command:

google-chrome --headless --no-sandbox --use-fake-ui-for-media-stream --start-maximized --kiosk --enabled --enable-logging --vmodule=*=3 --disable-infobars --alsa-output-device=plug:amix --screenshot=/recordings/out2.png https://dev-sip-jitsi.synzi.com/TestConference

I end up with a screen shot of the conference.

amcgowan · 2018-11-20 01:16:44 UTC

Was able to get this to work by adding --headless and --no-sandbox to the ChromeOptions in JibriSelenium.kt

Are others running into this issue or is this a problem specific to my setup?

Next problem is that FFMpeg wont start. End up with the message: Ffmpeg is running but doesn't appear to be encoding

amcgowan · 2018-11-23 20:39:45 UTC

Was able to resolve these issues to allow recording to work.

Headless flag cannot be used - chrome must actually launch in an x11 window. Even though I added the dependencies indicated on the Jibri install instructions, I had no active window for chrome to launch in.

Added Xvfb and prior to starting Jibri I’m running the following:

Xvfb :0 -ac -screen 0 1280x720x16 &

Chrome launches in this screen now and FFmpeg is able to record.

I also ended up needing to add the following flags to the command used to launch chrome --disable-setuid-sandbox --no-sandbox

So now that I’ve confirmed that recording works, I should be able to use the video in a sip call? Well not yet, this results in more issues.

First - I don’t have Pjsua installed. Nothing in the docs appear to indicate this is a dependency. Can’t seem to find a package to install for this, so installed the source and compiled.

Doing so has allowed making an audio only call. So after 4 days of effort I’m back at where I was with Jigasi.

After doing some further digging, I see that Pjsua needs to be compiled with various ffmpeg and sdl development libraries installed first. Installed those libraries and now I see in the compile output of Pjsua that it’s detecting ffmpeg and sdl. So I’m assuming I now have Pjsua compiled with proper video support.

I still don’t have video enabled on sip calls. When looking at the SDP in the Pjsua logs, it appears that it’s not included as an available stream.

Looking at the Jibri code, I don’t see how the video from display 0 is piped out through to Pjsua. I see the following comment on the start command in SipGatewayJibriService:

/**
     * Starting a [SipGatewayServiceParams] involves the following steps:
     * 1) Start selenium and join the web call on display :0
     * 2) Start the SIP client to join the SIP call on display :1
     * There are already ffmpeg daemons running which are capturing from
     * each of the displays and writing to video devices which selenium
     * and pjsua will use
     */

From what I can tell, there are no ffmpeg daemons running at this point to capture from selenium.

@damencho any insight on any of this? Is there a recommended version of Pjsua to use? What flags should be used during install and what additional packages should be installed ahead of time? I’m also confused about the sip call being on display :1… as far as I understand pjsua is a command line tool, why should it need a display?

damencho · 2018-11-24 19:10:24 UTC

Use this https://github.com/jitsi/pjproject. It uses video so it needs a display.

amcgowan · 2018-11-26 19:10:59 UTC

Thanks, I will give that a shot.