Jibri not working anymore [unknown error: DevToolsActivePort file doesn't exist]

Hi guys.

Just wanted to record a meeting, and … bam … does not work. :slight_smile: )
About version:

root@meetings:~# less /var/log/jitsi/jibri/log.0.txt
root@meetings:~# dpkg -l | grep jibri
ii  jibri                                 8.0-116-gca0c772-1                    all          Jibri
root@meetings:~# dpkg -l | grep jitsi
ii  jitsi-meet                            2.0.7001-1                            all          WebRTC JavaScript video conferences
ii  jitsi-meet-prosody                    1.0.5913-1                            all          Prosody configuration for Jitsi Meet
ii  jitsi-meet-turnserver                 1.0.5913-1                            all          Configures coturn to be used with Jitsi Meet
ii  jitsi-meet-web                        1.0.5913-1                            all          WebRTC JavaScript video conferences
ii  jitsi-meet-web-config                 1.0.5913-1                            all          Configuration for web serving of Jitsi Meet
ii  jitsi-videobridge2                    2.1-634-gff8609ad-1                   all          WebRTC compatible Selective Forwarding Unit (SFU)
root@meetings:~# 

About error:

2022-03-07 20:50:42.322 FINE: [44] [session_id=xhndnnfxrcqdfafv] FfmpegCapturer.<init>#88: Detected os as OS: LINUX
2022-03-07 20:50:42.939 SEVERE: [44] XmppApi.handleStartJibriIq#226: Error starting Jibri service 
org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: crashed.
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
  (Driver info: chromedriver=99.0.4844.51 (d537ec02474b5afe23684e7963d538896c63ac77-refs/branch-heads/4844@{#875}),platform=Linux 5.4.0-100-generic x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 241 milliseconds
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: 'meetings', ip: '', os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-100-generic', java.version: '1.8.0_312'
Driver info: driver.version: ChromeDriver

About config:

root@meetings:~# cat /etc/jitsi/jibri/jibri.conf
jibri {
chrome {
// The flags which will be passed to chromium when launching
flags = [
"--no-sandbox",
"--disable-dev-shm-usage"
]
}
}
root@meetings:~# 

Any hints how to get it to work again?

cheers, t.

What Chrome and chromedriver versions do you have? I see Chrome 99 there, did you update chromedriver accordingly?

Is that all output?

Yes. I had an empty jibri.conf first but after a little bit if internet search I dicided to put this two options in. But it did not help either.

Yes. It’s the same version for browser and driver.

Sounds like you’re still running Jibri on the old config.json that’s been deprecated. Take a look at this tutorial - you need to populate your jibri.conf. Use the sample there and adjust for your values.

99.0.4844.51 to be precise.

Just made the switch to jibri.conf but the error remains:

2022-03-08 00:03:52.145 FINE: [39] [session_id=pqezzylbrufmoiid] FfmpegCapturer.<init>#88: Detected os as OS: LINUX
2022-03-08 00:03:52.864 SEVERE: [39] XmppApi.handleStartJibriIq#226: Error starting Jibri service 
org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: crashed.
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashe
d.)
  (Driver info: chromedriver=99.0.4844.51 (d537ec02474b5afe23684e7963d538896c63ac77-refs/branch-heads/4844@{#875}),platform=Linux 5.4.0-100
-generic x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 323 milliseconds
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: 'meetings', ip: '', os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-100-generic', java.version: '1
.8.0_312'
Driver info: driver.version: ChromeDriver

The launcher:

root@meetings:~# cat /opt/jitsi/jibri/launch.sh 
#!/bin/bash

#exec java -Djava.util.logging.config.file=/etc/jitsi/jibri/logging.properties -Dconfig.file="/etc/jitsi/jibri/jibri.conf" -jar /opt/jitsi/jibri/jibri.jar --config "/etc/jitsi/jibri/config.json"
exec java -Djava.util.logging.config.file=/etc/jitsi/jibri/logging.properties -Dconfig.file="/etc/jitsi/jibri/jibri.conf" -jar /opt/jitsi/jibri/jibri.jar
root@meetings:~# 

Maybe I updated to the new jitsi release to soon?

What’s the status of Jibri?

systemctl status jibri

Also, share your full jibri.conf (edit out sensitive info).

First of all. Thanks for your help Freddie!

According to systemctl jibri and jibri-xorg are running fine.

Here my jibri.conf

himbeere@rather ~ $ cat jibri.conf 
jibri {
// A unique identifier for this Jibri
// TODO: eventually this will be required with no default
id = "fluffy"
// Whether or not Jibri should return to idle state after handling
// (successfully or unsuccessfully) a request. A value of ‘true’
// here means that a Jibri will NOT return back to the IDLE state
// and will need to be restarted in order to be used again.
single-use-mode = false
api {
http {
external-api-port = 2222
internal-api-port = 3333
 }
xmpp {
// See example_xmpp_envs.conf for an example of what is expected here
environments = [
{
name = "prod environment"
xmpp-server-hosts = ["meetings.and.so.on"]
xmpp-domain = "meetings.and.so.on"


            control-muc {
                domain = "internal.auth.meetings.and.so.on"
                room-name = "JibriBrewery"
                nickname = "jibri-nickname"
            }

            control-login {
                domain = "auth.meetings.and.so.on"
                username = "jibri"
                password = "removed-sensitive-data"
            }

            call-login {
                domain = "recorder.meetings.and.so.on"
                username = "recorder"
                password = "removed-sensitive-data"
            }

            strip-from-room-domain = "conference."
            usage-timeout = 0
            trust-all-xmpp-certs = true
         }
      ]
   }
 }
recording {
recordings-directory = "/opt/recordings"
# TODO: make this an optional param and remove the default
finalize-script = "/usr/local/bin/finalize_recording.sh"
 }
streaming {
// A list of regex patterns for allowed RTMP URLs. The RTMP URL used
// when starting a stream must match at least one of the patterns in
// this list.
rtmp-allow-list = [
// By default, all services are allowed
".*"
   ]
 }
ffmpeg {
resolution = "1920x1080"
// The audio source that will be used to capture audio on Linux
audio-source = "alsa"
// The audio device that will be used to capture audio on Linux
audio-device = "plug:bsnoop"
 }
chrome {
// The flags which will be passed to chromium when launching
flags = [
"--use-fake-ui-for-media-stream",
"--start-maximized",
"--kiosk",
"--enabled",
"--disable-infobars",
"--autoplay-policy=no-user-gesture-required"
"--no-sandbox"
"--disable-dev-shm-usage"
   ]
 }
stats {
enable-stats-d = true
 }
// webhook {
// A list of subscribers interested in receiving webhook events
// subscribers = ""
// }
jwt-info {
// The path to a .pem file which will be used to sign JWT tokens used in webhook
// requests. If not set, no JWT will be added to webhook requests.
# signing-key-path = “/path/to/key.pem”

// The kid to use as part of the JWT
# kid = "key-id"

// The issuer of the JWT
# issuer = "issuer"

// The audience of the JWT
# audience = "audience"

// The TTL of each generated JWT.  Can't be less than 10 minutes.
# ttl = 1 hour

 }
call-status-checks {
// If all clients have their audio and video muted and if Jibri does not
// detect any data stream (audio or video) comming in, it will stop
// recording after NO_MEDIA_TIMEOUT expires.
no-media-timeout = 30 seconds

// If all clients have their audio and video muted, Jibri consideres this
// as an empty call and stops the recording after ALL_MUTED_TIMEOUT expires.
all-muted-timeout = 10 minutes

// When detecting if a call is empty, Jibri takes into consideration for how
// long the call has been empty already. If it has been empty for more than
// DEFAULT_CALL_EMPTY_TIMEOUT, it will consider it empty and stop the recording.
default-call-empty-timeout = 30 seconds

   }
}
himbeere@rather ~ $ 

Your jibri.conf looks good.

Try downgrading your Chrome and Chromedriver to version 91 (or even 98, if that’s what you were previously running).

Downgraded chromedriver and google chrome to 98 but chrome is still crashing.

chromedriver.log (partial)

1646731132.579][INFO]: Launching chrome: /usr/bin/google-chrome --allow-pre-commit-input --autoplay-policy=no-user-gesture-required --disable-background-networking --disable-client-side-phishing-detection --disable-default-apps --disable-dev-shm-usage --disable-hang-monitor --disable-infobars --disable-popup-blocking --disable-prompt-on-repost --disable-sync --enable-automation --enable-blink-features=ShadowDOMV0 --enable-logging --enabled --kiosk --log-level=0 --no-first-run --no-sandbox --no-service-autorun --password-store=basic --remote-debugging-port=0 --start-maximized --test-type=webdriver --use-fake-ui-for-media-stream --use-mock-keychain --user-data-dir=/tmp/.com.google.Chrome.llb4dJ data:,
[1646731132.789][INFO]: [419a1daede9e171736c5bc05083bafc0] RESPONSE InitSession ERROR unknown error: Chrome failed to start: crashed.
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

I have this versions running on another system fine. :thinking:

Is openjdk fine?

root@meetings:~# dpkg -l | grep jdk
ii  openjdk-8-jre-headless:amd64          8u312-b07-0ubuntu1~20.04              amd64        OpenJDK Java runtime, using Hotspot JIT (headless)
root@meetings:~# 

I seem to recall there was some issue with this sometime back, if i’m not mistaken. I’d recommend adoptopenjdk-8-hotspot instead.

We now use Java 11 for everything except Jigasi.

Switched to java 11. Makes no difference. Is this supposed to work? I’m trying to find out why chrome crashes.

root@meetings:~# /usr/bin/google-chrome --allow-pre-commit-input --autoplay-policy=no-user-gesture-required --disable-background-networking --disable-client-side-phishing-detection --disable-default-apps --disable-dev-shm-usage --disable-hang-monitor --disable-infobars --disable-popup-blocking --disable-prompt-on-repost --disable-sync --enable-automation --enable-blink-features=ShadowDOMV0 --enable-logging --enabled --kiosk --log-level=0 --no-first-run --no-sandbox --no-service-autorun --password-store=basic --remote-debugging-port=0 --start-maximized --test-type=webdriver --use-fake-ui-for-media-stream --headless --use-mock-keychain --user-data-dir=/tmp/.com.google.Chrome.zEPH9i data:,
Segmentation fault (core dumped)
root@meetings:~#

On the machine where jibri is running fine google-chrome does not segfault. Maybe that’s the root problem?

Try to check where it crashes with gdb.

Broken Chrome Machine

himbeere@meetings:~$ google-chrome --version
Segmentation fault (core dumped)
himbeere@meetings:~$

Working one:

himbeere@viko:~$ google-chrome --version
Google Chrome 98.0.4758.80 
himbeere@viko:~$

So I guess the problem is 100% jibri unrelated. :slight_smile:

The output does not really help me. :slight_smile:

himbeere@meetings:/opt/google/chrome$ gdb ./chrome
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./chrome...
(No debugging symbols found in ./chrome)
(gdb) run
Starting program: /opt/google/chrome/chrome 

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7fe1f19 in ?? () from /lib64/ld-linux-x86-64.so.2
(gdb) backtrace
#0  0x00007ffff7fe1f19 in ?? () from /lib64/ld-linux-x86-64.so.2
#1  0x00007ffff7fe258d in ?? () from /lib64/ld-linux-x86-64.so.2
#2  0x00007ffff7fd0dcc in ?? () from /lib64/ld-linux-x86-64.so.2
#3  0x00007ffff7fecdb2 in ?? () from /lib64/ld-linux-x86-64.so.2
#4  0x00007ffff7fd32d4 in ?? () from /lib64/ld-linux-x86-64.so.2
#5  0x00007ffff7febc3b in ?? () from /lib64/ld-linux-x86-64.so.2
#6  0x00007ffff7fd104c in ?? () from /lib64/ld-linux-x86-64.so.2
#7  0x00007ffff7fd0108 in ?? () from /lib64/ld-linux-x86-64.so.2
#8  0x0000000000000001 in ?? ()
#9  0x00007fffffffe76d in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb)