Jibri prepare recording but fail to start. Time out waiting for call page to load

Hi there,

I guess I am in the same boat with some fellow installers.

Our team has been using jitsi (self hosted, conference only with no jibri) for quite a while and we want to upgrade to latest version for better UI and also recording/streaming with jibri

So we did a new install, Jitsi meet on 1 machine and Jibri on a second.
Both Ubuntu 20 installation with 4vGPU + 8G Ram to be on safe side.

New Jitsi meet install works well (we use CA signed cert), got it up accessible both internally and from external IP, tried 5-10 participants, works like a charm.

However I have struggle to get Jibri to work.
Steps to reproduce:

  1. Start jiti meet
  2. Single participant in the room
  3. press “Start recording” → shows “Prepare to record the meeting”
  4. After a while shows a red warning “Recording failed to start”

Checked jicofo.conf
Checked jibri.conf
Checked conf.js

Checked nginx access log (shows jibri IP connects with chrome)
Checked jibri log (shows connect to mcu at startup, then when start recording has the following:
----- severe selenium.pageobjects.CallPage.log() Timed out waiting for call page to load

Everything seems ok, but just can’t get this to work.

I will paste my conf and log file below, please help see if i have any config issue, and help point out a direction, appreciated.

jibri {
// A unique identifier for this Jibri
// TODO: eventually this will be required with no default
id = “”
// Whether or not Jibri should return to idle state after handling
// (successfully or unsuccessfully) a request. A value of ‘true’
// here means that a Jibri will NOT return back to the IDLE state
// and will need to be restarted in order to be used again.
single-use-mode = false
api {
http {
external-api-port = 2222
internal-api-port = 3333
}
xmpp {
// See example_xmpp_envs.conf for an example of what is expected here
environments = [
{
name = “prod environment”
xmpp-server-hosts = [“mydomain.com”]
xmpp-domain = “mydomain.com

            control-muc {
                domain = "internal.auth.mydomain.com"
                room-name = "JibriBrewery"
                nickname = "jibri-nickname"
            }

            control-login {
                domain = "auth.mydomain.com"
                username = "jibri"
                password = "passwd"
            }

            call-login {
                domain = "recorder.mydomain.com"
                username = "recorder"
                password = "passwd"
            }

            strip-from-room-domain = "conference."
            usage-timeout = 0
            trust-all-xmpp-certs = true
        }]
}

}
recording {
recordings-directory = “/recordings”
# TODO: make this an optional param and remove the default
finalize-script = “/etc/jitsi/jibri/record.sh”
}
streaming {
// A list of regex patterns for allowed RTMP URLs. The RTMP URL used
// when starting a stream must match at least one of the patterns in
// this list.
rtmp-allow-list = [
// By default, all services are allowed
“.*”
]
}
chrome {
// The flags which will be passed to chromium when launching
flags = [
“–use-fake-ui-for-media-stream”,
“–start-maximized”,
“–kiosk”,
“–enabled”,
“–disable-infobars”,
“–autoplay-policy=no-user-gesture-required”
“–no-sandbox”,
“–ignore-certificate-errors”
]
}
stats {
enable-stats-d = true
}
webhook {
// A list of subscribers interested in receiving webhook events
subscribers =
}
jwt-info {
// The path to a .pem file which will be used to sign JWT tokens used in webhook
// requests. If not set, no JWT will be added to webhook requests.
# signing-key-path = “/path/to/key.pem”

// The kid to use as part of the JWT
# kid = "key-id"

// The issuer of the JWT
# issuer = "issuer"

// The audience of the JWT
# audience = "audience"

// The TTL of each generated JWT.  Can't be less than 10 minutes.
# ttl = 1 hour

}
call-status-checks {
// If all clients have their audio and video muted and if Jibri does not
// detect any data stream (audio or video) comming in, it will stop
// recording after NO_MEDIA_TIMEOUT expires.
no-media-timeout = 30 seconds

// If all clients have their audio and video muted, Jibri consideres this
// as an empty call and stops the recording after ALL_MUTED_TIMEOUT expires.
all-muted-timeout = 10 minutes

// When detecting if a call is empty, Jibri takes into consideration for how
// long the call has been empty already. If it has been empty for more than
// DEFAULT_CALL_EMPTY_TIMEOUT, it will consider it empty and stop the recording.
default-call-empty-timeout = 30 seconds

}
}

jicofo/sip-communicator.properties

org.jitsi.jicofo.jibri.BREWERY=JibriBrewery@internal.auth.mydomain.com
org.jitsi.jicofo.jibri.PENDING_TIMEOUT=90

jicofo.conf

Jicofo HOCON configuration. See /usr/share/jicofo/jicofo.jar/reference.conf for

#available options, syntax, and default values.
jicofo {
xmpp: {
client: {
client-proxy: focus.mydomain.com
}
trusted-domains: [ “recorder.mydomain.com” ]
}
bridge: {
brewery-jid: “JvbBrewery@internal.auth.mydomain.com
}
}

This might be a resource issue. Although 4CPU/8GB should be sufficient to run Jibri, vCPUs don’t exactly pack the same punch, so may not be as powerful. Maybe try downscaling Jibri to 720p resolution and then try again? If it works, then that confirms it’s a resource constraint.

Right, this is possible, good call.
Le me try it out, so just add ffmpeg in jibri.conf? Do i need to change xorg as well? I suppose both files need to be edited as seem on some posts.

Will also try increase resource for the VM as well to see if this works too.

Thanks!

Yes, you’d need to make the change in both jibri.conf and xorg-dummy. Don’t forget to restart both services.

Just tried downscaling, still have the same issue.

I added this to jibri.conf
ffmpeg {
resolution = “1280x720”
// The audio source that will be used to capture audio on Linux
audio-source = “alsa”
// The audio device that will be used to capture audio on Linux
audio-device = “plug:bsnoop”
}

and changed on xorg-video-dummy.conf
Virtual 1280 720
#Virtual 1920 1080

Restarted both services, still have the same behavior.

This is jibir log

INFO: [26] org.jitsi.xmpp.mucclient.MucClient.log() Connected.
INFO: [26] org.jitsi.xmpp.mucclient.MucClient.log() Logging in.
INFO: [26] org.jitsi.xmpp.mucclient.MucClient.log() Joined MUC: jibribrewery@internal.auth.mydomain.com
INFO: [41] org.jitsi.jibri.api.xmpp.XmppApi.log() Received JibriIq from environment [MucClient id=mydomain.com hostname=mydomain.com]
INFO: [41] org.jitsi.jibri.api.xmpp.XmppApi.log() Received start request, starting service
INFO: [41] org.jitsi.jibri.api.xmpp.XmppApi.log() Parsed call url info: CallUrlInfo(baseUrl=https://mydomain.com, callName=test2, urlParams=)
INFO: [41] org.jitsi.jibri.JibriManager.log() Starting a file recording with params: FileRecordingRequestParams(callParams=CallParams(callUrlInfo=CallUrlInfo(baseUrl=https://mydomain.com, callName=test2, urlParams=), email=’’, passcode=null, callStatsUsernameOverride=, displayName=), sessionId=tnqicpujfykpkswc, callLoginParams=XmppCredentials(domain=recorder.mydomain.com, port=null, username=recorder, password=*****))
FINE: [41] org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.log() Detected os as OS: LINUX
INFO: [41] org.openqa.selenium.remote.ProtocolHandshake.createSession() Detected dialect: OSS
INFO: [41] org.jitsi.jibri.service.impl.FileRecordingJibriService.log() Writing recording to /recordings/tnqicpujfykpkswc, finalize script path /etc/jitsi/jibri/record.sh
FINE: [41] org.jitsi.jibri.statsd.JibriStatsDClient.log() Incrementing statsd counter: start:recording
INFO: [41] org.jitsi.jibri.status.JibriStatusManager.log() Busy status has changed: IDLE → BUSY
FINE: [41] org.jitsi.jibri.webhooks.v1.WebhookClient.log() Updating 0 subscribers of status
INFO: [41] org.jitsi.jibri.api.xmpp.XmppApi.log() Jibri reports its status is now JibriStatus(busyStatus=BUSY, health=OverallHealth(healthStatus=HEALTHY, details={})), publishing presence to connections
INFO: [41] org.jitsi.jibri.api.xmpp.XmppApi.log() Sending ‘pending’ response to start IQ
INFO: [52] org.jitsi.jibri.selenium.pageobjects.HomePage.log() Visiting url https://mydomain.com
INFO: [52] org.jitsi.jibri.selenium.pageobjects.HomePage.log() Waited 1.09s for driver to load page
INFO: [52] org.jitsi.jibri.selenium.pageobjects.CallPage.log() Visiting url https://mydomain.com/test2#config.iAmRecorder=true&config.externalConnectUrl=null&config.startWithAudioMuted=true&config.startWithVideoMuted=true&interfaceConfig.APP_NAME=“Jibri”&config.analytics.disabled=true&config.p2p.enabled=false&config.prejoinPageEnabled=false&config.requireDisplayName=false
INFO: [52] org.jitsi.jibri.selenium.pageobjects.CallPage.log() Waited 811ms for driver to load page
FINE: [52] org.jitsi.jibri.selenium.pageobjects.CallPage.log() Not joined yet: Cannot read properties of undefined (reading ‘isJoined’)
FINE: [52] org.jitsi.jibri.selenium.pageobjects.CallPage.log() Not joined yet: Cannot read properties of undefined (reading ‘isJoined’)
FINE: [52] org.jitsi.jibri.selenium.pageobjects.CallPage.log() Not joined yet: Cannot read properties of undefined (reading ‘isJoined’)
SEVERE: [52] org.jitsi.jibri.selenium.pageobjects.CallPage.log() Timed out waiting for call page to load
INFO: [52] org.jitsi.jibri.selenium.JibriSelenium.log() Transitioning from state Starting up to Error: FailedToJoinCall SESSION Failed to join the call
INFO: [52] org.jitsi.jibri.service.impl.FileRecordingJibriService.log() File recording service transitioning from state Starting up to Error: FailedToJoinCall SESSION Failed to join the call
INFO: [52] org.jitsi.jibri.api.xmpp.XmppApi.log() Current service had an error Error: FailedToJoinCall SESSION Failed to join the call, sending error iq
FINE: [52] org.jitsi.jibri.statsd.JibriStatsDClient.log() Incrementing statsd counter: stop:recording

Is this error statement accurate or does it suggest some other things?

Thanks

this is logs from nginx, so jitsi is getting post/get from jibri.
using chrome 96, not sure if this is a compatibility issue?

192.168.18.75 - - “POST /http-bind?room=test2 HTTP/1.1” 200 614 “https://mydomain.com/test2” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36”
192.168.18.75 - - “POST /http-bind?room=test2 HTTP/1.1” 200 384 “https://mydomain.com/test2” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36”
192.168.18.75 - - “POST /http-bind?room=test2 HTTP/1.1” 200 315 “https://mydomain.com/test2” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36”
192.168.18.75 - - “GET /pwa-worker.js HTTP/1.1” 200 1511 “https://mydomain.com/pwa-worker.js” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36”

I haven’t tried Chrome 96, so I’m unsure whether it could be an issue. From the logs, it looks like selenium is having problems joining the meeting.
Did you have audio and video muted, by the way?

i followed the guide, but maybe somewhere i did wrong an get the latest edition for chrome.
No, both audio and video are on when i try start recording.

It will say Prepare and log a bunch of those
“CallPage.log() Not joined yet: Cannot read properties of undefined (reading ‘isJoined’)”.
Then timeout and say Recording failed to start.

Does your Chromedriver version match Chrome 96?

i think so, i use the command on the tutorial.

CHROME_DRIVER_VERSION=curl -sS chromedriver.storage.googleapis.com/LATEST_RELEASE

wget -N http://chromedriver.storage.googleapis.com/$CHROME_DRIVER_VERSION/chromedriver_linux64.zip -P ~/

so both should have the same driver version?

To be safe, yes.

Check what version you have:

chromedriver -v

If it’s not 96, you need 96 (since you’ve upgraded Chrome to 96).

You can see the versions here and what versions of Chrome they support:

Sure, just double checked, this is from jibir status
So they are exact match.

Starting ChromeDriver 96.0.4664.45 (76e4c1bb2ab4671b8beba3444e61c0f17584b2fc-refs/branch-heads/4664@{#947}) on port 7152
ChromeDriver was started successfully.

root@recorder:/etc/jitsi/jibri# google-chrome --version
Google Chrome 96.0.4664.45

Ok, good. That rules out that possibility.

Check out the tutorial below and look through the “Recording fails after a few seconds” section. Try out the suggestions there:

Thanks, I have digged through this tutorial actually when trying to figure out what happen. But didn’t notice anything in particular, well at least all service on both jitsi and jibra runs fine, just cannot connect.

So yes there is a high chance the error is in the config. I am new to the jitsi config so its no surprise missing something in the config.

If you notice anything from my jibra.conf, please let me know. I really hope i don’t need to reinstall the setup, though i would like to scale up to a few more jibra for multiple conference recording in near future after it works.

Thanks!

My recommendation would be to just uninstall Jibri and reinstall it. It’s probably faster than trying to figure out the teeny-weeny details that could hav gone wrong. I know there were some fixes applied to Jibri recently, but I don’t know if that’s a factor here. Try uninstalling and re-installing. Sometimes, the 2nd time is a charm. :smiley:

P.S: you can also try installing a much earlier version of Jibri, just in case…

Thanks for the advices Freddie,
Ok, now i got some progress after digging into browser.0.txt in the logs.
Found that i actually have an issue with my recorder user password which i fixed.

But now i encounter this error here, haven’t figure out what it is yet, any help in understanding it would be appreciated.

On jibri log
All are info/fine log entries, except for
ffmpeg.FfmpegCapturer.log() Ffmpeg quit abruptly. Last output line: plug:bsnoop: Input/output error

this is a fine tag as well so not sure what’s wrong here.

INFO: [2663] org.jitsi.jibri.selenium.pageobjects.CallPage.log() Waited 1.00s for driver to load page
FINE: [2663] org.jitsi.jibri.selenium.pageobjects.CallPage.log() Not joined yet: Cannot read properties of undefined (reading ‘isJoined’)
FINE: [2663] org.jitsi.jibri.selenium.pageobjects.CallPage.log() Not joined yet: Cannot read properties of undefined (reading ‘isJoined’)
INFO: [2663] org.jitsi.jibri.selenium.pageobjects.CallPage.log() Waited 1.14s to join the conference
INFO: [2663] org.jitsi.jibri.selenium.status_checks.EmptyCallStatusCheck.log() Starting empty call check with a timeout of PT30S
INFO: [2663] org.jitsi.jibri.selenium.JibriSelenium.log() Transitioning from state Starting up to Running
INFO: [2663] org.jitsi.jibri.service.impl.FileRecordingJibriService.log() Selenium joined the call, starting the capturer
INFO: [2663] org.jitsi.jibri.util.JibriSubprocess.log() Starting ffmpeg with command ffmpeg -y -v info -f x11grab -draw_mouse 0 -r 30 -s 1920x1080 -thread_queue_size 4096 -i :0.0+0,0 -f alsa -thread_queue_size 4096 -i plug:bsnoop -acodec aac -strict -2 -ar 44100 -b:a 128k -af aresample=async=1 -c:v libx264 -preset veryfast -profile:v main -level 3.1 -pix_fmt yuv420p -r 30 -crf 25 -g 60 -tune zerolatency -f mp4 /recordings/abtcymmrskrhgvzb/test2_2021-11-21-04-04-01.mp4 ([ffmpeg, -y, -v, info, -f, x11grab, -draw_mouse, 0, -r, 30, -s, 1920x1080, -thread_queue_size, 4096, -i, :0.0+0,0, -f, alsa, -thread_queue_size, 4096, -i, plug:bsnoop, -acodec, aac, -strict, -2, -ar, 44100, -b:a, 128k, -af, aresample=async=1, -c:v, libx264, -preset, veryfast, -profile:v, main, -level, 3.1, -pix_fmt, yuv420p, -r, 30, -crf, 25, -g, 60, -tune, zerolatency, -f, mp4, /recordings/abtcymmrskrhgvzb/test2_2021-11-21-04-04-01.mp4])
FINE: [17] org.jitsi.jibri.webhooks.v1.WebhookClient.log() Updating 0 subscribers of status
FINE: [17] org.jitsi.jibri.util.ProcessStatePublisher.log() Process ffmpeg hasn’t written in 2 seconds, publishing periodic update
INFO: [2667] org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.log() Ffmpeg quit abruptly. Last output line: plug:bsnoop: Input/output error
INFO: [2667] org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.log() Ffmpeg capturer transitioning from state Starting up to Error: QuitUnexpectedly SESSION plug:bsnoop: Input/output error
INFO: [2667] org.jitsi.jibri.service.impl.FileRecordingJibriService.log() File recording service transitioning from state Starting up to Error: QuitUnexpectedly SESSION plug:bsnoop: Input/output error
INFO: [2667] org.jitsi.jibri.api.xmpp.XmppApi.log() Current service had an error Error: QuitUnexpectedly SESSION plug:bsnoop: Input/output error, sending error iq
FINE: [2667] org.jitsi.jibri.statsd.JibriStatsDClient.log() Incrementing statsd counter: stop:recording
INFO: [2667] org.jitsi.jibri.JibriManager.log() Stopping the current service
INFO: [2667] org.jitsi.jibri.service.impl.FileRecordingJibriService.log() Stopping capturer
INFO: [2667] org.jitsi.jibri.util.JibriSubprocess.log() Stopping ffmpeg process
INFO: [2667] org.jitsi.jibri.util.JibriSubprocess.log() ffmpeg exited with value 1
INFO: [2667] org.jitsi.jibri.service.impl.FileRecordingJibriService.log() Quitting selenium
INFO: [2667] org.jitsi.jibri.service.impl.FileRecordingJibriService.log() No media was recorded, deleting directory and skipping metadata file & finalize
INFO: [2667] org.jitsi.jibri.selenium.JibriSelenium.log() Leaving call and quitting browser
INFO: [2667] org.jitsi.jibri.selenium.JibriSelenium.log() Recurring call status checks cancelled

just figured out the issue!

Its issue with ALSA and Loopback Device.

Somehow after reboot the loopback isn’t running.
did the following and it worked, tested jibra with recording and got the file recorded!

root@recorder:/recordings# modprobe snd-aloop
root@recorder:/recordings# lsmod | grep snd_aloop
snd_aloop 24576 0
snd_pcm 106496 1 snd_aloop
snd 90112 3 snd_timer,snd_aloop,snd_pcm

Is there a way to start snd-aloop at boot/restart without having to do it manually?
Try to dig through the forum but haven’t found a way to do it. And i thought i should be
running by default. Any ideas?

echo options snd-aloop enable=1,1,1,1 index=0,1,2,3 >/etc/modprobe.d/alsa-loopback.conf