[SOLVED] Jibri with JWT token enabled Jitsi

#1

Hi,

I’m trying to setup recording on a Jitsi deployment with jwt-authentication. I am able to send request to Jibri which publishes the status as BUSY, but times out while trying to load the selenium page. From what I logically understand, the Jibri tries to join the conference, but as it is jwt-based, it cannot connect and times out. Please correct me if I’m wrong here.

What could be the solution here @damencho @bbaldino ?

Jibri logs:

2019-01-30 16:41:22.810 WARNING: [1] org.glassfish.jersey.internal.inject.Providers.checkProviderRuntime() A provider org.jitsi.jibri.api.http.HttpApi registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.jitsi.jibri.api.http.HttpApi will be ignored. 
2019-01-30 16:41:40.395 INFO: [27] org.jitsi.jibri.api.xmpp.XmppApi.handleJibriIq() Received JibriIq <iq to='jibri@auth.10.107.14.1/0bc770d8-01b1-4a39-af5b-c942a46b3925' from='jibribrewery@internal.auth.10.107.14.1/focus' id='amlicmlAYXV0aC4xMC4xMDcuMTQuMS8wYmM3NzBkOC0wMWIxLTRhMzktYWY1Yi1jOTQyYTQ2YjM5MjUAZGlXR0gtNzE5ACv3xyK5WZmKx9H+BXfirSA=' type='set'><jibri xmlns='http://jitsi.org/protocol/jibri' action='start' recording_mode='file' room='test@conference.10.107.14.1' session_id='wzgzpxyxkhaglvlz'/></iq> from environment prod environment
2019-01-30 16:41:40.397 INFO: [27] org.jitsi.jibri.api.xmpp.XmppApi.handleStartJibriIq() Received start request
2019-01-30 16:41:40.401 INFO: [27] org.jitsi.jibri.api.xmpp.XmppApi.handleStartJibriIq() Sending 'pending' response to start IQ
2019-01-30 16:41:40.401 INFO: [40] org.jitsi.jibri.api.xmpp.XmppApi.run() Starting service
2019-01-30 16:41:40.408 INFO: [40] org.jitsi.jibri.api.xmpp.XmppApi.handleStartService() Parsed call url info: CallUrlInfo(baseUrl=https://10.107.14.1, callName=test, urlParams=[])
2019-01-30 16:41:40.410 INFO: [40] org.jitsi.jibri.JibriManager.startFileRecording() Starting a file recording with params: FileRecordingRequestParams(callParams=CallParams(callUrlInfo=CallUrlInfo(baseUrl=https://10.107.14.1, callName=test, urlParams=[])), sessionId=wzgzpxyxkhaglvlz, callLoginParams=XmppCredentials(domain=recorder.10.107.14.1, username=recorder, password=jibri)) finalize script path: /path/to/finalize_recording.sh and recordings directory: /srv/recordings
2019-01-30 16:41:41.360 INFO: [40] org.openqa.selenium.remote.ProtocolHandshake.createSession() Detected dialect: OSS
2019-01-30 16:41:41.381 FINE: [40] org.jitsi.jibri.capture.ffmpeg.FfmpegCapturer.<init>() Detected os as OS: LINUX
2019-01-30 16:41:41.387 INFO: [40] org.jitsi.jibri.service.impl.FileRecordingJibriService.<init>() Writing recording to /srv/recordings/wzgzpxyxkhaglvlz
2019-01-30 16:41:41.403 FINE: [40] org.jitsi.jibri.statsd.JibriStatsDClient.incrementCounter() Incrementing statsd counter: start:recording
2019-01-30 16:41:41.423 INFO: [40] org.jitsi.jibri.status.JibriStatusManager.log() Busy status has changed: IDLE -> BUSY
2019-01-30 16:41:41.424 INFO: [40] org.jitsi.jibri.api.xmpp.XmppApi.invoke() Jibri reports its status is now JibriStatus(busyStatus=BUSY, health=OverallHealth(healthStatus=HEALTHY, details={})), publishing presence to connection prod environment
2019-01-30 16:41:44.585 FINE: [47] org.jitsi.jibri.selenium.pageobjects.CallPage.visit() Visiting url https://10.107.14.1/test#config.iAmRecorder=true&config.externalConnectUrl=null&config.startWithAudioMuted=true&config.startWithVideoMuted=true&interfaceConfig.APP_NAME="Jibri"
2019-01-30 16:42:16.648 SEVERE: [47] org.jitsi.jibri.selenium.pageobjects.CallPage.visit() Timed out waiting for call page to load
2019-01-30 16:42:16.650 INFO: [47] org.jitsi.jibri.selenium.JibriSelenium.onSeleniumStateChange() Transitioning from state Starting up to Error: SESSION Failed to join call
2019-01-30 16:42:16.652 INFO: [47] org.jitsi.jibri.service.impl.FileRecordingJibriService.onServiceStateChange() File recording service transitioning from state Starting up to Error: SESSION Failed to join call
2019-01-30 16:42:16.652 INFO: [47] org.jitsi.jibri.api.xmpp.XmppApi.invoke() Current service had an error, sending error iq <iq to='jibribrewery@internal.auth.10.107.14.1/focus' id='s5Dz4-35' type='set'><jibri xmlns='http://jitsi.org/protocol/jibri' status='off' failure_reason='error'/></iq>
2019-01-30 16:42:16.653 FINE: [47] org.jitsi.jibri.statsd.JibriStatsDClient.incrementCounter() Incrementing statsd counter: stop:recording
2019-01-30 16:42:16.654 INFO: [47] org.jitsi.jibri.JibriManager.stopService() Stopping the current service
2019-01-30 16:42:16.654 INFO: [47] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Stopping capturer
2019-01-30 16:42:16.655 INFO: [47] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() Stopping ffmpeg process
2019-01-30 16:42:16.655 INFO: [47] org.jitsi.jibri.util.JibriSubprocess.ffmpeg.stop() ffmpeg exited with value null
2019-01-30 16:42:16.656 INFO: [47] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Quitting selenium
2019-01-30 16:42:16.667 INFO: [47] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Participants in this recording: []
2019-01-30 16:42:16.761 INFO: [47] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 12 log entries for type browser
2019-01-30 16:42:16.977 INFO: [47] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 1105 log entries for type driver
2019-01-30 16:42:17.147 INFO: [47] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Got 0 log entries for type client
2019-01-30 16:42:17.148 INFO: [47] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Leaving web call
2019-01-30 16:42:17.154 INFO: [47] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Quitting chrome driver
2019-01-30 16:42:17.222 INFO: [47] org.jitsi.jibri.selenium.JibriSelenium.leaveCallAndQuitBrowser() Chrome driver quit
2019-01-30 16:42:17.223 INFO: [47] org.jitsi.jibri.service.impl.FileRecordingJibriService.stop() Finalizing the recording
2019-01-30 16:42:17.231 SEVERE: [47] org.jitsi.jibri.service.impl.FileRecordingJibriService.finalize() Failed to run finalize script: java.io.IOException: Cannot run program "/path/to/finalize_recording.sh": error=2, No such file or directory with stack: 
java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
org.jitsi.jibri.util.ProcessWrapper.start(ProcessWrapper.kt:79)
org.jitsi.jibri.service.impl.FileRecordingJibriService.finalize(FileRecordingJibriService.kt:203)
org.jitsi.jibri.service.impl.FileRecordingJibriService.stop(FileRecordingJibriService.kt:188)
org.jitsi.jibri.JibriManager.stopService(JibriManager.kt:237)
org.jitsi.jibri.JibriManager$startService$1.invoke(JibriManager.kt:205)
org.jitsi.jibri.JibriManager$startService$1.invoke(JibriManager.kt:84)
org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:37)
org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:29)
org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:53)
org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:29)
kotlin.collections.CollectionsKt__MutableCollectionsKt.filterInPlace$CollectionsKt__MutableCollectionsKt(MutableCollections.kt:196)
kotlin.collections.CollectionsKt__MutableCollectionsKt.retainAll(MutableCollections.kt:187)
org.jitsi.jibri.util.StatusPublisher.publishStatus(StatusPublisher.kt:53)
org.jitsi.jibri.service.impl.StatefulJibriService.onServiceStateChange(StatefulJibriService.kt:40)
org.jitsi.jibri.service.impl.StatefulJibriService.access$onServiceStateChange(StatefulJibriService.kt:26)
org.jitsi.jibri.service.impl.StatefulJibriService$1.invoke(StatefulJibriService.kt:35)
org.jitsi.jibri.service.impl.StatefulJibriService$1.invoke(StatefulJibriService.kt:26)
org.jitsi.jibri.util.NotifyingStateMachine.notify(NotifyingStateMachine.kt:26)
org.jitsi.jibri.service.JibriServiceStateMachine.access$notify(JibriServiceStateMachine.kt:46)
org.jitsi.jibri.service.JibriServiceStateMachine$stateMachine$1$5.invoke(JibriServiceStateMachine.kt:100)
org.jitsi.jibri.service.JibriServiceStateMachine$stateMachine$1$5.invoke(JibriServiceStateMachine.kt:46)
com.tinder.StateMachine.notifyOnTransition(StateMachine.kt:66)
com.tinder.StateMachine.transition(StateMachine.kt:23)
org.jitsi.jibri.service.JibriServiceStateMachine.transition(JibriServiceStateMachine.kt:112)
org.jitsi.jibri.service.impl.StatefulJibriService$registerSubComponent$1.invoke(StatefulJibriService.kt:46)
org.jitsi.jibri.service.impl.StatefulJibriService$registerSubComponent$1.invoke(StatefulJibriService.kt:26)
org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:37)
org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:29)
org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:53)
org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:29)
kotlin.collections.CollectionsKt__MutableCollectionsKt.filterInPlace$CollectionsKt__MutableCollectionsKt(MutableCollections.kt:196)
kotlin.collections.CollectionsKt__MutableCollectionsKt.retainAll(MutableCollections.kt:187)
org.jitsi.jibri.util.StatusPublisher.publishStatus(StatusPublisher.kt:53)
org.jitsi.jibri.selenium.JibriSelenium.onSeleniumStateChange(JibriSelenium.kt:167)
org.jitsi.jibri.selenium.JibriSelenium.access$onSeleniumStateChange(JibriSelenium.kt:100)
org.jitsi.jibri.selenium.JibriSelenium$1.invoke(JibriSelenium.kt:152)
org.jitsi.jibri.selenium.JibriSelenium$1.invoke(JibriSelenium.kt:100)
org.jitsi.jibri.util.NotifyingStateMachine.notify(NotifyingStateMachine.kt:26)
org.jitsi.jibri.selenium.SeleniumStateMachine.access$notify(SeleniumStateMachine.kt:34)
org.jitsi.jibri.selenium.SeleniumStateMachine$stateMachine$1$5.invoke(SeleniumStateMachine.kt:79)
org.jitsi.jibri.selenium.SeleniumStateMachine$stateMachine$1$5.invoke(SeleniumStateMachine.kt:34)
com.tinder.StateMachine.notifyOnTransition(StateMachine.kt:66)
com.tinder.StateMachine.transition(StateMachine.kt:23)
org.jitsi.jibri.selenium.SeleniumStateMachine.transition(SeleniumStateMachine.kt:84)
org.jitsi.jibri.selenium.JibriSelenium$joinCall$1.run(JibriSelenium.kt:243)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

2019-01-30 16:42:17.239 INFO: [47] org.jitsi.jibri.status.JibriStatusManager.log() Busy status has changed: BUSY -> IDLE
2019-01-30 16:42:17.240 INFO: [47] org.jitsi.jibri.api.xmpp.XmppApi.invoke() Jibri reports its status is now JibriStatus(busyStatus=IDLE, health=OverallHealth(healthStatus=HEALTHY, details={})), publishing presence to connection prod environment

Console Error:

20%20PM
Screen Shot 2019-01-30 at 6.35.20 PM.png1112×1396 343 KB

Abhijit

#2

Can anybody help me out here? This looks like a strange case @damencho @bbaldino
@Aaron_K_van_Meerten

Abhijit

#3

Have you checked prosody logs for errors?

#4

Hi @damencho

Yes, i checked the logs. The prosody logs does not show any error. The jibri MUC connection is successful

|Jan 31 15:22:02 c2sff09e0|info|Client disconnected: closed|
|---|---|---|
|Jan 31 15:22:04 c2s140b460|info|Client connected|
|Jan 31 15:22:04 c2s140b460|info|Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)|
|Jan 31 15:22:04 c2s140b460|info|Authenticated as jibri@auth.10.107.14.1|
|Jan 31 15:22:47 mod_bosh|info|New BOSH session, assigned it sid '00450df4-2c0f-43fd-82eb-059efffbd186'|

The selenium is failing to join the page. I don’t see any other error anywhere.

Abhijit

#5

Hi @bbaldino @damencho,

I did some digging in Jibri code base to identify the cause. I cloned the code, added a few logs and built and deployed in my local environment. The var “result” https://github.com/jitsi/jibri/blob/0e96134dcc5370d4668533e160d573f13436f591/src/main/kotlin/org/jitsi/jibri/selenium/pageobjects/CallPage.kt#L48
shows this error when printed when connected with Jitsi with JWT againt normal Jitsi.

Jibri when connected to Jitsi with JWT:

2019-01-31 17:56:19.769 INFO: [44] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Result: Cannot read property 'isJoined' of undefined

The above msg prints for 30 sec and then times out.

Jibri when connected to normal Jitsi

2019-01-31 18:04:58.012 INFO: [45] org.jitsi.jibri.selenium.pageobjects.CallPage.apply() Result: true

Abhijit

#6

Are you requiring JWT tokens for all users? It may be that you’ll also need to continue to configure prosody with the proper domain/user/password for the jibri user, since jibri uses jitsi-meet to provide a username/password combination to authenticate, rather than JWT.

#7

Hi,

The prosody configs look okay. The components are getting discovered on Jicofo. It also passes the recording request to Jibri. So I’m not sure where to do the changes. Cam you please give example?

Abhijit

Regards,
Abhijit Nathwani

#8

There’s two connections Jibri makes. the first is the control connection to the brewery MUC, which as you say is working as expected. However, once jibri receives a request to record, it launches a selenium chrome session and joins a specific call in an authenticated fashion. It does this using a feature of jitsi meet which allows username and password to be provided in local settings. However, this requires the jibri recorder user to exist on the prosody, and for that prosody domain to allow plain text login.

#9

Hi @Aaron_K_van_Meerten,

The prosody has the recorder component with authentication = “internal_plain”.

- internal muc component, meant to enable pools of jibri and jigasi clients
Component "internal.auth.10.107.14.1" "muc"
    modules_enabled = {
      "ping";
    }
    storage = "null"
    muc_room_cache_size = 1000

VirtualHost "recorder.10.107.14.1"
  modules_enabled = {
    "ping";
  }
  authentication = "internal_plain"

Anything else do you suggest to add here? The recorder user has also been registered on prosody using the prosodyctl.

Abhijit

#10

Has this scenario been tested? Jibri to Jitsi with JWT ? @damencho @bbaldino @Aaron_K_van_Meerten

Abhijit

#11

Yes it is, but with config allow_empty_token = true. Actually this how meet.jit.si is working today.

#12

So where do I add this

?

Abhijit

#13

Hi @damencho,

I modified the prosody conf as below, but still no good.

VirtualHost "domain"
        -- enabled = false -- Remove this line to enable this host
        authentication = "token"
        -- Properties below are modified by jitsi-meet-tokens package config
        -- and authentication above is switched to "token"
        app_id="appid"
        app_secret="appsecret"
        -- Assign this host a certificate for TLS, otherwise it would use the one
        -- set in the global section (if any).
        -- Note that old-style SSL on port 5223 only supports one certificate, and will always
        -- use the global one.
        ssl = {
                key = "/etc/prosody/certs/domain.key";
                certificate = "/etc/prosody/certs/domain.crt";
        }
        -- we need bosh
        modules_enabled = {
            "bosh";
            "pubsub";
            "ping"; -- Enable mod_ping
        }

        c2s_require_encryption = false
        allow_empty_token = true

The selenium call page still fails to load.

Abhijit

#14

Copy the page URL and try from a browser, what is the error you see is that 504 error from screenshot above?

#15

Hi @damencho,

The error 504 is resolved somehow. I guess that was some firewall issue. Currently, I don’t get any log on console when I start recording.

Abhijit

#16

If you open this url from your browser, what do you get?

#17

Hi @damencho,

A blank gray page. Does not join the conference.

Abhijit

#18

And what is the console log error?

#19

Hi @damencho

06%20PM
Screen Shot 2019-02-01 at 6.07.06 PM.png2410×1558 606 KB

after setting allow_empty_token = true

15%20PM
Screen Shot 2019-02-01 at 6.14.15 PM.png2442×1540 698 KB

I can join from browser, but still the error persists while clicking “Start recording” from the room.

Abhijit

#20

The first screenshot looks normal, as jibri set its login values in the localstorage and you do not have it in your browser when testing. So your jibri still cannot load the page?
No idea what is this error about.