After 10minutes recording was stopped

ravichandra · May 2, 2023, 2:17pm

Before all recording servers are working fine. Now recordings stopped in 10minutes. Now and before we are
using containerized jibri.

Jibri 2023-05-02 14:11:38.074 INFO: [70] JibriServiceFinalizeCommandRunner.doFinalize#44: Finalizing the jibri service operation using command [/config/finalize.sh, /config/recordings/mptoxqmpvfigwobz]
Jibri 2023-05-02 14:11:38.075 SEVERE: [70] JibriServiceFinalizeCommandRunner.doFinalize#63: Failed to run finalize script
java.io.IOException: Cannot run program “/config/finalize.sh”: error=2, No such file or directory
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
at org.jitsi.jibri.util.ProcessWrapper.start(ProcessWrapper.kt:88)
at org.jitsi.jibri.service.impl.JibriServiceFinalizeCommandRunner.doFinalize(JibriServiceFinalizeCommandRunner.kt:47)
at org.jitsi.jibri.service.impl.FileRecordingJibriService.stop(FileRecordingJibriService.kt:233)
at org.jitsi.jibri.JibriManager.stopService(JibriManager.kt:263)
at org.jitsi.jibri.JibriManager$startService$1.invoke(JibriManager.kt:218)
at org.jitsi.jibri.JibriManager$startService$1.invoke(JibriManager.kt:204)
at org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:37)
at org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:36)
at org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:53)
at org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:53)
at kotlin.collections.CollectionsKt__MutableCollectionsKt.filterInPlace$CollectionsKt__MutableCollectionsKt(MutableCollections.kt:285)
at kotlin.collections.CollectionsKt__MutableCollectionsKt.retainAll(MutableCollections.kt:276)
at org.jitsi.jibri.util.StatusPublisher.publishStatus(StatusPublisher.kt:53)
at org.jitsi.jibri.service.impl.StatefulJibriService.onServiceStateChange(StatefulJibriService.kt:40)
at org.jitsi.jibri.service.impl.StatefulJibriService.access$onServiceStateChange(StatefulJibriService.kt:26)
at org.jitsi.jibri.service.impl.StatefulJibriService$1.invoke(StatefulJibriService.kt:35)
at org.jitsi.jibri.service.impl.StatefulJibriService$1.invoke(StatefulJibriService.kt:35)
at org.jitsi.jibri.util.NotifyingStateMachine.notify(NotifyingStateMachine.kt:26)
at org.jitsi.jibri.service.JibriServiceStateMachine.access$notify(JibriServiceStateMachine.kt:46)
at org.jitsi.jibri.service.JibriServiceStateMachine$stateMachine$1$5.invoke(JibriServiceStateMachine.kt:100)
at org.jitsi.jibri.service.JibriServiceStateMachine$stateMachine$1$5.invoke(JibriServiceStateMachine.kt:95)
at com.tinder.StateMachine.notifyOnTransition(StateMachine.kt:65)
at com.tinder.StateMachine.transition(StateMachine.kt:23)
at org.jitsi.jibri.service.JibriServiceStateMachine.transition(JibriServiceStateMachine.kt:112)
at org.jitsi.jibri.service.impl.StatefulJibriService$registerSubComponent$1.invoke(StatefulJibriService.kt:46)
at org.jitsi.jibri.service.impl.StatefulJibriService$registerSubComponent$1.invoke(StatefulJibriService.kt:45)
at org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:37)
at org.jitsi.jibri.util.StatusPublisher$addStatusHandler$1.invoke(StatusPublisher.kt:36)
at org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:53)
at org.jitsi.jibri.util.StatusPublisher$publishStatus$1.invoke(StatusPublisher.kt:53)
at kotlin.collections.CollectionsKt__MutableCollectionsKt.filterInPlace$CollectionsKt__MutableCollectionsKt(MutableCollections.kt:285)
at kotlin.collections.CollectionsKt__MutableCollectionsKt.retainAll(MutableCollections.kt:276)
at org.jitsi.jibri.util.StatusPublisher.publishStatus(StatusPublisher.kt:53)
at org.jitsi.jibri.selenium.JibriSelenium.onSeleniumStateChange(JibriSelenium.kt:217)
at org.jitsi.jibri.selenium.JibriSelenium.access$onSeleniumStateChange(JibriSelenium.kt:167)
at org.jitsi.jibri.selenium.JibriSelenium$1.invoke(JibriSelenium.kt:202)
at org.jitsi.jibri.selenium.JibriSelenium$1.invoke(JibriSelenium.kt:202)
at org.jitsi.jibri.util.NotifyingStateMachine.notify(NotifyingStateMachine.kt:26)
at org.jitsi.jibri.selenium.SeleniumStateMachine.access$notify(SeleniumStateMachine.kt:35)
at org.jitsi.jibri.selenium.SeleniumStateMachine$stateMachine$1$5.invoke(SeleniumStateMachine.kt:86)
at org.jitsi.jibri.selenium.SeleniumStateMachine$stateMachine$1$5.invoke(SeleniumStateMachine.kt:81)
at com.tinder.StateMachine.notifyOnTransition(StateMachine.kt:65)
at com.tinder.StateMachine.transition(StateMachine.kt:23)
at org.jitsi.jibri.selenium.SeleniumStateMachine.transition(SeleniumStateMachine.kt:91)
at org.jitsi.jibri.selenium.JibriSelenium$startRecurringCallStatusChecks$transitionState$1.invoke$lambda-0(JibriSelenium.kt:238)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: error=2, No such file or directory
at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
at java.base/java.lang.ProcessImpl.(ProcessImpl.java:340)
at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:271)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
… 51 more
Jibri 2023-05-02 14:11:38.076 FINE: [70] MainKt$setupMetaconfigLogger$1.debug#232: FallbackSupplier: checking for value via suppliers:
LambdaSupplier: ‘JibriConfig::singleUseMode’
ConfigSourceSupplier: key: ‘jibri.single-use-mode’, type: ‘kotlin.Boolean’, source: ‘config’
Jibri 2023-05-02 14:11:38.076 FINE: [70] MainKt$setupMetaconfigLogger$1.debug#232: LambdaSupplier: Trying to retrieve value via JibriConfig::singleUseMode
Jibri 2023-05-02 14:11:38.077 FINE: [70] MainKt$setupMetaconfigLogger$1.debug#232: FallbackSupplier: failed to find value via LambdaSupplier: ‘JibriConfig::singleUseMode’: org.jitsi.metaconfig.ConfigException$UnableToRetrieve$Error: class java.lang.NullPointerException
Jibri 2023-05-02 14:11:38.077 FINE: [70] MainKt$setupMetaconfigLogger$1.debug#232: ConfigSourceSupplier: Trying to retrieve key ‘jibri.single-use-mode’ from source ‘config’ as type kotlin.Boolean
Jibri 2023-05-02 14:11:38.078 FINE: [70] MainKt$setupMetaconfigLogger$1.debug#232: ConfigSourceSupplier: Successfully retrieved key ‘jibri.single-use-mode’ from source ‘config’ as type kotlin.Boolean
Jibri 2023-05-02 14:11:38.078 FINE: [70] MainKt$setupMetaconfigLogger$1.debug#232: FallbackSupplier: value found via ConfigSourceSupplier: key: ‘jibri.single-use-mode’, type: ‘kotlin.Boolean’, source: ‘config’
Jibri 2023-05-02 14:11:38.078 INFO: [70] JibriStatusManager$special$$inlined$observable$1.afterChange#72: Busy status has changed: BUSY → IDLE
Jibri 2023-05-02 14:11:38.078 FINE: [70] WebhookClient$updateStatus$1.invokeSuspend#109: Updating 0 subscribers of status
Jibri 2023-05-02 14:11:38.079 INFO: [70] XmppApi.updatePresence#209: Jibri reports its status is now JibriStatus(busyStatus=IDLE, health=OverallHealth(healthStatus=HEALTHY, details={})), publishing presence to connections
Jibri 2023-05-02 14:11:38.079 FINE: [70] MucClientManager.setPresenceExtension#160: Setting a presence extension: org.jitsi.xmpp.extensions.jibri.JibriStatusPacketExt@7070f873
Jibri 2023-05-02 14:11:38.079 FINE: [70] MucClientManager.saveExtension#185: Replacing presence extension: org.jitsi.xmpp.extensions.jibri.JibriStatusPacketExt@1227f42a
Jibri 2023-05-02 14:11:38.097 FINE: [46] org.jitsi.xmpp.extensions.DefaultPacketExtensionProvider.parse: Could not add a provider for element busy-status from namespace http://jitsi.org/protocol/jibri
Jibri 2023-05-02 14:11:38.097 FINE: [46] org.jitsi.xmpp.extensions.DefaultPacketExtensionProvider.parse: Could not add a provider for element health-status from namespace http://jitsi.org/protocol/health

Please give me solution.

damencho · May 2, 2023, 2:42pm

What do the logs say before the finalize error?
The finalize error is that you need to configure it correctly:

github.com

jitsi/jibri/blob/1a4bb8e0e83082440a57664382393f74c4e80014/src/main/resources/reference.conf#L23


      
              internal-api-port = 3333
            }
            xmpp {
              // See example_xmpp_envs.conf for an example of what is expected here
              environments = []
            }
          }
          recording {
            recordings-directory = "/tmp/recordings"
            # TODO: make this an optional param and remove the default
            finalize-script = "/path/to/finalize"
          }
          streaming {
            // A list of regex patterns for allowed RTMP URLs.  The RTMP URL used
            // when starting a stream must match at least one of the patterns in
            // this list.
            rtmp-allow-list = [
              // By default, all services are allowed
              ".*"
            ]
          }

Freddie · May 2, 2023, 3:11pm

What are the specs (CPU, RAM) of the server?
Also, share your ffmpeg log.

ravichandra · May 2, 2023, 6:18pm

@Freddie 16GB RAM and 8 core CPU, but before it was working fine. we are using dockerized jibri using docker composefile. but finelized.sh file will not find inside the docker container. find the logs of ffmpeg logs.

==> ffmpeg.0.txt <==
Jibri 2023-05-02 17:37:23.279 INFO: [476] LoggingUtils$Companion$OutputLogger$1$1.call#42: [libx264 @ 0x5602b2071880] mb I I16…4: 95.2% 0.0% 4.8%
Jibri 2023-05-02 17:37:23.279 INFO: [476] LoggingUtils$Companion$OutputLogger$1$1.call#42: [libx264 @ 0x5602b2071880] mb P I16…4: 0.0% 0.0% 0.0% P16…4: 0.0% 0.0% 0.0% 0.0% 0.0% skip:100.0%
Jibri 2023-05-02 17:37:23.279 INFO: [476] LoggingUtils$Companion$OutputLogger$1$1.call#42: [libx264 @ 0x5602b2071880] coded y,uvDC,uvAC intra: 3.1% 4.3% 3.9% inter: 0.0% 0.0% 0.0%
Jibri 2023-05-02 17:37:23.279 INFO: [476] LoggingUtils$Companion$OutputLogger$1$1.call#42: [libx264 @ 0x5602b2071880] i16 v,h,dc,p: 81% 1% 17% 0%
Jibri 2023-05-02 17:37:23.279 INFO: [476] LoggingUtils$Companion$OutputLogger$1$1.call#42: [libx264 @ 0x5602b2071880] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 25% 19% 37% 3% 3% 4% 3% 3% 3%
Jibri 2023-05-02 17:37:23.279 INFO: [476] LoggingUtils$Companion$OutputLogger$1$1.call#42: [libx264 @ 0x5602b2071880] i8c dc,h,v,p: 97% 1% 1% 0%
Jibri 2023-05-02 17:37:23.279 INFO: [476] LoggingUtils$Companion$OutputLogger$1$1.call#42: [libx264 @ 0x5602b2071880] Weighted P-Frames: Y:0.0% UV:0.0%
Jibri 2023-05-02 17:37:23.279 INFO: [476] LoggingUtils$Companion$OutputLogger$1$1.call#42: [libx264 @ 0x5602b2071880] kb/s:71.94
Jibri 2023-05-02 17:37:23.287 INFO: [476] LoggingUtils$Companion$OutputLogger$1$1.call#42: [aac @ 0x5602b2072e00] Qavg: 65524.152
Jibri 2023-05-02 17:37:23.287 INFO: [476] LoggingUtils$Companion$OutputLogger$1$1.call#42: Exiting normally, received signal 2.

damencho · May 2, 2023, 7:30pm

You can check in the logs of jibri, I suspect you are hitting some of these timers:

github.com

jitsi/jibri/blob/1a4bb8e0e83082440a57664382393f74c4e80014/src/main/resources/reference.conf#L102


      
              # ttl = 1 hour
            }
            call-status-checks {
              // If not all clients have their audio and video muted, but Jibri does not
              // detect any data stream (audio or video) comming in, it will stop
              // recording after NO_MEDIA_TIMEOUT expires.
              no-media-timeout = 3 minutes
          
          
    // If all clients have their audio and video muted, Jibri consideres this
              // as an empty call and stops the recording after ALL_MUTED_TIMEOUT expires.
              all-muted-timeout = 10 minutes
          
          
    // When detecting if a call is empty, Jibri takes into consideration for how
              // long the call has been empty already. If it has been empty for more than
              // DEFAULT_CALL_EMPTY_TIMEOUT, it will consider it empty and stop the recording.
              default-call-empty-timeout = 30 seconds
          
          
    // If ICE hasn't completed, or stays in a state other than "connected" for this amount of time, Jibri will stop.
              ice-connection-timeout = 30 seconds
            }
          }

ravichandra · May 3, 2023, 9:47am

@damencho we are using jibri docker compose file. where i need to add time settings in the give docker compose file. below is my docker composefile.

version: ‘3’

services:
jibri1:
image: jitsi/jibri:stable
restart: ${RESTART_POLICY}
volumes:
- ${CONFIG}/jibri1:/config:Z
- /dev/shm:/dev/shm
- /root/jibri-record/config/.asoundrc1:/home/jibri/.asoundrc
- /root/jibri-record/recordings:/config/recordings
cap_add:
- SYS_ADMIN
- NET_BIND_SERVICE
devices:
- /dev/snd:/dev/snd
environment:
- XMPP_AUTH_DOMAIN
- XMPP_INTERNAL_MUC_DOMAIN
- XMPP_RECORDER_DOMAIN
- XMPP_SERVER
- XMPP_DOMAIN
- JIBRI_XMPP_USER
- JIBRI_XMPP_PASSWORD
- JIBRI_BREWERY_MUC
- JIBRI_RECORDER_USER
- JIBRI_RECORDER_PASSWORD
- JIBRI_RECORDING_DIR
- JIBRI_FINALIZE_RECORDING_SCRIPT_PATH
- JIBRI_STRIP_DOMAIN_JID
- JIBRI_LOGS_DIR
- DISPLAY=:0
- TZ

damencho · May 3, 2023, 11:40am

I don’t think that is currently possible with docker.
Have you verified woth the jibri logs what is going on?

Freddie · May 3, 2023, 1:09pm

Is that all the log you got in your 10min recording?