Jibri Failed if Running 2 Server in Same Time

Install & Config Record & stream - Jibri
1

Continuing the discussion from Jibri Failed to send start Jibri IQ: org.jitsi.jicofo.recording.jibri.JibriSession$StartException: Unexpected response:

We are already following this tutorial but still getting errors when we have jibri running on 2 different servers simultaneously.

Nickname is different, but the most important is, if we only running in single server , and click recording, all be fine.

But if we run in 2 server jibri, 1 server will fail, and 1 server Is recording in the same room id.
so if I have 3 rooms, we have 3 jibri running, and 3 jibri failed.
If I get stats of jibri, it return BUSY rather than IDLE, even if jibri is crash or something

JIBRI_URL="http://127.0.0.1:2222"
curl --silent "$JIBRI_URL/jibri/api/v1.0/health"| jq -j '.status'
{
  "busyStatus": "BUSY",
  "health": {
    "healthStatus": "HEALTHY",
    "details": {}
  }
}

Jicofo log

root@meet:/var/log/jitsi# tail -f jicofo.log
Jicofo 2023-02-09 09:04:10.470 INFO: [996] BridgeSelectionStrategy.select#132: Selected bridge Bridge[jid=jvbbrewery@internal.auth.meet.xxxx.com/jvb-09755752229ae5ceb.meet.xxxx.com, version=2.2.69-gad606ca2, relayId=108.136.239.214, region=ap-southeast-3, stress=0.01] with stress=0.0 for participantProperties=ParticipantProperties(region=ap-southeast-1, visitor=false)
Jicofo 2023-02-09 09:04:10.470 INFO: [996] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] ColibriV2SessionManager.allocate#323: Selected jvb-09755752229ae5ceb.meet.xxxx.com, session exists: true
Jicofo 2023-02-09 09:04:11.492 INFO: [995] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947 participant=5abeb6a5] ParticipantInviteRunnable.doInviteOrReinvite#392: Sending session-initiate to: devops@conference.meet.xxxx.com/5abeb6a5 sources={jvb=[audio=[4047405235], video=[1674184931], groups=[]]}
Jicofo 2023-02-09 09:04:11.502 INFO: [996] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947 participant=85f2635d] ParticipantInviteRunnable.doInviteOrReinvite#392: Sending session-initiate to: devops@conference.meet.xxxx.com/85f2635d sources={jvb=[audio=[4047405235], video=[1674184931], groups=[]]}
Jicofo 2023-02-09 09:04:11.786 INFO: [987] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947 participant=5abeb6a5] Participant$JingleRequestHandlerImpl.onSessionOrTransportAccept#437: Received session-accept
Jicofo 2023-02-09 09:04:11.787 INFO: [987] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] ColibriV2SessionManager.updateParticipant#523: Updating 5abeb6a5 with transport=org.jitsi.xmpp.extensions.jingle.IceUdpTransportPacketExtension@31778247, sources=[audio=[], video=[], groups=[]]
Jicofo 2023-02-09 09:04:11.839 INFO: [987] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947 participant=85f2635d] Participant$JingleRequestHandlerImpl.onSessionOrTransportAccept#437: Received session-accept
Jicofo 2023-02-09 09:04:11.840 INFO: [987] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] ColibriV2SessionManager.updateParticipant#523: Updating 85f2635d with transport=org.jitsi.xmpp.extensions.jingle.IceUdpTransportPacketExtension@74751a0d, sources=[audio=[], video=[], groups=[]]
Jicofo 2023-02-09 09:04:32.165 INFO: [993] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] ColibriV2SessionManager.updateParticipant#523: Updating 5abeb6a5 with transport=null, sources=[audio=[2398050608], video=[], groups=[]]
Jicofo 2023-02-09 09:04:32.166 INFO: [44] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947 participant=85f2635d] Participant.sendQueuedRemoteSources#329: Sending a queued source-add, sources={5abeb6a5=[audio=[2398050608], video=[], groups=[]]}
Jicofo 2023-02-09 09:07:16.929 INFO: [990] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] BaseJibri.handleJibriRequest#75: Accepted jibri request: <iq xmlns='jabber:client' to='focus@auth.meet.xxxx.com/focus' from='devops@conference.meet.xxxx.com/5abeb6a5' id='Zm9jdXNAYXV0aC5tZWV0LnBpbnRhci5jby9mb2N1cwA1ODcwMzFhZi1mMDBhLTQxM2MtYmY5OC04OTYzNmVkZTlmMjM6c2VuZElRAIg+8v8g0M75PbDcwY906rI=' type='set'><jibri xmlns='http://jitsi.org/protocol/jibri' action='start' recording_mode='file' app_data='{"file_recording_metadata":{"share":true}}'/></iq>
Jicofo 2023-02-09 09:07:16.930 INFO: [1000] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] JibriSession.startInternal#316: Starting session with Jibri jibribrewery@internal.auth.meet.xxxx.com/jibri-0dfcae189d79a34fb
Jicofo 2023-02-09 09:07:16.930 INFO: [1000] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] JibriSession.sendJibriStartIq#470: Starting Jibri jibribrewery@internal.auth.meet.xxxx.com/jibri-0dfcae189d79a34fb for stream ID: null in room: devops@conference.meet.xxxx.com
Jicofo 2023-02-09 09:07:31.931 SEVERE: [1000] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] JibriSession.sendJibriStartIq#514: Unexpected response to start request: null
Jicofo 2023-02-09 09:07:31.931 SEVERE: [1000] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] JibriSession.startInternal#322: Failed to send start Jibri IQ: org.jitsi.jicofo.jibri.JibriSession$StartException$UnexpectedResponse: Unexpected response
org.jitsi.jicofo.jibri.JibriSession$StartException$UnexpectedResponse: Unexpected response
        at org.jitsi.jicofo.jibri.JibriSession.sendJibriStartIq(JibriSession.java:518)
        at org.jitsi.jicofo.jibri.JibriSession.startInternal(JibriSession.java:318)
        at org.jitsi.jicofo.jibri.JibriSession.start(JibriSession.java:282)
        at org.jitsi.jicofo.jibri.JibriRecorder.handleStartRequest(JibriRecorder.kt:116)
        at org.jitsi.jicofo.jibri.BaseJibri.doHandleIQRequest(BaseJibri.kt:170)
        at org.jitsi.jicofo.jibri.BaseJibri.incomingIqQueue$lambda-0(BaseJibri.kt:56)
        at org.jitsi.utils.queue.PacketQueue$HandlerAdapter.handleItem(PacketQueue.java:416)
        at org.jitsi.utils.queue.AsyncQueueHandler$1.run(AsyncQueueHandler.java:136)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Jicofo 2023-02-09 09:07:31.931 INFO: [1000] JibriDetector.instanceFailed#88: Instance failed: jibribrewery@internal.auth.meet.xxxx.com/jibri-0dfcae189d79a34fb. Will not be selected for the next PT1M
Jicofo 2023-02-09 09:07:31.931 INFO: [1000] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] JibriSession.startInternal#316: Starting session with Jibri jibribrewery@internal.auth.meet.xxxx.com/jibri-06b710b25167ba9c7
Jicofo 2023-02-09 09:07:31.931 INFO: [1000] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] JibriSession.sendJibriStartIq#470: Starting Jibri jibribrewery@internal.auth.meet.xxxx.com/jibri-06b710b25167ba9c7 for stream ID: null in room: devops@conference.meet.xxxx.com
Jicofo 2023-02-09 09:07:31.932 INFO: [1000] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] JibriSession.reschedulePendingTimeout#553: Rescheduling pending timeout task for room: devops@conference.meet.xxxx.com
Jicofo 2023-02-09 09:07:41.116 INFO: [1000] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] JibriSession.processJibriIqFromJibri#431: Updating status from JIBRI: <iq xmlns='jabber:client' to='focus@auth.meet.xxxx.com/focus' from='jibribrewery@internal.auth.meet.xxxx.com/jibri-06b710b25167ba9c7' id='MA35C-64685' type='result'><jibri xmlns='http://jitsi.org/protocol/jibri' status='pending'/></iq> for devops@conference.meet.xxxx.com
Jicofo 2023-02-09 09:07:41.116 INFO: [1000] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] JibriSession.handleJibriStatusUpdate#619: Got Jibri status update: Jibri jibribrewery@internal.auth.meet.xxxx.com/jibri-06b710b25167ba9c7 has status pending and failure reason null, current Jibri jid is jibribrewery@internal.auth.meet.xxxx.com/jibri-06b710b25167ba9c7
Jicofo 2023-02-09 09:07:41.116 INFO: [1000] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] JibriRecorder.handleStartRequest#117: Started Jibri session
Jicofo 2023-02-09 09:07:43.736 INFO: [1000] ConferenceIqHandler.handleConferenceIq#69: Focus request for room: devops@conference.meet.xxxx.com
Jicofo 2023-02-09 09:07:43.909 INFO: [990] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] JitsiMeetConferenceImpl.onMemberJoined#589: Member joined:cac71f03 stats-id=jibri-0dfcae189d79a34fb region=ap-southeast-1 audioMuted=true videoMuted=true role=OWNER isJibri=true isJigasi=false
Jicofo 2023-02-09 09:07:43.910 INFO: [990] DiscoveryUtil.discoverParticipantFeatures#164: Discovered features for devops@conference.meet.xxxx.com/cac71f03 in 0 ms.
Jicofo 2023-02-09 09:07:43.910 WARNING: [1000] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947 participant=cac71f03] ParticipantInviteRunnable.lambda$doRun$0#194: Failed to convert ContentPacketExtension to Media: <content creator='initiator' name='data' senders='both'><transport xmlns='urn:xmpp:jingle:transports:ice-udp:1'><fingerprint xmlns='urn:xmpp:jingle:apps:dtls:0'/></transport><description xmlns='urn:xmpp:jingle:apps:rtp:1' media='application'/></content>
Jicofo 2023-02-09 09:07:43.910 INFO: [1000] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] ColibriV2SessionManager.allocate#277: Allocating for cac71f03
Jicofo 2023-02-09 09:07:43.911 INFO: [1000] BridgeSelectionStrategy.select#132: Selected bridge Bridge[jid=jvbbrewery@internal.auth.meet.xxxx.com/jvb-09755752229ae5ceb.meet.xxxx.com, version=2.2.69-gad606ca2, relayId=108.136.239.214, region=ap-southeast-3, stress=0.00] with stress=0.00128 for participantProperties=ParticipantProperties(region=ap-southeast-1, visitor=false)
Jicofo 2023-02-09 09:07:43.914 INFO: [1000] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] ColibriV2SessionManager.allocate#323: Selected jvb-09755752229ae5ceb.meet.xxxx.com, session exists: true
Jicofo 2023-02-09 09:07:43.942 INFO: [1000] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947 participant=cac71f03] ParticipantInviteRunnable.doInviteOrReinvite#392: Sending session-initiate to: devops@conference.meet.xxxx.com/cac71f03 sources={5abeb6a5=[audio=[2398050608], video=[], groups=[]], jvb=[audio=[4047405235], video=[1674184931], groups=[]]}
Jicofo 2023-02-09 09:07:44.623 INFO: [994] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947 participant=cac71f03] Participant$JingleRequestHandlerImpl.onSessionOrTransportAccept#437: Received session-accept
Jicofo 2023-02-09 09:07:44.623 INFO: [994] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] ColibriV2SessionManager.updateParticipant#523: Updating cac71f03 with transport=org.jitsi.xmpp.extensions.jingle.IceUdpTransportPacketExtension@1d5e232e, sources=[audio=[], video=[], groups=[]]
Jicofo 2023-02-09 09:07:45.621 INFO: [1000] ConferenceIqHandler.handleConferenceIq#69: Focus request for room: devops@conference.meet.xxxx.com
Jicofo 2023-02-09 09:07:45.794 INFO: [994] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] JitsiMeetConferenceImpl.onMemberJoined#589: Member joined:55a7ce21 stats-id=jibri-06b710b25167ba9c7 region=ap-southeast-1 audioMuted=true videoMuted=true role=OWNER isJibri=true isJigasi=false
Jicofo 2023-02-09 09:07:45.794 INFO: [994] DiscoveryUtil.discoverParticipantFeatures#164: Discovered features for devops@conference.meet.xxxx.com/55a7ce21 in 0 ms.
Jicofo 2023-02-09 09:07:45.795 WARNING: [1000] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947 participant=55a7ce21] ParticipantInviteRunnable.lambda$doRun$0#194: Failed to convert ContentPacketExtension to Media: <content creator='initiator' name='data' senders='both'><transport xmlns='urn:xmpp:jingle:transports:ice-udp:1'><fingerprint xmlns='urn:xmpp:jingle:apps:dtls:0'/></transport><description xmlns='urn:xmpp:jingle:apps:rtp:1' media='application'/></content>
Jicofo 2023-02-09 09:07:45.795 INFO: [1000] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] ColibriV2SessionManager.allocate#277: Allocating for 55a7ce21
Jicofo 2023-02-09 09:07:45.796 INFO: [1000] BridgeSelectionStrategy.select#132: Selected bridge Bridge[jid=jvbbrewery@internal.auth.meet.xxxx.com/jvb-09755752229ae5ceb.meet.xxxx.com, version=2.2.69-gad606ca2, relayId=108.136.239.214, region=ap-southeast-3, stress=0.01] with stress=0.00128 for participantProperties=ParticipantProperties(region=ap-southeast-1, visitor=false)
Jicofo 2023-02-09 09:07:45.796 INFO: [1000] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] ColibriV2SessionManager.allocate#323: Selected jvb-09755752229ae5ceb.meet.xxxx.com, session exists: true
Jicofo 2023-02-09 09:07:45.821 INFO: [1000] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947 participant=55a7ce21] ParticipantInviteRunnable.doInviteOrReinvite#392: Sending session-initiate to: devops@conference.meet.xxxx.com/55a7ce21 sources={5abeb6a5=[audio=[2398050608], video=[], groups=[]], jvb=[audio=[4047405235], video=[1674184931], groups=[]]}
Jicofo 2023-02-09 09:07:46.447 INFO: [999] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947 participant=55a7ce21] Participant$JingleRequestHandlerImpl.onSessionOrTransportAccept#437: Received session-accept
Jicofo 2023-02-09 09:07:46.448 INFO: [999] [room=devops@conference.meet.xxxx.com meeting_id=c16f176d-8f10-4f47-a891-dd867568c947] ColibriV2SessionManager.updateParticipant#523: Updating 55a7ce21 with transport=org.jitsi.xmpp.extensions.jingle.IceUdpTransportPacketExtension@4e2f7713, sources=[audio=[], video=[], groups=[]]

My question is:
Is jibri can’t run in multiple servers (IDLE) and if one room press the recording, it will pick 1 server rather than 2 idle servers?

2

Are you sure each Jibri has a different nickname? Post the nickname block of the jibri configuration for each of the 3 jibri instances here.

3

Yes. we are using ec2 startup script to create dynamic name by instance id and combine with fqdn

  • Jibri 1
ubuntu@jibri-01f8a760001eb1e18:~$ grep name /etc/jitsi/jibri/jibri.conf 
          name = "jibri-01f8a760001eb1e18"
              room-name = "JibriBrewery"
              nickname = "jibri-01f8a760001eb1e18"
              username = "jibri"
              username = "recorder"
  • Jibri 2
ubuntu@jibri-083f823085c419b48:~$ grep name /etc/jitsi/jibri/jibri.conf
          name = "jibri-083f823085c419b48"
              room-name = "JibriBrewery"
              nickname = "jibri-083f823085c419b48"
              username = "jibri"
              username = "recorder"
  • Jibri 3
ubuntu@jibri-0913bfd5e3a983a34:~$ grep name /etc/jitsi/jibri/jibri.conf
          name = "jibri-0913bfd5e3a983a34"
              room-name = "JibriBrewery"
              nickname = "jibri-0913bfd5e3a983a34"
              username = "jibri"
  • Jibri 4
$ grep name /etc/jitsi/jibri/jibri.conf
          name = "jibri-0944bbb367acc8126"
              room-name = "JibriBrewery"
              nickname = "jibri-0944bbb367acc8126"
              username = "jibri"
              username = "recorder"
  • Jibri 5
ubuntu@jibri-094fd27e5f9506803:~$ grep name /etc/jitsi/jibri/jibri.conf
          name = "jibri-094fd27e5f9506803"
              room-name = "JibriBrewery"
              nickname = "jibri-094fd27e5f9506803"
              username = "jibri"
              username = "recorder"
  • Jibri 6
ubuntu@jibri-0d4733be5e257f321:~$ grep name /etc/jitsi/jibri/jibri.conf
          name = "jibri-0d4733be5e257f321"
              room-name = "JibriBrewery"
              nickname = "jibri-0d4733be5e257f321"
              username = "jibri"

Our scripts

#!/bin/bash
# Instance Detail
INSTANCE_ID=$( curl -s http://169.254.169.254/latest/meta-data/instance-id )
INSTANCE_ID_PARSE=$( echo $INSTANCE_ID | cut -d"-" -f2 )
INSTANCE_IPv4_LOCAL=$( curl -s http://169.254.169.254/latest/meta-data/local-ipv4 )
INSTANCE_IPv4_PUBLIC=$( curl -s http://169.254.169.254/latest/meta-data/public-ipv4 )
DOMAIN="meet.xxxxx.com"
INSTANCE_NAME="jibri-${INSTANCE_ID_PARSE}"
INSTANCE_NAME_FQDN="${INSTANCE_NAME}.${DOMAIN}"
INSTANCE_REGION=$( curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region )

# Get the data from AWS Parameter Store
JITSI_MASTER_IP=$(aws ssm get-parameter --name /jitsi/master/ip --with-decryption --query Parameter.Value --output text --region $INSTANCE_REGION)
JIBRI_CONTROL_USERNAME=$(aws ssm get-parameter --name /jitsi/jibri/control/username --with-decryption --query Parameter.Value --output text --region $INSTANCE_REGION)
JIBRI_CONTROL_PASSWORD=$(aws ssm get-parameter --name /jitsi/jibri/control/password --with-decryption --query Parameter.Value --output text --region $INSTANCE_REGION)
JIBRI_RECORDER_USERNAME=$(aws ssm get-parameter --name /jitsi/jibri/recorder/username --with-decryption --query Parameter.Value --output text --region $INSTANCE_REGION)
JIBRI_RECORDER_PASSWORD=$(aws ssm get-parameter --name /jitsi/jibri/recorder/password --with-decryption --query Parameter.Value --output text --region $INSTANCE_REGION)

sudo tee -a /etc/jitsi/jibri/jibri.conf<<EOF
jibri {
  id = "$INSTANCE_NAME"
  single-use-mode = false
  api {
    http {
      external-api-port = 2222
      internal-api-port = 3333
    }
    xmpp {
      environments = [
        {
          name = "$INSTANCE_NAME"
          xmpp-server-hosts = ["$DOMAIN"]
          xmpp-domain = "$DOMAIN"

          control-muc {
              domain = "internal.auth.$DOMAIN"
              room-name = "JibriBrewery"
              nickname = "$INSTANCE_NAME"
          }

          control-login {
              domain = "auth.$DOMAIN"
              username = "$JIBRI_CONTROL_USERNAME"
              password = "$JIBRI_CONTROL_PASSWORD"
          }

          call-login {
              domain = "recorder.$DOMAIN"
              username = "$JIBRI_RECORDER_USERNAME"
              password = "$JIBRI_RECORDER_PASSWORD"
          }

          strip-from-room-domain = "conference."
          usage-timeout = 0
          trust-all-xmpp-certs = true
        }
      ]
    }
  }
  recording {
    recordings-directory = "/srv/recordings"
    finalize-script = "/srv/finalize.sh"
  }
  streaming {
    rtmp-allow-list = [
      ".*"
    ]
  }
  chrome {
    flags = [
      "--use-fake-ui-for-media-stream",
      "--start-maximized",
      "--kiosk",
      "--enabled",
      "--disable-infobars",
      "--autoplay-policy=no-user-gesture-required"
    ]
  }
  stats {
    enable-stats-d = true
  }
  webhook {
    subscribers = []
  }
  jwt-info {
    # signing-key-path = "/path/to/key.pem"
    # kid = "key-id"
    # issuer = "issuer"
    # audience = "audience"
    # ttl = 1 hour
  }
  call-status-checks {
    no-media-timeout = 90 seconds
    all-muted-timeout = 10 minutes
    default-call-empty-timeout = 30 seconds
  }
}
EOF
4

Does it run before the jibri service?

5

Correct.

In the end we start the jibri

echo "Fixed Jibri ICEWM"
tee -a /home/jibri/.icewm/startup <<EOF
#!/bin/bash

echo "Starting Jibri ICEWM" >> /tmp/icewm.log
date >> /tmp/icewm.log

/usr/bin/google-chrome https://meet.xxxxx.com &
(sleep 20 && pkill --oldest --signal TERM '(chromium|chrome|google-chrome)') &
EOF

echo "Fixed Jibri ICEWM Permission"
sudo chmod 755 /home/jibri/.icewm/startup
sudo chown jibri:jibri /home/jibri/.icewm/
sudo chown jibri:jibri /home/jibri/.icewm/*

echo "Start Jibri Server"
sudo systemctl start jibri
echo "Status Jibri Server"
sudo systemctl status jibri

Do i need run jibri-xorg or jibri-icewm separatly?

6

You don’t need to start jibri in this script. Did you enable jibri service?

systemctl enable jibri
systemctl start jibri
7

Edited:
It’s not enable by default in AMI.

since we using AWS EC2 with AMI,
Base AMI only setup audio and jibri installation, and we stop all of them: jibri, xorg and icewm

So when the EC2 startup script executes, jibri not running.
after all configuration is set, we start and enable jibri service

8

ok, we found the issue.

we check at jibri-icewm with an error

Can't open display: <none>. X must be running and $DISPLAY set.
9

Do you have an already running desktop on this server?

10

do you mean jibri-xorg or others?
and how to check desktop on jibri?

11

You may check the running processes by using

ps auxww
12

ok, I see icewm here. is that correct?

ubuntu@jibri-0d70e8ae4ebbce4ec:~$ sudo ps auxww
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.1 171416 13528 ?        Ss   06:00   0:03 /sbin/init
root           2  0.0  0.0      0     0 ?        S    06:00   0:00 [kthreadd]
root           3  0.0  0.0      0     0 ?        I<   06:00   0:00 [rcu_gp]
root           4  0.0  0.0      0     0 ?        I<   06:00   0:00 [rcu_par_gp]
root           6  0.0  0.0      0     0 ?        I<   06:00   0:00 [kworker/0:0H-kblockd]
root           8  0.0  0.0      0     0 ?        I<   06:00   0:00 [mm_percpu_wq]
root           9  0.0  0.0      0     0 ?        S    06:00   0:00 [ksoftirqd/0]
root          10  0.0  0.0      0     0 ?        I    06:00   0:00 [rcu_sched]
root          11  0.0  0.0      0     0 ?        S    06:00   0:00 [migration/0]
root          12  0.0  0.0      0     0 ?        S    06:00   0:00 [idle_inject/0]
root          14  0.0  0.0      0     0 ?        S    06:00   0:00 [cpuhp/0]
root          15  0.0  0.0      0     0 ?        S    06:00   0:00 [cpuhp/1]
root          16  0.0  0.0      0     0 ?        S    06:00   0:00 [idle_inject/1]
root          17  0.0  0.0      0     0 ?        S    06:00   0:00 [migration/1]
root          18  0.0  0.0      0     0 ?        S    06:00   0:00 [ksoftirqd/1]
root          20  0.0  0.0      0     0 ?        I<   06:00   0:00 [kworker/1:0H-kblockd]
root          21  0.0  0.0      0     0 ?        S    06:00   0:00 [cpuhp/2]
root          22  0.0  0.0      0     0 ?        S    06:00   0:00 [idle_inject/2]
root          23  0.0  0.0      0     0 ?        S    06:00   0:00 [migration/2]
root          24  0.0  0.0      0     0 ?        S    06:00   0:00 [ksoftirqd/2]
root          26  0.0  0.0      0     0 ?        I<   06:00   0:00 [kworker/2:0H-kblockd]
root          27  0.0  0.0      0     0 ?        S    06:00   0:00 [cpuhp/3]
root          28  0.0  0.0      0     0 ?        S    06:00   0:00 [idle_inject/3]
root          29  0.0  0.0      0     0 ?        S    06:00   0:00 [migration/3]
root          30  0.0  0.0      0     0 ?        S    06:00   0:00 [ksoftirqd/3]
root          32  0.0  0.0      0     0 ?        I<   06:00   0:00 [kworker/3:0H-kblockd]
root          33  0.0  0.0      0     0 ?        S    06:00   0:00 [kdevtmpfs]
root          34  0.0  0.0      0     0 ?        I<   06:00   0:00 [netns]
root          35  0.0  0.0      0     0 ?        S    06:00   0:00 [rcu_tasks_kthre]
root          36  0.0  0.0      0     0 ?        S    06:00   0:00 [kauditd]
root          37  0.0  0.0      0     0 ?        S    06:00   0:00 [khungtaskd]
root          38  0.0  0.0      0     0 ?        S    06:00   0:00 [oom_reaper]
root          39  0.0  0.0      0     0 ?        I<   06:00   0:00 [writeback]
root          40  0.0  0.0      0     0 ?        S    06:00   0:00 [kcompactd0]
root          41  0.0  0.0      0     0 ?        SN   06:00   0:00 [ksmd]
root          42  0.0  0.0      0     0 ?        SN   06:00   0:00 [khugepaged]
root          47  0.0  0.0      0     0 ?        I    06:00   0:00 [kworker/1:1-mm_percpu_wq]
root          89  0.0  0.0      0     0 ?        I<   06:00   0:00 [kintegrityd]
root          90  0.0  0.0      0     0 ?        I<   06:00   0:00 [kblockd]
root          91  0.0  0.0      0     0 ?        I<   06:00   0:00 [blkcg_punt_bio]
root          92  0.0  0.0      0     0 ?        I<   06:00   0:00 [tpm_dev_wq]
root          93  0.0  0.0      0     0 ?        I<   06:00   0:00 [ata_sff]
root          94  0.0  0.0      0     0 ?        I<   06:00   0:00 [md]
root          95  0.0  0.0      0     0 ?        I<   06:00   0:00 [edac-poller]
root          96  0.0  0.0      0     0 ?        I<   06:00   0:00 [devfreq_wq]
root          98  0.0  0.0      0     0 ?        S    06:00   0:00 [watchdogd]
root         100  0.0  0.0      0     0 ?        I    06:00   0:00 [kworker/2:1-mm_percpu_wq]
root         102  0.0  0.0      0     0 ?        S    06:00   0:00 [kswapd0]
root         103  0.0  0.0      0     0 ?        S    06:00   0:00 [ecryptfs-kthrea]
root         105  0.0  0.0      0     0 ?        I<   06:00   0:00 [kthrotld]
root         106  0.0  0.0      0     0 ?        I<   06:00   0:00 [acpi_thermal_pm]
root         107  0.0  0.0      0     0 ?        I<   06:00   0:00 [vfio-irqfd-clea]
root         108  0.0  0.0      0     0 ?        I<   06:00   0:00 [ipv6_addrconf]
root         118  0.0  0.0      0     0 ?        I<   06:00   0:00 [kstrp]
root         121  0.0  0.0      0     0 ?        I<   06:00   0:00 [kworker/u9:0]
root         134  0.0  0.0      0     0 ?        I<   06:00   0:00 [charger_manager]
root         182  0.0  0.0      0     0 ?        I<   06:00   0:00 [ena]
root         183  0.0  0.0      0     0 ?        I<   06:00   0:00 [nvme-wq]
root         184  0.0  0.0      0     0 ?        I<   06:00   0:00 [nvme-reset-wq]
root         185  0.0  0.0      0     0 ?        I    06:00   0:00 [kworker/3:2-memcg_kmem_cache]
root         186  0.0  0.0      0     0 ?        I<   06:00   0:00 [nvme-delete-wq]
root         187  0.0  0.0      0     0 ?        I<   06:00   0:00 [cryptd]
root         252  0.0  0.0      0     0 ?        I<   06:01   0:00 [raid5wq]
root         295  0.0  0.0      0     0 ?        S    06:01   0:00 [jbd2/nvme0n1p1-]
root         296  0.0  0.0      0     0 ?        I<   06:01   0:00 [ext4-rsv-conver]
root         331  0.0  0.0      0     0 ?        I<   06:01   0:00 [kworker/1:1H-kblockd]
root         355  0.0  0.0      0     0 ?        I<   06:01   0:00 [kworker/0:1H-kblockd]
root         370  0.0  0.2  46020 21056 ?        S<s  06:01   0:01 /lib/systemd/systemd-journald
root         399  0.0  0.0  23016  6520 ?        Ss   06:01   0:00 /lib/systemd/systemd-udevd
root         405  0.0  0.0      0     0 ?        I<   06:01   0:00 [kworker/2:1H-kblockd]
root         449  0.0  0.0      0     0 ?        I<   06:01   0:00 [nfit]
root         496  0.0  0.0      0     0 ?        I<   06:01   0:00 [kaluad]
root         497  0.0  0.0      0     0 ?        I<   06:01   0:00 [kmpath_rdacd]
root         498  0.0  0.0      0     0 ?        I<   06:01   0:00 [kmpathd]
root         499  0.0  0.0      0     0 ?        I<   06:01   0:00 [kmpath_handlerd]
root         500  0.0  0.2 280136 17948 ?        SLsl 06:01   0:00 /sbin/multipathd -d -s
root         508  0.0  0.0      0     0 ?        S<   06:01   0:00 [loop0]
root         516  0.0  0.0      0     0 ?        S<   06:01   0:00 [loop1]
root         518  0.0  0.0      0     0 ?        S<   06:01   0:00 [loop2]
root         519  0.0  0.0      0     0 ?        S<   06:01   0:00 [loop3]
root         520  0.0  0.0      0     0 ?        S<   06:01   0:00 [loop4]
root         524  0.0  0.0      0     0 ?        I<   06:01   0:00 [kworker/3:1H-kblockd]
systemd+     534  0.0  0.0  90652  5812 ?        Ssl  06:01   0:00 /lib/systemd/systemd-timesyncd
systemd+     626  0.0  0.0  27384  7812 ?        Ss   06:01   0:00 /lib/systemd/systemd-networkd
systemd+     639  0.0  0.1  24408 12028 ?        Ss   06:01   0:00 /lib/systemd/systemd-resolved
root         647  0.0  0.0      0     0 ?        I    06:01   0:00 [kworker/2:3-cgroup_destroy]
root         725  0.0  0.1 241048  9136 ?        Ssl  06:01   0:00 /usr/lib/accountsservice/accounts-daemon
root         726  0.0  0.0   2540   720 ?        Ss   06:01   0:00 /usr/sbin/acpid
root         730  0.0  0.0   8540  2776 ?        Ss   06:01   0:00 /usr/sbin/cron -f
message+     731  0.0  0.0   7680  4772 ?        Ss   06:01   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         749  0.0  0.0  81892  3572 ?        Ssl  06:01   0:00 /usr/sbin/irqbalance --foreground
root         750  0.0  0.2  29872 18304 ?        Ss   06:01   0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
root         751  0.0  0.1 236312  8768 ?        Ssl  06:01   0:00 /usr/lib/policykit-1/polkitd --no-debug
syslog       752  0.0  0.0 224492  5148 ?        Ssl  06:01   0:00 /usr/sbin/rsyslogd -n -iNONE
root         753  0.0  0.1 1379344 14540 ?       Ssl  06:01   0:01 /snap/amazon-ssm-agent/6312/amazon-ssm-agent
root         756  0.0  0.5 1096600 45264 ?       Ssl  06:01   0:01 /usr/lib/snapd/snapd
root         762  0.0  0.0  17220  7820 ?        Ss   06:01   0:00 /lib/systemd/systemd-logind
root         768  0.0  0.1 395484 13812 ?        Ssl  06:01   0:00 /usr/lib/udisks2/udisksd
daemon       773  0.0  0.0   3796  2296 ?        Ss   06:01   0:00 /usr/sbin/atd -f
root         788  0.0  0.0   7352  2420 ttyS0    Ss+  06:01   0:00 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220
root         794  0.0  0.0   5828  1816 tty1     Ss+  06:01   0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
root         813  0.0  0.2 108120 20756 ?        Ssl  06:01   0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
root         820  0.0  0.1 245092 13464 ?        Ssl  06:01   0:00 /usr/sbin/ModemManager
root         940  0.0  0.0  12176  6912 ?        Ss   06:01   0:00 sshd: /usr/sbin/sshd -D -o AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys %u %f -o AuthorizedKeysCommandUser ec2-instance-connect [listener] 0 of 10-100 startups
root        1009  0.0  0.2 1462260 22736 ?       Sl   06:01   0:00 /snap/amazon-ssm-agent/6312/ssm-agent-worker
root        7106  0.0  1.3 1615488 105292 ?      Ssl  06:04   0:07 /usr/local/bin/teleport start --pid-file=/run/teleport.pid
jibri       7217  0.0  0.8 425784 66324 ?        Ssl  06:04   0:00 /usr/lib/xorg/Xorg -nocursor -noreset +extension RANDR +extension RENDER -logfile /var/log/jitsi/jibri/xorg.log -config /etc/jitsi/jibri/xorg-video-dummy.conf :0
jibri       7218  0.0  0.0   5964  1660 ?        Ss   06:04   0:00 /usr/bin/icewm-session
jibri       7219  0.1  3.0 5684912 242592 ?      Ssl  06:04   0:17 java -Djava.util.logging.config.file=/etc/jitsi/jibri/logging.properties -Dconfig.file=/etc/jitsi/jibri/jibri.conf -jar /opt/jitsi/jibri/jibri.jar --config /etc/jitsi/jibri/config.json
jibri       7227  0.0  0.1  21880  9984 ?        SNs  06:04   0:00 /usr/bin/icewmbg
jibri       7228  0.0  0.1  27824 11064 ?        Ss   06:04   0:00 /usr/bin/icewm --notify
root        7344  0.0  0.0   8696  3552 ?        Ss   06:04   0:00 /bin/bash /root/stats.sh
root        7408  0.0  0.0   8696  3512 ?        Ss   06:04   0:00 /bin/bash /root/log-sync.sh
root       19596  0.0  0.0      0     0 ?        I<   06:06   0:00 [xfsalloc]
root       19600  0.0  0.0      0     0 ?        I<   06:06   0:00 [xfs_mru_cache]
root       19603  0.0  0.0      0     0 ?        S    06:06   0:00 [jfsIO]
root       19604  0.0  0.0      0     0 ?        S    06:06   0:00 [jfsCommit]
root       19605  0.0  0.0      0     0 ?        S    06:06   0:00 [jfsCommit]
root       19606  0.0  0.0      0     0 ?        S    06:06   0:00 [jfsCommit]
root       19607  0.0  0.0      0     0 ?        S    06:06   0:00 [jfsCommit]
root       19608  0.0  0.0      0     0 ?        S    06:06   0:00 [jfsSync]
root       29116  0.0  0.0      0     0 ?        I    06:12   0:00 [kworker/1:0]
root       29119  0.0  0.0      0     0 ?        I    06:12   0:00 [kworker/3:0-cgroup_destroy]
root       29621  0.0  0.0      0     0 ?        I    06:32   0:00 [kworker/0:1-cgroup_destroy]
root       31828  0.0  0.0      0     0 ?        I    08:15   0:00 [kworker/u8:1-events_power_efficient]
root       32194  0.0  0.0      0     0 ?        I    08:33   0:00 [kworker/u8:0-events_unbound]
root       32758  0.1  0.8 430428 69592 ?        Ssl  08:59   0:00 /usr/libexec/fwupd/fwupd
root       32767  0.0  0.1 242948  8800 ?        Ssl  08:59   0:00 /usr/lib/upower/upowerd
root       32792  0.0  0.0  83028  2728 ?        Ss   08:59   0:00 gpg-agent --homedir /var/lib/fwupd/gnupg --use-standard-socket --daemon
root       32805  0.0  0.0      0     0 ?        I    08:59   0:00 [kworker/0:0-events]
root       32807  0.0  0.0      0     0 ?        I    08:59   0:00 [kworker/u8:2-events_unbound]
root       32937  0.0  0.0   7228   584 ?        S    09:05   0:00 sleep 60
root       32940  0.2  1.0 1467536 81592 ?       Sl   09:05   0:00 /usr/local/bin/teleport exec
ubuntu     32950  0.0  0.0   8260  5200 pts/0    Ss   09:05   0:00 -bash
root       32969  0.0  0.0   7228   580 ?        S    09:06   0:00 sleep 60
root       32970  0.0  0.0   9256  4548 pts/0    S+   09:06   0:00 sudo ps auxww
root       32971  0.0  0.0  23016  4128 ?        S    09:06   0:00 /lib/systemd/systemd-udevd
root       32972  0.0  0.0   8888  3372 pts/0    R+   09:06   0:00 ps auxww
13

The process list seems OK. Where do you see the display error?

14

I didn’t see why. so we just downgraded from ubuntu 20.04 to 18.04 and now it’s working now