Jibri audio/video recording problem after upgrade to stable-6433

Hi Jitsi,

We recently upgraded our self-hosted Jitsi deployment to stable-6433 (thanks to contributors) unfortunately, Jibri can’t see participants’ videos nor hear their voices.

The interesting part is when we change the version of Jicofo to an older version Jibri works fine. But I can’t find a good connection between this problem and Jicofo.
I investigate logs of the Jicofo, JVB, and Prosody, nothing is interesting there.
Also, try to connect to Jibri using VNC and check the browser.logs

You can find some logs and screen-shots below

only interesting log in the browser.logs:

2021-12-07 21:53:27.616 INFO: [154] browser.leaveCallAndQuitBrowser() [2021-12-07T21:53:27+0330] [SEVERE] https://room.gharar.ir/libs/app.bundle.min.js?v=0.25.5 459:882435 "2021-12-07T18:23:20.209Z" "[modules/statistics/AvgRTPStatsReporter.js]" "\u003Cm.addNext>: " "bandwidth_download - invalid value for idx: 0" undefined

I’m able to debug Jibri and run js commands in the console, so if you have any idea about what to check please tell me.

Thank you for reading this
Greetings

you are running a recent (relatively, 6433 is no longer the latest stable) recent server with (I guess) recent clients, but is the Jibri browser up-to-date too ? If you use an old Chrome it will not be able to use Jitsi-Meet, so if you did not update Chrome and the Chrome driver in your Jibri it may be normal it can’t connect to media channels.

1 Like

Share your jicofo.conf

1 Like

thank you very much, I will consider your points. the chrome version must be updated according to jibri:stable-6433 image

here you go sir

jicofo {
  // Authentication with external services
  authentication {
    enabled = true
    // The type of authentication. Supported values are XMPP, JWT or SHIBBOLETH (default).
    type = XMPP

    // The pattern of authentication URL. See ShibbolethAuthAuthority for more information.
    login-url = "o3.room.gh.am"
    logout-url = "o3.room.gh.am"

    authentication-lifetime = 24 hours
    enable-auto-login = true
  }
  // Configuration related to jitsi-videobridge
  bridge {
    // The maximum number of participants in a single conference to put on one bridge (use -1 for no maximum).
    max-bridge-participants = -1
    // The assumed maximum packet rate that a bridge can handle.
    max-bridge-packet-rate = 50000
    // The assumed average packet rate per participant.
    average-participant-packet-rate-pps = 500
    // The default assumed average stress per participant. This value is only used when a bridge does not report its
    // own value.
    average-participant-stress = 0.01
    // The assumed time that an endpoint takes to start contributing fully to the load on a bridge. To avoid allocating
    // a burst of endpoints to the same bridge, the bridge stress is adjusted by adding the number of new endpoints
    // in the last [participant-rampup-time] multiplied by [average-participant-stress].
    participant-rampup-interval = 20 seconds
    // The stress level above which a bridge is considered overstressed.
    stress-threshold = 0.8
    // The amount of to wait before retrying using a failed bridge.
    failure-reset-threshold = 1 minute
    // The bridge selection strategy. The built-in strategies are:
    // SingleBridgeSelectionStrategy: Use the least loaded bridge, do not split a conference between bridges (Octo).
    // SplitBridgeSelectionStrategy: Use a separate bridge for each participant (for testing).
    // RegionBasedBridgeSelectionStrategy: Attempt to put each participant in a bridge in theam local region (i.e. use
    //    Octo for geo-location).
    // IntraRegionBridgeSelectionStrategy: Use additional bridges when a bridge becomes overloaded (i.e. use Octo for
    //    load balancing).
    //
    // Additionally, you can use the fully qualified class name for custom BridgeSelectionStrategy implementations.
    selection-strategy = SingleBridgeSelectionStrategy
    health-checks {
      // Whether jicofo should perform periodic health checks to the connected bridges.
      enabled = true
      // The interval at which to perform health checks.
      interval = 10 seconds
      // When a health checks times out, jicofo will retry and only consider it fail after the retry fails. This
      // configures the delay between the original health check timing out and the second health check being sent.
      // It is a duration and defaults to half the [interval].
      # retry-delay = 5 seconds
    }

    // The JID of the MUC to be used as a brewery for bridge instances.
    brewery-jid = "JvbBrewery@internal.auth.o3.room.gh.am"

    // The XMPP connection to use to communicate with Jitsi Videobridge instances.
    // Either `Client` or `Service` (case-sensitive). See the corresponding XMPP connection configuration under `xmpp`.
    // Note that if no separate Service connection has been configured, all services will automatically use the
    // Client connection.
    xmpp-connection-name = Service
  }
  // Configure the codecs and RTP extensions to be used in the offer sent to clients.
  codec {
    video {
      vp8 {
        enabled = true
        pt = 100
        // Payload type for the associated RTX stream. Set to -1 to disable RTX.
        rtx-pt = 96
        enable-remb = true
      }
      vp9 {
        enabled = true
        pt = 101
        // Payload type for the associated RTX stream. Set to -1 to disable RTX.
        rtx-pt = 97
        enable-remb = true
      }
      h264 {
        enabled = true
        pt = 107
        // Payload type for the associated RTX stream. Set to -1 to disable RTX.
        rtx-pt = 99
        enable-remb = true
      }
    }

    audio {
      isac-16000 {
        enabled = true
        pt = 103
      }
      isac-32000 {
        enabled = true
        pt = 104
      }
      opus {
        enabled = true
        pt = 111
        minptime = 10
        use-inband-fec = true
        red {
          enabled = false
          pt = 112
        }
      }
      telephone-event {
        enabled = true
        pt = 126
      }
    }

    // RTP header extensions
    rtp-extensions {
      audio-level {
        enabled = true
        id = 1
      }
      tof {
        // TOF is currently disabled, because we don't support it in the bridge
        // (and currently clients seem to not use it when abs-send-time is
        // available).
        enabled = false
        id = 2
      }
      abs-send-time {
        enabled = true
        id = 3
      }
      rid {
        enabled = false
        id = 4
      }
      tcc {
        enabled = true
        id = 5
      }
      video-content-type {
        enabled = false
        id = 7
      }
      framemarking {
        enabled = false
        id = 9
      }
    }
  }

  conference {
    // Whether to automatically grant the 'owner' role to the famst participant in the conference (and subsequently to
    // the next in line when the current owner leaves).
    enable-auto-owner = false

    // How long to wait for the initial participant in a conference.
    initial-timeout = 15 seconds

    // Whether jicofo should inject a random SSRC for endpoints which don't advertise any SSRCs. This is a temporary
    // workaround for an issue with signaling endpoints for Octo.
    inject-ssrc-for-recv-only-endpoints = false

    max-ssrcs-per-user = 20

    // How long a participant's media session will be kept alive once it remains the only participant in the room.
    single-participant-timeout = 20 seconds

    // The minimum number of participants requamed for the conference to be started.
    min-participants = 2

    // Experimental.
    enable-lip-sync = false

    shared-document {
      // If `true` the shared document uses a random name. Otherwise, it uses the conference name.
      use-random-name = false
    }
  }

  // Configuration for the internal health checks performed by jicofo.
  health {
    // Whether to perform health checks.
    enabled = false

    // The interval between health checks. If set to 0, periodic health checks will not be performed.
    interval = 10 seconds

    # The timeout for a health check
    timeout = 30 seconds

    # If performing a health check takes longer than this, it is considered unsuccessful.
    max-check-duration = 20 seconds

    # The prefix to use when creating MUC rooms for the purpose of health checks.
    room-name-prefix = "__jicofo-health-check"
  }

  jibri {
    // The JID of the MUC to be used as a brewery for jibri instances for streaming.
    brewery-jid = "JibriBrewery@internal.auth.o3.room.gh.am"

    // How many times to retry a given Jibri request before giving up. Set to -1 to allow infinite retries.
    num-retries = 5

    // How long to wait for Jibri to start recording from the time it accepts a START request.
    pending-timeout = 90 seconds

    // The XMPP connection to use to communicate with Jibri instances. Either `Client` or `Service` (case-sensitive).
    // See the corresponding XMPP connection configuration under `xmpp`.
    // Note that if no separate Service connection has been configured, all services will automatically use the
    // Client connection.
    xmpp-connection-name = Client
  }

  jibri-sip {
    // The JID of the MUC to be used as a brewery for jibri instances for SIP.
    # brewery-jid = "jibrisipbrewery@example.com"
  }

  jigasi {
    // The JID of the MUC to be used as a brewery for jigasi instances.
    brewery-jid = "JigasiBrewery@internal.auth.o3.room.gh.am"

    // The XMPP connection to use to communicate with Jigasi instances. Either `Client` or `Service` (case-sensitive).
    // See the corresponding XMPP connection configuration under `xmpp`.
    // Note that if no separate Service connection has been configured, all services will automatically use the
    // Client connection.
    xmpp-connection-name = Client
  }

  // The region in which the machine is running.
  #local-region="us-east-1"

  octo {
    // Whether or not to use Octo. Note that when enabled, its use will be determined by
    // $jicofo.bridge.selection-strategy. There's a corresponding flag in the JVB and these
    // two MUST be in sync (otherwise bridges will crash because they won't know how to
    // deal with octo channels).
    enabled = false

    // An identifier of the Jicofo instance, used for the purpose of generating conference IDs unique across a set of
    // Jicofo instances. Valid values are [1, 65535]. The value 0 is used when none is explicitly configured.
    id = 1234
  }

  rest {
    port = 8888
    tls-port = 8843
  }

  sctp {
    // Whether to allocate SCTP channels on the bridge (only when the client advertises support, and SCTP is
    // enabled in the per-conference configuration).
    enabled = true
  }

  xmpp {
    // The separate XMPP connection used for communication with clients (endpoints).
    client {
      enabled = true
      hostname = "ejabberd"
      port = 5222
      domain = "auth.o3.room.gh.am"
      username = "focus"
      password = "##########"
      // The JID/domain of the MUC service used for conferencing.
      conference-muc-jid = "conference.o3.room.gh.am"

      // How long to wait for a response to a stanza before giving up.
      reply-timeout = 15 seconds

      // A flag to suppress the TLS certificate verification.
      disable-certificate-verification = true

      // The JID of the mod_client_proxy component if used. It will be trusted to encode the JID of the original
      // sender in the resource part of the JID.
      client-proxy = "focus.o3.room.gh.am"

      // Use TLS between Jicofo and the XMPP server
      // Only disable this if your xmpp connection is on loopback!
      use-tls = true
    }
    // The separate XMPP connection used for internal services (currently only jitsi-videobridge).
    service {
      enabled = false
      hostname = "localhost"
      port = 6222
      #domain =
      #username =
      #password =

      // How long to wait for a response to a stanza before giving up.
      reply-timeout = 15 seconds

      // A flag to suppress the TLS certificate verification.
      disable-certificate-verification = false

      // Use TLS between Jicofo and the XMPP server
      // Only disable this if your xmpp connection is on loopback!
      use-tls = true
    }
  }
}

This bit is missing in your configuration:

1 Like

Thanks. I add trusted-domains but the problem remains.
In the Jicofo logs, a warning about trusted-domains turned into an info log so I assume I did it correctly.

before:
WARNING: Jibri signaled from a non-trusted domain: o3.room.gh.am. The domain can be configured as trusted with the jicofo.xmpp.trusted-domains property.

now:
INFO: trusted-domains: [o3.room.gh.am]

How did you add it?

Try different methods, but the last thing I do was add trusted-domains directly to Jicofo /src/main/resources/reference.conf

even, i changed org/jitsi/impl/protocol/xmpp/ChatMemberImpl.java:263 manually but no hope

ps: i have read about this trusted-domains issue with Jibri in an older topic and try it before too.

I actually was asking to see the line you added to Jicofo. You added it like this?

1 Like

After trying trusted-domains failed.
Just change the ChatMemberImpl.isJidTrusted() to always return true.

    private boolean isJidTrusted()
    {
        Jid jid = getJid();
        return jid != null && XmppConfig.config.getTrustedDomains().contains(jid.asDomainBareJid());
    }

This is when isJidTrusted() used:

            if (features.getFeatureExtensions().stream().anyMatch(
                    feature -> "http://jitsi.org/protocol/jibri".equals(feature.getVar())))
            {
                if (isJidTrusted())
                {
                    isJibri = true;
                }
                else
                {
                    Jid domain = getJid() == null ? null : getJid().asDomainBareJid();
                    logger.warn("Jibri signaled from a non-trusted domain: " + domain +
                            ". The domain can be configured as trusted with the jicofo.xmpp.trusted-domains property.");
                    isJibri = false;
                }
            }

I’m really just asking you to clarify that trusted domain line in jicofo.conf. Is that how you added it?

1 Like

add it like this trusted-domains = [o3.room.gh.am]

Okay, that’s wrong. It should be:

trusted-domains: [recorder.o3.room.gh.am]

1 Like

Thank you, I will update you if this works :boom:

setting trusted-domains = [recorder.o3.room.gh.am] did not solve the problem for me.
Is there any other stuff that i can check? thanks

jicofo log INFO: trusted-domains: [recorder.o3.room.gh.am]

Can you share your log for Jibri?

1 Like

BTW, hope you reverted any other changes you made beside the change to jicofo.conf and restarted all services before testing.

Additional - Also, I just checked the images you shared. Are you sure your JVB is even working? I’m looking at the screenshots now and it seems evident that you’re not able to successfully host a 3-party meeting. Am I mistaken?

1 Like

I very appreciate your cooperation. :love_letter:
The problem is fixed now.
I don’t know what was the root cause but it seems Jibri can record just fine. probably the problem was Jibri configurations.

greetings