Troubleshooting steps Jitsi Meet (also Jibri), my comprehensive tips for the beginner

Troubleshooting steps - Jitsi and Jibri.

So now we have a server for Jitsi Meet and possibly also anotherone with Jibri and we expect to start videoconferences, recordings and youtube-streams… OR AT LEAST WE THINK WE HAVE!

If things go awry, then the search is on! It is one thing to try the black box and if it doesn’t work (or part of it doesn’t) start searching the community forum or your favorite search engine. It is more helpful to find more specific information from the logfiles that are generated. It is important to understand the services that are involved:

Jitsi Meet

This server depends mainly on 3 services: prosody, videobridge2 (sometimes called jvb) and jicofo. After server reboot, these services should start in order like:

/etc/init.d/prosody start
/etc/init.d/jitsi-videobridge2 start
sleep 5
/etc/init.d/jicofo start

When the server is running, we can check their status and the results look something like:

service prosody status
Shows:

● prosody.service - Prosody XMPP Server
   Loaded: loaded (/lib/systemd/system/prosody.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-04-15 15:19:58 CEST; 1 day 9h ago
     Docs: https://prosody.im/doc
 Main PID: 3572 (lua5.2)
    Tasks: 1 (limit: 4581)
   Memory: 18.4M
   CGroup: /system.slice/prosody.service
           └─3572 lua5.2 /usr/bin/prosody

service jitsi-videobridge2 status
Shows:

● jitsi-videobridge2.service - Jitsi Videobridge
   Loaded: loaded (/lib/systemd/system/jitsi-videobridge2.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-04-15 13:11:15 CEST; 1 day 11h ago
  Process: 1106 ExecStartPost=/bin/bash -c echo $MAINPID > /var/run/jitsi-videobridge/jitsi-videobridge.pid (code=exited
 Main PID: 1105 (java)
    Tasks: 41 (limit: 65000)
   Memory: 276.3M
   CGroup: /system.slice/jitsi-videobridge2.service
           └─1105 java -Xmx3072m -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Dnet.jav

service jicofo status
Shows:

● jicofo.service - LSB: Jitsi conference Focus
   Loaded: loaded (/etc/init.d/jicofo; generated)
   Active: active (running) since Wed 2020-04-15 15:08:05 CEST; 1 day 9h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 3192 ExecStart=/etc/init.d/jicofo start (code=exited, status=0/SUCCESS)
    Tasks: 107 (limit: 4581)
   Memory: 218.2M
   CGroup: /system.slice/jicofo.service
           └─3200 java -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Dnet.java.sip.communicator.SC_HO

Jibri

This server depends on 1 service: jibri

service jibri status
Shows:

● jibri.service - Jibri Process
   Loaded: loaded (/etc/systemd/system/jibri.service; disabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-04-15 13:16:18 CEST; 1 day 11h ago
 Main PID: 749 (java)
    Tasks: 36 (limit: 4581)
   Memory: 471.4M
   CGroup: /system.slice/jibri.service
           └─749 java -Djava.util.logging.config.file=/etc/jitsi/jibri/logging.properties -jar /opt/jitsi/jibri/jibri.jar --config /etc/jitsi/jibri/config.json

Apr 15 13:16:18 jibri systemd[1]: Started Jibri Process.
Apr 15 13:16:20 jibri launch.sh[749]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
Apr 15 13:16:20 jibri launch.sh[749]: SLF4J: Defaulting to no-operation (NOP) logger implementation
Apr 15 13:16:20 jibri launch.sh[749]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Apr 15 13:18:37 jibri launch.sh[749]: Starting ChromeDriver 81.0.4044.69 (6813546031a4bc83f717a2ef7cd4ac6ec1199132-refs/branch-heads/4044@{#776}) on port 26333
Apr 15 13:18:37 jibri launch.sh[749]: Only local connections are allowed.
Apr 15 13:18:37 jibri launch.sh[749]: Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.

Log-files

locations,

Jitsi:
    /var/log/prosody
    /var/log/jitsi
Jibri:
    /var/log/jitsi/jibri

Investigation first part: check logfiles when services are started but idle
To investigate logfiles can be daunting, I suggest following approache(s):
Step 1: Stop all 3 services prosody, jitsi-videobridge2 and jicofo (or single service on jibri),
Step 2: Remove ALL files from the log-directories,
Step 3: Restart the services in order: prosody, jitsi-videobridge2 and jicofo
Wait approx. 1 minute and then Stop all 3 services again(!).

Now, the logfiles are not filling up anymore and they are not very large, which makes a first analysis more comfortable. Now aall lines with INFO messages can be removed to narrow the search even further down.

Investigation second part: check output when initialising/utilising a video conference
If you cannot find a cause for your problem, then a second phase should be considered to investigate more…
Step 1: If an error occurs, try to find the steps to reproduce the error. Do this several times to make sure you you know how to force the error!
Step 2: Stop all 3 services and Remove ALL files from the log-directories (like above),
Step 3: Restart the services in order: prosody, jitsi-videobridge2 and jicofo and rather soon proceed with the steps to force the error,
Step 4: Stop all 3 services again(!).

Now examine the logfiles and focus on the timing around the moment where the error was enforced. Also here I tend to remove all lines with INFO messages to narrow it even further down.

While writing this final document, I recognise that my jitsi-videobridge2 and jicofo are complaining with the same error:

CGroup: /system.slice/jicofo.service
└─3200 java -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Dnet.java.sip.communicator.SC_HO

So that’s all folks, happy hunting! (I’m off… Hunting for more clues about “+HeapDumpOnOutOfMemoryError”), seems I need to peek in Path=/tmp

Cheers, Igor

4 Likes

good