Ubuntu 18.04: Systemd unable to start videobridge2

Hello,

It appears with the latest upgrade that I’m unable to start the videobridge:

Sep 07 14:40:26 jitsi systemd[1]: jitsi-videobridge2.service: Converting job jitsi-videobridge2.service/restart -> jitsi-videobridge2.service/start
Sep 07 14:40:26 jitsi systemd[1]: jitsi-videobridge2.service: Failed to set blkio.weight: No such file or directory
Sep 07 14:40:26 jitsi systemd[1]: jitsi-videobridge2.service: Passing 0 fds to service
Sep 07 14:40:26 jitsi systemd[1]: jitsi-videobridge2.service: About to execute: /bin/bash -c 'exec /usr/share/jitsi-videobridge/jvb.sh ${JVB_OPTS} < /dev/null >> ${LOGFILE} 2>&1'
Sep 07 14:40:26 jitsi systemd[1]: jitsi-videobridge2.service: Forked /bin/bash as 13264
Sep 07 14:40:26 jitsi systemd[1]: jitsi-videobridge2.service: About to execute: /bin/bash -c 'echo $MAINPID > /var/run/jitsi-videobridge/jitsi-videobridge.pid'
Sep 07 14:40:26 jitsi systemd[1]: jitsi-videobridge2.service: Forked /bin/bash as 13265
Sep 07 14:40:26 jitsi systemd[1]: jitsi-videobridge2.service: Changed dead -> start-post
Sep 07 14:40:26 jitsi systemd[1]: Starting Jitsi Videobridge...
-- Subject: Unit jitsi-videobridge2.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit jitsi-videobridge2.service has begun starting up.
Sep 07 14:40:26 jitsi systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/jitsi_2dvideobridge2_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=37605 reply_cookie=0 signature=sa{sv}as 
Sep 07 14:40:26 jitsi systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/jitsi_2dvideobridge2_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=37606 reply_cookie=0 signature=sa{sv}as 
Sep 07 14:40:26 jitsi systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/job/140472 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=37607 reply_cookie=0 signature=sa{sv}as error-name=n/a error-mess
Sep 07 14:40:26 jitsi systemd[1]: jitsi-videobridge2.service: User lookup succeeded: uid=999 gid=1000
Sep 07 14:40:26 jitsi systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/jitsi_2dvideobridge2_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=37608 reply_cookie=0 signature=sa{sv}as 
Sep 07 14:40:26 jitsi systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/jitsi_2dvideobridge2_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=37609 reply_cookie=0 signature=sa{sv}as 
Sep 07 14:40:26 jitsi systemd[13265]: jitsi-videobridge2.service: Failed to apply ambient capabilities (before UID change): Invalid argument
Sep 07 14:40:26 jitsi systemd[13265]: jitsi-videobridge2.service: Failed at step CAPABILITIES spawning /bin/bash: Invalid argument
-- Subject: Process /bin/bash could not be executed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The process /bin/bash could not be executed and failed.
-- 
-- The error number returned by this process is 22.

If I start manually using:

export JVB_OPTS="--apis=,"
export LOGFILE=/var/log/jitsi/foo.log
/bin/bash -c 'exec /usr/share/jitsi-videobridge/jvb.sh ${JVB_OPTS} < /dev/null >> ${LOGFILE} 2>&1'

it works without issue

This is on Ubuntu 18.04:
jitsi-videobridge2 2.1-304-g8488f77d-1

Any ideas on what I can check next to get this working? Thanks in advance.

what is the output?

egrep -v '(HOSTNAME|SECRET)' /etc/jitsi/videobridge/config

Here’s the output:

root@jitsi:/etc/jitsi/jicofo# egrep -v '(HOSTNAME|SECRET)' /etc/jitsi/videobridge/config
# Jitsi Videobridge settings

# sets the XMPP domain (default: none)

# sets the hostname of the XMPP server (default: domain if set, localhost otherwise)
JVB_HOST=

# sets the port of the XMPP server (default: 5275)
JVB_PORT=5347

# sets the shared secret used to authenticate to the XMPP server

# extra options to pass to the JVB daemon
JVB_OPTS="--apis=,"

# adds java system props that are passed to jvb (default are for home and logging config file)
JAVA_SYS_PROPS="-Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/videobridge/logging.properties"

This file seems OK. There may be an unsupported line in /lib/systemd/system/jitsi-videobridge2.service

AmbientCapabilities=CAP_NET_BIND_SERVICE ?

Commenting that line out did the trick. It looks like it’s operational now. :slight_smile:

For some reason an apt upgrade replaced that line in /lib/systemd/system/jitsi-videobridge2.service. I had to comment it out again.

Wondering if there’s something wrong with the package?

Seems this keeps happening.

Looking at some posts on stack overflow that is because you need at least a 2.6.24 kernel.

root@jitsi:/etc# uname -a
Linux jitsi 4.1.5-x86_64-linode61 #7 SMP Mon Aug 24 13:46:31 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

Happens with each upgrade of the VideoBridge.

Is CONFIG_SECURITY_FILE_CAPABILITIES enabled for your kernel?

I’m using the Linode-supplied kernel so I’m going to assume not (not sure how to check that from the command-line).

for Ubuntu it could be something like

cat /boot/config-$(uname -a | cut -d" " -f 3)
root@jitsi:~# capsh --print
Current: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read+ep
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
uid=0(root)
gid=0(root)
groups=0(root)

(unfortunately the /boot/config-* doesn’t have the running kernel in there).

well, indeed it seems that for a modern kernel this is always on and as such not even listed.
however capsh seems to show that your kernel does support it. Is your VPS a full VM or a container ?

It’s a full VM.

if it’s a VM maybe you could upgrade the kernel ?

apt-cache show linux-image* | grep Package

BTW, it’s a bit strange that with an OS released in 2018 (Ubuntu LTS numbering come from the release year after all) your kernel has been compiled in 2015 from what uname -a is saying. I have a VPS with Ubuntu 18.04 too and uname -a gives me a date in 2021…

I can manage my own kernel, yes, but I’d prefer not to. And if someone else decides not to then they’re going to have a videobridge that’s not going to work without doing a search on the error to find this discussion (as I seem to do repeatedly).

from a very quick internet search, Linode’s approach to kernel management seem to be very different from what I have seen on either Ovh, Ionos or Scaleway:

the Linode manager says that indeed their approach is not common and that:

This of course also means that you can only use features of the kernel that we've explicitly built in, whereas on distribution-supplied kernels you can load kernel modules on the fly to extend its functionality. We try to account for all of our customers' use-cases when building our kernels, but occasionally we miss things and we encourage customers to let us know if there's something they'd like us to include in our kernels. We can build and deploy new kernels pretty quickly, so ultimately I think the trade-off is worth it.

I think that your best option is to take the problem with them.

What should I tell them? That Jitsi’s systemd line seems to keep trying to enable something that I keep commenting out? :slight_smile:

you should tell them that your system is refusing a systemd feature that is accepted without problem everywhere else.