Tutorial: How To Install FFMPEG GPU with Jibri On Ubuntu 20.04

I am working on an open-source gaming platform called Glitch: https://www.glitch.fun/ . The idea is something like Twitch but a user can restream their games to other platforms (Twitch, Facebook, Youtube). And its an open source project powered by…Jitsi!

Because its open source, here are the repos:

The Problem GPUs with FFMPEG Solves

One of the problems with streaming games is the quality of the video, and the requirements are different from standard video conferencing. The biggest difference is because of higher bitrate and preset requirements for decent video thats not choppy or pixelated, and the strain those requirements puts on CPUs. Therefore, the utilization of a GPU is night and day when streaming games for a good user experience.

In my case, Jibri is hosted on AWS on a gdad instance type, which uses AMD Radeon Pro V520 GPUs, and the drivers for Ubuntu 16/18 are either non-existent or do not work. Therefore Ubuntu 20.04 is required. To start the installation, read below.

Install Everything On An Ubuntu 18 Instance

You have to start with Ubuntu 18 because the snd_aloop does not work in Ubuntu 20 unless you change the Kernel. And when you do change the Kernel, it breaks the OS. But if you upgrade from 18 to 20, it works fine.

Let’s start with updating and installing a few packages including the Linux extra headers:

sudo apt-get update && sudo-apt get upgrade -y
sudo apt-get install -y default-jre-headless linux-image-extra-virtual curl alsa-utils icewm xdotool xserver-xorg-video-dummy ruby-hocon

Afterward, enable the snd_loop

echo "snd_aloop" >> /etc/modules
modprobe snd_aloop

Next, we have to update grub. Start by retrieving the kernel boot options:

grep -A200 submenu /boot/grub/grub.cfg |grep menuentry

You should get back something like this:

submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-e6dc6474-8016-46c6-8ab0-0a871bae4cd1' {
    menuentry 'Ubuntu, with Linux 4.4.0-1107-aws' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.4.0-1107-aws-advanced-e6dc6474-8016-46c6-8ab0-0a871bae4cd1' {
    menuentry 'Ubuntu, with Linux 4.4.0-1107-aws (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.4.0-1107-aws-recovery-e6dc6474-8016-46c6-8ab0-0a871bae4cd1' {
    menuentry 'Ubuntu, with Linux 4.4.0-1106-aws' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.4.0-1106-aws-advanced-e6dc6474-8016-46c6-8ab0-0a871bae4cd1' {
    menuentry 'Ubuntu, with Linux 4.4.0-1106-aws (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.4.0-1106-aws-recovery-e6dc6474-8016-46c6-8ab0-0a871bae4cd1' {
    menuentry 'Ubuntu, with Linux 4.4.0-1105-aws' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.4.0-1105-aws-advanced-e6dc6474-8016-46c6-8ab0-0a871bae4cd1' {
    menuentry 'Ubuntu, with Linux 4.4.0-1105-aws (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.4.0-1105-aws-recovery-e6dc6474-8016-46c6-8ab0-0a871bae4cd1' {
    menuentry 'Ubuntu, with Linux 4.4.0-179-generic' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.4.0-179-generic-advanced-e6dc6474-8016-46c6-8ab0-0a871bae4cd1' {
    menuentry 'Ubuntu, with Linux 4.4.0-179-generic (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.4.0-179-generic-recovery-e6dc6474-8016-46c6-8ab0-0a871bae4cd1' {

You are looking for two items that resemble something like this:

  1. The option that does not contain aws or generic: gnulinux-advanced-e6dc6474-8016-46c6-8ab0-0a871bae4cd1
  2. An option that contains the word generic and that resembles something like this: gnulinux-4.x.x-xxx-generic-advanced-e6dc6474-8016-46c6-8ab0-0a871bae4cd1

In the /etc/default/grub file, change the GRUB_DEFAULT as follows:

GRUB_DEFAULT="gnulinux-advanced-e6dc6474-8016-46c6-8ab0-0a871bae4cd1>gnulinux-4.4.0-179-generic-advanced-e6dc6474-8016-46c6-8ab0-0a871bae4cd1"

Update grub and reboot:

sudo update-grub
sudo reboot

When you ssh back into the server and run the following:

lsmod | grep snd_aloop

It should output the snd_aloop now. Next, let us install ffmpeg5.

sudo add-apt-repository ppa:savoury1/ffmpeg5
sudo add-apt-repository ppa:savoury1/ffmpeg4
sudo apt update
sudo apt full-upgrade

Next up, we are going to install Chrome:

curl https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo sh -c 'gpg --dearmor > /usr/share/keyrings/google-chrome-keyring.gpg'
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/google-chrome-keyring.gpg] http://dl.google.com/linux/chrome/deb/ stable main" | sudo tee /etc/apt/sources.list.d/google-chrome.list
apt-get -y update
apt-get -y install google-chrome-stable

Update the policies not to display warnings.

mkdir -p /etc/opt/chrome/policies/managed
echo '{ "CommandLineFlagSecurityWarningsEnabled": false }' >>/etc/opt/chrome/policies/managed/managed_policies.json

And then install the Chrome Driver.

CHROME_DRIVER_VERSION=`curl -sS chromedriver.storage.googleapis.com/LATEST_RELEASE`
wget -N http://chromedriver.storage.googleapis.com/$CHROME_DRIVER_VERSION/chromedriver_linux64.zip -P ~/
unzip ~/chromedriver_linux64.zip -d ~/
rm ~/chromedriver_linux64.zip
sudo mv -f ~/chromedriver /usr/local/bin/chromedriver
sudo chown root:root /usr/local/bin/chromedriver
sudo chmod 0755 /usr/local/bin/chromedriver

And finally, we are going to install Jibri:

curl https://download.jitsi.org/jitsi-key.gpg.key | sudo sh -c 'gpg --dearmor > /usr/share/keyrings/jitsi-keyring.gpg'
echo 'deb [signed-by=/usr/share/keyrings/jitsi-keyring.gpg] https://download.jitsi.org stable/' | sudo tee /etc/apt/sources.list.d/jitsi-stable.list > /dev/null
sudo apt-get update
sudo apt-get install jibri

Give Jibri permission to execute with the drivers and run as a normal user:

sudo usermod -aG adm,audio,video,plugdev jibri

This tutorial will not cover setting up the Jibri configuration file in /etc/jitsi/jibri/jibri.conf, but you should:

  1. Set up your configuration file
  2. Enable Jibri to start on the server boot

Now for the final part! Updating your distribution to Ubuntu 20.04.

sudo apt install update-manager-core
sudo do-release-upgrade

Follow the prompts and instructions the OS will give you on upgrading.

Ubuntu 20.04 Modifications and Hardware

After you have completed the upgrade, restart your server and ssh into the server. Test the snd_loop:

lsmod | grep snd_aloop

Sometimes, you will want to install drivers but might run into issues with your current Kernel version. Once again, run the command to get a list of boot items:

grep -A200 submenu /boot/grub/grub.cfg |grep menuentry

You want to find the latest generic version, something like gnulinux-5.x.x-xxx-generic-advanced-e6dc6474-8016-46c6-8ab0-0a871bae4cd1. Inside the /etc/default/grub file, replace the gnulinux-4.x.x-xxx-generic-advanced-e6dc6474-8016-46c6-8ab0-0a871bae4cd1 with the gnulinux-5.x.x-xxx-generic-advanced-e6dc6474-8016-46c6-8ab0-0a871bae4cd1 . Update grub and reboot!

sudo update-grub
sudo reboot

You should now be able to install your GPU drivers.

Using the GPU Driver with FFMPEG

This last part will vary depending on your GPU. But we need to modify FFmpeg calls. The problem is the current version of Jibri hard codes important variables such as libx264. This snippet is from Jibri source:

fun getFfmpegCommandLinux(ffmpegExecutorParams: FfmpegExecutorParams, sink: Sink): List<String> {
    return listOf(
        "ffmpeg", "-y", "-v", "info",
        "-f", "x11grab",
        "-draw_mouse", "0",
        "-r", ffmpegExecutorParams.framerate.toString(),
        "-s", ffmpegExecutorParams.resolution,
        "-thread_queue_size", ffmpegExecutorParams.queueSize.toString(),
        "-i", ":0.0+0,0",
        "-f", ffmpegExecutorParams.audioSource,
        "-thread_queue_size", ffmpegExecutorParams.queueSize.toString(),
        "-i", ffmpegExecutorParams.audioDevice,
        "-acodec", "aac", "-strict", "-2", "-ar", "44100", "-b:a", "128k",
        "-af", "aresample=async=1",
        "-c:v", "libx264", "-preset", ffmpegExecutorParams.videoEncodePreset,
        *sink.options, "-pix_fmt", "yuv420p", "-r", ffmpegExecutorParams.framerate.toString(),
        "-crf", ffmpegExecutorParams.h264ConstantRateFactor.toString(),
        "-g", ffmpegExecutorParams.gopSize.toString(), "-tune", "zerolatency",
        "-f", sink.format, sink.path
    )
}

To get around this, we are going to add a file name /usr/local/bin/ffmpeg and enter the following:

#!/bin/bash
echo ffmpeg in $0 #Comment this line after making sure, that running ffmpeg, points to this script.

ARGS=$@
ARGS=$(echo $ARGS | sed 's/-maxrate 2976k -bufsize 5952k/ -maxrate 14000k -bufsize 14000k/')
ARGS=$(echo $ARGS | sed 's/-g 120/-g 60/g')

ARGS=$(echo $ARGS | sed 's/libx264/h264_[some_encoder] -vf format=[some_format for the encoder]')
echo $ARGS >> /tmp/ffmpeg.log

exec /usr/bin/ffmpeg -[gpu_device] /dev/dri/[some_device] $ARGS

The gist of the above cod is we are using Linux string replace command “s/oldvalue/newvalue/” to find things we want to place in the ffmpeg command that Jibri will use. A lot of the variables you can tweak depending on the quality of the stream that you need but here are the key elements:

libx264/h264_[some_encoder]

Replace the [some_encoder] with the encoder you are using- vaapi, cudi, nvda, etc. In my case, it was h264_vaapi.

-vf format=[some_format for the encoder]

Typically the encoder requires some filtering and formatting. In my case I had to use -vf format=nv12|vaapi,hwupload, but everyone will differ.

-[gpu_device] /dev/dri/[some_device]

This is to the encoder select its device driver. In my case it was -vaapi_device /dev/dri/renderD128, but everyone will differ

And that’s it! Jibri will now use hardware encoding, which does wonders for processing at higher qualities with fewer resources.

1 Like

Thanks for sharing!