PJSUA with Jibri

amcgowan · 2018-11-23 20:57:50 UTC

@bbaldino @damencho I posted these questions in another thread (Jibri - Chrome fails to start / PJSUA issues) but want to bring them into a new thread since the original post was chrome specific.

What version of Pjsua should be used for SIP video calls in Jibri?

How should Pjsua be compiled to ensure it supports video?

How does Pjsua capture chrome on display :0 when running on display :1 - I’m not seeing anything that would trigger Pjsua to grab data from :0 in the code and I’m not seeing any additional ffmpeg processes running.

There is a comment in the code that indicates that Pjsua runs on display :1 selenium is on display :0 and there are ffmpeg processes to connect this. Why does Pjsua need a display if it’s a cli tool? How do I get the ffpmeg processes running to connect to Pjsua?

bbaldino · 2018-11-26 18:21:16 UTC

@Aaron_van_Meerten can probably be helpful with these, as many are related to the setup of Pjsua. I’m pretty sure we just install PJsua from apt, so I don’t know that any special compilation is needed and whatever the stable version is in should be fine I’d think?

amcgowan · 2018-11-26 19:15:46 UTC

I was directed to the fork of pjsua https://github.com/jitsi/pjproject. Looks like it is configured to build with video support. Giving that a shot.

When looking into installation instructions for Pjsua all I could find was directions for compiling it. Couldn’t find any indication of any packages available for apt.

bbaldino · 2018-11-26 19:23:15 UTC

Oops, my bad…don’t think I knew about that! Ok let us know.

amcgowan · 2018-11-27 22:45:21 UTC

I’ve got the forked PJProject library compiled and installed. I have the video menu option enabled now when I run Pjsua manually.

I’m still having issues getting Pjsua to send video.

When Pjsua starts up I see the following in the logs:

22:37:49.384    pjsua_vid.c  ..Initializing video subsystem..
22:37:49.386     v4l2_dev.c  ...Video4Linux2 has 0 devices
22:37:49.411      sdl_dev.c  ...SDL 2.0 initialized
22:37:49.411 colorbar_dev.c  ...Colorbar video src initialized with 2 device(s):
22:37:49.411 colorbar_dev.c  ... 0: Colorbar generator
22:37:49.411 colorbar_dev.c  ... 1: Colorbar-active
22:37:49.411 sip_endpoint.c  .Module "mod-evsub" registered

When I check the video device list in pjsua this is the ouput:

22:41:34.647 pjsua_app_comm  Video device list:
22:41:34.647 pjsua_app_comm  3 device(s) detected:
22:41:34.647 pjsua_app_comm   -2 SDL renderer [SDL][render] (default renderer device)
22:41:34.647 pjsua_app_comm      Supported capabilities: format, window, resize, wndflags
22:41:34.647 pjsua_app_comm      Supported formats: RGBA, RGB24, BGRA, DIB , YUY2, UYVY, YVYU, I420, YV12, I420JPG, I422JPG
22:41:34.647 pjsua_app_comm   -1 Colorbar generator [Colorbar][capture] (default capture device)
22:41:34.647 pjsua_app_comm      Supported capabilities: format
22:41:34.647 pjsua_app_comm      Supported formats: YUY2, UYVY, YVYU, RGBA, RGB24, BGRA, YV12, I420, I422, I420JPG, I422JPG
22:41:34.647 pjsua_app_comm    0 SDL renderer [SDL][render] 
22:41:34.647 pjsua_app_comm      Supported capabilities: format, window, resize, wndflags
22:41:34.647 pjsua_app_comm      Supported formats: RGBA, RGB24, BGRA, DIB , YUY2, UYVY, YVYU, I420, YV12, I420JPG, I422JPG
22:41:34.647 pjsua_app_comm    1 Colorbar generator [Colorbar][capture] 
22:41:34.647 pjsua_app_comm      Supported capabilities: format
22:41:34.647 pjsua_app_comm      Supported formats: YUY2, UYVY, YVYU, RGBA, RGB24, BGRA, YV12, I420, I422, I420JPG, I422JPG
22:41:34.647 pjsua_app_comm    2 Colorbar-active [Colorbar][capture] 
22:41:34.647 pjsua_app_comm      Supported capabilities: format
22:41:34.647 pjsua_app_comm      Supported formats: YUY2, UYVY, YVYU, RGBA, RGB24, BGRA, YV12, I420, I422, I420JPG, I422JPG

From that, which video device would map back to what’s currently on display :0? I am able to record that display using ffmpeg, but cannot seem to get Pjsua to use that display during a sip call.

In my psjua.config file it has --vcapture-dev=-1 and I see that Jibri is hardcoded to send --capture-dev=23 and --playback-dev=24 would these be correct for all environments or something that needs modification? How do I determine which device ids I should be using?

amcgowan · 2018-11-28 18:18:22 UTC

@Aaron_van_Meerten @bbaldino how is Pjsua instructed to access the video on display :0? If I use FFmpeg I can capture the display, but Pjsua doesn’t appear to be able to use it as a capture device.

bbaldino · 2018-11-28 19:57:46 UTC

--capture-dev and --playback-dev I believe are the original flags used for audio (see here). I’m guessing we override the --vcapture-dev in our config. @Aaron_van_Meerten can probably confirm.

amcgowan · 2018-11-28 20:20:28 UTC

Is there a way to confirm what the vcap device is supposed to be set to, or if the device is actually available? Right now I’m not sure if I’ve set the right number for pjsua, or if it can’t access it for some other reason.

amcgowan · 2018-11-29 00:58:25 UTC

@damencho @Aaron_van_Meerten what is the magic glue that enables Pjsua video of the display that that Chrome is running on? There must be something I’m missing as every device I try to use ends up reporting as inactive:

16:27:56.623          pjsua_media.c  ......Call 0: stream #1 (video) unchanged.
16:27:56.623          pjsua_media.c  ......Video updated, stream #1:  (inactive)
16:27:56.623            pjsua_app.c  .....Call 0 media 1 [type=video], status is None

These are the processes that I can see that are running under the Jibri user:

jibri     2108  0.5  1.2 111684 12892 ?        Sl   16:34   0:00 /usr/local/bin/chromedriver --port=24686 --log-path=/tmp/chromedriver.log
jibri     2120 40.3 15.7 838100 159748 ?       Rl   16:34   0:08 /usr/bin/google-chrome --alsa-input-device=plughw:1,1 --alsa-output-device=plug:amix --disable-background-networking --disable-client-side-phishing-detection --disable-default-apps --disable-hang-monitor --disable-infobars --disable-popup-blocking --disable-prompt-on-repost --disable-sync --disable-web-resources --enable-automation --enable-logging --enabled --force-fieldtrials=SiteIsolationExtensions/Control --ignore-certificate-errors --kiosk --load-extension=/tmp/.org.chromium.Chromium.SHqsne/internal --log-level=0 --metrics-recording-only --no-first-run --password-store=basic --remote-debugging-port=0 --start-maximized --test-type=webdriver --use-fake-ui-for-media-stream --use-mock-keychain --user-data-dir=/tmp/.org.chromium.Chromium.w9c3vb --vmodule=*=3 data:,
jibri     2125  0.0  0.0   7444   692 ?        S    16:34   0:00 cat
jibri     2126  0.0  0.0   7444   660 ?        S    16:34   0:00 cat
jibri     2129  0.1  4.7 410400 47764 ?        S    16:34   0:00 /opt/google/chrome/chrome --type=zygote --enable-logging --log-level=0 --vmodule=*=3 --user-data-dir=/tmp/.org.chromium.Chromium.w9c3vb
jibri     2130  0.0  0.4  25792  4196 ?        S    16:34   0:00 /opt/google/chrome/nacl_helper
jibri     2133  0.0  1.3 410400 13604 ?        S    16:34   0:00 /opt/google/chrome/chrome --type=zygote --enable-logging --log-level=0 --vmodule=*=3 --user-data-dir=/tmp/.org.chromium.Chromium.w9c3vb
jibri     2157  1.0  8.6 564432 87624 ?        Sl   16:34   0:00 /opt/google/chrome/chrome --type=gpu-process --field-trial-handle=8431131300081281736,14078744098482537308,131072 --enable-logging --log-level=0 --vmodule=*=3 --user-data-dir=/tmp/.org.chromium.Chromium.w9c3vb --gpu-preferences=KAAAAAAAAACAAABAAQAAAAAAAAAAAGAAAAAAAAAAAAAIAAAAAAAAAAgAAAAAAAAA --user-data-dir=/tmp/.org.chromium.Chromium.w9c3vb --enable-logging --log-level=0 --vmodule=*=3 --service-request-channel-token=7724248558749148209
jibri     2369  0.2  6.8 654556 69456 ?        Sl   16:34   0:00 /opt/google/chrome/chrome --type=renderer --enable-automation --enable-logging --log-level=0 --test-type=webdriver --use-fake-ui-for-media-stream --vmodule=*=3 --field-trial-handle=8431131300081281736,14078744098482537308,131072 --disable-gpu-compositing --service-pipe-token=18291684356167916442 --lang=en-US --user-data-dir=/tmp/.org.chromium.Chromium.w9c3vb --extension-process --disable-client-side-phishing-detection --enable-offline-auto-reload --enable-offline-auto-reload-visible-only --num-raster-threads=1 --service-request-channel-token=18
jibri     2557 63.0 22.9 957368 232372 ?       Rl   16:34   0:12 /opt/google/chrome/chrome --type=renderer --enable-automation --enable-logging --log-level=0 --test-type=webdriver --use-fake-ui-for-media-stream --vmodule=*=3 --field-trial-handle=8431131300081281736,14078744098482537308,131072 --disable-gpu-compositing --service-pipe-token=8267568253464799960 --lang=en-US --user-data-dir=/tmp/.org.chromium.Chromium.w9c3vb --disable-client-side-phishing-detection --enable-offline-auto-reload --enable-offline-auto-reload-visible-only --num-raster-threads=1 --service-request-channel-token=8267568253464799960 --r
jibri     2580  1.7  1.0 555680 10872 ?        Sl   16:34   0:00 /usr/bin/pulseaudio --start --log-target=syslog
jibri     2621 41.5  2.3 441800 24132 ?        SLl  16:34   0:04 pjsua --capture-dev=23 --playback-dev=24 --id TestSip123 <sip:jibri@127.0.0.1> --config-file /home/jibri/pjsua.config --log-file /tmp/pjsua.out sip:mcgowan@xxxxxx
mcgowan   2641  0.0  0.1  14228  1092 pts/0    S+   16:34   0:00 grep --color=auto jibri
jibri    32611  0.1  4.7 422152 47624 ?        Ssl  16:08   0:02 /usr/lib/xorg/Xorg -nocursor -noreset +extension RANDR +extension RENDER -logfile /tmp/xorg.log -config /etc/jitsi/jibri/xorg-video-dummy.conf :0
jibri    32679  0.0  0.0   6672   740 ?        Ss   16:10   0:00 /usr/bin/icewm-session
jibri    32685  0.0  0.5  76928  5504 ?        SNs  16:10   0:00 icewmbg
jibri    32686  0.0  1.0  93016 10188 ?        Ss   16:10   0:00 icewm --notify
jibri    32688  0.0  0.5  80120  5816 ?        Ss   16:10   0:00 icewmtray --notify
jibri    32766  0.8 14.4 2695668 146448 ?      Ssl  16:11   0:12 java -Djava.util.logging.config.file=/etc/jitsi/jibri/logging.properties -jar /opt/jitsi/jibri/jibri.jar --config /etc/jitsi/jibri/config.json

This is the pjsua.config file:

--video
--vcapture-dev=1
--no-color
--log-level=4
--app-log-level=4
--nameserver 8.8.8.8
--auto-update-nat 0
--disable-stun
--no-tcp
--dis-codec GSM
--dis-codec H263
--dis-codec iLBC
--dis-codec G722
--dis-codec speex
--dis-codec pcma
--dis-codec pcmu
--dis-codec opus
--add-codec pcmu
--add-codec pcma
--add-codec speex
--add-codec G722
--add-codec opus
--no-vad
--ec-tail 0
--quality 10
--capture-dev=-1
--playback-dev=-1

I’ve also tried running Pjsua outside of Jibri to see how one would attempt to use an xwindow as a capture device. The closest I’ve gotten is sending a green box. But this is not what is on the display, as when I just capture the display at random even with nothing running, the background is a cold blue.

It seems like there is something missing here to pipe the display that chrome is using to Pjsua. I was looking into Video4Linux, and looks like you could set up a null video device and then use ffmpeg to do an x11grab and pipe that out to the null device. Pjsua since it uses video4linux would then see that as a webcam capture device.

Is that what would be necessary to get video to work over sip using jibri? Or is there something fundamentally wrong with my install? (I’ve tried to set this up on 3 different versions of linux now - one an AWS debian instance, ubuntu 18 desktop (just to see if pjsua could capture the screen in a desktop environment), and ubuntu 16.04 to match what is indicated in the Jibri installation instructions.

Note that I’ve tried both installing from the Jibri source, and from the Jibri package. I’ve also used the latest PJProject 2.8, and the Jitsi fork of PJProject - end up with the same results in all scenarios.

amcgowan · 2018-12-03 20:40:38 UTC

To be completely clear, the red connections in the diagram below is what I either do not have configured, don’t understand, or are just missing from Jibri currently.

@Aaron_van_Meerten @damencho where should I investigate to determine how these components interact?

@Aaron_van_Meerten @damencho any additional details on this? From what I can tell, I’m either misunderstanding how Jibri works, or it’s missing functionality.

I’m working on implementing the piece that I think is missing, but would really like to get some confirmation first to make sure I don’t go down this path if something already exists to handle this.

Carlos_Aleman · 2018-12-11 14:04:59 UTC

Hi Team,

I am stucked in this same point. I managed to place a call from Jibri-pjsua but I can´t see any video input/output.

Carlos_Aleman · 2019-01-15 19:36:42 UTC

Could you please share some thoughts on this topic ?? Did someone could succeed to configure Jibri and video with pjsua ?? I am using two different VM forJitis and Jibri stuff

amcgowan · 2019-01-15 21:46:41 UTC

I had no success and ended up having to abandon the project and start developing a custom client to handle video.

Carlos_Aleman · 2019-01-16 10:21:45 UTC

Thnaks amcgowan, that would probably be the best idea