Framerate Not Being Set For Desktop Screensharing

Through Jitis Low-Level, I am customizing my FPS. It appears to work when I create a local tract for the user’s camera, but not for desktop. My code is as follows:

window.JitsiMeetJS.createLocalTracks({
                devices: ['desktop'],
                resolution: 1080,
                desktopSharingFrameRate: {
                    min: 60,
                    max: 90
                },
                minFps: 60,
                maxFps: 90,
                constraints : {
                    video: {
                        height: {
                            ideal: 1080,
                            max: 2160,
                            min: 240,
                        },
                        width: {
                            ideal: 1920,
                            max: 4096, 
                            min: 320,
                        },
                        frameRate: {
                            min: 60,
                            max : 90
                        }
                    },
                    audio: {
                        autoGainControl: false,
                        channelCount: 2,
                        echoCancellation: false,
                        latency: 0,
                        noiseSuppression: false,
                        sampleRate: 48000,
                        sampleSize: 16,
                        volume: 1.0
                    }
                }
                //resolution: '720'
            }).then(async (screenTracks) => {

});

My chrome://webrtc-internals look as such:

The first call is when I am calling the user camera and you will notice the FPS values and the resolution values are set in the getUserMedia call. In the second call in which the video Track is the window, nothing is set.

How do I set the FPS and other values for the desktop/screensharing?

Actually, did a little digging and I think there is a logical error. None of the normal video contraints work, but if I set an FPS using setDesktopSharingFrameRate:

room.setDesktopSharingFrameRate(90);

It sets the max rate to 90 in the chrome://webrtc-internals

I’m thinking this might be a logical error because there is no way of setting any of the other constraints such as minimum or ideal. ie, it DOES NOT except objects such as :

room.setDesktopSharingFrameRate({min: 60, max: 90});

This will NOT work and throw an invalid parameter error. Going deeper into the code:

/**
 * Configures the peerconnection so that a given framre rate can be achieved for desktop share.
 *
 * @param {number} maxFps The capture framerate to be used for desktop tracks.
 * @returns {boolean} true if the operation is successful, false otherwise.
 */

JitsiConference.prototype.setDesktopSharingFrameRate = function(maxFps) {
    if (typeof maxFps !== 'number' || isNaN(maxFps)) {
        logger.error(`Invalid value ${maxFps} specified for desktop capture frame rate`);

        return false;
    }

    this._desktopSharingFrameRate = maxFps;

    // Enable or disable simulcast for plan-b screensharing based on the capture fps.
    this.jvbJingleSession && this.jvbJingleSession.peerconnection.setDesktopSharingFrameRate(maxFps);

    // Set the capture rate for desktop sharing.
    this.rtc.setDesktopSharingFrameRate(maxFps);

    return true;
};

So looking at the above, I would say:

  1. This function is named wrong
  2. The functionality is incomplete only allowing maxFps to bet set.

Really Diving in, it seems the logic is in the ScreenObtainer class in lib-jitsi-meet. Starting with how it only allows the mutation of the maxFPS:


/**
     * Sets the max frame rate to be used for a desktop track capture.
     *
     * @param {number} maxFps capture frame rate to be used for desktop tracks.
     * @returns {void}
     */
    setDesktopSharingFrameRate(maxFps) {
        logger.info(`Setting the desktop capture rate to ${maxFps}`);

        this.options.desktopSharingFrameRate = {
            min: SS_DEFAULT_FRAME_RATE,
            max: maxFps
        };
    }
};

The default rate being an immutable const at the top:

/**

* The default frame rate for Screen Sharing.

*/

export const SS_DEFAULT_FRAME_RATE = 5;

And finally the constraints, which the majority are hard-coded:

const constraints = {
                            audio: audioConstraints,
                            video: {
                                mandatory: {
                                    chromeMediaSource: 'desktop',
                                    chromeMediaSourceId: streamId,
                                    minFrameRate: desktopSharingFrameRate?.min ?? SS_DEFAULT_FRAME_RATE,
                                    maxFrameRate: desktopSharingFrameRate?.max ?? SS_DEFAULT_FRAME_RATE,
                                    maxWidth: window.screen.width,
                                    maxHeight: window.screen.height
                                }
                            }
                        };

                        // We have to use the old API on Electron to get a desktop stream.
                        navigator.mediaDevices.getUserMedia(constraints)

The code is way too tightly coupled to a specific use case and doesn’t allow for mutability with the options for different scenarios.

High resolution desktop sharing is still a bit tricky, since it was originally devised for low FPS high resolution streams by browsers.

When you set the max of 90, what fps do you get for the stream?

@saghul Its not as tricky as it seems. You can tweak settings in all areas and here is what I got :slight_smile:

And this is at 120 fps. I think I can push it to 160/180 fps.

That might be the case today, we have been at this for a long long time, I’m not just making it up :slight_smile:

What is the source framerate? Is the game runs at 60fps there is not much point in capturing it at 120, is there?

What is the source framerate? Is the game runs at 60fps there is not much point in capturing it at 120, is there?

And that is where things get tricky. Break apart the system into 3 distinct sections and areas the developer has control over vs no control:

  • The Gamers Computer (no control)
  • The system “Jitsi, Jibri, The Bridge, FFmpeg, etc”
  • The Watcher (no control - to an extent)

If the gamers computer doesn’t support a certain FPS amount (30 fps, 60 fps, 120 etc), then the screen share quality will not go above what their computer can support. The gamer is the first bottle neck that we as the developer have no control over.

Now the watcher is a slightly different story because the developer can control how the content is delivered. We don’t want to screen share directly to them because thats a lot of bandwidth and we cannot control the quality of what their browser is able to accept. So we should live stream to them. This introduces latency but increases guarenteed quality at lower bandwidth.

Now on our systems, the developer can tweak and control the quality of the video - that’s what I meant when I said I could get this up to 180 fps. So with Jitsi, the bridge, jibri and everything else it is possible to stream or record a very high FPS and high bitrate IF the gamers computers allows it. Also protocols like VP9 and AVI do a good job at keeping quality high.

Is the game runs at 60fps there is not much point in capturing it at 120, is there?
It doesn’t hurt and the system should be setup to support users who can do higher FPS.

Looping back into the code above, it could use some modification to more easily allow for different use cases.

I see.

@jallamsetty would be more qualified here to provide guidance, but as a general rule, allowing transparent contraints for desktop sharing doesn’t sound like a bad idea.

Have you tried the desktopSharingFrameRate setting in config.js instead of passing it as options to JitsiMeetJS.createLocalTracks?

Please check the first post in the thread. It goes over all the settings that were passed into the track, including desktopSharingFrameRate. But looking at the source code on how Desktop tracks are created, this option is totally ignored.

@BingeWave Yes, I do see that the desktopSharingFrameRate options being passed to createLocalTracks are being ignored. The behavior changed after a refactor and the docs were not updated. However, setting desktopSharingFrameRate under config.js should continue to work. Did you give that a try ?

@jallamsetty In the site config, I did set the desktopSharingFrameRate.

Also for the refactor, has it been merged in? The latest version of the code still seams to be hard coded: lib-jitsi-meet/ScreenObtainer.js at master · jitsi/lib-jitsi-meet · GitHub

Also an easy way to check is using the chrome://webrtc-internals/ tools. According the tools, the parameters are not being applied except fro maxFPS.

Are you sure setting desktopSharingFrameRate in config.js doesn’t work ? I just tested it and its working as expected.
Screen Shot 2022-12-06 at 2.37.50 PM

@jallamsetty Yes, and your screenshot also is the repesnative of what I post. To be clear on the issue, the only variable that can be changed is the maxFPS. I’m thinking there should be the ability to set the other options such as:

  • Min bitrate
  • Resolution Min
  • Resolution Max
  • Ideal Resolution
  • Audio Bitrate

Constraints serve a different purpose in getDisplayMedia than they do in getUserMedia. They do not aid discovery, instead they are applied only after user-selection. So if your captured media doesn’t conform to your constraints, browser will not try to upscale or downscale it.
Setting the resolution height/width to 9999 does force Chrome to capture at the highest possible resolution and that is why those values have been hardcoded. Max fps is configurable through UI, API and through config.js. Min. fps has no effect on screen capture. All the constraints that work for camera capture don’t make sense for screen capture.

Screen Capture has the constrainable properties defined by w3c but not all of them are supported in all the browsers currently. There is no way to constrain the bitrate for screenshare. You can however pick a codec based on your b/w constraints.

1 Like

Screen Capture has the constrainable properties defined by w3c but not all of them are supported in all the browsers currently. There is no way to constrain the bitrate for screenshare. You can however pick a codec based on your b/w constraints.

The issue I am talking is about is not the current browser constraints, its that the current implementation on Jitsi Low Level is tightly coupled (The Difference Between Tight Coupling and Loose Coupling | Nordic APIs |). This prevents overriding that current implementation, limits the use cases, and doesn’t allow the leveraging of future browser APIs.

@BingeWave, the reason the current implementation is tightly coupled is because we’ve had a fair share of issues w.r.t screensharing video quality in the past and we were able to determine the best settings for the general use case after much experimentation.
Having said that, I agree developers should have an option to override the current implementation. This is something we plan to take up in the next 1-2 months based on our other product priorities so I cannot commit to a date as to when this will be available. You are more than welcome to submit a PR if you want to expedite this.