Delay with lib-jitsi-meet.js replaceTrack()

buddhiv · March 1, 2022, 5:39pm

Hi,
I am trying to implement a screen share function with lib-jitsi-meet.js. I use the following logic to replace the current video track with the new screen track.

var screenTracks = await JitsiMeetJS.createLocalTracks({
    devices: ['desktop'],
    resolution: '720'
});

var screenVideoTrack = screenTracks.filter(track => track.getType() === 'video')[0];

if (screenVideoTrack) {
    screenVideoTrack.addEventListener(JitsiMeetJS.events.track.LOCAL_TRACK_STOPPED, () => {
        //handle listener
    });

    if (localVideoTrack && localVideoTrack.stream) {
        await screenVideoTrack.setEffect(new PresenterEffect(localVideoTrack.stream));
    }

    if (localVideoTrack) {
        await room.replaceTrack(localVideoTrack, screenVideoTrack);
        await localVideoTrack.dispose()
    } else {
        await room.addTrack(screenVideoTrack);
    }
}

When the screen sharing participant creates a desktop track and replaces his own local video track in the conference, it takes around 30 seconds for the update to arrive in the other participants’ side. (Other participants to see the shared video. Until then they’ll have no content in the stream.) But the JitsiMeetJS.events.track.TRACK_VIDEOTYPE_CHANGED event fires immediately after the first participant starts the screen share.

IS THERE A P2P IMPACT?

I have tried with different number of participants, and this particular behaviour is observed only when the p2p mode is not there. (Tested with 2 or more participants and verified with chrome://webrtc-internals/)

When there are only two participants, the shared screen appear immediately on the other side.

Apparently, when the streams go through the servers, the replaceTrack function creates a huge delay in passing the new stream’s content.

Is this a known behaviour? (I tested the example application and it behaves in the same way.)
What can be done to make the application more responsive in this matter? (I suppose the asynchronous code represents exactly how the jitsi-meet handles the same behaviour with promises.)
Is there any server config that might affect this behaviour?

Thank you.

buddhiv · March 4, 2022, 4:14pm

UPDATE
Tested with the alpha.jitsi.net and beta.meet.jit.si servers. The same behaviour is there.

buddhiv · March 4, 2022, 5:36pm

UPDATE
I an using room.replaceTrack(oldTrack, newTrack);
Switching to desktop track from camera track takes more time than switching to camera track from desktop track.

room.replaceTrack(cameraTrack, desktopTrack) → around 20, 30 seconds
room.replaceTrack(desktopTrack, cameraTrack) → within 5 seconds

buddhiv · March 6, 2022, 3:27pm

REPRODUCIBLE DEMO

buddhiv · March 7, 2022, 10:57am

UPDATE

Following two methods will behave differently when adding a track to a conference using lib-jitsi-meet.

Can reproduce the above problem when written is this order.

var newScreenTracks = await JitsiMeetJS.createLocalTracks({
    devices: ['desktop'],
});
  
await room.removeTrack(localTracks[0]);
await localTracks[0].dispose();

localTracks = [];
screenTracks = newScreenTracks;
  
await room.addTrack(screenTracks[0]);
screenTracks[0].attach($(`#video-track-local`)[0]);

Adds the new desktop track without any delay when written is this order.

 
await room.removeTrack(localTracks[0]);
await localTracks[0].dispose();

var newScreenTracks = await JitsiMeetJS.createLocalTracks({
    devices: ['desktop'],
});

localTracks = [];
screenTracks = newScreenTracks;
  
await room.addTrack(screenTracks[0]);
screenTracks[0].attach($(`#video-track-local`)[0]);

damencho · March 7, 2022, 11:29am

What happens if you use replace track?

Do you reproduce the same when using jitsi-meet?

buddhiv · March 7, 2022, 11:40am

Hi, Thanks for looking at this.

replaceTrack has the same issue as I originally mentioned in the issue. In the jitsi meet instance this is not reproducible.

buddhiv · March 7, 2022, 11:42am

@damencho this is a strange behaviour that I encountered. This is the screen recording when the first code block is being used. Loom | Free Screen & Video Recording Software

You can see a noticeable delay in receiving the content of the screen track.

damencho · March 7, 2022, 11:59am

I think jitsi-meet does not use remove track - add track it uses replace track, you can try debugging it and see what is the difference with your code.

buddhiv · March 7, 2022, 12:09pm

Thanks. Yes jitsi-meet does use replaceTrack.

buddhiv:

var screenTracks = await JitsiMeetJS.createLocalTracks({
    devices: ['desktop'],
    resolution: '720'
});

var screenVideoTrack = screenTracks.filter(track => track.getType() === 'video')[0];

if (screenVideoTrack) {
    screenVideoTrack.addEventListener(JitsiMeetJS.events.track.LOCAL_TRACK_STOPPED, () => {
        //handle listener
    });

    if (localVideoTrack && localVideoTrack.stream) {
        await screenVideoTrack.setEffect(new PresenterEffect(localVideoTrack.stream));
    }

    if (localVideoTrack) {
        await room.replaceTrack(localVideoTrack, screenVideoTrack);
        await localVideoTrack.dispose()
    } else {
        await room.addTrack(screenVideoTrack);
    }
}

This is my implementation. This is the same as in the jitsi-meet. (Apart from the fact that I use async/await style instead of promises.) I reproduced the issue with this code.

Is disposing the oldTrack required?
Can there be anything specific in jitsi about the desktop tracks? Because this is only reproducible with replaceTrack when the newTrack is a desktop track.

damencho · March 7, 2022, 12:17pm

@jallamsetty any idea?

buddhiv · March 7, 2022, 1:25pm

@damencho What does Ro, Yo, Ao mean in the createLocalTracks array references? I noted that these are different in jitsi-meet and lib-jitsi-meet.

in jitsi-meet

in lib-jitsi-meet

buddhiv · March 9, 2022, 4:51am

Hi @damencho
I managed to resolve this by adding the videoQuality configs. This is the added config change. Now lib-jitsi-meet manages to show the content to the other participants without any delay after replaceTrack is called.

videoQuality: {
    preferredCodec: 'VP9',
    maxBitratesVideo: {
        VP8: {
            low: 200000,
            standard: 500000,
            high: 1500000
        },
        VP9: {
            low: 100000,
            standard: 300000,
            high: 1200000
        }
    },
}

May be the server takes time to analyze the codec when it is not specified?

Trace · April 12, 2022, 3:07pm

I am using lib-jitsi-meet and observing the same issue. Where did you add the videQuality? In the createLocalTrack options?

buddhiv · April 14, 2022, 6:11am

I set the above config in the main Jitsi config. This is the reference from the example.

(Put the above videoQuality config values in the options object.)

github.com

jitsi/lib-jitsi-meet/blob/d4682073558205289ec89720b452275c325ad652/doc/example/example.js#L3

      
        
            /* global $, JitsiMeetJS */
            
            
const options = {
                hosts: {
                    domain: 'jitsi-meet.example.com',
                    muc: 'conference.jitsi-meet.example.com' // FIXME: use XEP-0030
                },
                bosh: '//jitsi-meet.example.com/http-bind', // FIXME: use xep-0156 for that
            
            
    // The name of client node advertised in XEP-0115 'c' stanza
                clientNode: 'http://jitsi.org/jitsimeet'
            };

trix · April 21, 2022, 10:52am

Hi, I’m facing the same issue. But in my case, I’m using react-app. Whenever I replace the tracks it takes around 20-30secs before the remote user gets the stream. It’s the same even after I applied your fix.

Also, I cloned your angular app and ran it locally. it’s still the same. after clicking on stop share. the remote users cannot see the camera video immediately.

buddhiv · April 24, 2022, 7:12am

Hello,
Can I see your jitsi configuration object please?

trix · April 25, 2022, 7:20am

Hi, below is my conference object

const connectionOptions = {
  hosts: {
    domain: JITSI_SERVICE_URL,
    muc: `muc.${JITSI_SERVICE_URL}`
  },
  serviceUrl: `https://${JITSI_SERVICE_URL}/http-bind`,
  clientNode: 'http://jitsi.org/jitsimeet',

  videoQuality: {
    preferredCodec: 'VP8',
    maxBitratesVideo: {
      VP8: {
        low: 200000,
        standard: 500000,
        high: 1500000
      },
      VP9: {
        low: 100000,
        standard: 300000,
        high: 1200000
      }
    }
  }

};

buddhiv · May 2, 2022, 10:40am

let config = {
    hosts: {
        domain: 'beta.meet.jit.si',
        muc: 'conference.beta.meet.jit.si', // FIXME: use XEP-0030
    },
    disableSimulcast: false,
    constraints: {
        video: {
            height: {
                ideal: 720,
                max: 720,
                min: 180
            },
            width: {
                ideal: 1280,
                max: 1280,
                min: 320
            }
        }
    },
    enableP2P: true,
    useStunTurn: true,
    p2p: {
        enabled: true,
        enableUnifiedOnChrome: true,
        preferredCodec: 'VP9',
        disableH264: true,
        useStunTurn: true,
        stunServers: [
            { urls: 'stun:meet-jit-si-turnrelay.jitsi.net:443' }
        ],
    },
    bosh: '//beta.meet.jit.si/http-bind',
    // The name of client node advertised in XEP-0115 'c' stanza
    clientNode: 'http://jitsi.org/jitsimeet',
    videoQuality: {
        preferredCodec: 'VP9',
        maxBitratesVideo: {
            VP8: {
                low: 200000,
                standard: 500000,
                high: 1500000
            },
            VP9: {
                low: 100000,
                standard: 300000,
                high: 1200000
            }
        },
    },
}

Can you confirm if this is working?