[Virtual background] Apply background effect with 'jitsiTrack.setEffect()'

[ahn]

There are a total of two cases of applying virtual backgrounds in jitsiTrack.
(However, I failed the second method.)

1. Creating a video track for the first time - (I succeeded )
2. If I apply a virtual background using ‘setEffect’ to it after the video track has already been created - (I failed )
   jitsi-meet/actions.ts at master · jitsi/jitsi-meet · GitHub

---

[Questions]

If I want to change an existing video track, do I need additional steps?
(Example: Should I end the existing videotrack and then recreate a new one using the function ‘JitsiMeetJS.createLocalTracks()’?)

Or, is there something wrong with my jitsiTrack?

---

[How I tried the “1, 2” method]

1. The first method was easy to apply.
   I added an option called ‘effects’ as shown below when the video track was first created.

      const test = await _test();
      const tracks = await JitsiMeetJS.createLocalTracks(
        {
          devices,
          resolution: config.resolution || 720,
          cameraDeviceId,
          micDeviceId,
          effects: [test]
        },
        showPrompt
      );

When I check the value of [test] in the console log, I get the ‘[JitsiStreamBackgroundEffect]’.

2. In the case of the second method, the ‘jitsi open source’ was used as follows.

await jitsiTrack.setEffect(await createVirtualBackgroundEffect(virtualBackground, dispatch));

jitsi/jitsi-meet/blob/master/react/features/virtual-background/actions.ts#L27

So I tried the same for my project as well.
(I tried ‘track.setEffect()’ after getting the video track in Redux.)

const onClickTest = async () => {
    // console.log("videoTrack: ", videoTrack);
    const createVirtualBackgroundEffect = await _test();  // console.log("createVirtualBackgroundEffect: ", createVirtualBackgroundEffect);
    await videoTrack.setEffect(createVirtualBackgroundEffect);
  }

However, I am getting an error like below.

[modules/RTC/JitsiLocalTrack.js] Failed to switch to the new stream!
Track has been already disposed

[↑ If I expand the console log above]
When I checked the video track with the console log, the phrase ‘ka’ was attached at the beginning.

↓ However, the video track used by Jitsi open source has the letter ‘Zc’ attached to it.
So I was confused about the difference between ‘video track of my project’ and ‘video track of Jitsi open source’.

// https://github.com/jitsi/jitsi-meet/blob/master/react/features/virtual-background/actions.ts#L27

if (options.enabled) {
     const _createVirtualBackgroundEffect = await createVirtualBackgroundEffect(virtualBackground, dispatch)
     console.log("jitsiTrack: ", jitsiTrack);
     console.log("_createVirtualBackgroundEffect: ", _createVirtualBackgroundEffect);
                    
     await jitsiTrack.setEffect(_createVirtualBackgroundEffect);
}

I checked if jitsi open source creates a new video track.
So, after adding ‘redux-logger’, I changed the background.

But they didn’t change the video track separately.
Redux was only updating the information of the changed image.

(not change video track)

[other things i checked]

• jitsi startEffect stream: jitsi-meet/JitsiStreamBackgroundEffect.js at master · jitsi/jitsi-meet · GitHub
  

  

• my startEffect stream:
  

  

[summary]

A) An error occurs when a video track in my project uses the ‘setEffect()’ function.
(“Track has been already disposed”)

B) The “setEffect()” function works normally for the video track of “jitsi Open Source”.

C) Does Jitsi Open Source do anything special to import videotracks?

[saghul]

There is only one Jitsi

How are you trying to do this, by modifying your own Jitsi Meet instance or by using lib-jitsi-meet ?

[ahn]

1. When creating a new video track when the project first started, the virtual background was applied normally.
   when I use the JitsiMeetJS.createLocalTracks() function, the virtual background is normally applied.

2. However, there was a problem with ‘modifying a video track after it has already been created’.
   if I use jitsiTrack.setEffect(), I get an error.

Example) If you want to add a virtual background to an existing video track by clicking the Add Image button after a video track has already been created.

---

[Alternatives I think]
If I can’t use ‘jitsiTrack.setEffect()’, I’m going to try another method.

replaceTrack(oldTrack, newTrack)

• I am reviewing a way to create a new video track with a virtual background applied after removing the existing video track.
• if the existing videotrack cannot be modified with setEffect(), I would rather destroy it and recreate it.