[virtual-background] How do I use the "tflite.js" file in Jitsi Open Source?

[ahn]

jitsi-meet/react/features/stream-effects/virtual-background/vendor/tflite at...

After downloading the jitsi open source react project, I ran it like this.

1. npm install
2. yarn start
   
   

Upon analyzing the project, it was using tflite files.

import createTFLiteModule from './vendor/tflite/tflite';
import createTFLiteSIMDModule from './vendor/tflite/tflite-simd';

tflite folder structure:

tflite-simd.js
tflite-simd.wasm
tflite.js
tflite.wasm

However, when I moved it to another react project and ran it, the following error occurred.

./src/features/stream-effects/virtual-background/vendor/tflite/tflite.js
  Line 9:   'read' is not defined                     no-undef
  Line 9:   'readbuffer' is not defined               no-undef
  Line 9:   'read' is not defined                     no-undef
  Line 9:   'scriptArgs' is not defined               no-undef
  Line 9:   'quit' is not defined                     no-undef
  Line 9:   Unexpected use of 'print'                 no-restricted-globals
  Line 9:   Unexpected use of 'print'                 no-restricted-globals
  Line 9:   'printErr' is not defined                 no-undef
  Line 9:   Unexpected use of 'print'                 no-restricted-globals
  Line 9:   Unexpected use of 'self'                  no-restricted-globals
  Line 9:  'dateNow' is not defined                  no-undef
  Line 18:    'define' is not defined                   no-undef
  Line 19:     Expected imports instead of AMD define()  import/no-amd
  Line 19:     'define' is not defined                   no-undef

[Questions]

1. The tflite.js file that has already been created generates the above error.
   If I want to use the ‘already created tflite.js file’ in the jitsi open source in another project, do I need a separate setting?

2. If the already created tflite.js file does not work, should I use the wasm file directly?
   (I was confused whether I should use the wasm file directly, I referred to the official document below, but this also gave me an error.)

• Documents on using wasm that I referenced:
  Loading and running WebAssembly code - WebAssembly | MDN

• Error:
  Uncaught (in promise) TypeError: Failed to execute ‘compile’ on ‘WebAssembly’: Incorrect response MIME type. Expected ‘application/wasm’.

[summary]
I don’t know how to apply the virtual-background feature.

[saghul]

You need to use both the JS and the WASM files. The JS file will load the WASM file.

I’d suggest you look into GitHub - Volcomix/virtual-background: Demo on adding virtual background to a live video stream in the browser which is where we source them from.

[ahn]

If I want to use the tflite-simd.js file in another project, do I need to do ‘npm install’ additionally?
(ex: “@jitsi/rnnoise-wasm”, “@tensorflow/tfjs-backend-wasm”)

If I look at the package.json file, I will see something related to wasm.
I wasn’t sure if I should install these as well.

[ahn]

If I just copy the tflite-simd.js file, I get an error that the variable name cannot be found.
It expects to fetch that variable from outside.
However, I did not know where and how to get it, so I asked.

[ahn]

I looked at the git commit record of the source code, and it seems that only wasm-check was installed. (And I had that file installed in my project as well.)

If so, I don’t think it’s a problem with npm install, but I don’t know what options to add.

[saghul]

We vendor those dependencies, there is no npm package for it. As I said, you need both the JS and the WASM files. The JS files just make calls into the WASM loaded one.

[ahn]

Thank you for answer.
I checked the README.md file on github you mentioned.
There it was written to do ‘yarn build:tflite’ through docker to use the wasm file.

GitHub - Volcomix/virtual-background: Demo on adding virtual background to a...

ㅤ

Is jitsi also using docker to build?
I don’t know what docker is, so I’m checking this part.

Can 'public/tflite' files be used in other projects? (tflite.wasm, tflite.js ...)

[saghul]

That is necessary in case you want to build the WASM file yourself. Otherwise you can take the prebuilt assets in the repo.

[ahn]

I couldn’t understand what the ‘tflite.js, tflite-simd.js’ files included in the link above were.
So I looked for 3 different ways to use wasm files.
But I don’t know if this way is correct.

---

1. GitHub - bedcoding/wasm_study: study ...

• Import and use the ‘tflite.js, tflite-simd.js’ files included in the jitsi open source.
• Then an error occurs, so add “eslintIgnore”: [“tflite.js”, “tflite-simd.js”] to the package.json file.
• The ‘vendor > package-1.0.1.tgz’ file is also imported and used. But I don’t know what this is.

---

2. GitHub - BhuwanChhimal/wasm-import

• Use the WebAssembly.compile() function

---

3. GitHub - BhuwanChhimal/wasm_import

• Using the as-bind library

[ahn]

At first, I reasoned that the ‘tflite.js’ file with the error would only work in jitsi’s react project.
So, I thought it wouldn’t work in a new react project.

However, through the contents of the link below, I thought that I might be wrong.

Build TFLite Wasm/SIMD and run Google Meet Virtual Background

---

↑ However, I tried to unzip the ‘tflite.wasm’ file as per the above document, but it didn’t work.
(I typed “tar -xvf tflite.wasm” on my MacBook)

---

1. The ‘tflite.js’ file is not a file created in a specific React project.
   => That is, the wasm file itself was built.

2. If so, there is a high probability that there will be no problem using the ‘tflite.js, tflite-simd.js’ files provided by jitsi in my new project.
   => But I don’t know if my reasoning is correct.

I am checking if the reason why the tflite.js file is not working is because the react version of my project is low. However, as soon as I up the react version of my project, a lot of errors are occurring. I think I need to solve this problem first to check if my reasoning is correct.

[ahn]

I imported the tflite.js file like this.
But I don’t know if this is the right way.

1. Insert 4 files in public folder.
   tflite-simd.js
   tflite-simd.wasm
   tflite.js
   tflite.wasm
   → virtual-background/public/tflite at main · Volcomix/virtual-background · GitHub

2. Call js file in index.html file in public folder
   
   → virtual-background/index.html at main · Volcomix/virtual-background · GitHub

3. Use like a global variable
   const createTFLiteModule = window.createTFLiteModule;
   const createTFLiteSIMDModule = window.createTFLiteSIMDModule;
   jitsi-meet/index.js at master · jitsi/jitsi-meet · GitHub

However, if I do this, another error occurs in the startEffect() function of ‘JitsiStreamBackgroundEffect.js’.
I’m checking the cause.

---

I saw the code below, so I put the same in the public folder of my project.

const models = {
    modelLandscape: 'libs/selfie_segmentation_landscape.tflite'
};

Afterwards, after finding out the url information through the console log, if I enter the path in the Internet address bar, the file is downloaded.

const modelResponse = await fetch(models.modelLandscape);
        console.log("ㅇㅇㅇㅇㅇmodelResponse: ", modelResponse);

        if (!modelResponse.ok) {
            throw new Error('Failed to download tflite model!');
        }

        modelBuffer = await modelResponse.arrayBuffer();
        console.log("ㅇㅇㅇㅇㅇmodelBuffer: ", modelBuffer);

        ...

↓

So this part doesn’t seem to be a problem.

I’m still checking