2D map with 3D sound localization for online party

We are developing a video chat tool for parties. In a party, multiple people talk at once and they not only join the circle of a talk in front of them but also some loud voices from other circles of talks.
Social VRs realizes such party. But they are not as easy as video chat and people will prefer to eat and drink during online party. So we made a prototype with lib-jitsi-meet and here is a demo. The codes are at github.
We will continue to work on this and we will need to modify serverside also. But currently, our demo uses the servers for meet.jit.si.

I have several questions:

  1. Is it OK to keep this demo public on github server ? It may give some load on the server for meet.jit.si.
  2. Does jitsi community and/or 8x8 welcome this kind of client development ?
  3. Does any one know similar products ? I know Spatial Chat but it does not have 3D sound localization.
  4. Do you interested in 3D sound localization and/or online party application ?

We will work on:
A. UI for sharing screens/images/videos. Currently, it has only video and sound.
B. Extend JVB function to reduce relay packets. Currently all packets are forwarded to all clients.
We hope that our project is accepted by and get some interest from jitsi community.

3 Likes

I would love to be able to change server side component jvb (forwarding to everyone for now) for minimizing server load. and yeah… I heard about a project like DJ party app with jitsi recently and it is still in development I am sure. I have not heard about 2D/3D sound… is it stereo? which can locate right/left? if it is, stereo property is already in server config. I also wanna know about using meet.jitsi server for client development as it can cause bandwidth load issue.

Thank you for kind reply! I’m very glad to got positive message.
For the forwarding optimization, client side will send two sets of participant IDs, which will be rendered for video and audio, to the server. I saw some code on https://github.com/jitsi/jitsi-videobridge and feel we need similar to the “pinnedEndpointIds” but different one.
For the DJ party app, I’d like to contact with the developer if possible. We may be able to help their work.
The sound localization, or 3D sound, in our app is done at client side browsers. The audio streams on the net are monaural. Treatment of stereo stream is future work. We should not localize stereo source using HRTF in client side. Just changing the attenuation will be better. So the video and audio packets are almost the same to the original client. Our prototype also uses JitsiConference’s sendCommand API to share positions and orientations of participants in the 2D map.

Hello hasevr, a little off topic but it may be you can help me.

I’m a psychiatrist based in Mexico City and I do a type of therapy that uses bilateral alternating sound for trauma healing… I also love tech and I have little knowledge in programming (very basic) and I have a platform to give BLS and assist therapists and with covid crisis this platform has been a way to keep giving this kind of therapy online until zoom canceled stereo sound and made things complex… To make things fast Im wondering if I can install jitsi on a server (that I know I can) but what I can’t solve or find out so far is if I can share a browser tab with separate stereo channel (or any other means) so that the therapist can apply BLS to the patient on the other side under the control of the therapist (play,pause,change audio source etc)…

Hopefully you can help me with this one as I’m very interested in setting this facility on my page (blstherapy.com) to continue helping therapist on their task of healing trauma.
Thanks in advance!

Thank you for your interest and suggestion about possible application.
Sorry for my late reply.
I’ve a bit busy but continue to develop
Now, basic functions work but screen sharing it not implemented yet.
We move the demo to https://jitsipartyteam.github.io/jitsi-party/