Using Jitsi Videobridge in Upstage: some challenges we found

[Gloria_W]

Hi all. Upstage is an Open Source performance platform that has existed for over 15 years: GitHub - upstage-org/upstage: UpStage is a platform for cyberformance: remote players combine digital media in real-time for an online audience. All you need is a web browser! . In the recent rewrite, we created the concept of avatars: characters, drawings, animations, and Jitsi stream windows that can be created and dropped on stage. The audience watches the individual Jitsi Videobridge streams along with all other avatars on the stage, while the players control the avatars/streams.

We’re running into some issues, most likely because this is a nonstandard use of Jitsi. Here are notes from our front end dev, who is expert-level:

“For example, when you join a Jitsi meeting room. You got asked to allow microphone + camera access first, then join in. In UpStage, it would NOT be a meeting until some artist drags their avatar to the stage. In a traditional Jitsi meeting room there are participants. In UpStage we have players - audiences. Audiences should not be asked for camera/microphone when their role is to watch the stage. But in Videobridge, you must join in to hear/see the others, and when join in you are designed to talk/meet with them…”

I am hoping to work with some Jitsi devs to refine Videobridge to work well in this use case. If you are interested and have the time and energy, please reach out to me. We would be happy to contribute to your project if you can guide us, and work with us.

[saghul]

Hey there!

Interesting use case indeed.

Before we dive deeper, how are you interacting with the jitsi videobridge? Did you write your own REST API client? Are you using lib-jitsi-meet? Or the higher level iframe API?

[Gloria_W]

From our front end dev:

Yes we are using lib-jitsi-meet - the low level API for building individual mode. Here is what our configuration looks like:

JitsiMeetJS.init();
JitsiMeetJS.setLogLevel(JitsiMeetJS.logLevels.ERROR);
jitsi.connection = new JitsiMeetJS.JitsiConnection(null, null, {
hosts: {
domain: domain,
muc: conference.${domain},
focus: focus.${domain},
},
bosh: https://${domain}/http-bind,
});

jitsi.connection.addEventListener(
JitsiMeetJS.events.connection.CONNECTION_ESTABLISHED,
(e) => {
console.log(“Connection established”, e);
jitsi.room = jitsi.connection.initJitsiConference(stageUrl, {});
jitsi.room.on(JitsiMeetJS.events.conference.TRACK_ADDED, (track) => {
store.dispatch(“stage/addTrack”, track);
});
jitsi.room.on(JitsiMeetJS.events.conference.CONFERENCE_JOINED, (e) => {
console.log(“Conference joined”, e);
joined.value = true;
});

jitsi.room.join();

}
);
jitsi.connection.addEventListener(
JitsiMeetJS.events.connection.CONNECTION_FAILED,
(e) => {
console.error(“Connection failed”, e);
joined.value = false;
}
);
jitsi.connection.addEventListener(
JitsiMeetJS.events.connection.CONNECTION_DISCONNECTED,
(e) => {
console.error(“Connection disconnected”, e);
joined.value = false;
}
);

jitsi.connection.connect();

We do also have the higher level iframe API, as the Meeting tool. They are 2 different features: Meeting and Individual Streaming , but the Meeting is stable enough, you can focus on the low level lib-jitsi-meet only!

[saghul]

@Gloria_W

For example, when you join a Jitsi meeting room. You got asked to allow microphone + camera access first, then join in.

If you are using lib-jitsi-meet, you don’t need to create tracks before joining, that’s what will trigger the permission prompt. You could join the meeting without any tracks and add them once you decide the user will start sharing audio / video / screen.

For the iframe use case, you can use the disableInitialGUM config option, to prevent initial tracks from bring created.

[Gloria_W]

Thank you, we’ll give this a try. Maybe the tracks were causing another problem we saw, where some audience could not see some streams, and a refresh shifted the problem to other streams, or corrected it at times. It very inconsistent when there are multiple stream avatars.

Thanks again!

[saghul]

Give it a shot and let us know how it goes!

[Gloria_W]

Will do, thank you!

[Gloria_W]

Hi again,

We’re now trying to figure out what is causing the http-bind loop to happen and how to prevent it from happening in this chunk of code:

jitsi.connection = new JitsiMeetJS.JitsiConnection(null, null, {
  hosts: {
    domain: domain,
    muc: `conference.${domain}`,
    focus: `focus.${domain}`,
  },
  bosh: `https://${domain}/http-bind`,
});

Any help you could provide would be greatly appreciated.
Thank you!

[damencho]

What loop? Can you upload logs to show the error?

[Gloria_W]

From our dev:
the network tabs show a lot of http-bind requests to the Jitsi server. I’m not really sure what that endpoint does, and all about the BOSH stuff. So I cannot tell what went wrong!

[damencho]

This is completely normal. This is the xmpp communication with the server.
The js console logs, I’m asking for. Do you see anything there, what happens before you click join and after you click join?
Do you have any modifications? Does it work with latest from master?

[Gloria_W]

We have no modifications. I’ll have the dev make sure we’re using latest, but I’m pretty sure we are. More info re: joining coming soon.