Combining Web Audio API with Jitsi Audio Tracks

Developers Jitsi Meet API
#1

Hi there everyone :slight_smile:

we’re using jitsi in one of our projects and after a lot of trial and error, I’m stuck and I would love to have someone that could push me the right way.

The scenario is the following:
A person (sort of moderator) is lecturing the other participants. The video signal (webcam i.e.) from the lecturer should be send to aximmetry (for rendering him into a virtual studio) and then it should be returned to jitsi (rendered). The audio signal’s still sent via jitsi, so the video and audio signal aren’t sync anymore.

The current approach is, to add a delay to the audio (on the receiving client side). I don’t know if this is the best possible way, but at the moment, it’s all I have.

I want to add the delay via Web Audio API, but I can’t get it to work, the stream seems to be muted.

Here’s the code:
…
function _onTrackAdded(track) {
…
const _streamContainer = document.querySelector(#${_streamName});

  const _audioContext = new AudioContext();
  const _streamSource = _audioContext.createMediaStreamSource(track.stream);
  const _streamDelay = _audioContext.createDelay(2);
  const _streamDestination = _audioContext.createMediaStreamDestination();

  _streamDelay.delayTime.value = 2;

  _streamSource.connect(_streamDelay);
  _streamDelay.connect(_streamDestination);

… HOW TO ADD TO JitsiTrack?..

}
…

Combining Web Audio API with Jitsi Audio Tracks
#2

Hi @damencho , I am using the latest jitsi code on web and trying to call on an earlier sdk version in apps, but I am not able to view thw video stream.So, I wanted to know if this issue is because of the version mismatch on apps and web. I am using my own jitsi deployment

#3

@Gunika_Gupta please do not tag people in your requests for help unless they’re already helping you with the issue at hand. Also, you’d do best to create a new topic for your issue rather than hijacking an existing one.

#4

If anyone is interested, here’s the current progress (if someone faces the same problem):
I was able to add a delay via the web audio api for the receiving side, but as a strange side effect also the sending client is hearing himself now with a delay (that’s very helpful to drive someone to pure insanity within a few seconds =))

Here’s the code (for the sake of simplicity I omitted a few function arguments and further event listener calls etc.):

JitsiMeetJS.init()
connection = new JitsiMeetJS.JitsiConnection();

connection.addEventListener(JitsiMeetJS.events.connection.CONNECTION_ESTABLISHED, function() {

	room = connection.initJitsiConference();

	room.on(JitsiMeetJS.events.conference.TRACK_ADDED, function(track) {

		const _container = $(`#${id}`)[0];
		track.attach(_container);

		if(track.getType() !== 'video' && !track.isLocal()) {

			_container.muted = true;

			const _audioContext = new AudioContext();
			const _streamSource = _audioContext.createMediaStreamSource(_container.srcObject.clone())
			const _streamDelay = _audioContext.createDelay(2)
			const _streamDestination = _audioContext.createMediaStreamDestination();

			_streamDelay.delayTime.value = 2;

			_streamSource.connect(_streamDelay);
			_streamDelay.connect(_streamDestination);

			$('body').append(`<audio autoplay='1' id='${_container.id}--modified' />`);
			$(`#${_container.id}--modified`)[0].srcObject = _streamDestination.stream;
		}
	});
});

So, what I don`t understand by now is, why the remote stream is delayed too, because I cloned it and added the delay to the cloned stream oO

#5

Really, nobody?
It doesn’t need to be for free, if that’s the problem :slight_smile:

#6

You should probably create a thread in the “Paid Work” section for visibility.

#7

Done, thank you!

See link for new topic

#8

This is presumably caused by the receiving participants’ microphones picking up the delayed audio. Echo cancellation won’t work correctly once the delay is introduced. Try muting the microphones of the receiving participants.

#9

Even when we’re talking with each other through headphones? I’ll actually test it and give feedback as soon as I have results. Thank you!

#10

If you’re both using headphones then echo cancellation is not needed and the microphones should not be picking up the delayed audio (unless the headphones are turned up very loud!)