Combining Web Audio API with Jitsi Audio Tracks

Duplicate of https://community.jitsi.org/t/combining-web-audio-api-with-jitsi-audio-tracks/114296

The scenario is the following:

A person (sort of moderator) is lecturing the other participants. The video signal (webcam i.e.) from the lecturer should be send to aximmetry (for rendering him into a virtual studio) and then it should be returned to jitsi (rendered). The audio signal’s still sent via jitsi, so the video and audio signal aren’t sync anymore.

The current approach is, to add a delay to the audio (on the receiving client side). I don’t know if this is the best possible way, but at the moment, it’s all I have.

I was able to add a delay via the web audio api for the receiving side, but as a strange side effect also the sending client is hearing himself now with a delay (that’s very helpful to drive someone to pure insanity within a few seconds =))

Here’s the code (for the sake of simplicity I omitted a few function arguments and further event listener calls etc.):

JitsiMeetJS.init()
connection = new JitsiMeetJS.JitsiConnection();

connection.addEventListener(JitsiMeetJS.events.connection.CONNECTION_ESTABLISHED, function() {

	room = connection.initJitsiConference();

	room.on(JitsiMeetJS.events.conference.TRACK_ADDED, function(track) {

		const _container = $(`#${id}`)[0];
		track.attach(_container);

		if(track.getType() !== 'video' && !track.isLocal()) {

			_container.muted = true;

			const _audioContext = new AudioContext();
			const _streamSource = _audioContext.createMediaStreamSource(_container.srcObject.clone())
			const _streamDelay = _audioContext.createDelay(2)
			const _streamDestination = _audioContext.createMediaStreamDestination();

			_streamDelay.delayTime.value = 2;

			_streamSource.connect(_streamDelay);
			_streamDelay.connect(_streamDestination);

			$('body').append(`<audio autoplay='1' id='${_container.id}--modified' />`);
			$(`#${_container.id}--modified`)[0].srcObject = _streamDestination.stream;
		}
	});
});

I personally think this approach is problematic. Your delay estimation have a high possibility of failure. Because you have lots of unknown parameters such as network speed to/from client and also rendering device, rendering output time in rendering device, video image size/bitrate so on…

I guess, you should seek for better approach like keeping track of your video processing time and then re-sync audio and video later when it is processed.

Hi @TA_AS,
We sent you a DM. Kindly refer to that and reach us for more clarifications.
Thank you.

Best Regards,
Team Meetrix