JVB2.1 missing endpoints in lastN list

Hi,

I’ve been testing with the jvb version 2.1-137-g07c5cc83-1. I’m seeing the bridge send our custom client a empty last-n list even when there are clients sharing their video. If other users join they will see the video but they will get an empty last-n list at some point during the meeting as well. If video is stopped then restarted all clients will get the video but again they will get a last-n list without the endpoint that is sharing video at some point during the meeting.

The client only does video so it doesn’t use active speaker currently. I’ve also disabled last-n (set to -1) and still see the same behavior. All the clients in the test are running in chrome (80)

I also disabled cc.trust-bwe and that seemed to fix the issue. But I’m not sure if that’s a best practice.
Any tips on how to debug this more?

Thanks!

We recently discovered an issue with last-n in JVB 2.1, we’ve got a PR here that we’ll probably merge soon.

I’m not sure what would be causing this, non-last-n should work fine. But did you say you’re not doing any active speaker? Maybe an event isn’t being triggered…that’s not a use case we’ve tried.

Yeah, you should definitely try and get things working without this. The fact that this works I think just suggests you either don’t have clients sending a ‘selected’ message (which is needed to receive the HD stream) or something else in the prioritization logic in BitrateController.

Thanks for the quick response.

That is correct we don’t do any active speaker event but we do send a pinned endpoint message to the bridge.

Is the ‘selected’ message in regards to simulcast? The client also is not doing simulcast.

I also did a test with only Firefox clients and there were no issues. Could this be something related to TCC vs REMB?

So you have clients not doing simulcast, last-n is turned off, clients send pinned messages and they’re not seeing any video?

They see video at the start but at some point (usually within 30mins) the consuming clients get a last-n message without the video sharers endpoint. Here is the datachannel messages my clients are seeing:

DataChannelService dc:incoming {"colibriClass":"DominantSpeakerEndpointChangeEvent","dominantSpeakerEndpoint":"2ebe0570-6547-11ea-b88d-314db5365485"}

10:25:00.336debug.js:267 DataChannelService dc:incoming {"colibriClass":"ServerHello"}

10:25:02.251debug.js:267 DataChannelService dc:incoming {"colibriClass":"LastNEndpointsChangeEvent","lastNEndpoints":["2ae1b5a0-6547-11ea-9685-f5682c3cb995"],"endpointsEnteringLastN":["2ae1b5a0-6547-11ea-9685-f5682c3cb995"],"conferenceEndpoints":["2ae1b5a0-6547-11ea-9685-f5682c3cb995"]}

10:38:03.184debug.js:267 DataChannelService dc:incoming {"colibriClass":"LastNEndpointsChangeEvent","lastNEndpoints":[],"endpointsEnteringLastN":[],"conferenceEndpoints":["2ae1b5a0-6547-11ea-9685-f5682c3cb995"]}

There is a known issue with last-n (the PR i linked above) that could be causing that, but you said this was also happening without last-n enabled?

yes I see this even without last-n enabled

So:

  1. Clients don’t do simulcast
  2. Last-n disabled
  3. Clients sending pinned endpoint messages

But no one sees any video?

That is correct.

They see the video but the video will stop anywhere from 1 minute to 40 minutes into the video share.

Do the connection stats show issues? Is there loss? Delay?

Sorry for the delay. I’m not seeing anything that stands out in the connection stats.

Consumer:

Publisher:

Those stats are from from, say, participant A when participant A is not seeing any video? The data does appear to stop being received there.

The top picture is from the participant that stops seeing video. The bottom picture is from the participant that is sending the video.

Odd. Can you include some JVB logs from when this happens? Maybe we can find something there.

Ive included a snippet of when it happens. I’m just using the default logging level. maybe a debug level would be better?jvb2.log (14.0 KB)

Yeah that doesn’t help a whole lot I guess. I’d put these in your logging.properties:

org.jitsi.videobridge.Endpoint.level=FINE
org.jitsi.videobridge.Conference.level=FINEST

And, when an endpoint stops seeing video, let it go for a minute or so and then have it leave the call, then collect the logs (we dump some extra stats when an endpoint leaves).

Sorry for the delay. I was able to get a snippet of logs from when the issue occurs with the increased verbosity. jvb2.log (861.3 KB)

One thing that’s interesting there is that it looks like you’re connected via TCP. Is that expected?

that is not expected. interesting.