Forcing client to always send same resolution at expense of frame rate or bit rate to stop jumping resolution?

,

Hi all,

I know the answer to this is “use an MCU and buy a more powerful server” but…

I have one user who has a poor internet connection (WIMAX based, going over the countryside to a mountain via an antenna). She gets around 4 to 5Mbps upload (20 down). There’s a possibility - maybe - of increasing it to 7… find out tomorrow but even that isn’t great.

My users are connecting for the sole purpose of recording video and/or live streaming video. The problem is - no matter what I try in the settings, sometimes her resolution changes mid-stream. I have been trying to fix it to 720/30 but may have to go lower. But I’ve even seen the odd occasion when my own resolution has dropped slightly from 1920x to 1440x, and I’m on the same 1Gbps local network as the server.

What I really need is to be able to force JVB to always send a fixed size to me regardless of anything else (ie, drop the frame rate if needed, or drop the bitrate). I appreciate it’s an SFU and doesn’t touch the data, so this may be impossible as things stand, but I thought I’d ask anyway (and no offence to jbg on the IRC channel who has given me amazing help for weeks - months? - he really knows his stuff, and we’ve discussed this in detail -I’m sure he is right but it never hurts to get a second opinion).

At the moment I save the raw mediastream data that JVB sends, so that is why the resolution changes in my saved file. On the screen, where I send it to an HDMI capture card, it stays fixed because the html5 video element automatically resizes it on-the-fly.

But capturing that seems to only capture the original stream so same problem.

The only way I’ve got it to work is to redraw the stream onto an html 5 canvas element. That keeps it full size, and I can capture that and save it to disk, but the Raspberry PI 4s I’m using for this have finally said “come on, we’re already doing enough no?” and won’t give me a good enough frame rate to make it work - even at 720, never mind 1080 which I’d prefer for the other users.

The only other thing I can think of - apart from moving to an MCU/new server etc which I don’t want to do - is if there was something I could use to take the stream and resize it on the fly before I save it without the overhead of drawing it on the screen in a canvas and capturing that data. I’ll speak to the fmmpeg guys about that.

Otherwise, is there any way with Jitisi I can say “please always send me 720 (or 480 even) - and if the internet has a blip, or the client computer is struggling for some reason - just slow down the frame rate or bitrate” ?

I’m using a local deployment of Jitisi including a new installation of the source code of JVB from github yesterday. Server is running on a mini pc ubuntu machine with 8gb.

Many thanks

Have you tried setting the resolution to 480p and disabling simulcast? Depending on the number of people you have on the call, the user with the low bandwidth may experience problems with the receive stream. Otherwise, if it’s just a handful of people on the call, it might work.

Thanks Freddie. No, I’ve not had the chance to try 480p yet. That was something jbg suggested yesterday but my colleague hasn’t been available to test today. But I have tried it with simulcast off and the resolution still drops sometimes. Whether that is the browser dropping it, or JVB, I don’t know.

I’m happy to turn it off simulcast for her - there are only a max of 5 people at once - and force her to a lower resolution, but even then I’ve seen her go down as low as 320 (it can be very briefly, but it messes up the recording). And as I said, I’ve even seen my own 1080p drop a bit on the local network sometimes so I ideally need to eliminate the possibility of this happening at all for anyone, rather than just lower the resolution, if possible.

The following constraints in your config.js should also work.

constraints: {
        video: {
             height: {
                 ideal: 480,
                 max: 480,
                 min: 480
             }
         }
     },

Be aware though that setting a fixed resolution (either through constraints of by disabling simulcast) puts the client at risk of losing video completely if their bandwidth falls below what’s required to maintain the resolution.

Thanks again Freddie. I have tried this (not at 480p yet though) but I’m sure I’ve still encountered times when the resolution has dropped down despite setting min/max/ideal to 720. I could be wrong, and I will retest today, but I’m sure it has dropped down to 640 and 320 despite simulcast being off and constraints fixed to 720. Is it possible Chrome on the senders end is making a decision during the call to send a lower resolution than first negotiated?

As for the video being lost completely, is there no way using Jitsi to tell it to simple reduce the frame rate or bitrate to maintain resolution? Obviously I appreciate if the internet is extremely bad or down, the picture will go, but I’m not sure that’s what’s happening here. It’s as if something is making a judgement (and I’ve tried BWE off too). Maybe Chrome is making the decision to send a smaller resolution?

Another alternative I’m looking at, although not sure it will work yet, is to pipe the output of mediarecorder to ffmpeg and recode to a fixed resolution. Previously I’ve been using mediarecorder and saving the chunks of data to a file with FS.write but last night I changed it to use a pipe (well, I think FS.write is technically using a pipe but I mean my own pipe, this is a new area to me) and today I’m going to test using GitHub - phaux/node-ffmpeg-stream: Node.js bindings to ffmpeg command, exposing stream based API to pipe this to ffmpeg, transcode and save the result at a fixed resolution.

The recording doesn’t need to be live, so it doesn’t need to be real-time transcoding.

But even with this method, I except freezes in the video at the point Jitsi/Chrome/whatever is changing resolution so this would at best be a “fudge” to keep everything at the size dimensions rather than solving the underlying problem. I’d still rather prefer if JVB could keep resolution and degrade framefrate or bitrate first.

Thanks

Sorry, I just realised my logic here is flawed.

Using ffmpeg to transcode as I save the stream is pointless. I can process the files afterwards to the same affect without the overhead of doing it while saving the stream. At this point in the process, it serves no real purpose.

What matters is the image on the Raspberry PI clients are full screen - and consistently full screen - for the HDMI capture out to my mac. The canvas redrawing solves this I think (again, more testing today if my colleague is free).

Where I was struggling with this solution was getting the canvas to redraw and then capture the canvas to save it to disk, instead of saving the original mediastream. It’s the capturing of the canvas that the RPi just can’t keep up with.

I guess, in order of preference, my solutions seem to be

  1. jvb always sending the RPi the same resolution mediastream, even if it means dropping the frame rate etc. But so far I can’t find any way to do this. I will try the fixed constraints again, but as I’ve said, I’m fairly certain I’ve tried that and it still dropped below sometimes. I will check again of course. This way I could display it at a fixed resolution on the RPi for HDMI capture, and save it at a fixed resolution too.

  2. jvb sending a stream that may change in resolution, but me transcoding it to a fixed resolution either on the same server I am running Jitsi, before sending it to the RPI, or on the RPI itself if it could cope, and then displaying that output full screen and saving this transcoded full screen stream.

  3. jitsi sending a stream that may change in resolution, but redrawing it full screen on the canvas, and then just saving the original stream that will change in resolution at some points, and fixing it afterwards as best I can.

  4. If it is only affecting the one user (I’m expecting all other users to have better internet - this colleague lives in the Spanish mountains), then I could use the redrawing on the canvas solution to get a consistent resolution, and then when I send the HDMI capture to my mac, record that camera input separately on the mac (the mac will be doing other things - ie, running ecamm live etc but if it’s just one, it may be ok). Absolute worst case, I could use an HDMI splitter and run it to my macbook and capture there. Not ideal of course but it is important this one user is captured ok and short of a miraculous installation of fibre (or even copper!) to her house, I simply have to work with her bandwidth.

So I think I have solutions, it’s just they are all hacky, and if I could just get JVB to always send the same resolution video (even if it’s 480) regardless of any blips in the internet and downgrade the frame rate or bit rate if needed, I could live with the results and not have to employ any of these hacks.

Thanks, and sorry for my confusion

A test today with her side locked down to 480 via the url variables worked fine except for one brief time when it blanked out, and that was when I asked her to click on the connection/network status icon to check the bandwidth etc! But it lasted a second maybe.

But I think this will do. Only prolonged testing/use will show if it ever changes size again but if not, great. 480p isn’t ideal but we have to live within our means!

Thanks for the help.

OK, done some testing and the problem is not solved.

She is using this link…

https://xxxxxxx.xxx/TestRoom#&config.constraints.video.framerate.max=25&config.constraints.video.framerate.min=15&config.constraints.video.framerate.ideal=25&config.resolution=480&config.constraints.video.width.ideal=852&config.constraints.video.width.max=852&config.constraints.video.width.min=852&config.constraints.video.height.ideal=480&config.constraints.video.height.max=480&config.constraints.video.height.min=480

When we tested it yesterday, it was fine.

But during a test now with three people (me, her and someone else), she was variously at 318x180, and 426x240. I don’t think she ever went up to 480. I thought she should go blank ?

Now her internet wasn’t as good as it can be during this period, I know that because she did a speed test just after the call and it was a bit lower than normal, but what worries me is why it is dropping down when simulcast is turned off for all in the main config file and she is set to a fixed rate in the url variables?

I’m going to speak to her internet company (the account is actually in my name anyway) and I’ve ordered her a replacement USBC to ethernet adaptor too because her wifi is actually better than the wired connection (the internet company put that wire in last week but she says the dongle is dodgy and loose).

The other caller was stable in terms of 1280x720 (macbook top resolution), but twice during the call he went “inactive” on jitsi meet front end. However, when I checked my recordings of the stream, they were fine. At the points where he goes and says “I’ve gone inactive?” etc, he is recorded fine (one was recorded by my own node.js app saving the mediastream to disk, and one recorded by Ecamm Live on the mac which is taking the HDMI out from the raspberry PI which is displaying the same mediastream in an html video element).

So that looks like Jitsi meet is deciding not to draw the stream for some reason. That’s not the end of the world as both my Ecamm feed and recording are fine, but it’s disconcerting to the guests when it happens. Any idea why it would happen and what can be done it stop it?

But more importantly, why is the fixed 480 one dropping down to 180?!

Thanks

If there is a bandwidth or CPU load issue on the client side, jitsi may drop down the resolution. Search for org.jitsi.videobridge.TRUST_BWE

BWE means "bandwitdh estimation

Thanks. Yes, I’ve previously tried that but not at this stage of tweaking. I’ll give it a try and see if it is any better.