Yesterday I made some test and try to find a way to tell our brand new scaler (which use libswscale) to resize at a "user" desired resolution (instead to output at the capture resolution).
And I found one . In MediaDeviceSession::setFormat(Processor processor, Format format), format has just the encoding set (H264/RTP) and we can add the desired size in it (I can made a less intrusive way to do that by override this method in VideoMediaDeviceSession). But (yes there is a but) the results in really bad (see desktop_1280x1024_resized_to_1000x800_rgbconverter.png). The fact that the images is first resized from RGB 1280x1024 to RGB 1000x800 via JMF's RGBConverter/JavaRGBConverter and then change format via our new SwScaler.
I try to remove JavaRGBConverter/RGBConverter from JMF plugin and play with supportedInput/outputFormats but filter graph failed. So today I will look at the way that RGBConverter works and try to resize/scale in one call.
So that is the bad thing, now a better new for our SwScaler . If I hardcode output format in SwScaler (let's say 1000x800 to have right ratio with 1280x1024) I have a wonderful resized image . See desktop_1280x1024_resized_to_1000x800.png, desktop_1280x1024_resized_to_1000x800_video.png and desktop_1280x1024_resized_to_1000x800_video_fullscreen.png. (Note I use the FullScreenLayout's overlay set to true to have image fit all to the screen).
A few word in case I don't manage to make RGB to YUV rescaling with desired resolution via JMF/filtergraph stuff, we can made an accessor for setting SwScaler's outputFormat but yes, I know it is pretty bad and I will do my best to find another way.
BTW for negocating resolution in SIP/SDP we can inspire us of this ietf-draft: http://tools.ietf.org/html/draft-ietf-mmusic-image-attributes-04