[jitsi-dev] Video Rendering speed on Windows


#1

Hi,
I logged a ticket on JIRA a few days ago regarding video latency on windows which seems to be related to scaling. The ticket URL is below:-
http://java.net/jira/browse/JITSI-1079

Since logging the ticket I have grabbed the source snapshot, eclipse and had a little dig around. The problem appears to be in how the JNawtrenderer is constructed.

First removing the native dll completely on my system improved performance by around 50%. I presume this just falls back to the default AWT renderer. It did mess up the scaling in full screen mode however.

Next I found if I replaced the following code in d3d_surface.cpp:-
    ret = D3DXLoadSurfaceFromMemory(m_surface, NULL, &rect, data,
            D3DFMT_A8R8G8B8, width * 4, NULL, &rect, D3DX_FILTER_NONE, NULL);

With:-
       D3DLOCKED_RECT lockrect;
    /* load image from memory */
       ret = m_surface->LockRect(&lockrect, &rect, 0);
       if(!FAILED(ret))
       {
              memcpy(lockrect.pBits, data, width*height*4);
              ret = m_surface->UnlockRect();
       }

Performance was improved significantly again - down from around 15-20% cpu in full screen to 4-6%. However on less powerful machines full screen still seems to present an issue.

So I have dug further and discovered that I cannot do hardware scaling as the source data is already scaled when sent to the renderer.

My knowledge of Java is rusty and I have never worked extensively with either AWT or the Jitsi codebase. I am wondering if anyone can point me to which files I would need to goto to disable the scaling of the image which I believe is done with FFMPEG currently. I would like to try using Direct3d's own hw scaling as I think this would improve matters again.

Best Regards
James

Regards,

James M. Martin
Director of Development
ESP Technologies Limited - Delivering Total IT Solutions to Government
[cid:image003.jpg@01CDA0BD.4E3F5950]
Email:

JamesM@esptl.com<mailto:JamesM@esptl.com>

Website:

www.esptl.com<http://www.esptl.com>

Mobile:

+353 (0) 87 777 6422

Head Office:

Auburn House, Ard na Greine, Ennis, Co. Clare, Ireland

Dublin Office:

Phoenix Centre, Block 6, Broomhall Business Park,
Rathnew, Co. Wicklow, Ireland.

ESP Technologies Limited is registered with the Registrar of Companies, Dublin - No. 367933
Registered Address: Auburn House, Ard na Greine, Ennis, Co. Clare, Ireland

This email is intended for the addressees named above and any other use is prohibited. It may contain confidential information. If you received this email in error, please contact the sender by return email. ESP does not accept legal responsibility for the contents of this message if it has reached you via the Internet, as Internet communications are not secure. Any opinions expressed are those of the author and are not necessarily endorsed by ESP. All ESP communications are scanned for malware & virus infection using the latest anti-viral utilities.


#2

I logged a ticket on JIRA a few days ago regarding video latency on windows which seems to be related to scaling. The ticket URL is below:-
http://java.net/jira/browse/JITSI-1079

In the future, please follow our bug/issue reporting procedure as described at https://jitsi.org/index.php/Development/BugsAndIssues.

First removing the native dll completely on my system improved performance by around 50%. I presume this just falls back to the default AWT renderer. It did mess up the scaling in full screen mode however.

Yes, removing jnawtrenderer.dll will cause a fallback to a renderer from FMJ i.e. the media framework/library that we build upon and plug into.

To add to the problem with "the scaling in full screen mode" that you've noticed, a renderer other than our own will completely prevent us to provide an optimization which keeps the video data/frame out of the Java heap. So we'd like to eventually resolve issues with our renderer rather than default to an FMJ renderer.

Next I found if I replaced the following code in d3d_surface.cpp:-
    ret = D3DXLoadSurfaceFromMemory(m_surface, NULL, &rect, data,
            D3DFMT_A8R8G8B8, width * 4, NULL, &rect, D3DX_FILTER_NONE, NULL);

With:-
       D3DLOCKED_RECT lockrect;
    /* load image from memory */
       ret = m_surface->LockRect(&lockrect, &rect, 0);
       if(!FAILED(ret))
       {
              memcpy(lockrect.pBits, data, width*height*4);
              ret = m_surface->UnlockRect();
       }

Performance was improved significantly again – down from around 15-20% cpu in full screen to 4-6%. However on less powerful machines full screen still seems to present an issue.

Thank you for your feedback! Could you please add the above finding of yours to http://java.net/jira/browse/JITSI-1079? We'd very much like to further investigate your proposal but we're currently focused on preparing a stable release.

So I have dug further and discovered that I cannot do hardware scaling as the source data is already scaled when sent to the renderer.

My knowledge of Java is rusty and I have never worked extensively with either AWT or the Jitsi codebase. I am wondering if anyone can point me to which files I would need to goto to disable the scaling of the image which I believe is done with FFMPEG currently. I would like to try using Direct3d’s own hw scaling as I think this would improve matters again.

The class that you're interested in is org.jitsi.impl.neomedia.codec.video.SwScaler in the libjitsi library project. I'm afraid we cannot guarantee at this time that disabling the scaling it performs will be easy. You could begin by removing the initializations of its instances in the classes in MediaServiceImpl.java and VideoMediaDeviceSession.java.

···

On 02.10.2012, at 18:46, James M. Martin <jamesm@esptl.com> wrote:


#3

Lubomir,
Thank you for your feedback and pointers. I will indeed update the ticket with the solution I have found. I believe this is the correct method to use in this case as you will find multiple references to D3DXLoadSurfaceFromMemory on google about it being slow when used for this purpose.

LockRect is the fast way to update the surface pixels as it allows direct access, this is particularly good when your image data is in the correct format to begin with which it is with the current calling pattern. I also believe this code is relatively safe.

Removing the scaling is a different matter as you point out :slight_smile: I'm actually stuck trying to get libjitsi.jar to build and provide the same output. At the moment it does not seem to be bundling the fmj classes into the JAR and hence Jitsi refuses to load. Should I just be executing the Rebuild task from within eclipse or is there more to building libjitsi?

Regards
James

···

-----Original Message-----

From: Lubomir Marinov [mailto:lubo@sip-communicator.org] On Behalf Of Lyubomir Marinov

Sent: 02 October 2012 20:59
To: dev@jitsi.java.net
Subject: [jitsi-dev] Re: Video Rendering speed on Windows

On 02.10.2012, at 18:46, James M. Martin <jamesm@esptl.com> wrote:

I logged a ticket on JIRA a few days ago regarding video latency on windows which seems to be related to scaling. The ticket URL is below:-
http://java.net/jira/browse/JITSI-1079

In the future, please follow our bug/issue reporting procedure as described at https://jitsi.org/index.php/Development/BugsAndIssues.

First removing the native dll completely on my system improved performance by around 50%. I presume this just falls back to the default AWT renderer. It did mess up the scaling in full screen mode however.

Yes, removing jnawtrenderer.dll will cause a fallback to a renderer from FMJ i.e. the media framework/library that we build upon and plug into.

To add to the problem with "the scaling in full screen mode" that you've noticed, a renderer other than our own will completely prevent us to provide an optimization which keeps the video data/frame out of the Java heap. So we'd like to eventually resolve issues with our renderer rather than default to an FMJ renderer.

Next I found if I replaced the following code in d3d_surface.cpp:-
    ret = D3DXLoadSurfaceFromMemory(m_surface, NULL, &rect, data,
            D3DFMT_A8R8G8B8, width * 4, NULL, &rect, D3DX_FILTER_NONE, NULL);

With:-
       D3DLOCKED_RECT lockrect;
    /* load image from memory */
       ret = m_surface->LockRect(&lockrect, &rect, 0);
       if(!FAILED(ret))
       {
              memcpy(lockrect.pBits, data, width*height*4);
              ret = m_surface->UnlockRect();
       }

Performance was improved significantly again - down from around 15-20% cpu in full screen to 4-6%. However on less powerful machines full screen still seems to present an issue.

Thank you for your feedback! Could you please add the above finding of yours to http://java.net/jira/browse/JITSI-1079? We'd very much like to further investigate your proposal but we're currently focused on preparing a stable release.

So I have dug further and discovered that I cannot do hardware scaling as the source data is already scaled when sent to the renderer.

My knowledge of Java is rusty and I have never worked extensively with either AWT or the Jitsi codebase. I am wondering if anyone can point me to which files I would need to goto to disable the scaling of the image which I believe is done with FFMPEG currently. I would like to try using Direct3d's own hw scaling as I think this would improve matters again.

The class that you're interested in is org.jitsi.impl.neomedia.codec.video.SwScaler in the libjitsi library project. I'm afraid we cannot guarantee at this time that disabling the scaling it performs will be easy. You could begin by removing the initializations of its instances in the classes in MediaServiceImpl.java and VideoMediaDeviceSession.java.


#4

Oh, it's done as follows:

1. Run the Ant target rebuild (or make) of the libjitsi project. It will prepare a libjitsi.jar in the libjitsi project right next to the build.xml file.
2. Copy libjitsi.jar from the libjitsi project directory into Jitsi's lib/installer-exclude/ (overwriting the existing libjitsi.jar there).
3. Run the Ant target rebuild (or make) of the Jitsi project. It will produce a libjitsi.jar in the sc-bundles/ directory of the Jitsi project. That last one is an OSGi bundle which contains the libjitsi classes, the necessary dependencies such as FMJ and a BundleActivator.

···

On 03.10.2012, at 00:09, "James M. Martin" <jamesm@esptl.com> wrote:

Removing the scaling is a different matter as you point out :slight_smile: I'm actually stuck trying to get libjitsi.jar to build and provide the same output. At the moment it does not seem to be bundling the fmj classes into the JAR and hence Jitsi refuses to load. Should I just be executing the Rebuild task from within eclipse or is there more to building libjitsi?


#5

Thanks that worked for me. I've managed to do a hack which keeps the frame the right size until it hits D3D. I achieved this by forcing the output size set on PlayerScaler to be the same as the input video size and then changed the D3D calls to handle scaling correctly.

This seems to work very well. I have now done more tests which have been performed with the following attributes:-
1) Processor is i7-2820QM (sandybridge) at 2.3ghz.
2) Call is initiated from this machine using Bria in HD widescreen mode (1280x720) with bandwidth and quality settings set to max.
3) Recipient is Jitsi running on the same machine, only decoding and not sending video.

The results are:-

The latest nightly with no changes

···

-----------------------------------------
CPU: 14-18% (avg 15% with BIG spikes)
Sync Issues: Yes

With the LockRect modification
--------------------------------------
CPU: 5-8% (avg 7%)
Sync Issues: This machine No, on other machines Yes.

LockRect+StetchRect+LibJitsi changes
----------------------------------------------
CPU: 3-6% (avg 4%)
Sync Issues: This machine No, other machines yet to be checked.

-James

-----Original Message-----

From: Lubomir Marinov [mailto:lubo@sip-communicator.org] On Behalf Of Lyubomir Marinov

Sent: 02 October 2012 22:29
To: dev@jitsi.java.net
Subject: [jitsi-dev] Re: Video Rendering speed on Windows

On 03.10.2012, at 00:09, "James M. Martin" <jamesm@esptl.com> wrote:

Removing the scaling is a different matter as you point out :slight_smile: I'm actually stuck trying to get libjitsi.jar to build and provide the same output. At the moment it does not seem to be bundling the fmj classes into the JAR and hence Jitsi refuses to load. Should I just be executing the Rebuild task from within eclipse or is there more to building libjitsi?

Oh, it's done as follows:

1. Run the Ant target rebuild (or make) of the libjitsi project. It will prepare a libjitsi.jar in the libjitsi project right next to the build.xml file.
2. Copy libjitsi.jar from the libjitsi project directory into Jitsi's lib/installer-exclude/ (overwriting the existing libjitsi.jar there).
3. Run the Ant target rebuild (or make) of the Jitsi project. It will produce a libjitsi.jar in the sc-bundles/ directory of the Jitsi project. That last one is an OSGi bundle which contains the libjitsi classes, the necessary dependencies such as FMJ and a BundleActivator.