[jitsi-dev] Re: Video Rendering speed on Windows - Update & Question


#1

Lubomir,

Since the last email I have done more work on this issue and some real world tests. Firstly I have managed to get solve the problem with the video frame being transited through the Java heap. This was surprisingly easy, it involves the following steps:-

1. Add AVFrameFormat to the list of supported input codecs for the JAWTRenderer.

2. Alter the JAWTRender Java logic to pass the AVFrame pointer rather than the frame buffer

3. Add support for directly rendering the frame from the AVFrame structure to the native renderer dll.

This approach appears to achieve the aim of never converting the frame from its native representation until the point of blitting it to the screen while not breaking the existing architecture.

I am about to investigate a similar approach for the JNIEncoder which is still receiving a frame which has passed through the SwScaler.

As for performance the real world improvement is dramatic. Jitsi was previous only usable on the highest spec machines, on all other machines it was only reliable for audio only calls or very low definition video.

The patched Jitsi is however consistently outperforming the commercial SIP client that we have been using up until now. We are managing 720p @ 30fps on most configurations available to us. One machine required the configuration to be dropped to 640x480 but then an excellent quality call continued for several hours :).

The conclusion for us is that Jitsi is now a realistic replacement for the commercial product we have been using. We are now aiming to do 1080p as combined with decent conference microphones this allows "telepresence" without usual price tag :).

So I have two questions.

Question 1 - Call statistics

Somehow I have managed to disable the video information (jitter, packet loss and encodings) from being propagated to the call information panel.

If possible I would appreciate a quick pointer on how this information is propagated out from libjitsi. E.g. where I should start looking to fix this.

Question 2 - Patches/Contributing

We are very keen that our work can be integrated into the mainline Jitsi client. However it is clear that a great deal of care and attention has been gone into Jitsi and we do not want to "step on toes" by not understanding your design or having different development goals.

At present the changes are "unfinished" code but I would like to understand what the usual procedure is and how best we can facilitate getting the finished code included. Do you have a code standards document for example?

I should note that from our point of view the current nightly build and release version of Jitsi do not function well in regard to video calling on Windows except when conditions are excellent.

As a result I strongly believe that at least a partial solution should be considered for Jitsi 1.1 (or is it 2.0?). One suggestion is I can provide some direct support to one of the dev team regarding a quick fix either on IRC or phone.

Best Regards

James

Regards,

James M. Martin
Director of Development
ESP Technologies Limited - Delivering Total IT Solutions to Government
[cid:image001.jpg@01CDA70A.6F1CEA90]
Website:

www.esptl.com<http://www.esptl.com>

Head Office:

Auburn House, Ard na Greine, Ennis, Co. Clare, Ireland

Dublin Office:

Phoenix Centre, Block 6, Broomhall Business Park,
Rathnew, Co. Wicklow, Ireland.

ESP Technologies Limited is registered with the Registrar of Companies, Dublin - No. 367933
Registered Address: Auburn House, Ard na Greine, Ennis, Co. Clare, Ireland

This email is intended for the addressees named above and any other use is prohibited. It may contain confidential information. If you received this email in error, please contact the sender by return email. ESP does not accept legal responsibility for the contents of this message if it has reached you via the Internet, as Internet communications are not secure. Any opinions expressed are those of the author and are not necessarily endorsed by ESP. All ESP communications are scanned for malware & virus infection using the latest anti-viral utilities.


#2

Hey James,

We are currently busy finalizing support for multi-party video
conferences and we'll come back to this later (probably early next
week). A few quick answers below:

Lubomir,

Since the last email I have done more work on this issue and some real
world tests. Firstly I have managed to get solve the problem with the
video frame being transited through the Java heap. This was surprisingly
easy, it involves the following steps:-

1. Add AVFrameFormat to the list of supported input codecs for the
JAWTRenderer.

2. Alter the JAWTRender Java logic to pass the AVFrame pointer
rather than the frame buffer

3. Add support for directly rendering the frame from the AVFrame
structure to the native renderer dll.

This approach appears to achieve the aim of never converting the frame
from its native representation until the point of blitting it to the
screen while not breaking the existing architecture.

I am about to investigate a similar approach for the JNIEncoder which is
still receiving a frame which has passed through the SwScaler.

As for performance the real world improvement is dramatic. Jitsi was
previous only usable on the highest spec machines, on all other machines
it was only reliable for audio only calls or very low definition video.

The patched Jitsi is however consistently outperforming the commercial
SIP client that we have been using up until now. We are managing 720p @
30fps on most configurations available to us. One machine required the
configuration to be dropped to 640x480 but then an excellent quality
call continued for several hours J.

The conclusion for us is that Jitsi is now a realistic replacement for
the commercial product we have been using. We are now aiming to do
1080p as combined with decent conference microphones this allows
“telepresence” without usual price tag J.

So I have two questions.

Question 1 – Call statistics

Somehow I have managed to disable the video information (jitter, packet
loss and encodings) from being propagated to the call information panel.

If possible I would appreciate a quick pointer on how this information
is propagated out from libjitsi. E.g. where I should start looking to
fix this.

org.jitsi.service.neomedia.MediaStreamStats and
org.jitsi.impl.neomedia.MediaStreamStatsImpl

Question 2 – Patches/Contributing

We are very keen that our work can be integrated into the mainline Jitsi
client.

Glad to hear this. We've been looking at hardware decoding for a while
now. I even think Seb has made some progress on this but I think he does
not currently have the time to continue. Seb, feel free to chime in.

However it is clear that a great deal of care and attention has
been gone into Jitsi

Thanks for noticing :slight_smile:

and we do not want to “step on toes” by not
understanding your design or having different development goals.

At present the changes are “unfinished” code but I would like to
understand what the usual procedure is and how best we can facilitate
getting the finished code included.

That's it actually. Discussing your intentions here and agreeing on a
solution is the first step (I apologise again that you'll need to wait
until next week for answers).

Next step is sending a patch:
    https://jitsi.org/faq/patch

(The first time you do this we'll ask you to sign our contributor
agreement: http://bluejimp.com/bca.pdf)

Eventually, after several patch iterations you'll have the option of
getting commit access:

    https://jitsi.org/commitaccess/

Do you have a code standards document for example?

    Yes:

    https://jitsi.org/codeconvention

Cheers,
Emil

···

On 10.10.12, 18:14, James M. Martin wrote:

I should note that from our point of view the current nightly build and
release version of Jitsi do not function well in regard to video calling
on Windows except when conditions are excellent.

As a result I strongly believe that at least a partial solution should
be considered for Jitsi 1.1 (or is it 2.0?). One suggestion is I can
provide some direct support to one of the dev team regarding a quick fix
either on IRC or phone.

Best Regards

James

Regards,

James M. Martin
Director of Development
ESP Technologies Limited - Delivering Total IT Solutions to Government

Website:

www.esptl.com <http://www.esptl.com>

Head Office:

Auburn House, Ard na Greine, Ennis, Co. Clare, Ireland

Dublin Office:

Phoenix Centre, Block 6, Broomhall Business Park,
Rathnew, Co. Wicklow, Ireland.

ESP Technologies Limited is registered with the Registrar of Companies,
Dublin - No. 367933
Registered Address: Auburn House, Ard na Greine, Ennis, Co. Clare, Ireland

This email is intended for the addressees named above and any other use
is prohibited. It may contain confidential information. If you received
this email in error, please contact the sender by return email. ESP does
not accept legal responsibility for the contents of this message if it
has reached you via the Internet, as Internet communications are not
secure. Any opinions expressed are those of the author and are not
necessarily endorsed by ESP. All ESP communications are scanned for
malware & virus infection using the latest anti-viral utilities.

--
Emil Ivov, Ph.D. 67000 Strasbourg,
Project Lead France
Jitsi
emcho@jitsi.org PHONE: +33.1.77.62.43.30
https://jitsi.org FAX: +33.1.77.62.47.31


#3

Emil,

Many thanks for your quick response. I am *delighted* to hear the news about multi-party video. You can be assured we will put this through its paces heavily as soon as it's functional. This is an extremely important feature to us.

Have read quickly the URLs you provided, nothing that worries me on first read. Once we have a legal opinion on the contributor agreement I'll arrange it to be signed and return to you this week/next. I'll correspond privately with you on this.

Feel free to drop me a line next week when it suits.

Best Regards

James

···

-----Original Message-----

From: Emil Ivov [mailto:emcho@jitsi.org]

Sent: 10 October 2012 18:14
To: dev@jitsi.java.net
Cc: James M. Martin
Subject: Re: [jitsi-dev] Re: Video Rendering speed on Windows - Update & Question

Hey James,

We are currently busy finalizing support for multi-party video conferences and we'll come back to this later (probably early next week). A few quick answers below:

On 10.10.12, 18:14, James M. Martin wrote:

Lubomir,

Since the last email I have done more work on this issue and some real

world tests. Firstly I have managed to get solve the problem with

the video frame being transited through the Java heap. This was

surprisingly easy, it involves the following steps:-

1. Add AVFrameFormat to the list of supported input codecs for the

JAWTRenderer.

2. Alter the JAWTRender Java logic to pass the AVFrame pointer

rather than the frame buffer

3. Add support for directly rendering the frame from the AVFrame

structure to the native renderer dll.

This approach appears to achieve the aim of never converting the frame

from its native representation until the point of blitting it to the

screen while not breaking the existing architecture.

I am about to investigate a similar approach for the JNIEncoder which

is still receiving a frame which has passed through the SwScaler.

As for performance the real world improvement is dramatic. Jitsi was

previous only usable on the highest spec machines, on all other

machines it was only reliable for audio only calls or very low definition video.

The patched Jitsi is however consistently outperforming the commercial

SIP client that we have been using up until now. We are managing 720p

@ 30fps on most configurations available to us. One machine required

the configuration to be dropped to 640x480 but then an excellent

quality call continued for several hours J.

The conclusion for us is that Jitsi is now a realistic replacement for

the commercial product we have been using. We are now aiming to do

1080p as combined with decent conference microphones this allows

"telepresence" without usual price tag J.

So I have two questions.

Question 1 - Call statistics

Somehow I have managed to disable the video information (jitter,

packet loss and encodings) from being propagated to the call information panel.

If possible I would appreciate a quick pointer on how this information

is propagated out from libjitsi. E.g. where I should start looking to

fix this.

org.jitsi.service.neomedia.MediaStreamStats and org.jitsi.impl.neomedia.MediaStreamStatsImpl

Question 2 - Patches/Contributing

We are very keen that our work can be integrated into the mainline

Jitsi client.

Glad to hear this. We've been looking at hardware decoding for a while now. I even think Seb has made some progress on this but I think he does not currently have the time to continue. Seb, feel free to chime in.

However it is clear that a great deal of care and attention has been

gone into Jitsi

Thanks for noticing :slight_smile:

and we do not want to "step on toes" by not understanding your design

or having different development goals.

At present the changes are "unfinished" code but I would like to

understand what the usual procedure is and how best we can facilitate

getting the finished code included.

That's it actually. Discussing your intentions here and agreeing on a solution is the first step (I apologise again that you'll need to wait until next week for answers).

Next step is sending a patch:

    https://jitsi.org/faq/patch

(The first time you do this we'll ask you to sign our contributor

agreement: http://bluejimp.com/bca.pdf)

Eventually, after several patch iterations you'll have the option of getting commit access:

    https://jitsi.org/commitaccess/

Do you have a code standards document for example?

    Yes:

    https://jitsi.org/codeconvention

Cheers,

Emil

I should note that from our point of view the current nightly build

and release version of Jitsi do not function well in regard to video

calling on Windows except when conditions are excellent.

As a result I strongly believe that at least a partial solution should

be considered for Jitsi 1.1 (or is it 2.0?). One suggestion is I can

provide some direct support to one of the dev team regarding a quick

fix either on IRC or phone.

Best Regards

James

Regards,

James M. Martin

Director of Development

ESP Technologies Limited - Delivering Total IT Solutions to Government

Website:

www.esptl.com<http://www.esptl.com> <http://www.esptl.com>

Head Office:

Auburn House, Ard na Greine, Ennis, Co. Clare, Ireland

Dublin Office:

Phoenix Centre, Block 6, Broomhall Business Park, Rathnew, Co.

Wicklow, Ireland.

ESP Technologies Limited is registered with the Registrar of

Companies, Dublin - No. 367933 Registered Address: Auburn House, Ard

na Greine, Ennis, Co. Clare, Ireland

This email is intended for the addressees named above and any other

use is prohibited. It may contain confidential information. If you

received this email in error, please contact the sender by return

email. ESP does not accept legal responsibility for the contents of

this message if it has reached you via the Internet, as Internet

communications are not secure. Any opinions expressed are those of the

author and are not necessarily endorsed by ESP. All ESP communications

are scanned for malware & virus infection using the latest anti-viral utilities.

--

Emil Ivov, Ph.D. 67000 Strasbourg,

Project Lead France

Jitsi

emcho@jitsi.org<mailto:emcho@jitsi.org> PHONE: +33.1.77.62.43.30

https://jitsi.org FAX: +33.1.77.62.47.31


#4

Hi,

···

Le mercredi 10 octobre 2012, Emil Ivov <emcho@jitsi.org> a écrit :

Hey James,

We are currently busy finalizing support for multi-party video
conferences and we'll come back to this later (probably early next
week). A few quick answers below:

On 10.10.12, 18:14, James M. Martin wrote:

Lubomir,

Since the last email I have done more work on this issue and some real
world tests. Firstly I have managed to get solve the problem with the
video frame being transited through the Java heap. This was surprisingly
easy, it involves the following steps:-

1. Add AVFrameFormat to the list of supported input codecs for the
JAWTRenderer.

2. Alter the JAWTRender Java logic to pass the AVFrame pointer
rather than the frame buffer

3. Add support for directly rendering the frame from the AVFrame
structure to the native renderer dll.

This approach appears to achieve the aim of never converting the frame
from its native representation until the point of blitting it to the
screen while not breaking the existing architecture.

I am about to investigate a similar approach for the JNIEncoder which is
still receiving a frame which has passed through the SwScaler.

As for performance the real world improvement is dramatic. Jitsi was
previous only usable on the highest spec machines, on all other machines
it was only reliable for audio only calls or very low definition video.

The patched Jitsi is however consistently outperforming the commercial
SIP client that we have been using up until now. We are managing 720p @
30fps on most configurations available to us. One machine required the
configuration to be dropped to 640x480 but then an excellent quality
call continued for several hours J.

The conclusion for us is that Jitsi is now a realistic replacement for
the commercial product we have been using. We are now aiming to do
1080p as combined with decent conference microphones this allows
“telepresence” without usual price tag J.

So I have two questions.

Question 1 – Call statistics

Somehow I have managed to disable the video information (jitter, packet
loss and encodings) from being propagated to the call information panel.

If possible I would appreciate a quick pointer on how this information
is propagated out from libjitsi. E.g. where I should start looking to
fix this.

org.jitsi.service.neomedia.MediaStreamStats and
org.jitsi.impl.neomedia.MediaStreamStatsImpl

Question 2 – Patches/Contributing

We are very keen that our work can be integrated into the mainline Jitsi
client.

Glad to hear this. We've been looking at hardware decoding for a while
now. I even think Seb has made some progress on this but I think he does
not currently have the time to continue. Seb, feel free to chime in.

I ported my modification to libjitsi/jitsi but I lack time to test.
Basically the hw decoding works on Linux with VAAPI but still no luck with
Windows's DXVA :(. I will try to fully test modifications in order to post
a patch. And maybe allow someone else to continue this feature.

Regards,
--
Seb

However it is clear that a great deal of care and attention has
been gone into Jitsi

Thanks for noticing :slight_smile:

and we do not want to “step on toes” by not
understanding your design or having different development goals.

At present the changes are “unfinished” code but I would like to
understand what the usual procedure is and how best we can facilitate
getting the finished code included.

That's it actually. Discussing your intentions here and agreeing on a
solution is the first step (I apologise again that you'll need to wait
until next week for answers).

Next step is sending a patch:
    https://jitsi.org/faq/patch

(The first time you do this we'll ask you to sign our contributor
agreement: http://bluejimp.com/bca.pdf)

Eventually, after several patch iterations you'll have the option of
getting commit access:

    https://jitsi.org/commitaccess/

Do you have a code standards document for example?

    Yes:

    https://jitsi.org/codeconvention

Cheers,
Emil

I should note that from our point of view the current nightly build and
release version of Jitsi do not function well in regard to video calling
on Windows except when conditions are excellent.

As a result I strongly believe that at least a partial solution should
be considered for Jitsi 1.1 (or is it 2.0?). One suggestion is I can
provide some direct support to one of the dev team regarding a quick fix
either on IRC or phone.

Best Regards

James

Regards,

James M. Martin
Director of Development
ESP Technologies Limited - Delivering Total IT Solutions to Government

Website:

www.esptl.com <http://www.esptl.com>

Head Office:

Auburn House, Ard na Greine, Ennis, Co. Clare, Ireland

Dublin Office:

Phoenix Centre, Block 6, Broomhall Business Park,
Rathnew, Co. Wicklow, Ireland.

ESP Technologies Limited is registered with the Registrar of Companies,
Dublin - No. 367933
Registered Address: Auburn House, Ard na Greine, Ennis, Co. Clare,

Ireland

This email is intended for the addressees named above and any other use
is prohibited. It may contain confidential information. If you received
this email in error, please contact the sender by return email. ESP does
not accept legal responsibility for the contents of this message if it
has reached you via the Internet, as Internet communications are not
secure. Any opinions expressed are those of the author and are not
necessarily endorsed by ESP. All ESP communications are scanned for
malware & virus infection using the latest anti-viral utilities.

--
Emil Ivov, Ph.D. 67000 Strasbourg,
Project Lead France
Jitsi
emcho@jitsi.org PHONE: +33.1.77.62.43.30
https://jitsi.org FAX: +33.1.77.62.47.31