[jitsi-dev] Java 7+, OSX and Video


#1

Hey

So, I've spent a considerable amount of time debugging the broken video
support on OSX with Java 7+.

- We have most of the infrastructure to support JAWT with the "new" CALayer
model
- The attaching/detaching of the CALayer from Java in our code is broken
- We have problems with the z-Order (because AWT uses lower numbers as the
top-most index, contrary to every other layering system including
CoreAnimation (and ironically Swing as well))
- JAWT is a heavyweight component while Swing components are lightweight, so
it will most likely never be possible again to position a button on top of
the videos, despite the announcement in [1]

Given all these troubles, I wonder why we don't just resort to draw the
received image with Java2D, which internally uses CoreAnimation and OpenGL
anyway. I didn't notice a major difference in performance while swearing at
OSX.

The attached patch
- Implements a plain Java rendering which can be enabled with a property
(almost fit for commit)
- Brings the JAWT rendering back to life against Java 8 (with some hacks,
definitely not fit for commit)

Lyubo, you wrote most of that code. What do you think?
And all the other devs would be most welcome to state their opinion as well
and not remain silent.

Ingo

[1]
http://www.oracle.com/technetwork/articles/java/mixing-components-433992.htm
l

libjitsi-java8-osx.patch (14.8 KB)


#2

Hello, Ingo!

- We have most of the infrastructure to support JAWT with the "new" CALayer
model

I'm afraid I don't fully understand. If I remember correctly, the
"new" CALayer model was fully working...

- The attaching/detaching of the CALayer from Java in our code is broken

Could I please ask you for details? I see that you've introduced a
id<JAWT_SurfaceLayers> sl but it appears to only be assigned to and
never read.

- We have problems with the z-Order (because AWT uses lower numbers as the
top-most index, contrary to every other layering system including
CoreAnimation (and ironically Swing as well))

Is it easy to fix (in the context of JAWTRenderer, of course)?

- JAWT is a heavyweight component while Swing components are lightweight, so
it will most likely never be possible again to position a button on top of
the videos, despite the announcement in [1]

We don't try to address that with JAWTRenderer.

Given all these troubles, I wonder why we don't just resort to draw the
received image with Java2D, which internally uses CoreAnimation and OpenGL
anyway.

We introduced JAWTRenderer for the purposes of performance. It gives
us the ability to one day keep the whole video frames out of the Java
heap and into native memory.

I didn't notice a major difference in performance while swearing at
OSX.

Could you please share a bit more details about the scenarios that you observed?

The attached patch
- Implements a plain Java rendering which can be enabled with a property
(almost fit for commit)

FMJ implements pure-Java video Renderers
(./src/net/sf/fmj/media/renderer/video/). If memory serves me right,
one of them kicks in and does a good job if JAWTRenderer fails to
register. Have you looked at them? If one of them works for us
alright, could we please not re-implement it in JAWTRenderer?

- Brings the JAWT rendering back to life against Java 8 (with some hacks,
definitely not fit for commit)

I will appreciate explanations here because I do not understand the
patch. (I guess the changes to src/native/build.xml are OK because you
supposedly used them to rebuild the jnawtrenderer binary.)

Lyubo, you wrote most of that code. What do you think?
And all the other devs would be most welcome to state their opinion as well
and not remain silent.

I'm very happy that we're looking into moving to Java 8.

Please consider my comments complementing, not opposing.

Best regards,
Lyubomir

···

2014-11-16 23:20 GMT+02:00 Ingo Bauersachs <ingo@jitsi.org>:


#3

Hey Lyubo

Thanks for looking at it!

- We have most of the infrastructure to support JAWT with the "new"

CALayer

model

I'm afraid I don't fully understand. If I remember correctly, the
"new" CALayer model was fully working...

That was more a note that we're already on the new model and shouldn't need
to replace NSView anymore, like other projects still had to.

- The attaching/detaching of the CALayer from Java in our code is broken

Could I please ask you for details? I see that you've introduced a
id<JAWT_SurfaceLayers> sl but it appears to only be assigned to and
never read.

Sure. There are multiple issues:
- If the CALayer is repeatedly attached from within paint(..), it doesn't
work at all. The only thing that worked was to attach it from within
addNotify. This makes sense because the Javadoc of addNotify says that it is
called from the framework when the native counterpart is ready.
- The sl field is read when closing the native part, line 57 in the patch
- Without that, the CALayer will stay around for as long as the window is
still open
- Closing has to be called from removeNotify

- We have problems with the z-Order (because AWT uses lower numbers as

the

top-most index, contrary to every other layering system including
CoreAnimation (and ironically Swing as well))

Is it easy to fix (in the context of JAWTRenderer, of course)?

Well, easier than all the rest I guess. The problem is that we don't really
know how many components will be added to the container, so a
Abs(count()-index()) doesn't do the job and why I hacked a string comparison
for "LOCAL" into the Java part of the JAWT component to set a high index.

- JAWT is a heavyweight component while Swing components are
lightweight, so it will most likely never be possible again to position
a button on top of the videos, despite the announcement in [1]

We don't try to address that with JAWTRenderer.

No, but we run into it because JAWTRenderer is heavyweight and the button to
close the local video preview is lightweight. However, I would consider this
button the least of our concerns as we still have the toolbar button to
show/hide the local video.

Given all these troubles, I wonder why we don't just resort to draw the
received image with Java2D, which internally uses CoreAnimation and

OpenGL

anyway.

We introduced JAWTRenderer for the purposes of performance. It gives
us the ability to one day keep the whole video frames out of the Java
heap and into native memory.

Sure, but we don't have the hardware decoding feature ready (Sebs old branch
for that doesn't even have an attempt for OSX). Given that we want to move
away from Swing/AWT at some point and the current focus is entirely on Meet,
I don't really see anyone addressing that in the foreseeable future.

I didn't notice a major difference in performance while swearing at
OSX.

Could you please share a bit more details about the scenarios that you
observed?

This was just a one to one call and certainly not exhaustive testing. But
AFAIK the conferences have always been problematic anyway?

What makes a difference though is that Java 8 now uses OpenGL and CALayer
internally for drawing with Graphics, so there is no difference anymore in
what the JRE does over what we're doing.

The attached patch
- Implements a plain Java rendering which can be enabled with a property
(almost fit for commit)

FMJ implements pure-Java video Renderers
(./src/net/sf/fmj/media/renderer/video/). If memory serves me right,
one of them kicks in and does a good job if JAWTRenderer fails to
register. Have you looked at them? If one of them works for us
alright, could we please not re-implement it in JAWTRenderer?

Nope, didn't know about this. Will take a look, thanks for the hint.

- Brings the JAWT rendering back to life against Java 8 (with some hacks,
definitely not fit for commit)

I will appreciate explanations here because I do not understand the
patch. (I guess the changes to src/native/build.xml are OK because you
supposedly used them to rebuild the jnawtrenderer binary.)

Well to be honest I don't understand all of it either. Without the patch, I
only see a black window. So I took Apple's JAWTExample, saw that they only
register the CALayer once, adapted this to libjitsi and got a picture again
(apparently this is new to Java 8 as in Java 7 it was only the z-index that
caused problems). But with these changes, the layer stayed on the screen
even if the components were closed - so I had to set the layer to nil (hence
the sl field).

Hope that makes a bit more sense...?

The build.xml changes are for compiling against Java 8, but I didn't use
them much as I created an XCode project to be able to debug the native code.

Lyubo, you wrote most of that code. What do you think?
And all the other devs would be most welcome to state their opinion as

well

and not remain silent.

I'm very happy that we're looking into moving to Java 8.

Please consider my comments complementing, not opposing.

Oh not at all! :slight_smile:

Best regards,
Lyubomir

Ingo

···

2014-11-16 23:20 GMT+02:00 Ingo Bauersachs <ingo@jitsi.org>:


#4

Just to be clear: this is not related to the efforts for hardware decoding.

It is about the possibility to get bytes from the RTP stack into native-land and to move them through the chain as native up until the renderer without ever bringing them back into the Java heap.

As far as I understand, this breaks when we use a Java renderer that forces them into the heap and then shoots them back to native for OpenGL rendering.

Now of course, it is all about choosing which sacrifice will be bigger: throwing away the possibility of a native chain or releasing without Java 7 compatibility.

Emil

···

On 18.11.14, 0:39, Ingo Bauersachs wrote:

Sure, but we don't have the hardware decoding feature ready (Sebs old branch
for that doesn't even have an attempt for OSX).

--
https://jitsi.org


#5

Sure, but we don't have the hardware decoding feature ready (Sebs old
branch for that doesn't even have an attempt for OSX).

Just to be clear: this is not related to the efforts for hardware

decoding.

It is about the possibility to get bytes from the RTP stack into
native-land and to move them through the chain as native up until the
renderer without ever bringing them back into the Java heap.

Okay, but the efforts to make this possible would most likely be related.
And as far as I can tell from my observations, the resource demand from
copying is negligible compared to the H264 encoding/decoding.

As far as I understand, this breaks when we use a Java renderer that
forces them into the heap and then shoots them back to native for OpenGL
rendering.

Yes, unless there is a possibility with JNI to access a char* from within
Java as a byte[] without copying it into the Java heap.

Now of course, it is all about choosing which sacrifice will be bigger:
throwing away the possibility of a native chain or releasing without
Java 7 compatibility.

We don't throw that away, we would rather postpone the effort of getting the
JAWT plugin back to work. And the JAWT will work again in Java 8 with some
modifications, the hacks I made prove this to some point - but these hacks
are not primetime ready.

Unless I have overlooked something - which is very well possible given that
this was the first (and hopefully last) time I did anything with native code
on a Mac - it still seems advantageous to upgrade to Java 8 given that new
Yosemite installs apparently don't even have the option for a Java 6
installation anymore (don't if this is true, there was a post recently on
the list mentioning this).

@Lyubo: of course this is okay. I hope my hacks can give you a jump start.

Emil

Ingo

···

On 18.11.14, 0:39, Ingo Bauersachs wrote:


#6

If that's OK with everyone, I'd really like to have a thorough look at
all this on Friday.


#7
···

Am 19.11.2014 um 11:45 schrieb Ingo
Bauersachs:

On 18.11.14, 0:39, Ingo Bauersachs wrote:
Sure, but we don't have the hardware decoding feature ready (Sebs old
branch for that doesn't even have an attempt for OSX).
Just to be clear: this is not related to the efforts for hardware
decoding.
It is about the possibility to get bytes from the RTP stack into
native-land and to move them through the chain as native up until the
renderer without ever bringing them back into the Java heap.
Okay, but the efforts to make this possible would most likely be related.
And as far as I can tell from my observations, the resource demand from
copying is negligible compared to the H264 encoding/decoding.
As far as I understand, this breaks when we use a Java renderer that
forces them into the heap and then shoots them back to native for OpenGL
rendering.
Yes, unless there is a possibility with JNI to access a char* from within
Java as a byte[] without copying it into the Java heap.

` If I remember it right, there is shared memory type of data
transfer between

  JNI and Java through memory mapped files.

`

Now of course, it is all about choosing which sacrifice will be bigger:
throwing away the possibility of a native chain or releasing without
Java 7 compatibility.

We don't throw that away, we would rather postpone the effort of getting the
JAWT plugin back to work. And the JAWT will work again in Java 8 with some
modifications, the hacks I made prove this to some point - but these hacks
are not primetime ready.
Unless I have overlooked something - which is very well possible given that
this was the first (and hopefully last) time I did anything with native code
on a Mac - it still seems advantageous to upgrade to Java 8 given that new
Yosemite installs apparently don't even have the option for a Java 6
installation anymore (don't if this is true, there was a post recently on
the list mentioning this).
@Lyubo: of course this is okay. I hope my hacks can give you a jump start.
Emil


Ingo
_______________________________________________
dev mailing list
Unsubscribe instructions and other list options:

dev@jitsi.orghttp://lists.jitsi.org/mailman/listinfo/dev


#8

Whichever way we go with JAWTRenderer, is there work to be done on the
application launcher in order to employ Java 7+?


#9

It is about the possibility to get bytes from the RTP stack into
>native-land and to move them through the chain as native up until the
>renderer without ever bringing them back into the Java heap.

Okay, but the efforts to make this possible would most likely be related.

In as much as they would happen in the same parts of the code, yes. Ultimately however one is about integrating a new decoder while the other is about improving the way we are moving data through the FMJ codec chain.

And as far as I can tell from my observations, the resource demand from
copying is negligible compared to the H264 encoding/decoding

Not really. This is decoded/raw data so we are talking about megabytes for every single frame (i.e., 25 to 30 times a second).

Eliminating copies of raw image data in the past has always resulted in a significant performance gain.

Emil

···

On 19.11.14, 11:45, Ingo Bauersachs wrote:

--
https://jitsi.org


#10

Just a little :slight_smile: Splashscreen and testing on macosx 10.6 32bit, cause
launcher crashes there and do not work.

···

On Wed, Nov 19, 2014 at 12:54 PM, Lyubomir Marinov <lyubomir.marinov@jitsi.org> wrote:

Whichever way we go with JAWTRenderer, is there work to be done on the
application launcher in order to employ Java 7+?

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#11

I haven't tested it with java8, by the way, so this one will also need
some attention.

···

On Wed, Nov 19, 2014 at 12:58 PM, Damian Minkov <damencho@jitsi.org> wrote:

Just a little :slight_smile: Splashscreen and testing on macosx 10.6 32bit, cause
launcher crashes there and do not work.

On Wed, Nov 19, 2014 at 12:54 PM, Lyubomir Marinov > <lyubomir.marinov@jitsi.org> wrote:

Whichever way we go with JAWTRenderer, is there work to be done on the
application launcher in order to employ Java 7+?

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#12

Just a little :slight_smile: Splashscreen and testing on macosx 10.6 32bit, cause
launcher crashes there and do not work.

I couldn't create an x86 universal dylib while testing. Is there a Java 7+
32bit available at all?

Ingo


#13

Hi,

Well, back then I was testing to first replace current launcher with
the new one and make it possible to start and java6.
It seems that these are only macbooks released May 16, 2006 (Early
2006) MacBook1,1.
Late 2006 macbooks MacBook2,1 are with 64bit processors.
If we decide we will drop these, we can significantly lower the size
of the binaries. Of course no need to do it right now and recompile
all of them, but we can just remove the instructions from
native/src/build.xml and in the future will replace them one by one.

Regards
damencho

···

On Wed, Nov 19, 2014 at 2:13 PM, Ingo Bauersachs <ingo@jitsi.org> wrote:

Just a little :slight_smile: Splashscreen and testing on macosx 10.6 32bit, cause
launcher crashes there and do not work.

I couldn't create an x86 universal dylib while testing. Is there a Java 7+
32bit available at all?

Ingo

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#14

We can do that, yes. A 2006 macboom would have trouble doing a video
conversation anyway.

--sent from my mobile

···

On 19 Nov 2014 1:32 PM, "Damian Minkov" <damencho@jitsi.org> wrote:

Hi,

Well, back then I was testing to first replace current launcher with
the new one and make it possible to start and java6.
It seems that these are only macbooks released May 16, 2006 (Early
2006) MacBook1,1.
Late 2006 macbooks MacBook2,1 are with 64bit processors.
If we decide we will drop these, we can significantly lower the size
of the binaries. Of course no need to do it right now and recompile
all of them, but we can just remove the instructions from
native/src/build.xml and in the future will replace them one by one.

Regards
damencho

On Wed, Nov 19, 2014 at 2:13 PM, Ingo Bauersachs <ingo@jitsi.org> wrote:
>> Just a little :slight_smile: Splashscreen and testing on macosx 10.6 32bit, cause
>> launcher crashes there and do not work.
>
> I couldn't create an x86 universal dylib while testing. Is there a Java
7+
> 32bit available at all?
>
> Ingo
>
>
> _______________________________________________
> dev mailing list
> dev@jitsi.org
> Unsubscribe instructions and other list options:
> http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#15

I agree with the dropping of 32-bit on OS X. If I understand the news
correctly, Chrome 39/stable is 64-bit only as well.

···

2014-11-19 14:34 GMT+02:00 Emil Ivov <emcho@jitsi.org>:

We can do that, yes. A 2006 macboom would have trouble doing a video
conversation anyway.


#16

Hi devs,

I just saw my notes about java7 problems. Every now and then I have
problem running Jitsi from source and sip calling under macosx. And
when I check what is going on, my local address is 127.0.0.1 (this in
the call info) and no rtp can be seen going out from my connection.
This one also needs some attention.

Regards
damencho

···

On Wed, Nov 19, 2014 at 3:53 PM, Lyubomir Marinov <lyubomir.marinov@jitsi.org> wrote:

2014-11-19 14:34 GMT+02:00 Emil Ivov <emcho@jitsi.org>:

We can do that, yes. A 2006 macboom would have trouble doing a video
conversation anyway.

I agree with the dropping of 32-bit on OS X. If I understand the news
correctly, Chrome 39/stable is 64-bit only as well.

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#17

I just saw my notes about java7 problems. Every now and then I have
problem running Jitsi from source and sip calling under macosx. And
when I check what is going on, my local address is 127.0.0.1 (this in
the call info) and no rtp can be seen going out from my connection.
This one also needs some attention.

I guess you don't have a clue how to reproduce that?

Regards
damencho

Ingo


#18

Hi,

I reproduce it on every call I made through the asterisk installation we use.
Every inbound and outbound call. I will try to take a look now.

Regards
damencho

···

On Sat, Nov 22, 2014 at 1:41 PM, Ingo Bauersachs <ingo@jitsi.org> wrote:

I just saw my notes about java7 problems. Every now and then I have
problem running Jitsi from source and sip calling under macosx. And
when I check what is going on, my local address is 127.0.0.1 (this in
the call info) and no rtp can be seen going out from my connection.
This one also needs some attention.

I guess you don't have a clue how to reproduce that?

Regards
damencho

Ingo

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#19

Hi,

seem a line in /etc/hosts is causing this. But it works with java6.
127.0.0.1<----->damencho-dev
The problem comes from InetAddress.getLocalHost() (Thanks Bobi for the
help and java8 tests:) ).

Without the problem line printing result of InetAddress.getLocalHost():
java1.6: 10.0.0.117
java1.7: 10.0.0.117
java1.8: 10.0.0.117
With that line in /etc/hosts:
java1.6: 10.0.0.117
java1.7: 127.0.0.1
java1.8: 10.0.0.117

Not sure whether that line is ok there, but without it everything works.

Regards
damencho

···

On Mon, Nov 24, 2014 at 12:15 PM, Damian Minkov <damencho@jitsi.org> wrote:

Hi,

I reproduce it on every call I made through the asterisk installation we use.
Every inbound and outbound call. I will try to take a look now.

Regards
damencho

On Sat, Nov 22, 2014 at 1:41 PM, Ingo Bauersachs <ingo@jitsi.org> wrote:

I just saw my notes about java7 problems. Every now and then I have
problem running Jitsi from source and sip calling under macosx. And
when I check what is going on, my local address is 127.0.0.1 (this in
the call info) and no rtp can be seen going out from my connection.
This one also needs some attention.

I guess you don't have a clue how to reproduce that?

Regards
damencho

Ingo

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#20

seem a line in /etc/hosts is causing this. But it works with java6.
127.0.0.1<----->damencho-dev The problem comes from
InetAddress.getLocalHost() (Thanks Bobi for the help and java8 tests:) ).

Without the problem line printing result of InetAddress.getLocalHost():
java1.6: 10.0.0.117
java1.7: 10.0.0.117
java1.8: 10.0.0.117
With that line in /etc/hosts:
java1.6: 10.0.0.117
java1.7: 127.0.0.1
java1.8: 10.0.0.117

What was the exact version of Java 7? This might have been a regression...
Given that it works again with Java 8, why not just skip 7 entirely and
deliver it with 8 embedded?

Not sure whether that line is ok there, but without it everything works.

Is it present on other Mac systems by default? I'll check mine in the
evening, although I might have modified hosts and have forgotten about it.

Regards
damencho

Ingo