[jitsi-dev] New video on android (was Jitsi-android with openfire)


#1

Hi,

Hi Pawel,
    This problem has been fixed until i add the two user in one team, but
the video doesn't work even the call is connected and time is passing.
    Another thing i found is that the settings menu is different among
devices ,some devices support both "mediaRecorder" and "Android camera" and
the others doesn't , as well as the "use hardware encoding" item, why?
Regards,
Ray

Yes, it depends on the device. This commit added new device system
"Android camera", so now we have:

1. Mediarecorder - uses android.media.MediaRecorder to record video.
It extracts H264 NALs and it's probably hardware accelerated. Those
H264 frames are then packetized and send via rtp. It should be
supported by all devices.

2. "AndroidCamera" - it's a new device system that works with
android.hardware.Camera. It can work with libjitsi software codecs or
Android's android.media.MediaCodec.

2. A) software mode - camera preview callback is used to capture YV12
frames and those are converted to YUV420Planar format that is accepted
by libjitsi. H264 and H263 are currently compiled for Android, VP8
will not work for now.

Conversion from YV12 to YUV420P is usually only about swaping U and V
planes which takes 3 copy operations. For some sizes YV12 padding has
to be removed and those should be avoided. In the docs we can find
info about image formats:
http://developer.android.com/reference/android/graphics/ImageFormat.html#YV12
YV12: "This format is guaranteed to be supported for camera preview
images since API level 12; for earlier API versions,
checkgetSupportedPreviewFormats()."
I guess that you've tried device below AIP12, which doesn't support
this format and that is why "AndroidCamera" wasn't listed as one of
devices. You can try to debug and see what formats are supported on
this device and try converting it to YUV420Planar. You can do this in:
org.jitsi.impl.neomedia.jmfext.media.protocol.androidcamera.PreviewStream

2. B) "hardware" mode - this requires API16 that comes with MediaCodec
class. If device has API18, then Surface can be used to feed the
encoder and skip image format converstion which is much faster.
Surface can be used to read from decoders which is supported since
API16.

Each API16 device can have many codecs for the same media type. Those
can be listed in Settings->MediaCodecs. It displays built-in media
encoders/decoders for different media types(H264,H263, VP8). Items on
the list are displayed in order of appearance and 3 colors are used:
a) blue - indicates that this codec will be used for encoding or
decoding media type printed in parentheses
b) gery - this codec is disabled and will not be taken into account
during encoder/decoder selection
c) white - this codecs is enabled, but there is another one that
appears before it so this one won't be used.
Click on codec to disable/enable it. After restart this list is
restored to defaults.

You can try experiment with different codecs. I will be happy to see
some results from different devices. Some of encoders crash for
unknown reasons(to me) and they are disabled by default. I would try
to send/receive video between Android and desktop Jitsi first. The
list will be disabled once we'll have an idea how to make it work
reliable for most of devices.

Known issues:
- "Hardware" decoding works only for 352x244 resultion for now.
- Sometimes it takes a lot of time before video from remote party is
displayed. It speeds up when debugger is connected to the device -
really strange.
- Some of "hardware" encoders/decoders crash
- "Hardware" is not always hardware accelerated. I figured out that
usually encodes with "google" in name are software.
- VP8 encoder with "google" in name works only with 176x144 resolution.

Regards,
Pawel

···

On Thu, Nov 21, 2013 at 6:55 AM, MAXWELL <95143848@qq.com> wrote:


#2

Added to the list of knows issues:
- device rotation is not supported in video call(work in progress)

···

On Thu, Nov 21, 2013 at 9:59 AM, Paweł Domas <pawel.domas@jitsi.org> wrote:

Hi,

On Thu, Nov 21, 2013 at 6:55 AM, MAXWELL <95143848@qq.com> wrote:

Hi Pawel,
    This problem has been fixed until i add the two user in one team, but
the video doesn't work even the call is connected and time is passing.
    Another thing i found is that the settings menu is different among
devices ,some devices support both "mediaRecorder" and "Android camera" and
the others doesn't , as well as the "use hardware encoding" item, why?
Regards,
Ray

Yes, it depends on the device. This commit added new device system
"Android camera", so now we have:

1. Mediarecorder - uses android.media.MediaRecorder to record video.
It extracts H264 NALs and it's probably hardware accelerated. Those
H264 frames are then packetized and send via rtp. It should be
supported by all devices.

2. "AndroidCamera" - it's a new device system that works with
android.hardware.Camera. It can work with libjitsi software codecs or
Android's android.media.MediaCodec.

2. A) software mode - camera preview callback is used to capture YV12
frames and those are converted to YUV420Planar format that is accepted
by libjitsi. H264 and H263 are currently compiled for Android, VP8
will not work for now.

Conversion from YV12 to YUV420P is usually only about swaping U and V
planes which takes 3 copy operations. For some sizes YV12 padding has
to be removed and those should be avoided. In the docs we can find
info about image formats:
http://developer.android.com/reference/android/graphics/ImageFormat.html#YV12
YV12: "This format is guaranteed to be supported for camera preview
images since API level 12; for earlier API versions,
checkgetSupportedPreviewFormats()."
I guess that you've tried device below AIP12, which doesn't support
this format and that is why "AndroidCamera" wasn't listed as one of
devices. You can try to debug and see what formats are supported on
this device and try converting it to YUV420Planar. You can do this in:
org.jitsi.impl.neomedia.jmfext.media.protocol.androidcamera.PreviewStream

2. B) "hardware" mode - this requires API16 that comes with MediaCodec
class. If device has API18, then Surface can be used to feed the
encoder and skip image format converstion which is much faster.
Surface can be used to read from decoders which is supported since
API16.

Each API16 device can have many codecs for the same media type. Those
can be listed in Settings->MediaCodecs. It displays built-in media
encoders/decoders for different media types(H264,H263, VP8). Items on
the list are displayed in order of appearance and 3 colors are used:
a) blue - indicates that this codec will be used for encoding or
decoding media type printed in parentheses
b) gery - this codec is disabled and will not be taken into account
during encoder/decoder selection
c) white - this codecs is enabled, but there is another one that
appears before it so this one won't be used.
Click on codec to disable/enable it. After restart this list is
restored to defaults.

You can try experiment with different codecs. I will be happy to see
some results from different devices. Some of encoders crash for
unknown reasons(to me) and they are disabled by default. I would try
to send/receive video between Android and desktop Jitsi first. The
list will be disabled once we'll have an idea how to make it work
reliable for most of devices.

Known issues:
- "Hardware" decoding works only for 352x244 resultion for now.
- Sometimes it takes a lot of time before video from remote party is
displayed. It speeds up when debugger is connected to the device -
really strange.
- Some of "hardware" encoders/decoders crash
- "Hardware" is not always hardware accelerated. I figured out that
usually encodes with "google" in name are software.
- VP8 encoder with "google" in name works only with 176x144 resolution.

Regards,
Pawel


#3

To get better video quality bitrate should be adjusted to used video
resoultion. Also I'd recommend to disable call encryption as it takes
too much resources for now.

···

On Thu, Nov 21, 2013 at 10:24 AM, Paweł Domas <pawel.domas@jitsi.org> wrote:

Added to the list of knows issues:
- device rotation is not supported in video call(work in progress)

On Thu, Nov 21, 2013 at 9:59 AM, Paweł Domas <pawel.domas@jitsi.org> wrote:

Hi,

On Thu, Nov 21, 2013 at 6:55 AM, MAXWELL <95143848@qq.com> wrote:

Hi Pawel,
    This problem has been fixed until i add the two user in one team, but
the video doesn't work even the call is connected and time is passing.
    Another thing i found is that the settings menu is different among
devices ,some devices support both "mediaRecorder" and "Android camera" and
the others doesn't , as well as the "use hardware encoding" item, why?
Regards,
Ray

Yes, it depends on the device. This commit added new device system
"Android camera", so now we have:

1. Mediarecorder - uses android.media.MediaRecorder to record video.
It extracts H264 NALs and it's probably hardware accelerated. Those
H264 frames are then packetized and send via rtp. It should be
supported by all devices.

2. "AndroidCamera" - it's a new device system that works with
android.hardware.Camera. It can work with libjitsi software codecs or
Android's android.media.MediaCodec.

2. A) software mode - camera preview callback is used to capture YV12
frames and those are converted to YUV420Planar format that is accepted
by libjitsi. H264 and H263 are currently compiled for Android, VP8
will not work for now.

Conversion from YV12 to YUV420P is usually only about swaping U and V
planes which takes 3 copy operations. For some sizes YV12 padding has
to be removed and those should be avoided. In the docs we can find
info about image formats:
http://developer.android.com/reference/android/graphics/ImageFormat.html#YV12
YV12: "This format is guaranteed to be supported for camera preview
images since API level 12; for earlier API versions,
checkgetSupportedPreviewFormats()."
I guess that you've tried device below AIP12, which doesn't support
this format and that is why "AndroidCamera" wasn't listed as one of
devices. You can try to debug and see what formats are supported on
this device and try converting it to YUV420Planar. You can do this in:
org.jitsi.impl.neomedia.jmfext.media.protocol.androidcamera.PreviewStream

2. B) "hardware" mode - this requires API16 that comes with MediaCodec
class. If device has API18, then Surface can be used to feed the
encoder and skip image format converstion which is much faster.
Surface can be used to read from decoders which is supported since
API16.

Each API16 device can have many codecs for the same media type. Those
can be listed in Settings->MediaCodecs. It displays built-in media
encoders/decoders for different media types(H264,H263, VP8). Items on
the list are displayed in order of appearance and 3 colors are used:
a) blue - indicates that this codec will be used for encoding or
decoding media type printed in parentheses
b) gery - this codec is disabled and will not be taken into account
during encoder/decoder selection
c) white - this codecs is enabled, but there is another one that
appears before it so this one won't be used.
Click on codec to disable/enable it. After restart this list is
restored to defaults.

You can try experiment with different codecs. I will be happy to see
some results from different devices. Some of encoders crash for
unknown reasons(to me) and they are disabled by default. I would try
to send/receive video between Android and desktop Jitsi first. The
list will be disabled once we'll have an idea how to make it work
reliable for most of devices.

Known issues:
- "Hardware" decoding works only for 352x244 resultion for now.
- Sometimes it takes a lot of time before video from remote party is
displayed. It speeds up when debugger is connected to the device -
really strange.
- Some of "hardware" encoders/decoders crash
- "Hardware" is not always hardware accelerated. I figured out that
usually encodes with "google" in name are software.
- VP8 encoder with "google" in name works only with 176x144 resolution.

Regards,
Pawel