[sip-comm-dev] Handling of video images


#1

Hello all!

Is there already a plan on how to pass the video image of a video SIP call
from the CallSessionImpl to the GUI front-end? I am currently working on
embedding the SIP communicator into our application and need to integrate
the video component. If not, I could propose a way and start implementing it
if it is accepted (at least the back-end side).

We at Rowa need an embeddable Java video conference libary and I am working
on using SIP communicator as a basis. We'll release the (modified) sources
in accordance with the LGPL of course, but since I had to restructure it
pretty much I don't know if any of it will be useful for integration with
the main SIP Communicator. I hope that I can help out with some things,
however (mainly robustness of the SIP backend).

Once I am through with embedding SIP Communicator, I'll post a report of
what I have done.

Thanks to all contributors for all the great work you have done so far.

Regards
Michael Koch


#2

hello everybody,
im not sure if anyone has every thought about anything like that or maybe there already is some kind of solution for this. im thinking about a quality of service implementation for the SC doing two different things. the probably more easy part to write a custom jmf rtpconnector to be able to define the tos field for the ip pakets. after looking only vaguely over the rtpmanager initialization in SC i saw that you are using the addtarget functions which will be a problem when initializing with a custom rtpconnector.

else has anyone ever thought about some kind of buffering the incoming data and then selecting which to send out. let me give you any example. you are doing a file transfer and have a audio phone conversation at the same time. both data is being put in two different buffers like this

  file | | audio |
tranfer | | conversation |
          > > >

-------- ---------------- \
         \ <--- selector
          \

and then you have some kind of selecting mechanism to decide which data to send out first. the easiest way for this is probably to define for example audio as priority 1 and a file transfer as prio 2 and then the selector chooses the buffer prio 1 for as long as there is data in that buffer. when there is nothing in there is checks buffer prio 2 and so on.

does anyone have any clue if this idea is actually realizable to program into the SC. i mean in a one man job and with a time frame of about 1 month. ill be thankful for any kind of help. where to start or anything that could help

thanks alot for any help/time!

regards

kolja

···

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#3

Hello Emil!

I don't know SIP communicator much, and especially haven't looked at the GUI
part, so I'm not really qualified to say which would be the best approach.
I'll try anyway :slight_smile:

I think that, from the GUI point of view, the video image has to be
associated with the CallParticipant who is shown. This way, when there is a
conference call or multiple simultaneous calls, the GUI can show the name of
the caller. Since the incoming media is thus associated with the
CallParticipant, a media listener which can be registered on objects of that
class may be useful. It would make the implementation kind of difficult,
since the component has to be passed from media to protocol, and from there
to gui.

Your solution 1 (extending the CallSession interface) has the drawback that
the interface is currently used for the low-level tasks like handling the
SDP data, which the GUI usually doesn't care about. I don't know if the GUI
currently uses the media bundle directly at all?

Solutions 2 and 3 add more GUI knowledge to the media bundle. Of course I'm
not the one to make the decisions, but for me it doesn't feel right. On the
other hand, audio rendering and control is also done in media, so perhaps it
would not be too bad after all.

On UI lib independence, I think it is also a matter of what the media
library supplies to the media bundle. The players of JMF and FMJ both use
some java.awt.Component subclass (I think), so I guess that is all SIP
communicator needs to support. Since java.awt.Component has been part of
Java since 1.0, I guess all alternative GUI libraries will support it in
some way.

I don't know what would be the best approach. One probably has to think
about the GUI for video calls first, and then decide how the needed
information is best passed from back-end to front-end. If someone wants to
do this part, I could help out with the back-end implementation. For my
short-term needs, I have extended the MediaEvent/MediaListener mechanism
which was already there and pass the component around in the event. I don't
know if this is such a good solution, because:
* I'm not sure if I am using the event for the purpose it was intended for
* It hooks the GUI directly to the media bundle, which I don't know if this
is supposed to be done
but it works for me :slight_smile:
(I could create a patch if there is interest).

Regards
Michael Koch

···

Right now the media service is simply popping out a frame
rendering the
video flow. I guess you've noticed this already. This of
course is only
temporary and we'll have to think of a more elaborate way to handle it
in the future.

Possible solutions may include:

1. Adding a method to CallSession that allows it to return a visual
component that the UI (or any other bundle) would be able to
display. We
would also have to add methods that would allow users to register
listeners and get notified when such a component becomes available.
(Pretty much the way things work with the JMF Player interface)

2. Allowing other modules to register container components where the
media service would be supposed render all visual content.

3. Do not pass components across modules, and instead, define
an API in
CallSession that would give you some control over its visual
components
(move, resize, dock).

The tricky part for 1 and 2 is how to handle this in a way that is
independent from a particular UI lib. I don't really see a
way to do so
unless we settle for AWT components and hope that they will
be supported
by other UI libs (which is the case with swing and swt)

3 on the other hand, would give us the independence but would
add a lot
of complexity to the implementation of Call Sessions.

Personally, I am not sure which is the best way to go. Do you have any
ideas?

Emil


#4

Hello Kolja,

(see inline)

Kolja Ehlers wrote:

hello everybody,
im not sure if anyone has every thought about anything like that or maybe
there already is some kind of solution for this. im thinking about a quality
of service implementation for the SC doing two different things. the
probably more easy part to write a custom jmf rtpconnector to be able to
define the tos field for the ip pakets. after looking only vaguely over the
rtpmanager initialization in SC i saw that you are using the addtarget
functions which will be a problem when initializing with a custom
rtpconnector.

This is easily changed, and besides, we might have to change it anyway
since I heard Su Bing (who is working on SRTP) is also having problems
with the same thing.

else has anyone ever thought about some kind of buffering the incoming data
and then selecting which to send out. let me give you any example. you are
doing a file transfer and have a audio phone conversation at the same time.
both data is being put in two different buffers like this

> file | | audio |
> tranfer | | conversation |
> > > >
-------- ----------------
        \
         \ <--- selector
          \

and then you have some kind of selecting mechanism to decide which data to
send out first. the easiest way for this is probably to define for example
audio as priority 1 and a file transfer as prio 2 and then the selector
chooses the buffer prio 1 for as long as there is data in that buffer. when
there is nothing in there is checks buffer prio 2 and so on.

does anyone have any clue if this idea is actually realizable to program
into the SC. i mean in a one man job and with a time frame of about 1 month.
ill be thankful for any kind of help.

Depends on what your goal is. In case you are only working on a research
project and all you want to have at the end is a proof-of-concept demo,
then it might feasible.

However, in case you want to use this in real world conditions then it
is probably not the best approach. You'd have to modify all stacks that
support file transfer, and make sure that they use your QoS queues, and
it would still not prevent other applications from occupying all
available bandwidth.

For a more reliable implementation you'd probably have to do this in the
OS kernel itself. I think there is a fair amount of related art so you
might want to look around for existing implementations.

Hope this helps
Emil

···

where to start or anything that could
help

thanks alot for any help/time!

regards

kolja

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#5

Hi Kolja,

I'm working on SRTP support for SC. Since we are going to use our
customized RTPConnector to handle RTP traffic, we are responsible for
managing the "connection" (stream targets, etc). So, there are some
logic we should take care. But if you have only one target, you can
simply discard the addTarget() call and set your target session
address in your RTPConnector, e.g. constructor. If you will have
multiple destinations, you can use a vector to store the addresses and
send every packets to all the targets.

Best regards,
Su

···

On 7/16/07, Emil Ivov <emcho@emcho.com> wrote:

Hello Kolja,

(see inline)

Kolja Ehlers wrote:
> hello everybody,
> im not sure if anyone has every thought about anything like that or maybe
> there already is some kind of solution for this. im thinking about a quality
> of service implementation for the SC doing two different things. the
> probably more easy part to write a custom jmf rtpconnector to be able to
> define the tos field for the ip pakets. after looking only vaguely over the
> rtpmanager initialization in SC i saw that you are using the addtarget
> functions which will be a problem when initializing with a custom
> rtpconnector.

This is easily changed, and besides, we might have to change it anyway
since I heard Su Bing (who is working on SRTP) is also having problems
with the same thing.

> else has anyone ever thought about some kind of buffering the incoming data
> and then selecting which to send out. let me give you any example. you are
> doing a file transfer and have a audio phone conversation at the same time.
> both data is being put in two different buffers like this
>
> > file | | audio |
> > tranfer | | conversation |
> > > > >
> -------- ----------------
> \
> \ <--- selector
> \
>
> and then you have some kind of selecting mechanism to decide which data to
> send out first. the easiest way for this is probably to define for example
> audio as priority 1 and a file transfer as prio 2 and then the selector
> chooses the buffer prio 1 for as long as there is data in that buffer. when
> there is nothing in there is checks buffer prio 2 and so on.
>
> does anyone have any clue if this idea is actually realizable to program
> into the SC. i mean in a one man job and with a time frame of about 1 month.
> ill be thankful for any kind of help.

Depends on what your goal is. In case you are only working on a research
project and all you want to have at the end is a proof-of-concept demo,
then it might feasible.

However, in case you want to use this in real world conditions then it
is probably not the best approach. You'd have to modify all stacks that
support file transfer, and make sure that they use your QoS queues, and
it would still not prevent other applications from occupying all
available bandwidth.

For a more reliable implementation you'd probably have to do this in the
OS kernel itself. I think there is a fair amount of related art so you
might want to look around for existing implementations.

Hope this helps
Emil

> where to start or anything that could
> help
>
> thanks alot for any help/time!
>
> regards
>
> kolja
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
> For additional commands, e-mail: dev-help@sip-communicator.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#6

hi all, sorry for getting back to this with such a delay but i was on holiday and will start on this now. i would like to ask about emils responds:

However, in case you want to use this in real world conditions then it
is probably not the best approach. You'd have to modify all stacks that
support file transfer, and make sure that they use your QoS queues, and
it would still not prevent other applications from occupying all
available bandwidth.

i dont really understand. of course the problem with other applications will still hold but i dont want to look at that. i just want the sip communicator and its sip services to run over the QoS queues. so are there not just the 2 stacks that i would have to modify, the one for audio and the one for video?

it sounds pretty simple to me to just redirect the audio and the video data into these buffers and not to send them straight out. the selector does decide which is the next package to be send out.

thanks alot for your help

kolja

···

----- Original Message -----

From: "Emil Ivov" <emcho@emcho.com>

To: <dev@sip-communicator.dev.java.net>
Sent: Monday, July 16, 2007 3:55 PM
Subject: Re: [sip-comm-dev] Quality of Service

Hello Kolja,

(see inline)

Kolja Ehlers wrote:

hello everybody,
im not sure if anyone has every thought about anything like that or maybe
there already is some kind of solution for this. im thinking about a quality
of service implementation for the SC doing two different things. the
probably more easy part to write a custom jmf rtpconnector to be able to
define the tos field for the ip pakets. after looking only vaguely over the
rtpmanager initialization in SC i saw that you are using the addtarget
functions which will be a problem when initializing with a custom
rtpconnector.

This is easily changed, and besides, we might have to change it anyway
since I heard Su Bing (who is working on SRTP) is also having problems
with the same thing.

else has anyone ever thought about some kind of buffering the incoming data
and then selecting which to send out. let me give you any example. you are
doing a file transfer and have a audio phone conversation at the same time.
both data is being put in two different buffers like this

> file | | audio |
> tranfer | | conversation |
> > > >
-------- ---------------- \
         \ <--- selector
          \

and then you have some kind of selecting mechanism to decide which data to
send out first. the easiest way for this is probably to define for example
audio as priority 1 and a file transfer as prio 2 and then the selector
chooses the buffer prio 1 for as long as there is data in that buffer. when
there is nothing in there is checks buffer prio 2 and so on.

does anyone have any clue if this idea is actually realizable to program
into the SC. i mean in a one man job and with a time frame of about 1 month.
ill be thankful for any kind of help.

Depends on what your goal is. In case you are only working on a research
project and all you want to have at the end is a proof-of-concept demo,
then it might feasible.

However, in case you want to use this in real world conditions then it
is probably not the best approach. You'd have to modify all stacks that
support file transfer, and make sure that they use your QoS queues, and
it would still not prevent other applications from occupying all
available bandwidth.

For a more reliable implementation you'd probably have to do this in the
OS kernel itself. I think there is a fair amount of related art so you
might want to look around for existing implementations.

Hope this helps
Emil

where to start or anything that could
help

thanks alot for any help/time!

regards

kolja

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#7

Hey Kolja,

Yes you could certainly do this and it would not be difficult, but I
don't see what it would give you.

What else is going to use your queues so that you would assign lower
priority to it?

Besides, in today's world a video conversation with SIP Communicator
would in most cases only take a relatively small portion of the
bandwidth available to your station.

So, even provided that on a saturated link all traffic comes from your
PC, it is quite probable that the better part of it would not be
generated by SC but by another application (e.g. sftp, http ...).
Therefore chances are that a QoS implementation inside SC would not
really help improve the situation.

Regards
Emil

Kolja Ehlers wrote:

···

hi all, sorry for getting back to this with such a delay but i was on
holiday and will start on this now. i would like to ask about emils
responds:

However, in case you want to use this in real world conditions then it
is probably not the best approach. You'd have to modify all stacks that
support file transfer, and make sure that they use your QoS queues, and
it would still not prevent other applications from occupying all
available bandwidth.

i dont really understand. of course the problem with other applications will
still hold but i dont want to look at that. i just want the sip communicator
and its sip services to run over the QoS queues. so are there not just the 2
stacks that i would have to modify, the one for audio and the one for
video?

it sounds pretty simple to me to just redirect the audio and the video data
into these buffers and not to send them straight out. the selector does
decide which is the next package to be send out.

thanks alot for your help

kolja

----- Original Message -----
From: "Emil Ivov" <emcho@emcho.com>
To: <dev@sip-communicator.dev.java.net>
Sent: Monday, July 16, 2007 3:55 PM
Subject: Re: [sip-comm-dev] Quality of Service

Hello Kolja,

(see inline)

Kolja Ehlers wrote:

hello everybody,
im not sure if anyone has every thought about anything like that or maybe
there already is some kind of solution for this. im thinking about a
quality
of service implementation for the SC doing two different things. the
probably more easy part to write a custom jmf rtpconnector to be able to
define the tos field for the ip pakets. after looking only vaguely over
the
rtpmanager initialization in SC i saw that you are using the addtarget
functions which will be a problem when initializing with a custom
rtpconnector.

This is easily changed, and besides, we might have to change it anyway
since I heard Su Bing (who is working on SRTP) is also having problems
with the same thing.

else has anyone ever thought about some kind of buffering the incoming
data
and then selecting which to send out. let me give you any example. you
are
doing a file transfer and have a audio phone conversation at the same
time.
both data is being put in two different buffers like this

> file | | audio |
> tranfer | | conversation |
> > > >
-------- ----------------
        \
         \ <--- selector
          \

and then you have some kind of selecting mechanism to decide which data
to
send out first. the easiest way for this is probably to define for
example
audio as priority 1 and a file transfer as prio 2 and then the selector
chooses the buffer prio 1 for as long as there is data in that buffer.
when
there is nothing in there is checks buffer prio 2 and so on.

does anyone have any clue if this idea is actually realizable to program
into the SC. i mean in a one man job and with a time frame of about 1
month.
ill be thankful for any kind of help.

Depends on what your goal is. In case you are only working on a research
project and all you want to have at the end is a proof-of-concept demo,
then it might feasible.

However, in case you want to use this in real world conditions then it
is probably not the best approach. You'd have to modify all stacks that
support file transfer, and make sure that they use your QoS queues, and
it would still not prevent other applications from occupying all
available bandwidth.

For a more reliable implementation you'd probably have to do this in the
OS kernel itself. I think there is a fair amount of related art so you
might want to look around for existing implementations.

Hope this helps
Emil

where to start or anything that could
help

thanks alot for any help/time!

regards

kolja

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#8

hi emil, thanks for the mail. i just want to experminent with it. could you maybe point me to the direction from where i can access the sip audio and the video stream. i want to start off with just prioritising these two sip streams.

i guess i have to figure out jmf for doing this. so it is possible to grab the data pakets genereated by jmf and put them in these buffers and not let jmf send them out. then the selector does the sending.

maybe you could help me and tell me where to start. i really appreciate it

thanks

kolja

···

----- Original Message -----

From: "Emil Ivov" <emcho@emcho.com>

To: <dev@sip-communicator.dev.java.net>
Sent: Tuesday, July 31, 2007 3:50 AM
Subject: Re: [sip-comm-dev] Quality of Service

Hey Kolja,

Yes you could certainly do this and it would not be difficult, but I
don't see what it would give you.

What else is going to use your queues so that you would assign lower
priority to it?

Besides, in today's world a video conversation with SIP Communicator
would in most cases only take a relatively small portion of the
bandwidth available to your station.

So, even provided that on a saturated link all traffic comes from your
PC, it is quite probable that the better part of it would not be
generated by SC but by another application (e.g. sftp, http ...).
Therefore chances are that a QoS implementation inside SC would not
really help improve the situation.

Regards
Emil

Kolja Ehlers wrote:

hi all, sorry for getting back to this with such a delay but i was on
holiday and will start on this now. i would like to ask about emils
responds:

However, in case you want to use this in real world conditions then it
is probably not the best approach. You'd have to modify all stacks that
support file transfer, and make sure that they use your QoS queues, and
it would still not prevent other applications from occupying all
available bandwidth.

i dont really understand. of course the problem with other applications will
still hold but i dont want to look at that. i just want the sip communicator
and its sip services to run over the QoS queues. so are there not just the 2
stacks that i would have to modify, the one for audio and the one for
video?

it sounds pretty simple to me to just redirect the audio and the video data
into these buffers and not to send them straight out. the selector does
decide which is the next package to be send out.

thanks alot for your help

kolja

----- Original Message ----- From: "Emil Ivov" <emcho@emcho.com>
To: <dev@sip-communicator.dev.java.net>
Sent: Monday, July 16, 2007 3:55 PM
Subject: Re: [sip-comm-dev] Quality of Service

Hello Kolja,

(see inline)

Kolja Ehlers wrote:

hello everybody,
im not sure if anyone has every thought about anything like that or maybe
there already is some kind of solution for this. im thinking about a
quality
of service implementation for the SC doing two different things. the
probably more easy part to write a custom jmf rtpconnector to be able to
define the tos field for the ip pakets. after looking only vaguely over
the
rtpmanager initialization in SC i saw that you are using the addtarget
functions which will be a problem when initializing with a custom
rtpconnector.

This is easily changed, and besides, we might have to change it anyway
since I heard Su Bing (who is working on SRTP) is also having problems
with the same thing.

else has anyone ever thought about some kind of buffering the incoming
data
and then selecting which to send out. let me give you any example. you
are
doing a file transfer and have a audio phone conversation at the same
time.
both data is being put in two different buffers like this

> file | | audio |
> tranfer | | conversation |
> > > >
-------- ---------------- \
         \ <--- selector
          \

and then you have some kind of selecting mechanism to decide which data
to
send out first. the easiest way for this is probably to define for
example
audio as priority 1 and a file transfer as prio 2 and then the selector
chooses the buffer prio 1 for as long as there is data in that buffer.
when
there is nothing in there is checks buffer prio 2 and so on.

does anyone have any clue if this idea is actually realizable to program
into the SC. i mean in a one man job and with a time frame of about 1
month.
ill be thankful for any kind of help.

Depends on what your goal is. In case you are only working on a research
project and all you want to have at the end is a proof-of-concept demo,
then it might feasible.

However, in case you want to use this in real world conditions then it
is probably not the best approach. You'd have to modify all stacks that
support file transfer, and make sure that they use your QoS queues, and
it would still not prevent other applications from occupying all
available bandwidth.

For a more reliable implementation you'd probably have to do this in the
OS kernel itself. I think there is a fair amount of related art so you
might want to look around for existing implementations.

Hope this helps
Emil

where to start or anything that could
help

thanks alot for any help/time!

regards

kolja

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net