[sip-comm-dev] Call Recording: Week 1


#1

Hi all,

I don't have any progress to report on call recording yet, but rather
some questions and comments about the project.
As Emil suggested, I started to browse the source of the neomedia
package to see how the audio mixing works. The picture is not
completely clear to me, but here's what I found: the central class is
MediaServiceImpl that can create an audio mixer (which is a
MediaDevice) from some capture device (another MediaDevice) in
createMixer(). That method returns a AudioMixerMediaDevice class that
wraps the initial capture device and also has an AudioMixer instance.
AudioMixer can mix together multiple input streams into a single
output stream, it has an instance of AudioMixingPushBufferDataSource
that is a mix of all input data sources. The
AudioMixingPushBufferStream is the resulting stream.
There are many more classes and relations between them, but I think
these are more or less central to call recording. If anyone can give a
general overview of how the neomedia system works, that would be
great.
Also, should I set up some kind of environment for testing calls? Or
can I just use some echo service? I tried calling echo@iptel.org but
could not connect.

Any help appreciated,
Dmitri

···

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#2

If echo@iptel.org does not work for you (it doesn't seem to work for
me as well), you could for example create an account at sip2sip.info
and use its echo number 4444 (I just tested it and it seems to work as
expected).

Another option when you have just one computer and no other
SIP-enabled device is to run two instances of SIP Communicator with
different configuration folders (by setting the configuration
properties net.java.sip.communicator.SC_HOME_DIR_LOCATION and
net.java.sip.communicator.SC_HOME_DIR_NAME in the Ant target run). But
that will likely be more difficult to comprehend when working on
audio-related functionality.

···

On Wed, Jul 7, 2010 at 1:28 PM, Dmitri Melnikov <dmitri807@gmail.com> wrote:

Also, should I set up some kind of environment for testing calls? Or
can I just use some echo service? I tried calling echo@iptel.org but
could not connect.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#3

In order to get to know how the neomedia system works, I'd suggest
starting with its public interface. While its interfaces are in
service/neomedia, it will probably be easier to understand them while
looking at their usage. A great example is the way we use it in the
SIP protocol implementation and, more specifically, its
impl/protocol/sip/CallPeerMediaHandler.java. While the code there
isn't trivial itself, one should probably just focus on the dealing
with the neomedia interfaces and classes such as creating a
MediaStream, setting its properties. From there I expect one to also
understand how MediaDevices relate to MediaStreams. It's probably good
in the case to also look at the use of MediaService#createMixer in
CallSipImpl but that could happen at a later stage after getting to
know the basics of the public API of neomedia.

Once the terms MediaStream, MediaDevice, MediaFormat are clear from
the standpoint of service/neomedia, diving into impl/neomedia should
be easier. The new thing in impl/neomedia will likely be JMF which is
itself a non-trivial system. For the purposes of call recording, I'd
expect knowledge on the subject of MediaStream, MediaStreamImpl,
MediaDevice, MediaDeviceImpl and, especially, MediaDeviceSession,
AudioMediaDeviceSession and AudioMixerMediaDevice. MediaDeviceSession
encapsulates most of the dealing with JMF and AudioMixerMediaDevice
does a great deal of the magic with audio-mixing. Having an idea about
what goes over the wire into a MediaStream and out of it to the remote
peer in the terms of neomedia and JMF - ReceiveStream, SendStream,
DataSource, PushBufferDataSource, PullBufferDataSource,
PushBufferStream, PullBufferStream, CaptureDevice - is necessary as
well because the call recording will want to mix them.

As far as understand the requirements of the call recording project
and what Emil has told me, its implementation is not required to be
outside the neomedia bundle. Which means that the functionality will
have access to pretty much everything realted to media in terms of
neomedia and JMF. Call recoding will want to have the audio coming
from the CaptureDevice and the SendStream also wants it so that it can
send it to the remote peer. Because the call recording in a call is to
be started and stopped at arbitrary points in the life of the call, I
guess it means that neomedia will have to be ready at all times to
enable call recording i.e. AudioMediaDeviceSession#captureDevice will
need to be made available to two simultaneous uses - AudioMixer can do
(and in fact does for the JavaSound and PortAudio CaptureDevices) such
sharing of a single CaptureDevice. The same AudioMixer will also take
input from the ReceiveStreams (but they'll also be wanted for the
playback so their DataSources may need cloning) and output a
AudioMixingPushBufferDataSource which contains the mix of the send and
the received audio. At all times, we have to keep in mind that we may
need to add or modify functionality in AudioMixer in order to filly
configure what gets and what doesn't get mixed into a given
AudioMixingPushBufferDataSource.

···

On Wed, Jul 7, 2010 at 1:28 PM, Dmitri Melnikov <dmitri807@gmail.com> wrote:

If anyone can give a
general overview of how the neomedia system works, that would be
great.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#4

Now that I've had the time to clearly think about it, it seems to me
that we have pretty much everything regarding the mixing in our
conference calls: their CaptureDevice is AudioMixer-enabled and the
AudioMixer implementation already provides you a way to get a mix of
everything (i.e. capture and playback) -
AudioMixer#createOutputDataSource(). So it seems to me that it should
be much easier for you to start implementing call recoding in
conference calls. Instead of doing your testing with one-to-one calls
started by typing the number in the search field, create a call to
another person by going to Tools > Create a conference call... and add
just one person - you'll get a one-to-one call that is in fact a
conference call. Examine the neomedia code that's involved in such a
conference call and it should be easy to get its AudioMixer and
createOutputDataSource.

···

On Fri, Jul 9, 2010 at 4:19 PM, Lubomir Marinov <lubo@sip-communicator.org> wrote:

As far as understand the requirements of the call recording project
and what Emil has told me, its implementation is not required to be
outside the neomedia bundle. Which means that the functionality will
have access to pretty much everything realted to media in terms of
neomedia and JMF. Call recoding will want to have the audio coming
from the CaptureDevice and the SendStream also wants it so that it can
send it to the remote peer. Because the call recording in a call is to
be started and stopped at arbitrary points in the life of the call, I
guess it means that neomedia will have to be ready at all times to
enable call recording i.e. AudioMediaDeviceSession#captureDevice will
need to be made available to two simultaneous uses - AudioMixer can do
(and in fact does for the JavaSound and PortAudio CaptureDevices) such
sharing of a single CaptureDevice. The same AudioMixer will also take
input from the ReceiveStreams (but they'll also be wanted for the
playback so their DataSources may need cloning) and output a
AudioMixingPushBufferDataSource which contains the mix of the send and
the received audio. At all times, we have to keep in mind that we may
need to add or modify functionality in AudioMixer in order to filly
configure what gets and what doesn't get mixed into a given
AudioMixingPushBufferDataSource.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#5

Hey Dmitri, apologies for not responding earlier, I had to investigate
for such an environment my self.

So, another approach -closely related to what Lubomir suggests but maybe
slightly more convenient- is to download && build && run the pjsua SIP
User Agent, instead of a second SIP Communicator instance.

pjsua is an (open source) command line SIP user agent packed with some
interesting features like looping the incoming RTP stream to the
outgoing RTP stream (echo) and/or play an audio file for incoming calls.

Briefly, here are the steps I took to setup the environment:

1. wget http://www.pjsip.org/release/1.6/pjproject-1.6.tar.bz2 && tar xf
pjproject-1.6.tar.bz2 && cd pjproject-1.6 && ./configure && make dep &&
make clean && make && ./pjsip-apps/bin/pjsua-i686-pc-linux-gnu
--null-audio --auto-answer 200 --auto-loop

2. Launch SIP Communicator && add a local sip account && call 127.0.0.1

Normally, it should (just) work.

Cheers,
George

···

On 07/09/2010 02:04 PM, Lubomir Marinov wrote:

On Wed, Jul 7, 2010 at 1:28 PM, Dmitri Melnikov <dmitri807@gmail.com> wrote:

Also, should I set up some kind of environment for testing calls? Or
can I just use some echo service? I tried calling echo@iptel.org but
could not connect.

If echo@iptel.org does not work for you (it doesn't seem to work for
me as well), you could for example create an account at sip2sip.info
and use its echo number 4444 (I just tested it and it seems to work as
expected).

Another option when you have just one computer and no other
SIP-enabled device is to run two instances of SIP Communicator with
different configuration folders (by setting the configuration
properties net.java.sip.communicator.SC_HOME_DIR_LOCATION and
net.java.sip.communicator.SC_HOME_DIR_NAME in the Ant target run). But
that will likely be more difficult to comprehend when working on
audio-related functionality.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net