[jitsi-dev] Streaming Audio Data from raw File URGENT


#1

Hi developers,

I have integrated jitsi in a servlet container and want to send recorded raw
audio data from files to a sip client.
The format for the raw audio data is PCM (8000Hz, 16bit, mono).

To achieve this, I have implemented a "virtual" device, which extends
org.jitsi.impl.neomedia.device.AudioMediaDeviceImpl and provides the
reference to the datasource of the raw audio file.
This datasource extends
org.jitsi.impl.neomedia.jmfext.media.protocol.AbstractPullBufferCaptureDevic
e and uses internally
javax.media.protocol.audiorecord.DataSource.RAFPullSourceStream to read from
the raw file.
Even with having defined the Audioformat explicitly in the form
new AudioFormat(AudioFormat.LINEAR /* PCM */, 8000 /* sampleRate */, 16 /*
sampleSizeInBits */, 1 /* channels */, AudioFormat.LITTLE_ENDIAN,
AudioFormat.SIGNED) };
for both media device and datasource, I am still having troubles in getting
the audio stream initialized after CallPeerSipImpl.processInviteOK()
(AbstractPullBufferCaptureDevice not connected -> see trace below)

I don't understand, why this "AbstractPullBufferCaptureDevice not connected"
error is thrown, because I verified in debug mode, that the connect() method
of the datasource gets successfully called and the source file is available.

Please help!!!!!

(if there exists an alternative/better approach for sending raw audio data,
any hints would be appreciated!)

1000 thanks in advance and kind regards

Christian

Stack Trace:

INFO: Starting RTPSourceStream.
Apr 15, 2013 3:11:46 AM net.sf.fmj.media.Log error
SEVERE: Failed to configure: net.sf.fmj.media.ProcessEngine@1ec2baa
Apr 15, 2013 3:11:46 AM net.sf.fmj.media.Log error
SEVERE: IO exception:
org.jitsi.impl.neomedia.jmfext.media.protocol.AbstractPullBufferCaptureDevic
e$1 not connected

Apr 15, 2013 3:11:46 AM net.sf.fmj.media.Log error
SEVERE: Failed to configure: net.sf.fmj.media.ProcessEngine@147f9e
Apr 15, 2013 3:11:46 AM net.sf.fmj.media.Log error
SEVERE: IO exception:
org.jitsi.impl.neomedia.jmfext.media.protocol.AbstractPullBufferCaptureDevic
e$1 not connected

Apr 15, 2013 3:11:46 AM net.sf.fmj.media.Log error
SEVERE: Error: Unable to configure net.sf.fmj.media.ProcessEngine@1ec2baa
Apr 15, 2013 3:11:46 AM net.sf.fmj.media.Log error
SEVERE: Error: Unable to configure net.sf.fmj.media.ProcessEngine@147f9e
Apr 15, 2013 3:11:46 AM org.jitsi.util.Logger warn
WARNING:
javax.media.ResourceUnavailableEvent[source=com.sun.media.processor.unknown.
Handler@fe03ae,message=Failed to configure: failed to parse the input
media.]
Apr 15, 2013 3:11:46 AM org.jitsi.util.Logger warn
WARNING: ControllerErrorEvent:
javax.media.ResourceUnavailableEvent[source=com.sun.media.processor.unknown.
Handler@fe03ae,message=Failed to configure: failed to parse the input
media.]
Apr 15, 2013 3:11:46 AM org.jitsi.util.Logger warn
WARNING:
javax.media.ControllerClosedEvent[source=com.sun.media.processor.unknown.Han
dler@fe03ae]
Apr 15, 2013 3:11:46 AM org.jitsi.util.Logger debug
FINE: ControllerClosedEvent:
javax.media.ControllerClosedEvent[source=com.sun.media.processor.unknown.Han
dler@fe03ae]
Apr 15, 2013 3:11:46 AM org.jitsi.util.Logger debug
FINE: registering format ALAW/rtp, 8000.0 Hz, 8-bit, Mono, Signed with RTP
manager
Apr 15, 2013 3:11:46 AM org.jitsi.util.Logger debug
FINE: registering format g722/rtp, 8000.0 Hz, Mono with RTP manager
Apr 15, 2013 3:11:46 AM net.sf.fmj.media.Log warning
WARNING: Input DataSource:
org.jitsi.impl.neomedia.conference.AudioMixingPushBufferDataSource@2beef8
Apr 15, 2013 3:11:46 AM net.sf.fmj.media.Log warning
WARNING: Failed with IO exception: Got a empty stream array from the
DataSource
Apr 15, 2013 3:11:46 AM org.jitsi.util.Logger error
SEVERE: Failed to create Processor for
org.jitsi.impl.neomedia.conference.AudioMixingPushBufferDataSource@2beef8
java.io.IOException: Got a empty stream array from the DataSource
  at
net.sf.fmj.media.parser.RawPushBufferParser.setSource(RawPushBufferParser.ja
va:774)
  at
net.sf.fmj.media.BasicSourceModule.createDemultiplexer(BasicSourceModule.jav
  at
net.sf.fmj.media.BasicSourceModule.createModule(BasicSourceModule.java:105)
  at
net.sf.fmj.media.PlaybackEngine.setSource(PlaybackEngine.java:2048)
  at
net.sf.fmj.media.MediaProcessor.setSource(MediaProcessor.java:145)
  at javax.media.Manager.createProcessor(Manager.java:768)
  at javax.media.Manager.createProcessor(Manager.java:746)
  at
org.jitsi.impl.neomedia.device.MediaDeviceSession.createProcessor(MediaDevic
eSession.java:512)
  at
org.jitsi.impl.neomedia.device.MediaDeviceSession.getProcessor(MediaDeviceSe
ssion.java:1044)
  at
org.jitsi.impl.neomedia.device.MediaDeviceSession.startedDirectionChanged(Me
diaDeviceSession.java:2046)
  at
org.jitsi.impl.neomedia.device.MediaDeviceSession.start(MediaDeviceSession.j
ava:2024)
  at
org.jitsi.impl.neomedia.MediaStreamImpl.start(MediaStreamImpl.java:1914)
  at
org.jitsi.impl.neomedia.MediaStreamImpl.start(MediaStreamImpl.java:1877)
  at
net.java.sip.communicator.service.protocol.media.CallPeerMediaHandler.start(
CallPeerMediaHandler.java:1790)
  at
net.java.sip.communicator.impl.protocol.sip.CallPeerMediaHandlerSipImpl.star
t(CallPeerMediaHandlerSipImpl.java:1350)
  at
net.java.sip.communicator.impl.protocol.sip.CallPeerSipImpl.processInviteOK(
CallPeerSipImpl.java:849)
  at
net.java.sip.communicator.impl.protocol.sip.OperationSetBasicTelephonySipImp
l.processInviteOK(OperationSetBasicTelephonySipImpl.java:788)
  at
net.java.sip.communicator.impl.protocol.sip.OperationSetBasicTelephonySipImp
l.processResponse(OperationSetBasicTelephonySipImpl.java:411)
  at
net.java.sip.communicator.impl.protocol.sip.ProtocolProviderServiceSipImpl.p
rocessResponse(ProtocolProviderServiceSipImpl.java:660)
  at
net.java.sip.communicator.impl.protocol.sip.SipStackSharing.processResponse(
SipStackSharing.java:704)
  at
gov.nist.javax.sip.EventScanner.deliverEvent(EventScanner.java:296)
  at
gov.nist.javax.sip.SipProviderImpl.handleEvent(SipProviderImpl.java:196)
  at
gov.nist.javax.sip.DialogFilter.processResponse(DialogFilter.java:1480)
  at
gov.nist.javax.sip.stack.SIPClientTransaction.inviteClientTransaction(SIPCli
entTransaction.java:885)
  at
gov.nist.javax.sip.stack.SIPClientTransaction.processResponse(SIPClientTrans
action.java:548)
  at
gov.nist.javax.sip.stack.SIPClientTransaction.processResponse(SIPClientTrans
action.java:1554)
  at
gov.nist.javax.sip.stack.TCPMessageChannel.processMessage(TCPMessageChannel.
java:749)
  at
gov.nist.javax.sip.parser.PipelinedMsgParser.run(PipelinedMsgParser.java:470
)
  at java.lang.Thread.run(Thread.java:722)

···

a:70)


#2

I'd suggest that you'd better figure out how "Got a empty stream array
from the DataSource" is caused by and/or affects your code.

···

2013/4/15 Christian Seeberger <c.seeberger@web.de>:

Apr 15, 2013 3:11:46 AM net.sf.fmj.media.Log warning
WARNING: Input DataSource:
org.jitsi.impl.neomedia.conference.AudioMixingPushBufferDataSource@2beef8
Apr 15, 2013 3:11:46 AM net.sf.fmj.media.Log warning
WARNING: Failed with IO exception: Got a empty stream array from the
DataSource
Apr 15, 2013 3:11:46 AM org.jitsi.util.Logger error
SEVERE: Failed to create Processor for
org.jitsi.impl.neomedia.conference.AudioMixingPushBufferDataSource@2beef8
java.io.IOException: Got a empty stream array from the DataSource


#3

Hi, thanks very much for your advice!

The problem is, that is not trivial to identify the "Got a empty stream
array from the DataSource" problem, otherwise I would not have posted this
issue :wink:
I will keep on debugging and analyzing...

In the meantime it would be very helpful, if you could confirm that my
approach for sending raw audio files (extending
AbstractPullBufferCaptureDevice and AudioMediaDeviceImpl) is correct, or if
you would recommend a different way for realizing this goal.

Kind regards

···

-----Ursprüngliche Nachricht-----
Von: lubo@sip-communicator.org [mailto:lubo@sip-communicator.org] Im Auftrag
von Lyubomir Marinov
Gesendet: Montag, 15. April 2013 11:08
An: dev@jitsi.java.net
Betreff: [jitsi-dev] Re: Streaming Audio Data from raw File URGENT

2013/4/15 Christian Seeberger <c.seeberger@web.de>:

Apr 15, 2013 3:11:46 AM net.sf.fmj.media.Log warning
WARNING: Input DataSource:
org.jitsi.impl.neomedia.conference.AudioMixingPushBufferDataSource@2be
ef8 Apr 15, 2013 3:11:46 AM net.sf.fmj.media.Log warning
WARNING: Failed with IO exception: Got a empty stream array from the
DataSource
Apr 15, 2013 3:11:46 AM org.jitsi.util.Logger error
SEVERE: Failed to create Processor for
org.jitsi.impl.neomedia.conference.AudioMixingPushBufferDataSource@2be
ef8
java.io.IOException: Got a empty stream array from the DataSource

I'd suggest that you'd better figure out how "Got a empty stream array from
the DataSource" is caused by and/or affects your code.


#4

Your approach appears to be the right way to go about it, the
experienced problem sounds like a glitch in your implementation.

···

2013/4/15 Christian Seeberger <c.seeberger@web.de>:

In the meantime it would be very helpful, if you could confirm that my
approach for sending raw audio files (extending
AbstractPullBufferCaptureDevice and AudioMediaDeviceImpl) is correct, or if
you would recommend a different way for realizing this goal.


#5

Unfortunately, I am still stuck with the problem.
Can you please confirm that the AudioMixingPushBufferDataSource is only
intended for local playback of the call?
If I only want to send mono PCM audio to the client, could it be a solution,
not to use AudioMixingPushBufferDataSource, and how could this be achieved?
Thanks for your help!

···

-----Ursprüngliche Nachricht-----
Von: lubo@sip-communicator.org [mailto:lubo@sip-communicator.org] Im Auftrag
von Lyubomir Marinov
Gesendet: Montag, 15. April 2013 13:17
An: dev@jitsi.java.net
Betreff: [jitsi-dev] Re: Streaming Audio Data from raw File URGENT

2013/4/15 Christian Seeberger <c.seeberger@web.de>:

In the meantime it would be very helpful, if you could confirm that my
approach for sending raw audio files (extending
AbstractPullBufferCaptureDevice and AudioMediaDeviceImpl) is correct,
or if you would recommend a different way for realizing this goal.

Your approach appears to be the right way to go about it, the experienced
problem sounds like a glitch in your implementation.

.


#6

No, I cannot confirm that statement. Jitsi creates multiple instances
of AudioMixingPushBufferDataSource for a single AudioMixer: one is
indeed intended for local playback in the call, a second one is
encoded and sent to the remote user/peer in the call (there is an
AudioMixingPushBufferDataSource instance per remote user/peer,
actually, in a conference call) and a third one will be created if the
call is recorded into a file.

Anyway, you likely do not want to do anything explicit with the
AudioMixer class and its related (i.e. AudioMixer*, AudioMixing*)
classes in your scenario.
.

···

2013/4/15 Christian Seeberger <c.seeberger@web.de>:

Can you please confirm that the AudioMixingPushBufferDataSource is only
intended for local playback of the call?


#7

Thanks for your advice. Indeed, for just sending recorded audio data, the
mixer does not make sense...
The problem is: How do I get rid of the mixer? As I remember, the mixing
device is internally created during call peer creation...

···

-----Ursprüngliche Nachricht-----
Von: lubo@sip-communicator.org [mailto:lubo@sip-communicator.org] Im Auftrag
von Lyubomir Marinov
Gesendet: Montag, 15. April 2013 21:35
An: dev@jitsi.java.net
Betreff: [jitsi-dev] Re: Streaming Audio Data from raw File URGENT

2013/4/15 Christian Seeberger <c.seeberger@web.de>:

Can you please confirm that the AudioMixingPushBufferDataSource is
only intended for local playback of the call?

No, I cannot confirm that statement. Jitsi creates multiple instances of
AudioMixingPushBufferDataSource for a single AudioMixer: one is indeed
intended for local playback in the call, a second one is encoded and sent to
the remote user/peer in the call (there is an
AudioMixingPushBufferDataSource instance per remote user/peer, actually, in
a conference call) and a third one will be created if the call is recorded
into a file.

Anyway, you likely do not want to do anything explicit with the AudioMixer
class and its related (i.e. AudioMixer*, AudioMixing*) classes in your
scenario.

.


#8

The only approach to ridding yourself of the AudioMixer is to modify
the source code of Jitsi and remove the invocation of
org.jitsi.service.neomedia.MediaService.createMixer(MediaDevice) there
in the case of audio.
.

···

2013/4/15 Christian Seeberger <c.seeberger@web.de>:

The problem is: How do I get rid of the mixer? As I remember, the mixing
device is internally created during call peer creation...


#9

OK, I'll check this out! Thank you so far.

···

-----Ursprüngliche Nachricht-----
Von: lubo@sip-communicator.org [mailto:lubo@sip-communicator.org] Im Auftrag
von Lyubomir Marinov
Gesendet: Montag, 15. April 2013 22:36
An: dev@jitsi.java.net
Betreff: [jitsi-dev] Re: Streaming Audio Data from raw File URGENT

2013/4/15 Christian Seeberger <c.seeberger@web.de>:

The problem is: How do I get rid of the mixer? As I remember, the
mixing device is internally created during call peer creation...

The only approach to ridding yourself of the AudioMixer is to modify the
source code of Jitsi and remove the invocation of
org.jitsi.service.neomedia.MediaService.createMixer(MediaDevice) there in
the case of audio.

.


#10

As it is not so easy to strip out the mixer stuff, I am thinking about a
different approach:
In fact, as it is no problem to send audio data from a wav file (via file
url device locator)), the streaming should be in principle also be able for
raw files.
However the stream set up is failing, when using e.g. the ".raw" extension
for the file as there is no match in the mime type mappings for this
extension.
Question: Does there a dedicated file extension exist in order to indicate a
raw audio type and if not, is there a possibility to add this mapping?
Cheers

···

-----Ursprüngliche Nachricht-----
Von: Christian Seeberger [mailto:c.seeberger@web.de]
Gesendet: Montag, 15. April 2013 22:45
An: dev@jitsi.java.net
Betreff: [jitsi-dev] Re: Streaming Audio Data from raw File URGENT

OK, I'll check this out! Thank you so far.

-----Ursprüngliche Nachricht-----
Von: lubo@sip-communicator.org [mailto:lubo@sip-communicator.org] Im Auftrag
von Lyubomir Marinov
Gesendet: Montag, 15. April 2013 22:36
An: dev@jitsi.java.net
Betreff: [jitsi-dev] Re: Streaming Audio Data from raw File URGENT

2013/4/15 Christian Seeberger <c.seeberger@web.de>:

The problem is: How do I get rid of the mixer? As I remember, the
mixing device is internally created during call peer creation...

The only approach to ridding yourself of the AudioMixer is to modify the
source code of Jitsi and remove the invocation of
org.jitsi.service.neomedia.MediaService.createMixer(MediaDevice) there in
the case of audio.


#11

As I did still have troubles with the raw data, I implemented a workaround
with creating wav files from raw audio and declaring them in the custom
audio device as medialocator file url datasource.
Thanks for your help!

···

-----Ursprüngliche Nachricht-----
Von: Christian Seeberger [mailto:c.seeberger@web.de]
Gesendet: Dienstag, 16. April 2013 00:10
An: 'dev@jitsi.java.net'
Betreff: AW: [jitsi-dev] Re: Streaming Audio Data from raw File URGENT

As it is not so easy to strip out the mixer stuff, I am thinking about a
different approach:
In fact, as it is no problem to send audio data from a wav file (via file
url device locator)), the streaming should be in principle also be able for
raw files.
However the stream set up is failing, when using e.g. the ".raw" extension
for the file as there is no match in the mime type mappings for this
extension.
Question: Does there a dedicated file extension exist in order to indicate a
raw audio type and if not, is there a possibility to add this mapping?
Cheers

-----Ursprüngliche Nachricht-----
Von: Christian Seeberger [mailto:c.seeberger@web.de]
Gesendet: Montag, 15. April 2013 22:45
An: dev@jitsi.java.net
Betreff: [jitsi-dev] Re: Streaming Audio Data from raw File URGENT

OK, I'll check this out! Thank you so far.

-----Ursprüngliche Nachricht-----
Von: lubo@sip-communicator.org [mailto:lubo@sip-communicator.org] Im Auftrag
von Lyubomir Marinov
Gesendet: Montag, 15. April 2013 22:36
An: dev@jitsi.java.net
Betreff: [jitsi-dev] Re: Streaming Audio Data from raw File URGENT

2013/4/15 Christian Seeberger <c.seeberger@web.de>:

The problem is: How do I get rid of the mixer? As I remember, the
mixing device is internally created during call peer creation...

The only approach to ridding yourself of the AudioMixer is to modify the
source code of Jitsi and remove the invocation of
org.jitsi.service.neomedia.MediaService.createMixer(MediaDevice) there in
the case of audio.

.


#12

Hi, unfortunately I have one further question:

As I am using now a datasource with "file" media locator for sending audio to a sip peer, I would like to detect the end of the file input stream and hang up the peer afterwards. Right now, the peer hears just silence after the end of the streamed audio input file.

Is there a smart way to acheive this goal (messaging, callback hook, etc)?

···

Am 17.04.2013 um 02:27 schrieb "Christian Seeberger" <c.seeberger@web.de>:

As I did still have troubles with the raw data, I implemented a workaround
with creating wav files from raw audio and declaring them in the custom
audio device as medialocator file url datasource.
Thanks for your help!

-----Ursprüngliche Nachricht-----
Von: Christian Seeberger [mailto:c.seeberger@web.de]
Gesendet: Dienstag, 16. April 2013 00:10
An: 'dev@jitsi.java.net'
Betreff: AW: [jitsi-dev] Re: Streaming Audio Data from raw File URGENT

As it is not so easy to strip out the mixer stuff, I am thinking about a
different approach:
In fact, as it is no problem to send audio data from a wav file (via file
url device locator)), the streaming should be in principle also be able
for raw files.
However the stream set up is failing, when using e.g. the ".raw" extension
for the file as there is no match in the mime type mappings for this
extension.
Question: Does there a dedicated file extension exist in order to indicate
a raw audio type and if not, is there a possibility to add this mapping?
Cheers

-----Ursprüngliche Nachricht-----
Von: Christian Seeberger [mailto:c.seeberger@web.de]
Gesendet: Montag, 15. April 2013 22:45
An: dev@jitsi.java.net
Betreff: [jitsi-dev] Re: Streaming Audio Data from raw File URGENT

OK, I'll check this out! Thank you so far.

-----Ursprüngliche Nachricht-----
Von: lubo@sip-communicator.org [mailto:lubo@sip-communicator.org] Im
Auftrag von Lyubomir Marinov
Gesendet: Montag, 15. April 2013 22:36
An: dev@jitsi.java.net
Betreff: [jitsi-dev] Re: Streaming Audio Data from raw File URGENT

2013/4/15 Christian Seeberger <c.seeberger@web.de>:

The problem is: How do I get rid of the mixer? As I remember, the
mixing device is internally created during call peer creation...

The only approach to ridding yourself of the AudioMixer is to modify the
source code of Jitsi and remove the invocation of
org.jitsi.service.neomedia.MediaService.createMixer(MediaDevice) there in
the case of audio.