[sip-comm-dev] Three Issues


#1

I have found a few things I believe are issues with the SIP Communicator source
and wanted to post them here.

The first issue arose when attempting to create a one-way, video-only call where
the SIP-Communicator was recvonly and the other end was sendonly. If you take a
look in net.java.sip.communicator.impl.neomedia.device.MediaDeviceSession.java,
there is a method called startedDirectionChanged, which is called during the set
up process and has the (indirect?) affect of activating the thread that reads
RTP packets. Currently the method begins something like:
if(newValue.allowsSending()) {
    ... // start processor
} else if (...) {
    ... // stop processor
}

Note that if SIPCommunicator is recvonly, the allowsSending method will return
false and the communicator will never read the RTP packets.

I am not too familiar with the code, however, by adding an allowsReceiving else
if, which also starts the processor, I was able to get the code to read the RTP
packets. This may not be the appropriate way to fix it, but its probably useful
to know that it changes the behavior. If there is a more appropriate change you
can let me know or I'll read it off the dev list.

The 2nd issue involves sprop-parameter-sets for H.264 video. It appears that SIP
Communicator assumes that sequence and picture parameter sets are included
inline in the RTP data, ie in the form of NAL units that are sent before any
video-layer H.264 RTP data. However, RFC 3984 specifies that one should not send
parameter sets in the RTP stream if a non-reliable transport is being used. The
RFC only suggests this, so it is not illegal. However, the alternative to
in-band parameter sets is out of band parameter sets, i.e., inclusion in a
session control protocol using a reliable transport, eg SIP over TCP (actually
even SIP over UDP can be considered reliable due to its handshake-like INVITE
process). I noticed that even when I include the parameter sets in the fmtp
line of the SDP, that this is not used to configure the ffmpeg decoder, which
means, if the parameter set is not included in the RTP stream, the decoder will
never decode any video (I have seen many emails to the reflectors about video
decoding problems and this might be a good first suggestions at things to look
at for those people). I guess this could be by design, ie perhaps a SIP document
states that in-band is the way to go, however, it seems odd that that aspect of
the SDP would be ignored if it were available. To be fair, the way the code is
designed, it may not be trivial to make SDP parameters available at the time of
access unit decoding. In any case, this can almost always be overcome by
including the parameter set in the RTP stream.

The last issue is related to firewall (punch hole) packets. Although I often
ran SIP Communicator with a debugger running, and thus it may have been slower
than normal, I noticed a few times where the punch hole packets did not seem to
be generated in time to receive the first media packets. To be honest, I dont
think it was the network that was preventing the media packets from getting
through (I wasnt using a NAT in these tests), rather I think the code flow that
generates these packets only happens after the ACK is generated, which could
mean in some circumstances that the ACK makes it to the other endpoint and RTP
data is returned before the punch hole packets code is exercised. And I believe
the code that begins reading RTP packets isnt 'enabled' until after the punch
hole code is called. To be fair, I never saw this issue when I wasnt running a
debugger, so it may not be a critical issue, however, it does indicate a race
condition, which implies something about the code flow. This issue can be
especially problematic if the first RTP packets are in fact parameter sets,
since if no parameter sets are received, the decoder will never decode any
video and, obviously, no video will ever be displayed.

I hope these descriptions make sense. If you have any questions, please feel
free to ask.

thanks,
  
Mike

···

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#2

Mike, thank you very much for the clear and detailed reports! I'd like to share my understanding (and confirm your expectation) that starting the processor when !startedDirection.allowsSending() doesn't look right. The processor is the functionality which captures video from the camera, encodes it in the negotiated format and packetizes it in RTP packets to be sent to the remote peer i.e. the processor is utilised only when sending. When receiving, the functionality which depacketizes RTP packets received from the remote peer, decodes them in a format suitable for display and renders them is the player. That is why the scenario of !startedDirection.allowsSending() && startedDirection.allowsReceiving() is supposed to have the processor not started and the player started.

Seb and Emil, wasn't the NAT and RTP proxy hole punching packet meant to resolve this issue? The comments and code in CallPeerMediaHandler.start() state the following and seem to me to describe the scenario reported by Mike:

// send empty packet to deblock some kind of RTP proxy to let just
// one user sends its video
if(stream instanceof VideoMediaStream
         && !isLocalVideoTransmissionEnabled())
{
     sendHolePunchPacket(stream.getTarget());
}

···

On 09/29/2010 09:35 PM, severa@nbjsystems.com wrote:

The first issue arose when attempting to create a one-way, video-only call where
the SIP-Communicator was recvonly and the other end was sendonly. If you take a
look in net.java.sip.communicator.impl.neomedia.device.MediaDeviceSession.java,
there is a method called startedDirectionChanged, which is called during the set
up process and has the (indirect?) affect of activating the thread that reads
RTP packets. Currently the method begins something like:
if(newValue.allowsSending()) {
    ... // start processor
} else if (...) {
    ... // stop processor
}

Note that if SIPCommunicator is recvonly, the allowsSending method will return
false and the communicator will never read the RTP packets.

I am not too familiar with the code, however, by adding an allowsReceiving else
if, which also starts the processor, I was able to get the code to read the RTP
packets. This may not be the appropriate way to fix it, but its probably useful
to know that it changes the behavior. If there is a more appropriate change you
can let me know or I'll read it off the dev list.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#3

Hey folks,

На 29.09.10 22:18, Lubomir Marinov написа:

The first issue arose when attempting to create a one-way, video-only call where
the SIP-Communicator was recvonly and the other end was sendonly. If you take a
look in net.java.sip.communicator.impl.neomedia.device.MediaDeviceSession.java,
there is a method called startedDirectionChanged, which is called during the set
up process and has the (indirect?) affect of activating the thread that reads
RTP packets. Currently the method begins something like:
if(newValue.allowsSending()) {
    ... // start processor
} else if (...) {
    ... // stop processor
}

Note that if SIPCommunicator is recvonly, the allowsSending method will return
false and the communicator will never read the RTP packets.

I am not too familiar with the code, however, by adding an allowsReceiving else
if, which also starts the processor, I was able to get the code to read the RTP
packets. This may not be the appropriate way to fix it, but its probably useful
to know that it changes the behavior. If there is a more appropriate change you
can let me know or I'll read it off the dev list.

Mike, thank you very much for the clear and detailed reports! I'd like
to share my understanding (and confirm your expectation) that starting
the processor when !startedDirection.allowsSending() doesn't look right.

I might be missing something but it appears to me that we are doing
exactly the opposite (and hence the right) thing:

    if (newValue.allowsSending())
    {
        Processor processor = getProcessor();

        if (processor != null)
            startProcessorInAccordWithDirection(processor);
    }

and then in startProcessorInAccordWithDirection() we have:

    if (startedDirection.allowsSending()
        && (processor.getState() != Processor.Started))
    {
        processor.start();
        ...
    }

so I am really missing the issue here ... or, Lubo, did you also mean to
say that there was no issue?

Mike, if things worked our in your case then that's simply because, by
starting the processor, you started streaming video which then allowed
your server to send you the media from your peer. You could achieve the
same result with the trunk version of SC, by pushing the video button.

The processor is the functionality which captures video from the camera,
encodes it in the negotiated format and packetizes it in RTP packets to
be sent to the remote peer i.e. the processor is utilised only when
sending. When receiving, the functionality which depacketizes RTP
packets received from the remote peer, decodes them in a format suitable
for display and renders them is the player. That is why the scenario of
!startedDirection.allowsSending() && startedDirection.allowsReceiving()
is supposed to have the processor not started and the player started.

Seb and Emil, wasn't the NAT and RTP proxy hole punching packet meant to
resolve this issue? The comments and code in
CallPeerMediaHandler.start() state the following and seem to me to
describe the scenario reported by Mike:

It is indeed and it does work for some cases (e.g. ippi.com, running an
OpenSIPS + RTPProxy combo).

Sometimes however, like with asterisk, it doesn't have the desired
effect. I haven't been looking deeply enough into this but my guess is
that asterisk doesn't start sending media to a party until it actually
gets some. The empty hole punch packet probably doesn't reach its media
layer and hence doesn't trigger the stream.

This is wild speculation though so we'd need to investigate properly at
some point.

Cheers,
Emil

···

On 09/29/2010 09:35 PM, severa@nbjsystems.com wrote:

// send empty packet to deblock some kind of RTP proxy to let just
// one user sends its video
if(stream instanceof VideoMediaStream
         && !isLocalVideoTransmissionEnabled())
{
     sendHolePunchPacket(stream.getTarget());
}

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

--
Emil Ivov, Ph.D. 67000 Strasbourg,
Project Lead France
SIP Communicator
emcho@sip-communicator.org PHONE: +33.1.77.62.43.30
http://sip-communicator.org FAX: +33.1.77.62.47.31

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#4

I shared the opinion that we were already doing the right thing with
the processor and that Mike's suggestion to start the processor when
!startedDirection.allowsSending() &&
startedDirection.allowsReceiving() didn't seem right.

···

On Thu, Sep 30, 2010 at 12:53 AM, Emil Ivov <emcho@sip-communicator.org> wrote:

so I am really missing the issue here ... or, Lubo, did you also mean to
say that there was no issue?

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net