[jitsi-users] ICE connected with no Audio with org.jitsi.impl.neomedia.rtp.translator.OutputDataStreamImpl.warn() Will not translate RTP packet.


#1

Hi,
I am using standalone JVB without jicofo and jitsi-meet. Sometimes the ICE connects but no packets are received on the browser’s end. Whenever this happens “org.jitsi.impl.neomedia.rtp.translator.OutputDataStreamImpl.warn() Will not translate RTP packet.” appears in the log continuously. Also in my recording metadata.json, for that particular call the endpointId does not get recorded and for all further calls in the same conference the start time in metadata file is recorded very inaccurately. Please find attached the logs for the situation. The first call works fine in the logs but the second one has problems -

I am using video bridge tag 627.

jvb2.log (84 KB)


#2

We fixed an issue with exactly the same symptoms recently (after 627).

Regards,
Boris

···

On 17/02/16 07:50, Somil Bansal wrote:

Hi,
I am using standalone JVB without jicofo and jitsi-meet. Sometimes the ICE connects but no packets are received on the browser’s end. Whenever this happens “org.jitsi.impl.neomedia.rtp.translator.OutputDataStreamImpl.warn() Will not translate RTP packet.” appears in the log continuously. Also in my recording metadata.json, for that particular call the endpointId does not get recorded and for all further calls in the same conference the start time in metadata file is recorded very inaccurately. Please find attached the logs for the situation. The first call works fine in the logs but the second one has problems -

I am using video bridge tag 627.


#3

Hi Boris,

As per your suggestion, I updated my JVB. But the DTLS fixes in the new JVB seem to have a problem. In a video call, if Client Hello message arrives before createDataOutputStream is called for the video content, then it tries to create another DataOutputStream but without any target, which causes any further DTLS responses like Server Hello to be failed to send. One solution is to make sure that this new OutputStream is not automatically created on a DTLS message receipt. I added the create parameter false in doSend function(transform/dtls/DatagramTransportImpl.java) in outputStream = connector.getDataOutputStream(false). This makes sure that DTLS messages are ignored until OutputStream is created for video content. This solved the problem in Chrome. But Firefox is still facing issue. Can you look into it for any possible problems/error.

···

On 17-Feb-2016, at 7:20 PM, Somil Bansal <somilbansal12@gmail.com> wrote:

Hi,
I am using standalone JVB without jicofo and jitsi-meet. Sometimes the ICE connects but no packets are received on the browser’s end. Whenever this happens “org.jitsi.impl.neomedia.rtp.translator.OutputDataStreamImpl.warn() Will not translate RTP packet.” appears in the log continuously. Also in my recording metadata.json, for that particular call the endpointId does not get recorded and for all further calls in the same conference the start time in metadata file is recorded very inaccurately. Please find attached the logs for the situation. The first call works fine in the logs but the second one has problems -

I am using video bridge tag 627.

<jvb2.log>


#4

Hi Somil,

Hi Boris,

As per your suggestion, I updated my JVB. But the DTLS fixes in the
new JVB seem to have a problem. In a video call, if Client Hello
message arrives before createDataOutputStream is called for the video
content, then it tries to create another DataOutputStream but without
any target, which causes any further DTLS responses like Server Hello
to be failed to send. One solution is to make sure that this new
OutputStream is not automatically created on a DTLS message receipt.
I added the create parameter false in doSend
function(transform/dtls/DatagramTransportImpl.java) in outputStream =
connector.getDataOutputStream(false). This makes sure that DTLS
messages are ignored until OutputStream is created for video content.
This solved the problem in Chrome. But Firefox is still facing issue.
Can you look into it for any possible problems/error.

I can't quite understand what the issue is. Can you be more specific as to what is not initialized at the time of reception of the first DTLS packet? If you are getting an exception, please share the stack trace.

We have been using this code for a few weeks now, and we haven't run into something like this so far. So the problem is likely triggered by something that is different in your environment.

Regards,
Boris

···

On 01/03/16 13:36, Somil Bansal wrote:


#5

Hi Boris,

I’ll try to explain. The DTLS Client hello can be sent by the browser before the ICE state being connected. Following function in libjitsi/src/org/jitsi/impl/neomedia/AbstractRTPConnector.java might be causing the race condition -

public RTPConnectorOutputStream getDataOutputStream(boolean create)
        throws IOException
    {
        if ((dataOutputStream == null) && create) {
            dataOutputStream = createDataOutputStream();
        }
        return dataOutputStream;
    }

There are two paths from which this function for the the same object can be executed but in different threads. One path is when ICE state gets connected and a new target address is set for the MediaStream -

Somil: org.jitsi.impl.neomedia.AbstractRTPConnector.getDataOutputStream(AbstractRTPConnector.java:335)
Somil: org.jitsi.impl.neomedia.AbstractRTPConnector.getDataOutputStream(AbstractRTPConnector.java:315)
Somil: org.jitsi.impl.neomedia.AbstractRTPConnector.addTarget(AbstractRTPConnector.java:113)
Somil: org.jitsi.impl.neomedia.MediaStreamImpl.doSetTarget(MediaStreamImpl.java:1242)
Somil: org.jitsi.impl.neomedia.MediaStreamImpl.setTarget(MediaStreamImpl.java:2790)
Somil: org.jitsi.videobridge.RtpChannel.maybeStartStream(RtpChannel.java:1064)
Somil: org.jitsi.videobridge.Channel.transportConnected(Channel.java:776)
Somil: org.jitsi.videobridge.IceUdpTransportManager.onIceConnected(IceUdpTransportManager.java:1784)
Somil: org.jitsi.videobridge.IceUdpTransportManager.access$800(IceUdpTransportManager.java:47)
Somil: org.jitsi.videobridge.IceUdpTransportManager$3.run(IceUdpTransportManager.java:1826)

One path can be when a DTLS packet is being sent from the DTLS server in a different thread

Somil: org.jitsi.impl.neomedia.AbstractRTPConnector.getDataOutputStream(AbstractRTPConnector.java:335)
Somil: org.jitsi.impl.neomedia.AbstractRTPConnector.getDataOutputStream(AbstractRTPConnector.java:315)
Somil: org.jitsi.impl.neomedia.transform.dtls.DatagramTransportImpl.doSend(DatagramTransportImpl.java:185)
Somil: org.jitsi.impl.neomedia.transform.dtls.DatagramTransportImpl.flush(DatagramTransportImpl.java:226)
Somil: org.jitsi.impl.neomedia.transform.dtls.DatagramTransportImpl.send(DatagramTransportImpl.java:541)
Somil: org.bouncycastle.crypto.tls.DTLSRecordLayer.sendRecord(Unknown Source)
Somil: org.bouncycastle.crypto.tls.DTLSRecordLayer.send(Unknown Source)
Somil: org.bouncycastle.crypto.tls.DTLSReliableHandshake$RecordLayerBuffer.sendToRecordLayer(Unknown Source)
Somil: org.bouncycastle.crypto.tls.DTLSReliableHandshake.writeHandshakeFragment(Unknown Source)
Somil: org.bouncycastle.crypto.tls.DTLSReliableHandshake.writeMessage(Unknown Source)
Somil: org.bouncycastle.crypto.tls.DTLSReliableHandshake.sendMessage(Unknown Source)
Somil: org.bouncycastle.crypto.tls.DTLSServerProtocol.serverHandshake(Unknown Source)
Somil: org.bouncycastle.crypto.tls.DTLSServerProtocol.accept(Unknown Source)
Somil: org.jitsi.impl.neomedia.transform.dtls.DtlsPacketTransformer.runInConnectThread(DtlsPacketTransformer.java:981)
Somil: org.jitsi.impl.neomedia.transform.dtls.DtlsPacketTransformer.access$000(DtlsPacketTransformer.java:38)
Somil: org.jitsi.impl.neomedia.transform.dtls.DtlsPacketTransformer$1.run(DtlsPacketTransformer.java:1212)

These two can occur at the same time when early DTLS packets arrive but from different threads. The problem is that in the above function without any synchronisation, the dataOutputStream may be null and may get created for both (dataOutputStream = createDataOutputStream(); ). In case the DTLS path creates the final dataOutputStream, then it will not have any target addresses set to it. Which will cause any further packet sending failure due to lack of target address.

it’s a very clear race condition, weird that no-one else is facing it. If you have any doubt about the explanation then you can ask me.

Regards
Somil Bansal

···

On 02-Mar-2016, at 5:16 AM, Boris Grozev <boris@jitsi.org> wrote:

Hi Somil,

On 01/03/16 13:36, Somil Bansal wrote:

Hi Boris,

As per your suggestion, I updated my JVB. But the DTLS fixes in the
new JVB seem to have a problem. In a video call, if Client Hello
message arrives before createDataOutputStream is called for the video
content, then it tries to create another DataOutputStream but without
any target, which causes any further DTLS responses like Server Hello
to be failed to send. One solution is to make sure that this new
OutputStream is not automatically created on a DTLS message receipt.
I added the create parameter false in doSend
function(transform/dtls/DatagramTransportImpl.java) in outputStream =
connector.getDataOutputStream(false). This makes sure that DTLS
messages are ignored until OutputStream is created for video content.
This solved the problem in Chrome. But Firefox is still facing issue.
Can you look into it for any possible problems/error.

I can't quite understand what the issue is. Can you be more specific as to what is not initialized at the time of reception of the first DTLS packet? If you are getting an exception, please share the stack trace.

We have been using this code for a few weeks now, and we haven't run into something like this so far. So the problem is likely triggered by something that is different in your environment.

Regards,
Boris

_______________________________________________
dev mailing list
dev@jitsi.org <mailto:dev@jitsi.org>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#6

Hi Boris,

Did you have a chance to look at it?

Regards
Somil Bansal

···

On 03-Mar-2016, at 6:06 PM, Somil Bansal <somilbansal12@gmail.com> wrote:

Hi Boris,

I’ll try to explain. The DTLS Client hello can be sent by the browser before the ICE state being connected. Following function in libjitsi/src/org/jitsi/impl/neomedia/AbstractRTPConnector.java might be causing the race condition -

public RTPConnectorOutputStream getDataOutputStream(boolean create)
        throws IOException
    {
        if ((dataOutputStream == null) && create) {
            dataOutputStream = createDataOutputStream();
        }
        return dataOutputStream;
    }

There are two paths from which this function for the the same object can be executed but in different threads. One path is when ICE state gets connected and a new target address is set for the MediaStream -

Somil: org.jitsi.impl.neomedia.AbstractRTPConnector.getDataOutputStream(AbstractRTPConnector.java:335)
Somil: org.jitsi.impl.neomedia.AbstractRTPConnector.getDataOutputStream(AbstractRTPConnector.java:315)
Somil: org.jitsi.impl.neomedia.AbstractRTPConnector.addTarget(AbstractRTPConnector.java:113)
Somil: org.jitsi.impl.neomedia.MediaStreamImpl.doSetTarget(MediaStreamImpl.java:1242)
Somil: org.jitsi.impl.neomedia.MediaStreamImpl.setTarget(MediaStreamImpl.java:2790)
Somil: org.jitsi.videobridge.RtpChannel.maybeStartStream(RtpChannel.java:1064)
Somil: org.jitsi.videobridge.Channel.transportConnected(Channel.java:776)
Somil: org.jitsi.videobridge.IceUdpTransportManager.onIceConnected(IceUdpTransportManager.java:1784)
Somil: org.jitsi.videobridge.IceUdpTransportManager.access$800(IceUdpTransportManager.java:47)
Somil: org.jitsi.videobridge.IceUdpTransportManager$3.run(IceUdpTransportManager.java:1826)

One path can be when a DTLS packet is being sent from the DTLS server in a different thread

Somil: org.jitsi.impl.neomedia.AbstractRTPConnector.getDataOutputStream(AbstractRTPConnector.java:335)
Somil: org.jitsi.impl.neomedia.AbstractRTPConnector.getDataOutputStream(AbstractRTPConnector.java:315)
Somil: org.jitsi.impl.neomedia.transform.dtls.DatagramTransportImpl.doSend(DatagramTransportImpl.java:185)
Somil: org.jitsi.impl.neomedia.transform.dtls.DatagramTransportImpl.flush(DatagramTransportImpl.java:226)
Somil: org.jitsi.impl.neomedia.transform.dtls.DatagramTransportImpl.send(DatagramTransportImpl.java:541)
Somil: org.bouncycastle.crypto.tls.DTLSRecordLayer.sendRecord(Unknown Source)
Somil: org.bouncycastle.crypto.tls.DTLSRecordLayer.send(Unknown Source)
Somil: org.bouncycastle.crypto.tls.DTLSReliableHandshake$RecordLayerBuffer.sendToRecordLayer(Unknown Source)
Somil: org.bouncycastle.crypto.tls.DTLSReliableHandshake.writeHandshakeFragment(Unknown Source)
Somil: org.bouncycastle.crypto.tls.DTLSReliableHandshake.writeMessage(Unknown Source)
Somil: org.bouncycastle.crypto.tls.DTLSReliableHandshake.sendMessage(Unknown Source)
Somil: org.bouncycastle.crypto.tls.DTLSServerProtocol.serverHandshake(Unknown Source)
Somil: org.bouncycastle.crypto.tls.DTLSServerProtocol.accept(Unknown Source)
Somil: org.jitsi.impl.neomedia.transform.dtls.DtlsPacketTransformer.runInConnectThread(DtlsPacketTransformer.java:981)
Somil: org.jitsi.impl.neomedia.transform.dtls.DtlsPacketTransformer.access$000(DtlsPacketTransformer.java:38)
Somil: org.jitsi.impl.neomedia.transform.dtls.DtlsPacketTransformer$1.run(DtlsPacketTransformer.java:1212)

These two can occur at the same time when early DTLS packets arrive but from different threads. The problem is that in the above function without any synchronisation, the dataOutputStream may be null and may get created for both (dataOutputStream = createDataOutputStream(); ). In case the DTLS path creates the final dataOutputStream, then it will not have any target addresses set to it. Which will cause any further packet sending failure due to lack of target address.

it’s a very clear race condition, weird that no-one else is facing it. If you have any doubt about the explanation then you can ask me.

Regards
Somil Bansal

On 02-Mar-2016, at 5:16 AM, Boris Grozev <boris@jitsi.org <mailto:boris@jitsi.org>> wrote:

Hi Somil,

On 01/03/16 13:36, Somil Bansal wrote:

Hi Boris,

As per your suggestion, I updated my JVB. But the DTLS fixes in the
new JVB seem to have a problem. In a video call, if Client Hello
message arrives before createDataOutputStream is called for the video
content, then it tries to create another DataOutputStream but without
any target, which causes any further DTLS responses like Server Hello
to be failed to send. One solution is to make sure that this new
OutputStream is not automatically created on a DTLS message receipt.
I added the create parameter false in doSend
function(transform/dtls/DatagramTransportImpl.java) in outputStream =
connector.getDataOutputStream(false). This makes sure that DTLS
messages are ignored until OutputStream is created for video content.
This solved the problem in Chrome. But Firefox is still facing issue.
Can you look into it for any possible problems/error.

I can't quite understand what the issue is. Can you be more specific as to what is not initialized at the time of reception of the first DTLS packet? If you are getting an exception, please share the stack trace.

We have been using this code for a few weeks now, and we haven't run into something like this so far. So the problem is likely triggered by something that is different in your environment.

Regards,
Boris

_______________________________________________
dev mailing list
dev@jitsi.org <mailto:dev@jitsi.org>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#7

Hi Somil,

Sorry for the long delay, it's been a busy few days.

Hi Boris,

I’ll try to explain. The DTLS Client hello can be sent by the browser
before the ICE state being connected. Following function in
libjitsi/src/org/jitsi/impl/neomedia/AbstractRTPConnector.java might be
causing the race condition -

public RTPConnectorOutputStream getDataOutputStream(boolean create)
         throws IOException
     {
         if ((dataOutputStream == null) && create) {
             dataOutputStream = createDataOutputStream();
         }
         return dataOutputStream;
     }

There are two paths from which this function for the the same object can
be executed but in different threads. One path is when ICE state gets
connected and a new target address is set for the MediaStream -

Somil:
org.jitsi.impl.neomedia.AbstractRTPConnector.getDataOutputStream(AbstractRTPConnector.java:335)
Somil:
org.jitsi.impl.neomedia.AbstractRTPConnector.getDataOutputStream(AbstractRTPConnector.java:315)
Somil:
org.jitsi.impl.neomedia.AbstractRTPConnector.addTarget(AbstractRTPConnector.java:113)
Somil:
org.jitsi.impl.neomedia.MediaStreamImpl.doSetTarget(MediaStreamImpl.java:1242)
Somil:
org.jitsi.impl.neomedia.MediaStreamImpl.setTarget(MediaStreamImpl.java:2790)
Somil:
org.jitsi.videobridge.RtpChannel.maybeStartStream(RtpChannel.java:1064)
Somil: org.jitsi.videobridge.Channel.transportConnected(Channel.java:776)
Somil:
org.jitsi.videobridge.IceUdpTransportManager.onIceConnected(IceUdpTransportManager.java:1784)
Somil:
org.jitsi.videobridge.IceUdpTransportManager.access$800(IceUdpTransportManager.java:47)
Somil:
org.jitsi.videobridge.IceUdpTransportManager$3.run(IceUdpTransportManager.java:1826)

One path can be when a DTLS packet is being sent from the DTLS server in
a different thread

Somil:
org.jitsi.impl.neomedia.AbstractRTPConnector.getDataOutputStream(AbstractRTPConnector.java:335)
Somil:
org.jitsi.impl.neomedia.AbstractRTPConnector.getDataOutputStream(AbstractRTPConnector.java:315)
Somil:
org.jitsi.impl.neomedia.transform.dtls.DatagramTransportImpl.doSend(DatagramTransportImpl.java:185)
Somil:
org.jitsi.impl.neomedia.transform.dtls.DatagramTransportImpl.flush(DatagramTransportImpl.java:226)
Somil:
org.jitsi.impl.neomedia.transform.dtls.DatagramTransportImpl.send(DatagramTransportImpl.java:541)
Somil: org.bouncycastle.crypto.tls.DTLSRecordLayer.sendRecord(Unknown
Source)
Somil: org.bouncycastle.crypto.tls.DTLSRecordLayer.send(Unknown Source)
Somil:
org.bouncycastle.crypto.tls.DTLSReliableHandshake$RecordLayerBuffer.sendToRecordLayer(Unknown
Source)
Somil:
org.bouncycastle.crypto.tls.DTLSReliableHandshake.writeHandshakeFragment(Unknown
Source)
Somil:
org.bouncycastle.crypto.tls.DTLSReliableHandshake.writeMessage(Unknown
Source)
Somil:
org.bouncycastle.crypto.tls.DTLSReliableHandshake.sendMessage(Unknown
Source)
Somil:
org.bouncycastle.crypto.tls.DTLSServerProtocol.serverHandshake(Unknown
Source)
Somil: org.bouncycastle.crypto.tls.DTLSServerProtocol.accept(Unknown Source)
Somil:
org.jitsi.impl.neomedia.transform.dtls.DtlsPacketTransformer.runInConnectThread(DtlsPacketTransformer.java:981)
Somil:
org.jitsi.impl.neomedia.transform.dtls.DtlsPacketTransformer.access$000(DtlsPacketTransformer.java:38)
Somil:
org.jitsi.impl.neomedia.transform.dtls.DtlsPacketTransformer$1.run(DtlsPacketTransformer.java:1212)

Thank you very much for the detailed explanation! I think I get it now.

These two can occur at the same time when early DTLS packets arrive but
from different threads. The problem is that in the above function
without any synchronisation, the dataOutputStream may be null and may
get created for both (dataOutputStream = createDataOutputStream(); ). In
case the DTLS path creates the final dataOutputStream, then it will not
have any target addresses set to it. Which will cause any further packet
sending failure due to lack of target address.

I think that the DTLS connection thread should not run until a target has been set. And I think that if we set the target before the DTLS transformer is started both problems will disappear, because setting the target has the side effect of creating the RTPConnector's output streams (as your stack trace shows).

Based on this idea, I made this attempt for a fix:
https://github.com/bgrozev/libjitsi/tree/allow-set-target-without-connector
https://github.com/bgrozev/jitsi-videobridge/tree/allow-set-target-without-connector

None of this is tested yet, but I would appreciate feedback.

it’s a very clear race condition, weird that no-one else is facing it.
If you have any doubt about the explanation then you can ask me.

A hypothesis: In our environment the bridge is always the controlling ICE agent. So after the bridge decides which pair to use (and triggers onIceConnected()), at least one RTT passes before the first DTLS packet arrives so we never see the race. However, if you are running the bridge as a controlled ICE agent onIceConnected() will be triggered on reception of a USE-CANDIDATE STUN packet, which may be immediately followed by DTLS ClientHello and the race (or just the DTLS thread causing createDataOutputStream()) will be much more likely.

Would you be able to test with my changes above (you will need to adjust jitsi-videobridge's pom.xml)?

Regards,
Boris

···

On 03/03/16 06:36, Somil Bansal wrote:


#8

Not yet, but I will.

Boris

···

On 03/03/16 22:09, Somil Bansal wrote:

Hi Boris,

Did you have a chance to look at it?


#9

Hi Boris,

I have made a fix for the mentioned problem. You can check it at https://github.com/jitsi/libjitsi/compare/master...somilbansal:patch-1 .
I just made a function synchronized to avoid the mentioned race condition.

···

On 04-Mar-2016, at 10:36 AM, Boris Grozev <boris@jitsi.org> wrote:

On 03/03/16 22:09, Somil Bansal wrote:

Hi Boris,

Did you have a chance to look at it?

Not yet, but I will.

Boris

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#10

HI Boris,

Thanks for the fix. I am on a vacation right now and will test the changes on Tuesday.

In my case also, the bridge is the ICE_CONTROLLING agent. It sends the USE_CANDIDATE packet. But the browser starts sending early DTLS in accordance with https://tools.ietf.org/html/draft-ietf-rtcweb-sdp-00 . As mentioned there when browser responds with active role in its answer sdp, it implies that it is allowed to carry out DTLS handshake in parallel.

···

On 10-Mar-2016, at 9:37 AM, Boris Grozev <boris@jitsi.org> wrote:

Hi Somil,

Sorry for the long delay, it's been a busy few days.

On 03/03/16 06:36, Somil Bansal wrote:

Hi Boris,

I’ll try to explain. The DTLS Client hello can be sent by the browser
before the ICE state being connected. Following function in
libjitsi/src/org/jitsi/impl/neomedia/AbstractRTPConnector.java might be
causing the race condition -

public RTPConnectorOutputStream getDataOutputStream(boolean create)
        throws IOException
    {
        if ((dataOutputStream == null) && create) {
            dataOutputStream = createDataOutputStream();
        }
        return dataOutputStream;
    }

There are two paths from which this function for the the same object can
be executed but in different threads. One path is when ICE state gets
connected and a new target address is set for the MediaStream -

Somil:
org.jitsi.impl.neomedia.AbstractRTPConnector.getDataOutputStream(AbstractRTPConnector.java:335)
Somil:
org.jitsi.impl.neomedia.AbstractRTPConnector.getDataOutputStream(AbstractRTPConnector.java:315)
Somil:
org.jitsi.impl.neomedia.AbstractRTPConnector.addTarget(AbstractRTPConnector.java:113)
Somil:
org.jitsi.impl.neomedia.MediaStreamImpl.doSetTarget(MediaStreamImpl.java:1242)
Somil:
org.jitsi.impl.neomedia.MediaStreamImpl.setTarget(MediaStreamImpl.java:2790)
Somil:
org.jitsi.videobridge.RtpChannel.maybeStartStream(RtpChannel.java:1064)
Somil: org.jitsi.videobridge.Channel.transportConnected(Channel.java:776)
Somil:
org.jitsi.videobridge.IceUdpTransportManager.onIceConnected(IceUdpTransportManager.java:1784)
Somil:
org.jitsi.videobridge.IceUdpTransportManager.access$800(IceUdpTransportManager.java:47)
Somil:
org.jitsi.videobridge.IceUdpTransportManager$3.run(IceUdpTransportManager.java:1826)

One path can be when a DTLS packet is being sent from the DTLS server in
a different thread

Somil:
org.jitsi.impl.neomedia.AbstractRTPConnector.getDataOutputStream(AbstractRTPConnector.java:335)
Somil:
org.jitsi.impl.neomedia.AbstractRTPConnector.getDataOutputStream(AbstractRTPConnector.java:315)
Somil:
org.jitsi.impl.neomedia.transform.dtls.DatagramTransportImpl.doSend(DatagramTransportImpl.java:185)
Somil:
org.jitsi.impl.neomedia.transform.dtls.DatagramTransportImpl.flush(DatagramTransportImpl.java:226)
Somil:
org.jitsi.impl.neomedia.transform.dtls.DatagramTransportImpl.send(DatagramTransportImpl.java:541)
Somil: org.bouncycastle.crypto.tls.DTLSRecordLayer.sendRecord(Unknown
Source)
Somil: org.bouncycastle.crypto.tls.DTLSRecordLayer.send(Unknown Source)
Somil:
org.bouncycastle.crypto.tls.DTLSReliableHandshake$RecordLayerBuffer.sendToRecordLayer(Unknown
Source)
Somil:
org.bouncycastle.crypto.tls.DTLSReliableHandshake.writeHandshakeFragment(Unknown
Source)
Somil:
org.bouncycastle.crypto.tls.DTLSReliableHandshake.writeMessage(Unknown
Source)
Somil:
org.bouncycastle.crypto.tls.DTLSReliableHandshake.sendMessage(Unknown
Source)
Somil:
org.bouncycastle.crypto.tls.DTLSServerProtocol.serverHandshake(Unknown
Source)
Somil: org.bouncycastle.crypto.tls.DTLSServerProtocol.accept(Unknown Source)
Somil:
org.jitsi.impl.neomedia.transform.dtls.DtlsPacketTransformer.runInConnectThread(DtlsPacketTransformer.java:981)
Somil:
org.jitsi.impl.neomedia.transform.dtls.DtlsPacketTransformer.access$000(DtlsPacketTransformer.java:38)
Somil:
org.jitsi.impl.neomedia.transform.dtls.DtlsPacketTransformer$1.run(DtlsPacketTransformer.java:1212)

Thank you very much for the detailed explanation! I think I get it now.

These two can occur at the same time when early DTLS packets arrive but
from different threads. The problem is that in the above function
without any synchronisation, the dataOutputStream may be null and may
get created for both (dataOutputStream = createDataOutputStream(); ). In
case the DTLS path creates the final dataOutputStream, then it will not
have any target addresses set to it. Which will cause any further packet
sending failure due to lack of target address.

I think that the DTLS connection thread should not run until a target has been set. And I think that if we set the target before the DTLS transformer is started both problems will disappear, because setting the target has the side effect of creating the RTPConnector's output streams (as your stack trace shows).

Based on this idea, I made this attempt for a fix:
https://github.com/bgrozev/libjitsi/tree/allow-set-target-without-connector
https://github.com/bgrozev/jitsi-videobridge/tree/allow-set-target-without-connector

None of this is tested yet, but I would appreciate feedback.

it’s a very clear race condition, weird that no-one else is facing it.
If you have any doubt about the explanation then you can ask me.

A hypothesis: In our environment the bridge is always the controlling ICE agent. So after the bridge decides which pair to use (and triggers onIceConnected()), at least one RTT passes before the first DTLS packet arrives so we never see the race. However, if you are running the bridge as a controlled ICE agent onIceConnected() will be triggered on reception of a USE-CANDIDATE STUN packet, which may be immediately followed by DTLS ClientHello and the race (or just the DTLS thread causing createDataOutputStream()) will be much more likely.

Would you be able to test with my changes above (you will need to adjust jitsi-videobridge's pom.xml)?

Regards,
Boris

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#11

You are right, and now that I checked, I do see the same with Jitsi-Meet. So I don't know why we're not running into the same issue.

Boris

···

On 09/03/16 23:48, Somil Bansal wrote:

HI Boris,

Thanks for the fix. I am on a vacation right now and will test the
changes on Tuesday.

In my case also, the bridge is the ICE_CONTROLLING agent. It sends the
USE_CANDIDATE packet. But the browser starts sending early DTLS in
accordance with https://tools.ietf.org/html/draft-ietf-rtcweb-sdp-00 .
As mentioned there when browser responds with active role in its answer
sdp, it implies that it is allowed to carry out DTLS handshake in parallel.


#12

Hi Boris,

I didn’t test the changes but I was looking into your changes in libjitsi - https://github.com/jitsi/libjitsi/compare/master...bgrozev:allow-set-target-without-connector?w=1 <https://github.com/jitsi/libjitsi/compare/master...bgrozev:allow-set-target-without-connector?w=1> .

You are using the same rtpConnector but now with a new variable. Even though you are setting the target before the connector in video bridge code but still the target is set to the connector in the libjitsi code, which can’t happen because target is set to the connector which is not set yet.

···

On 10-Mar-2016, at 9:51 PM, Boris Grozev <boris@jitsi.org> wrote:

On 09/03/16 23:48, Somil Bansal wrote:

HI Boris,

Thanks for the fix. I am on a vacation right now and will test the
changes on Tuesday.

In my case also, the bridge is the ICE_CONTROLLING agent. It sends the
USE_CANDIDATE packet. But the browser starts sending early DTLS in
accordance with https://tools.ietf.org/html/draft-ietf-rtcweb-sdp-00 .
As mentioned there when browser responds with active role in its answer
sdp, it implies that it is allowed to carry out DTLS handshake in parallel.

You are right, and now that I checked, I do see the same with Jitsi-Meet. So I don't know why we're not running into the same issue.

Boris

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#13

Hi Boris,

I didn’t test the changes but I was looking into your changes in
libjitsi -
https://github.com/jitsi/libjitsi/compare/master...bgrozev:allow-set-target-without-connector?w=1 .

You are using the same rtpConnector but now with a new variable. Even
though you are setting the target before the connector in video bridge
code but still the target is set to the connector in the libjitsi code,
which can’t happen because target is set to the connector which is not
set yet.

Thanks for looking into it! The intention is the following: if the connector is null while setting the target, the check on line 1253 will succeed, and targetIsSet will be set to true. This will cause the rtpConnectorTarget field to be set on line 1289. When the connector is later set, doSetTarget will execute again and this time it will set the target of the connector. Am I missing something?

Regards,
Boris

···

On 15/03/16 06:46, Somil Bansal wrote:

On 10-Mar-2016, at 9:51 PM, Boris Grozev <boris@jitsi.org >> <mailto:boris@jitsi.org>> wrote:

On 09/03/16 23:48, Somil Bansal wrote:

HI Boris,

Thanks for the fix. I am on a vacation right now and will test the
changes on Tuesday.

In my case also, the bridge is the ICE_CONTROLLING agent. It sends the
USE_CANDIDATE packet. But the browser starts sending early DTLS in
accordance with https://tools.ietf.org/html/draft-ietf-rtcweb-sdp-00 .
As mentioned there when browser responds with active role in its answer
sdp, it implies that it is allowed to carry out DTLS handshake in
parallel.

You are right, and now that I checked, I do see the same with
Jitsi-Meet. So I don't know why we're not running into the same issue.

Boris

_______________________________________________
dev mailing list
dev@jitsi.org <mailto:dev@jitsi.org>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#14

Hi Boris,

It seems good. I missed the change in rtpConnectorChanged hence the confusion. Will test whenever I get the chance.

···

On 15-Mar-2016, at 10:52 PM, Boris Grozev <boris@jitsi.org> wrote:

On 15/03/16 06:46, Somil Bansal wrote:

Hi Boris,

I didn’t test the changes but I was looking into your changes in
libjitsi -
https://github.com/jitsi/libjitsi/compare/master...bgrozev:allow-set-target-without-connector?w=1 <https://github.com/jitsi/libjitsi/compare/master...bgrozev:allow-set-target-without-connector?w=1> .

You are using the same rtpConnector but now with a new variable. Even
though you are setting the target before the connector in video bridge
code but still the target is set to the connector in the libjitsi code,
which can’t happen because target is set to the connector which is not
set yet.

Thanks for looking into it! The intention is the following: if the connector is null while setting the target, the check on line 1253 will succeed, and targetIsSet will be set to true. This will cause the rtpConnectorTarget field to be set on line 1289. When the connector is later set, doSetTarget will execute again and this time it will set the target of the connector. Am I missing something?

Regards,
Boris

On 10-Mar-2016, at 9:51 PM, Boris Grozev <boris@jitsi.org <mailto:boris@jitsi.org> >>> <mailto:boris@jitsi.org <mailto:boris@jitsi.org>>> wrote:

On 09/03/16 23:48, Somil Bansal wrote:

HI Boris,

Thanks for the fix. I am on a vacation right now and will test the
changes on Tuesday.

In my case also, the bridge is the ICE_CONTROLLING agent. It sends the
USE_CANDIDATE packet. But the browser starts sending early DTLS in
accordance with https://tools.ietf.org/html/draft-ietf-rtcweb-sdp-00 .
As mentioned there when browser responds with active role in its answer
sdp, it implies that it is allowed to carry out DTLS handshake in
parallel.

You are right, and now that I checked, I do see the same with
Jitsi-Meet. So I don't know why we're not running into the same issue.

Boris

_______________________________________________
dev mailing list
dev@jitsi.org <mailto:dev@jitsi.org> <mailto:dev@jitsi.org <mailto:dev@jitsi.org>>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org <mailto:dev@jitsi.org>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org <mailto:dev@jitsi.org>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#15

Hi Boris,

I tested the changes and it’s working fine for me. I fixed a log in jvb and also added a log in libjitsi to check if targets list is empty to make sure that issue gets into notice if it happens again.

https://github.com/bgrozev/libjitsi/pull/1/commits/c5bc8768ed31d5543120a539b5ebf52d6cd1eb00
https://github.com/bgrozev/jitsi-videobridge/pull/1/commits/db1a8a2c32a3358cbb2723fda1291ac1000d2254

···

On 16-Mar-2016, at 12:14 PM, Somil Bansal <somilbansal12@gmail.com> wrote:

Hi Boris,

It seems good. I missed the change in rtpConnectorChanged hence the confusion. Will test whenever I get the chance.

On 15-Mar-2016, at 10:52 PM, Boris Grozev <boris@jitsi.org <mailto:boris@jitsi.org>> wrote:

On 15/03/16 06:46, Somil Bansal wrote:

Hi Boris,

I didn’t test the changes but I was looking into your changes in
libjitsi -
https://github.com/jitsi/libjitsi/compare/master...bgrozev:allow-set-target-without-connector?w=1 <https://github.com/jitsi/libjitsi/compare/master...bgrozev:allow-set-target-without-connector?w=1> .

You are using the same rtpConnector but now with a new variable. Even
though you are setting the target before the connector in video bridge
code but still the target is set to the connector in the libjitsi code,
which can’t happen because target is set to the connector which is not
set yet.

Thanks for looking into it! The intention is the following: if the connector is null while setting the target, the check on line 1253 will succeed, and targetIsSet will be set to true. This will cause the rtpConnectorTarget field to be set on line 1289. When the connector is later set, doSetTarget will execute again and this time it will set the target of the connector. Am I missing something?

Regards,
Boris

On 10-Mar-2016, at 9:51 PM, Boris Grozev <boris@jitsi.org <mailto:boris@jitsi.org> >>>> <mailto:boris@jitsi.org <mailto:boris@jitsi.org>>> wrote:

On 09/03/16 23:48, Somil Bansal wrote:

HI Boris,

Thanks for the fix. I am on a vacation right now and will test the
changes on Tuesday.

In my case also, the bridge is the ICE_CONTROLLING agent. It sends the
USE_CANDIDATE packet. But the browser starts sending early DTLS in
accordance with https://tools.ietf.org/html/draft-ietf-rtcweb-sdp-00 .
As mentioned there when browser responds with active role in its answer
sdp, it implies that it is allowed to carry out DTLS handshake in
parallel.

You are right, and now that I checked, I do see the same with
Jitsi-Meet. So I don't know why we're not running into the same issue.

Boris

_______________________________________________
dev mailing list
dev@jitsi.org <mailto:dev@jitsi.org> <mailto:dev@jitsi.org <mailto:dev@jitsi.org>>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org <mailto:dev@jitsi.org>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org <mailto:dev@jitsi.org>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#16

Great, thank you for testing and for the patches! I've merged your commits to my repo, and I've created PRs upstream:
https://github.com/jitsi/libjitsi/pull/114
https://github.com/jitsi/jitsi-videobridge/pull/181

Before we can merge these however, we need you to sign our contributor agreement (individual[0] or corporate[1]).

Regards,
Boris

[0] https://jitsi.org/icla
[1] https://jitsi.org/ccla

···

On 21/03/16 03:42, Somil Bansal wrote:

Hi Boris,

I tested the changes and it’s working fine for me. I fixed a log in jvb
and also added a log in libjitsi to check if targets list is empty to
make sure that issue gets into notice if it happens again.

https://github.com/bgrozev/libjitsi/pull/1/commits/c5bc8768ed31d5543120a539b5ebf52d6cd1eb00
https://github.com/bgrozev/jitsi-videobridge/pull/1/commits/db1a8a2c32a3358cbb2723fda1291ac1000d2254


#17

Done!

···

On 21-Mar-2016, at 10:33 PM, Boris Grozev <boris@jitsi.org> wrote:

On 21/03/16 03:42, Somil Bansal wrote:

Hi Boris,

I tested the changes and it’s working fine for me. I fixed a log in jvb
and also added a log in libjitsi to check if targets list is empty to
make sure that issue gets into notice if it happens again.

https://github.com/bgrozev/libjitsi/pull/1/commits/c5bc8768ed31d5543120a539b5ebf52d6cd1eb00
https://github.com/bgrozev/jitsi-videobridge/pull/1/commits/db1a8a2c32a3358cbb2723fda1291ac1000d2254

Great, thank you for testing and for the patches! I've merged your commits to my repo, and I've created PRs upstream:
https://github.com/jitsi/libjitsi/pull/114
https://github.com/jitsi/jitsi-videobridge/pull/181

Before we can merge these however, we need you to sign our contributor agreement (individual[0] or corporate[1]).

Regards,
Boris

[0] https://jitsi.org/icla
[1] https://jitsi.org/ccla

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#18

Hi Boris, when can I expect the fix to be merged?

···

On 21-Mar-2016, at 10:56 PM, Somil Bansal <somilbansal12@gmail.com> wrote:

Done!

On 21-Mar-2016, at 10:33 PM, Boris Grozev <boris@jitsi.org> wrote:

On 21/03/16 03:42, Somil Bansal wrote:

Hi Boris,

I tested the changes and it’s working fine for me. I fixed a log in jvb
and also added a log in libjitsi to check if targets list is empty to
make sure that issue gets into notice if it happens again.

https://github.com/bgrozev/libjitsi/pull/1/commits/c5bc8768ed31d5543120a539b5ebf52d6cd1eb00
https://github.com/bgrozev/jitsi-videobridge/pull/1/commits/db1a8a2c32a3358cbb2723fda1291ac1000d2254

Great, thank you for testing and for the patches! I've merged your commits to my repo, and I've created PRs upstream:
https://github.com/jitsi/libjitsi/pull/114
https://github.com/jitsi/jitsi-videobridge/pull/181

Before we can merge these however, we need you to sign our contributor agreement (individual[0] or corporate[1]).

Regards,
Boris

[0] https://jitsi.org/icla
[1] https://jitsi.org/ccla

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#19

Hi Boris,

Any update on this?

···

On 23-Mar-2016, at 1:30 AM, Somil Bansal <somilbansal12@gmail.com> wrote:

Hi Boris, when can I expect the fix to be merged?

On 21-Mar-2016, at 10:56 PM, Somil Bansal <somilbansal12@gmail.com> wrote:

Done!

On 21-Mar-2016, at 10:33 PM, Boris Grozev <boris@jitsi.org> wrote:

On 21/03/16 03:42, Somil Bansal wrote:

Hi Boris,

I tested the changes and it’s working fine for me. I fixed a log in jvb
and also added a log in libjitsi to check if targets list is empty to
make sure that issue gets into notice if it happens again.

https://github.com/bgrozev/libjitsi/pull/1/commits/c5bc8768ed31d5543120a539b5ebf52d6cd1eb00
https://github.com/bgrozev/jitsi-videobridge/pull/1/commits/db1a8a2c32a3358cbb2723fda1291ac1000d2254

Great, thank you for testing and for the patches! I've merged your commits to my repo, and I've created PRs upstream:
https://github.com/jitsi/libjitsi/pull/114
https://github.com/jitsi/jitsi-videobridge/pull/181

Before we can merge these however, we need you to sign our contributor agreement (individual[0] or corporate[1]).

Regards,
Boris

[0] https://jitsi.org/icla
[1] https://jitsi.org/ccla

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#20

It needs to be reviewed before it's merged. It's quite busy here, so I don't when it will happen, but it's on the list.

Boris

···

On 24/03/16 10:34, Somil Bansal wrote:

Hi Boris,

Any update on this?