[jitsi-dev] libjitsi inconsistant SctpTransferTest


#1

Hi,

I've been building libjitsi and have noticed that the SctpTransferTest often fails to receive the expected data over the link. When this happens tt appears that data is not going across the link even though previous attempts were successful.

Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.014 sec <<< FAILURE! - in org.jitsi.sctp4j.SctpTransferTest
testSocketBrokenLink(org.jitsi.sctp4j.SctpTransferTest) Time elapsed: 5.013 sec <<< FAILURE!
java.lang.AssertionError: actual array was null
at org.junit.Assert.fail(Assert.java:88)
at org.junit.internal.ComparisonCriteria.assertArraysAreSameLength(ComparisonCriteria.java:66)
at org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:32)
at org.junit.Assert.internalArrayEquals(Assert.java:473)
at org.junit.Assert.assertArrayEquals(Assert.java:294)
at org.junit.Assert.assertArrayEquals(Assert.java:305)
at org.jitsi.sctp4j.SctpTransferTest.testTransferPart(SctpTransferTest.java:141)
at org.jitsi.sctp4j.SctpTransferTest.testSocketBrokenLink(SctpTransferTest.java:101)

Is this a common occurrence or a known faulty test?

Thanks for you help!
      Justin
This email message is for the sole use of the intended recipient(s) and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. Any unauthorized review, use, copying, disclosure or dissemination is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.


#2

My experience is that it’s a common occurrence. I’m not aware of whether it’s a known faulty test or of anyone looking into it for that matter.

···

On Jan 19, 2016, at 11:50 AM, Justin Martinez <justin.martinez@readytalk.com> wrote:
Is this a common occurrence or a known faulty test?


#3

Hi Justin,

Hi,

I've been building libjitsi and have noticed that the SctpTransferTest
often fails to receive the expected data over the link. When this
happens tt appears that data is not going across the link even though
previous attempts were successful.

This is the result I experience too. Somewhere around one in three times
it will succeed and I go on my merry way.

Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.014
sec <<< FAILURE! - in org.jitsi.sctp4j.SctpTransferTest
testSocketBrokenLink(org.jitsi.sctp4j.SctpTransferTest) Time elapsed:
5.013 sec <<< FAILURE!
java.lang.AssertionError: actual array was null
at org.junit.Assert.fail(Assert.java:88)
at
org.junit.internal.ComparisonCriteria.assertArraysAreSameLength(ComparisonCriteria.java:66)
at
org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:32)
at org.junit.Assert.internalArrayEquals(Assert.java:473)
at org.junit.Assert.assertArrayEquals(Assert.java:294)
at org.junit.Assert.assertArrayEquals(Assert.java:305)
at
org.jitsi.sctp4j.SctpTransferTest.testTransferPart(SctpTransferTest.java:141)
at
org.jitsi.sctp4j.SctpTransferTest.testSocketBrokenLink(SctpTransferTest.java:101)

Is this a common occurrence or a known faulty test?

I'm guessing yes to both. Maybe there is non-determinism in the test.

Jesse

···

On Tue, 2016-01-19 at 17:50 +0000, Justin Martinez wrote:

Thanks for you help!
      Justin
This email message is for the sole use of the intended recipient(s)
and may contain information that is privileged, confidential, and
exempt from disclosure under applicable law. Any unauthorized review,
use, copying, disclosure or dissemination is prohibited. If you are
not the intended recipient, please contact the sender by reply email
and destroy all copies of the original message.
_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#4

Hi !

···

On Tue, Jan 19, 2016 at 7:44 PM, Jesse <bickelj@gmail.com> wrote:

Hi Justin,

On Tue, 2016-01-19 at 17:50 +0000, Justin Martinez wrote:

Hi,

I've been building libjitsi and have noticed that the SctpTransferTest
often fails to receive the expected data over the link. When this
happens tt appears that data is not going across the link even though
previous attempts were successful.

This is the result I experience too. Somewhere around one in three times
it will succeed and I go on my merry way.

Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.014
sec <<< FAILURE! - in org.jitsi.sctp4j.SctpTransferTest
testSocketBrokenLink(org.jitsi.sctp4j.SctpTransferTest) Time elapsed:
5.013 sec <<< FAILURE!
java.lang.AssertionError: actual array was null
at org.junit.Assert.fail(Assert.java:88)
at
org.junit.internal.ComparisonCriteria.assertArraysAreSameLength(ComparisonCriteria.java:66)
at
org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:32)
at org.junit.Assert.internalArrayEquals(Assert.java:473)
at org.junit.Assert.assertArrayEquals(Assert.java:294)
at org.junit.Assert.assertArrayEquals(Assert.java:305)
at
org.jitsi.sctp4j.SctpTransferTest.testTransferPart(SctpTransferTest.java:141)
at
org.jitsi.sctp4j.SctpTransferTest.testSocketBrokenLink(SctpTransferTest.java:101)

Is this a common occurrence or a known faulty test?

I'm guessing yes to both. Maybe there is non-determinism in the test.

Yes, this test "testSocketBrokenLink is emulating random packet loss
and probably requires some adjustments.

Regards,
Pawel


#5

By the way when I was investigating the test failures I swapped out the TestLink with a DirectLink under the hypothesis that without the broken link the test would always pass since no packets would be dropped. After a few runs I was able to reproduce the same failure.

Thanks for the information all.

Justin

···

________________________________________
From: dev [dev-bounces@jitsi.org] on behalf of Paweł Domas [pawel.domas@jitsi.org]
Sent: Thursday, January 21, 2016 8:38 AM
To: Jitsi Developers
Subject: Re: [jitsi-dev] libjitsi inconsistant SctpTransferTest

Hi !

On Tue, Jan 19, 2016 at 7:44 PM, Jesse <bickelj@gmail.com> wrote:

Hi Justin,

On Tue, 2016-01-19 at 17:50 +0000, Justin Martinez wrote:

Hi,

I've been building libjitsi and have noticed that the SctpTransferTest
often fails to receive the expected data over the link. When this
happens tt appears that data is not going across the link even though
previous attempts were successful.

This is the result I experience too. Somewhere around one in three times
it will succeed and I go on my merry way.

Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.014
sec <<< FAILURE! - in org.jitsi.sctp4j.SctpTransferTest
testSocketBrokenLink(org.jitsi.sctp4j.SctpTransferTest) Time elapsed:
5.013 sec <<< FAILURE!
java.lang.AssertionError: actual array was null
at org.junit.Assert.fail(Assert.java:88)
at
org.junit.internal.ComparisonCriteria.assertArraysAreSameLength(ComparisonCriteria.java:66)
at
org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:32)
at org.junit.Assert.internalArrayEquals(Assert.java:473)
at org.junit.Assert.assertArrayEquals(Assert.java:294)
at org.junit.Assert.assertArrayEquals(Assert.java:305)
at
org.jitsi.sctp4j.SctpTransferTest.testTransferPart(SctpTransferTest.java:141)
at
org.jitsi.sctp4j.SctpTransferTest.testSocketBrokenLink(SctpTransferTest.java:101)

Is this a common occurrence or a known faulty test?

I'm guessing yes to both. Maybe there is non-determinism in the test.

Yes, this test "testSocketBrokenLink is emulating random packet loss
and probably requires some adjustments.

Regards,
Pawel

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev
This email message is for the sole use of the intended recipient(s) and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. Any unauthorized review, use, copying, disclosure or dissemination is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.


#6

Oh that's interesting - thanks for investigating ! Then it means there
must be some other bug there. Unfortunately we're currently busy with
other stuff, so no a big chance of having it fixed soon.

Regards,
Pawel

···

On Fri, Jan 22, 2016 at 9:54 AM, Justin Martinez <justin.martinez@readytalk.com> wrote:

By the way when I was investigating the test failures I swapped out the TestLink with a DirectLink under the hypothesis that without the broken link the test would always pass since no packets would be dropped. After a few runs I was able to reproduce the same failure.

Thanks for the information all.

Justin


#7

Hi Justin and Pawel,

By the way when I was investigating the test failures I swapped out the TestLink with a DirectLink under the hypothesis that without the broken link the test would always pass since no packets would be dropped. After a few runs I was able to reproduce the same failure.

Thanks for the information all.

Justin

________________________________________
From: dev [dev-bounces@jitsi.org] on behalf of Paweł Domas [pawel.domas@jitsi.org]
Sent: Thursday, January 21, 2016 8:38 AM
To: Jitsi Developers
Subject: Re: [jitsi-dev] libjitsi inconsistant SctpTransferTest

Hi !

> Hi Justin,
>
>> Hi,
>>
>>
>> I've been building libjitsi and have noticed that the SctpTransferTest
>> often fails to receive the expected data over the link. When this
>> happens tt appears that data is not going across the link even though
>> previous attempts were successful.
>>
>
> This is the result I experience too. Somewhere around one in three times
> it will succeed and I go on my merry way.
>>
>> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.014
>> sec <<< FAILURE! - in org.jitsi.sctp4j.SctpTransferTest
>> testSocketBrokenLink(org.jitsi.sctp4j.SctpTransferTest) Time elapsed:
>> 5.013 sec <<< FAILURE!
>> java.lang.AssertionError: actual array was null
>> at org.junit.Assert.fail(Assert.java:88)
>> at
>> org.junit.internal.ComparisonCriteria.assertArraysAreSameLength(ComparisonCriteria.java:66)
>> at
>> org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:32)
>> at org.junit.Assert.internalArrayEquals(Assert.java:473)
>> at org.junit.Assert.assertArrayEquals(Assert.java:294)
>> at org.junit.Assert.assertArrayEquals(Assert.java:305)
>> at
>> org.jitsi.sctp4j.SctpTransferTest.testTransferPart(SctpTransferTest.java:141)
>> at
>> org.jitsi.sctp4j.SctpTransferTest.testSocketBrokenLink(SctpTransferTest.java:101)
>>
>>
>> Is this a common occurrence or a known faulty test?
>
> I'm guessing yes to both. Maybe there is non-determinism in the test.

Yes, this test "testSocketBrokenLink is emulating random packet loss
and probably requires some adjustments.

Regards,
Pawel

Pull request here may fix the issue:
https://github.com/jitsi/libjitsi/pull/77

The premise is that we want random data to be sent, and random failures
to be tested, but the same test to run. So each class got its own Random
object and a seed hard-coded. Also a source of problems can be
synchronization. I consulted JCIP and found a different form for the
synchronizer in use.

Try it out and let see if it helps. I think the test is still probably
testing what it intends, but takes less time and gets consistent passage
with the current code.

Jesse

···

On Fri, 2016-01-22 at 15:54 +0000, Justin Martinez wrote:

On Tue, Jan 19, 2016 at 7:44 PM, Jesse <bickelj@gmail.com> wrote:
> On Tue, 2016-01-19 at 17:50 +0000, Justin Martinez wrote: