[sip-comm-dev] volunteers for improving the testing utility?


#1

Hey all,

While I was looking at Brian's modifications on the build.xml, I thought
that the mechanism we currently use for specifying the list of tests to
include in the CC buildloop, is rather clumsy.

So I wonder, whether someone would be willing to contribute an extra
feature that would hopefully improve this.

Currently the SlickRunner bundle would read the
net.java.sip.communicator.slick.runner.TEST_LIST property and extract a
whitespace separated list of test names out of it. Each of these names
is a service.pid so the SlickRunner would run a search for OSGI services
corresponding to each of these names and execute them one by one.

The nasty thing about all that is the fact that the list of tests to run
is growing bigger and bigger so the build.xml is obviously not its place
(not to mention the fact that it is virtually impossible to run that
from the command line).

So I belieave that it would be a good idea to have this list in a separate file and replace the
net.java.sip.communicator.slick.runner.TEST_LIST property with
net.java.sip.communicator.slick.runner.TEST_LIST_FILE.
(And if possibly without breaking support for TEST_LIST)

Any takers and/or comments?

Emil

···

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#2

While I was looking at Brian's modifications on the build.xml, I thought
that the mechanism we currently use for specifying the list of tests to
include in the CC buildloop, is rather clumsy.

I thought that too, but took the attitude "if it aint broke, dont fix it"

Currently the SlickRunner bundle would read the
net.java.sip.communicator.slick.runner.TEST_LIST property and extract a
whitespace separated list of test names out of it. Each of these names
is a service.pid so the SlickRunner would run a search for OSGI services
corresponding to each of these names and execute them one by one.

I had to fight the temptation to research the mechanism. I could see it
wasn't straightforward, but felt tinkering with it was outside the scope
of my own task.

The nasty thing about all that is the fact that the list of tests to run
is growing bigger and bigger so the build.xml is obviously not its place
(not to mention the fact that it is virtually impossible to run that
from the command line).

I agree. build.xml should be highly stable because it tends to be very
complex in any serious project. If someone breaks a build, everyone
shares the pain.

So I belieave that it would be a good idea to have this list in a
separate file and replace the
net.java.sip.communicator.slick.runner.TEST_LIST property with
net.java.sip.communicator.slick.runner.TEST_LIST_FILE.
(And if possibly without breaking support for TEST_LIST)

Some of my own projects still rely on a explicit lists of test classes. They
are stable, so I don't worry too much about the risks. However, I can still
remember situations where I wasted a lot of time because I'd forgotten
to add a new test class to the list.

netbeans has 2 (well, four really) kinds of projects... those like ours are
called "with existing ant script", while the others have their build.xml created
by a netbeans wizard. I'm fairly hazy about the mechanism, but a netbeans
project has an <ide-action> that runs all unit tests... this is "wired" to a
build.xml target that uses introspection to create the list of unit test classes
"on the fly".

I don't know how hard it would be to do this kind of thing for slick.runner, but
a little extra effort now would save a lot of effort in the long run.

a) I assume all our unit tests will be found in the "test" project folder.

b) does each test class have a superclass (or implement an interface)
that uniquely identifies it as a class to be included in the "all tests" set?
(e.g. jUnit test classes always inherit from junit.framework.TestCase).

c) does each test method have an attribute that uniquely identifies it as a
member of the "all test methods" set within a test class? (e.g. jUnit test
methods for automatic execution have the signature...
public void testnnnnnnn ( ), where nnnnnn is an arbitray string).

d) does each test class have the equivalent of the junit setup() and
teardown() methods, or not need such things?

If the answers to these 4 questions are mostly affirmative, then there is
a strong case to create a TestRunner that class that uses introspection
to run all our tests. Isn't there one in the Oscar toolkit already? If not,
the simplest jUnit TestRunner ought to be a useful starting point as a
design pattern.

p.s. I just took a quick look at the service.protocol package.... each
test is in a different package, but I don't think that would be a problem.
Each test class extends junit.framework.TestCase already!!! I still
haven't "joined up all the dots" in my mind, but surely jUnit already
has an answer for our problem?

Do we need to clone the junit.textui.TestRunner class to make a
new junit.textui.OscarTestRunner? Is that it?
Regards,

Brian

···

On Tue, 06 Dec 2005 04:42:32 +0100, Emil Ivov wrote:

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#3

Hello Brian,

Brian Burch wrote:

I thought that too, but took the attitude "if it aint broke, dont fix it"

And that is a great attitude!

I don't know how hard it would be to do this kind of thing for slick.runner, but
a little extra effort now would save a lot of effort in the long run.

Quite easy actually. It all comes down to reading a file and extracting all whitespace separate names. Apart from that nothing else needs to be changed.

a) I assume all our unit tests will be found in the "test" project folder.

Yes.

b) does each test class have a superclass (or implement an interface)
that uniquely identifies it as a class to be included in the "all tests" set?
(e.g. jUnit test classes always inherit from junit.framework.TestCase).

Yes they all extend TestCase.

c) does each test method have an attribute that uniquely identifies it as a
member of the "all test methods" set within a test class? (e.g. jUnit test
methods for automatic execution have the signature...
public void testnnnnnnn ( ), where nnnnnn is an arbitray string).

d) does each test class have the equivalent of the junit setup() and
teardown() methods, or not need such things?

If the answers to these 4 questions are mostly affirmative, then there is
a strong case to create a TestRunner that class that uses introspection
to run all our tests.

The answers are all affirmative and SlickRunner is exactly the TestRunner you are talking about. SlickRunner is a JUnit test runner (a combination of the knopflerfish GRUNT and the apache-ant junit runner) that would connect to the OSGI framework and discover all registered services whose service.pid property is in the specified (currently by the TEST_LIST property) list of tests. Once all of those have been discovered they would simply be run like standard JUnit testsuites.

It's only problem right now is that it reads its TestSuite list from a java property and not a file.

p.s. I just took a quick look at the service.protocol package....

Most tests are located in the slick package. Those in service.protocol are just ... exceptions. They aren't even being run right now.

Cheers
Emil

···

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#4

Hey I just saw that the ant property task can actually load its contents from a file using the location attribute. This gives us exactly the type of functionality that I was thinking about.

I guess that settles the matter.

Cheers
Emil

Emil Ivov wrote:

···

Hello Brian,

Brian Burch wrote:

I thought that too, but took the attitude "if it aint broke, dont fix it"

And that is a great attitude!

I don't know how hard it would be to do this kind of thing for slick.runner, but
a little extra effort now would save a lot of effort in the long run.

Quite easy actually. It all comes down to reading a file and extracting all whitespace separate names. Apart from that nothing else needs to be changed.

a) I assume all our unit tests will be found in the "test" project folder.

Yes.

b) does each test class have a superclass (or implement an interface)
that uniquely identifies it as a class to be included in the "all tests" set?
(e.g. jUnit test classes always inherit from junit.framework.TestCase).

Yes they all extend TestCase.

c) does each test method have an attribute that uniquely identifies it as a
member of the "all test methods" set within a test class? (e.g. jUnit test
methods for automatic execution have the signature...
public void testnnnnnnn ( ), where nnnnnn is an arbitray string).

d) does each test class have the equivalent of the junit setup() and
teardown() methods, or not need such things?

If the answers to these 4 questions are mostly affirmative, then there is
a strong case to create a TestRunner that class that uses introspection
to run all our tests.

The answers are all affirmative and SlickRunner is exactly the TestRunner you are talking about. SlickRunner is a JUnit test runner (a combination of the knopflerfish GRUNT and the apache-ant junit runner) that would connect to the OSGI framework and discover all registered services whose service.pid property is in the specified (currently by the TEST_LIST property) list of tests. Once all of those have been discovered they would simply be run like standard JUnit testsuites.

It's only problem right now is that it reads its TestSuite list from a java property and not a file.

p.s. I just took a quick look at the service.protocol package....

Most tests are located in the slick package. Those in service.protocol are just ... exceptions. They aren't even being run right now.

Cheers
Emil

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#5

Yes, that would be an immediate improvement because your list will
become a resource in the test folder. I guess it would be best if you
made the change to build.xml, given that I've finished my own changes.

(I'll still keep my other suggestion in mind. If I have time, I'll look into
the possibility of creating a reflection-based test runner that dynamically
builds a set of all known test classes.)

Regards,

Brian

···

On Thu, 15 Dec 2005 02:48:39 +0100, Emil Ivov wrote:

Hey I just saw that the ant property task can actually load its contents
from a file using the location attribute. This gives us exactly the type
of functionality that I was thinking about.

I guess that settles the matter.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#6

Brian Burch wrote:

(I'll still keep my other suggestion in mind. If I have time, I'll look into
the possibility of creating a reflection-based test runner that dynamically
builds a set of all known test classes.)

Cool, just ping when ready so that we could discuss it.

Cheers
Emil

···

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net