The complete family of xxxUnit testing frameworks have their roots in Kent Beck’s sUnit testing Framework for Smalltalk.
While jUnit (the Java flavor) and sUnit were very similar for a long time, jUnit “lately” added a lot of stuff that is helpful in Java, but quite useless in Smalltalk (like Java Annotations), mostly with jUnit Version 4.
There is, however, one difference between sUnit and jUnit that I think should be overcome. I am speaking of the TestFailure Class in jUnit that has not been added to sUnit. I don’t know exactly when the jUnit implementation got TestFailure, but I miss it.
Why do I miss TestFailure? Because its non-existence makes post-testing reporting very hard, unless you write your own TestRunner.
But let’s start from the beginning.
I’ve played with the Hudson Build Server (which will soon be known as Jenkins, but that’s another ugly Oracle-related story) lately in order to have a nice frontend for automatic builds of the VA Smalltalk projects of a customer (you can learn more about the beginnings of this project here) .
One thing that I wanted to do is use my TestResults and report them back to Hudson. Which is quite easy once you find out about a suitable xml format for TestResult. The gold standard here is again an artefact that’s coming out of the jUnit project.
Since my build machine works like a batch processor, and since I wanted to simply use the TestResult of a Suite to create an xml file that can be read by Hudson, I was hoping for an easy way to marshal TestResult to XML. So the first idea was to add a step to my batch that accepts the TestResult from the Test runner step and simply convert the results to XML.
Unfortunately, sUnit does not keep the output of failed tests anywhere, it just keeps the test class and test selector of a failed test.
So if you have an assertion like
self assert: (1>2) description: ’1 should be greater than two’.
and the assertion fails, you will not find the description anywhere in the TestResult. The testwill be recorded as failed, but there will be no way of finding out what went wrong. You’ll have to rerun the test in the sUnit Browser and take a look at the Exception that gets thrown.
I took a look at Lukas’ implementation of his Hudson build integration for Pharo. He wrote his own TestRunner that immediately outputs the jUnit XML file, but I think this is not the right solution. The right solution is what jUnit does: keep an instance of TestFailure for each failed test that includes the description. (you can take a look at TestFailure API documentation – it’s lean and easy)
Why do I think so? Because writing your own TestRunner will probably decouple you from future sUnit development, and because writing an xml output is probably not the only place in which you might need the description.
What do others think? Should we change sUnit and add a TestFailure Class?
[UPDATE]: I just found out that sUnit V4 has a TestFailure class, as a subclass of Exception. From the comments on the SUnit Config Map in VAST 8.02:
- The major 4.0 change is to dispatch on the exception (see #sunitAnnounce:toResult:). This allows
user-defined subclasses of TestFailure or Error to plugin different behaviour in specific cases. (For
robust behaviour in the face of old code using pre-ANSI-error-raising, Signal also responds to it.)
So I now have to take a closer look at V4 to see whether and/or how I can collect them in a TestResult.
Still I think the design and implementation of #sunitAnnounce:toResult: is wrong or insuffcicient for reporting purposes. #sunitAnnounce:toResult: adds the TestCase to the list of failures, but what I’d really want is some extra infor to which I can add the error description so that it is accessible from the TestResult.
One approach (which seems obvious) would be to add the TestCase to the TestFailure Exception and add the exception to the failures collection. Thus the failures list could access all info that’s needed for Reporting.
I’ll have to Play with this a little, and hope to find some time in the next few days…
