What sUnit could learn from jUnit [UPDATE]

The complete family of xxxUnit testing frameworks have their roots in Kent Beck’s sUnit testing Framework for Smalltalk.

While jUnit (the Java flavor) and sUnit were very similar for a long time, jUnit “lately” added a lot of stuff that is helpful in Java, but quite useless in Smalltalk (like Java Annotations), mostly with jUnit Version 4.

There is, however, one difference between sUnit and jUnit that I think should be overcome. I am speaking of the TestFailure Class in jUnit that has not been added to sUnit. I don’t know exactly when the jUnit implementation got TestFailure, but I miss it.

Why do I miss TestFailure? Because its non-existence makes post-testing reporting very hard, unless you write your own TestRunner.

But let’s start from the beginning.

I’ve played with the Hudson Build Server (which will soon be known as Jenkins, but that’s another ugly Oracle-related story) lately in order to have a nice frontend for automatic builds of the VA Smalltalk projects of a customer (you can learn more about the beginnings of this project here) .

One thing that I wanted to do is use my TestResults and report them back to Hudson. Which is quite easy once you find out about a suitable xml format for TestResult. The gold standard here is again an artefact that’s coming out of the jUnit project.

Since my build machine works like a batch processor, and since I wanted to simply use the TestResult of a Suite to create an xml file that can be read by Hudson, I was hoping for an easy way to marshal TestResult to XML. So the first idea was to add a step to my batch that accepts the TestResult from the Test runner step and simply convert the results to XML.

Unfortunately, sUnit does not keep the output of failed tests anywhere, it just keeps the test class and test selector of a failed test.

So if you have an assertion like

self assert: (1>2) description: ‘1 should be greater than two’.

and the assertion fails, you will not find the description anywhere in the TestResult. The testwill be recorded as failed, but there will be no way of finding out what went wrong. You’ll have to rerun the test in the sUnit Browser and take a look at the Exception that gets thrown.

I took a look at Lukas’ implementation of his Hudson build integration for Pharo. He wrote his own TestRunner that immediately outputs the jUnit XML file, but I think this is not the right solution. The right solution is what jUnit does: keep an instance of TestFailure for each failed test that includes the description. (you can take a look at TestFailure API documentation – it’s lean and easy)

Why do I think so? Because writing your own TestRunner will probably decouple you from future sUnit development, and because writing an xml output is probably not the only place in which you might need the description.

What do others think? Should we change sUnit and add a TestFailure Class?

[UPDATE]: I just found out that sUnit V4 has a TestFailure class, as a subclass of Exception. From the comments on the SUnit Config Map in VAST 8.02:

– The major 4.0 change is to dispatch on the exception (see #sunitAnnounce:toResult:).  This allows
user-defined subclasses of TestFailure or Error to plugin different behaviour in specific cases.  (For
robust behaviour in the face of old code using pre-ANSI-error-raising, Signal also responds to it.)

So I now have to take a closer look at V4 to see whether and/or how I can collect them in a TestResult.

Still I think the design and implementation of #sunitAnnounce:toResult: is wrong or insuffcicient for reporting purposes. #sunitAnnounce:toResult: adds the TestCase to the list of failures, but what I’d really want is some extra infor to which I can add the error description so that it is accessible from the TestResult.

One approach (which seems obvious) would be to add the TestCase to the TestFailure Exception and add the exception to the failures collection. Thus the failures list could access all info that’s needed for Reporting.

I’ll have to Play with this a little, and hope to find some time in the next few days…

4 thoughts on “What sUnit could learn from jUnit [UPDATE]

  1. If you look closely, SUnit is not the same on all platforms. For example, SUnit on Squeak/Pharo (at least) has support for expected failures. This is really useful for cross-platform test suites where some of the cases are not expected to run on all platforms since it filters out the expected red/yellow balls in the SUnit browser, thus reducing the clutter and enabling us to focus on the real failures.

    Additionally, SUnit is not really an open source project since it doesn’t have a public website where issues such as this can be discussed (yes, I know about sunit.sourceforge.net, but it is almost totally inactive) — things happen to SUnit because Niall Ross gets interested in some feature (and feeds the results out to the various platforms who can choose to take it or not), or a platform developer (such as me) gets interested in some feature (and feeds the results back to Niall for inclusion in the “official” release).

    I would certainly like to see more transparency in the progression of SUnit. I have suggested SUnit as a CampSmalltalk project at Smalltalk Solutions this year. I don’t know if Niall is going to be there, but maybe if some of us “interested parties” get together, we can make some stuff happen.

    1. John,

      thanks for answering. Seems my picture of SUnit is somewhat wrong. I was under the impression that Niall is closely working with the Pharo Tribe on carrying it forward. The sunit page on sourceforge is in a similarly sad stage as glorp.org.

      Okay, so I will probably just go ahead and see what I can change on VAST and send Niall a mail describing my changes when I’m done. My first ideas so far would most probably break the SUnit Browser, which is what I’d like to avoid, or I need to fix that as well. Thinking of it, having a way of seing the error texts in the SUnit Browser would probably not be too bad, so well, maybe I even need t take a look at that. Which makes the likeliness of anything happening even smaller 😦

Comments are closed.