Little Addition to SUnit: keep a failure text with errors and failures

I’ve posted about a problem I’m having with SUnit before: a TestResult does not hold the description Strings of failures and errors, so it is not easy to log unit test results with bare SUnit.

I need this for our Hudson Build Server integration on our project. While some developers have solved this problem by using their own Test runner implementation that writes a log file in jUnit Format right during the run of a TestSuite, I’d prefer to first run the tests and harvest all the Results from the TestResult afterwards. It may be a mere question of Taste but I think mixing test running code with XML generation has a certain smell.

For quite a while we simply ignored the problem and fed an XML file back to hudson which contained the text “The test failed – please rerun it in the SUnit Browser for details”. Which is of course exactly what you have to do when you want to fix that test or the code it tests: rerun it in the Test Browser and maybe debug or step it to see what’s wrong. But it felt strange. We had spent so much effort into automated build tools and hudson integration batch files and stuff just to see a bloody placeholder text in our statistics. It just feels like being a whining coward, not a redneck programmer. But what’s even worse is that the developers tend to not add descriptions to their assertions, because they’re useless anyways. So this little glitch in SUnit may lead to tests that tend to be cryptic. Part of my consulting job is to convince long-time Smalltalkers about the usefulness of unit testing in legacy projects. There’s not much that’s more depressing than to see how people finally use unit tests and suddenly realize they give up on descriptive assertions for that reason.

So there’s always been the idea of “fixing” SUnit to make it keep the description Strings for failed assertions. It turned out that it’s neither easy nor elegant, at least none of the ideas I tried.

The best and most natural place for an extension first seemed to be TestResult>>#sunitAnnounce:toResult:  which was introduced in SUnit 4. Here’s what the release notes for SUnit 4  say:

TestResult now double-dispatches via the exception (see #sunitAnnounce:toResult:). This makes it easier for users to plugin specific behaviour.

Unfortunately, there are two problems with this method:

  1. The exception thrown or failed assertion is not handed over to this method, so you can’t store it anywhere
  2. If you simply create and instance of a TestCase and send it #runCase to run it, there is no TestResult involved in running it

Another problem is that even if we mostly run our tests in the form of a TestSuite which always produces a TestResult, the collections #failures and #errors do not store some kind of test result object, but simply the TestCase instance that failed/error’d.

So the simplest thing that could possibly work is to add an instance variable to TestCase to hold a String (well, it can of course hold anything, we’re using Smalltalk after all). This variable would be set to an empty String just before a test is run and be filled with either an exception’s description if test execution results in an error, or with the description text of a failed assertion.

This gives us the ability to not change much about the way TestResult works, not break the SUnit Browser or any other SUnit-related tool, but keep the error texts.

It turns out this worked quite well for our project, and there was very little I needed to change in SUnit:

  1. Add an instance variable #failureDescription to TestAsserter (with getter/setter)
  2. Change TestCase performTest to:
      "Implemented and tested on VA Smalltalk 8.5, but should be portable"
          self failureDescription: ''.
          self perform: testSelector sunitAsSymbol]
            sunitOn: TestResult failure , TestResult error
            do: [:ex |
                self failureDescription: ex description.
                ex pass]

And that’s it. I can now iterate over the #failures or #errors of a TestResult and harvest their decriptions to do whatever I want with them. In my case I want to add them as XML tags into our jUnit-xml for the hudson server. So far we’ve not had any problems with this change.

I’m well aware that this is neither an extraordinarily clever nor galactically elegant solution, especially since it involves changes to the SUnit code base. But I like the effects I see so far. The feedback from the team is: “I can now tell right from the hudson log what my bug is in many cases”.

Before I am putting this change up to, I’d like to hear what other people think about this change. Is it going in the wrong direction? Too invasive into the concepts of SUnit? Would people rather put this kind of thing into the logging portion of SUnit (Be aware that logFailure* methods are only executed for failures, not errors!) or do they like this approach? Any better ideas out there? How did you solve this problem?