This is just a small idea, but I’d like to share and discuss it: A test method that contains no assertions should turn yellow by default.
Why on earth?
Well, you know, in theory, everybody starts writing tests with assertions. And as we all know, we all write tests first and all that stuff.
In practice, the grass is a little less green and I admit freely that if I do test-first (or at least test-together-eith-evolving-code), I often start by writing a test method that simply contains a method send to the real method, just to see if it runs at all, or if I have some very stupid logic error in there. Sometimes, I even don’t really know what I’d expect as a result from the error, I just work myself towards the logic of looping over a stream or whatever. The approach of using a TestCase for the first experimental runs of an eveolving method is a good thing for me, because I can run my code and collect all that’s necessary to feed into it for its runs as setup code in the TestCase from the very start. I’ll have to implement that setup later anyways, so this saves me time and makes writing tests a very natural experience (just swap out the workspace for a test case).
Okay, so what?
So I start writing the method (most of the times, I must admit, I start with writing a draft method before I write the test method, and then start with a test driver for it to see if my idea for the method’s logic is any good) and then write a test method that simply calls my method. That’s all good and well, and I like the way this helps me write code.
BUT far too often, I jump away from this test method and work on something else, because I like the results of my method and since I have a test that’s green, all is perfect.
But wait: it’s green, but this doesn’t mean it meets any requierements, because I’ve not formulated any. Remember, my test method has no assert: deny: or anything like that, it simply runs my method. So it doesn’t really test anything other than whether the method crashes, because that would turn the test result to red.
It has happened to me more than once now that I was proud to have green tests and having done everything so much better than most developers ever do, just to find out that even though I had a bunch of green tests, the end result simply was wrong, because the green tests simply were green because they didn’t test anything.
But hey, if I did it right, this would not be a problem!
That’s true. If I always started by formulating a (probably useless or sure-to-fail) assertion around my call to the method, the test would fail from the very start and I couldn’t forget to turn the test green before shipping the code.
But there are several major problems with this approach:
- I am bad in breaking habits – good or bad
- Having to think about the result of a method before I even know how the logic of the method will be, and even before I know if it’s going to be broken up into several methods in just a few methods, interrupts my train of thought. But if this assertion should always fail anyways, why can’t SUnit fail for me (see also next argument)
- Isn’t a tool made for making life easier? So shouldn’t it check whether it’s checked anything, because if it didn’t it simply lies to me when it turns a test green!
- I guess I would write an assertion to the end of each test method that makes no sense, just to turn the test yellow. This would only help me develop a new bad habit instead of support my in my good intentions. And: SUnit can do that for me and thus enable me to concentrate on assertions that make sense while it reminds me I still need to write an assertion.
So what am I asking for ?
In fact, I am asking for a default state for a test to be yellow until a first assertion is made (or an error occurs, of course). Because green is a very dangerous color for a test method that didn’t check anything. It gives me the illusion that I am on the right track, and this is dangerous and can be expensive
So what do others think? Should we change SUnit or am I the only one with this problem?
