Coalmine Canary Tests

I was pairing the other day and we started talking about the scope of tests. I'm a huge advocate of writing the simplest, just slightly more than trivial tests when doing test-driven development, especially in a dynamic language (we were using Ruby). My pair, Kristy, and I got into a discussion of around 2 areas: just how simple should they be and should you keep them around after you've written the code? Lots of developers use these ultra simple tests, then make the test more robust once they've gotten the first incarnation to pass. I'm a fan of what I call "Coalmine Canary" tests. Back in the old days, coal miners used to take caged canaries down in the mines with them. The birds were much more sensitive to the gas that can build up in mines; if the bird died or starting struggling for breath, the coal miners knew it was time to high-tail it out of the mine. In the same way, very simple tests help you judge the safety and semantics of the code under test. We created a very simple test (basically, verify a method that included a semi-complicated ActiveRecord query returned a non-zero number of rows). We decided to keep it around, because it verified that we were calling the infrastructure correctly (in other words, that the semantics of the method were correct). Sure enough, as we started adding more code to the class, we introduced bugs. The first test we ran after making changes was the Canary test. If it failed, we knew that we had a fundamental problem with our infrastructure and didn't even bother looking at the more complex, nuanced behavior we have added until the Canary test passed again.

Having (and keeping) the simple Coalmine Canary test is a great sanity check on your code. If it fails, you've got serious problems, and you'd better fix those before you do any more mining in your code.