When Tests Fight Back

In the last couple of days, we were working on new API which rely on the registry for data storage and retrieval. Ohad and I paired for this development task.

We used TDD in this task. And we used Isolator for mocking the registry calls. Basically, we wrote unit tests for interactions – we set expectations on the registry calls in the test code, and then wrote the code that filled those expectations. We used Isolator’s CheckArguments API to make sure that we were writing and reading the correct keys. And finally we called MockManager.Verify to verify the calls were made correctly, with the correct arguments.

We were actually doing what classic unit testing does – abstracting the environment, and focusing on the inside. Tests passed as wrote them, and due to the nature of tests (the expectations we set were detailed enough so we can write the code correctly) we didn’t spend too much time on debugging. (Yay TDD!)

For a latter API we could not abstract the registry calls any longer, and so we wrote integration tests for it, using the registry API. And following that,  wrote examples for the new API, which uncovered a few bugs that the unit tests hid. When we fixed the bugs, the unit tests we wrote failed, and we had to fix those also.

We were discussing today what we can learn from this. Here are a couple of ideas:

  • We were focusing on unit tests, while not having enough integration tests.
  • We didn’t check for all the arguments, and amazingly, these were the ones where the bug was.
  • On the other hand, our tests are very specific already, and fixing them was our “reward”. Maybe with less details, we would spend less time on their maintenance.

Unit tests are great, but they are not enough. Integration tests should accompany and test the “real” environment. And the tools we use (in this case, mocking with Isolator) should help us in writing simple tests. Detailed tests are ok, as long as you are ready to pay the price.

Did you ever get lost in that magical zone between integration and unit tests?

TOP