In the last couple of days, we were working on new API which rely on the registry for data storage and retrieval. Ohad and I paired for this development task.
We used TDD in this task. And we used Isolator for mocking the registry calls. Basically, we wrote unit tests for interactions - we set expectations on the registry calls in the test code, and then wrote the code that filled those expectations. We used Isolator's CheckArguments API to make sure that we were writing and reading the correct keys. And finally we called MockManager.Verify to verify the calls were made correctly, with the correct arguments.
We were actually doing what classic unit testing does - abstracting the environment, and focusing on the inside. Tests passed as wrote them, and due to the nature of tests (the expectations we set were detailed enough so we can write the code correctly) we didn't spend too much time on debugging. (Yay TDD!)
For a latter API we could not abstract the registry calls any longer, and so we wrote integration tests for it, using the registry API. And following that, wrote examples for the new API, which uncovered a few bugs that the unit tests hid. When we fixed the bugs, the unit tests we wrote failed, and we had to fix those also.
We were discussing today what we can learn from this. Here are a couple of ideas:
- We were focusing on unit tests, while not having enough integration tests.
- We didn't check for all the arguments, and amazingly, these were the ones where the bug was.
- On the other hand, our tests are very specific already, and fixing them was our "reward". Maybe with less details, we would spend less time on their maintenance.
Unit tests are great, but they are not enough. Integration tests should accompany and test the "real" environment. And the tools we use (in this case, mocking with Isolator) should help us in writing simple tests. Detailed tests are ok, as long as you are ready to pay the price.
Did you ever get lost in that magical zone between integration and unit tests?
2 comments:
Yes I did get lost. As probably any other newbie, I started with integration tests for everything. I really hated it. The application worked, but the code was a mess, since it wasn't TDD but rather test-first development.
Next, I tried something called Need-Driven Development -- concentrating on a small unit and mocking everything else that I didn't want to code at the moment. After I coded everything and got all greens, I launched the demo app and discovered that everything is wrong.
Now I'm using an approach that is the best for me. First, I choose a simple story and test-drive it. What I get is an integration test and a small feature that works. Next, I look at my test and figure out how to "refactor" it into smaller tests, at the same time refactoring my code into smaller units. So, it's a sort of test-driven refactoring. Smaller units doesn't mean classes but rather something that has sense in terms of user requirements. When I add a new feature, I don't write a new functional test for it, usually I already have a unit that can implement it. So, I have an integration test making sure everything is linked together correctly, and a couple of unit tests that reflect the requirements. I can always refactor further, making the code more flexible, but I don't test-drive it. This is why unless the requirements change, my tests don't break.
With this approach, the integration test verifies that the arguments are being passed, and the unit tests verify that the arguments are correct.
ulu
@ulu:
Your method looks very interesting. Can you give an example on how you refactor test to smaller ones? Does the "resolution change" makes you lose some details or cases?
Post a Comment