TDD Research

Here’s a very nice case study conducted at Microsoft on two projects:

http://research.microsoft.com/manuvir/papers/isese-fp17288-Bhat.pdf

Here are two tables showing the difference when using TDD

Table 1: Project A Outcome Measures

Metric Description Value
Actual defects/KLOC (using TDD) X
Defects/KLOC of comparable team in org but not using TDD 2.6X

Increase in time taken to code the feature because of TDD (%) [Management estimates]

25-30%

Table 2 Project B Outcome Measures

Metric Description Value
Actual defects/KLOC (using TDD) Y
Defects/KLOC of comparable team in org but not using TDD 4.2Y

Increase in time taken to code the feature because of TDD (%) [PM/DevLEAD/MGR estimates]

15%

I think that the numbers speak for themselves.

  • Jacob Proffitt

    While they might speak for themselves, what they say turns out to not be that useful. The problem comes in what they give us from the comparable project metrics. Or rather, what they don’t give us. The only metrics we get from the non-TDD projects are LOC, dev time and team size. Conspicuously absent is test LOC and % block coverage. I couldn’t find anywhere in that case study where it says that the comparable projects used unit tests at all. So is this comparing TDD vs. no unit testing? That’s not a win for TDD, that’s a win for unit testing. That’s a problem I’ve had before with these studies.

  • Lior Friedman

    Jacob, actually that was my intended point in posting this. I think that the win is for AUTOMATED unit test over (probably) manual testing (at least this is an empirical result that can be quoted). Although the article doesn’t explicitly mention it, I think its safe to assume that some sort of testing was conducted in the “other” projects.
    That being said, I would still favor doing TDD, at the least its a good mechanism to make sure that the the tests are actually getting written (I find it so much harder, mentally speaking, to do it after)

  • Jacob Proffitt

    See, that’s why I want these things to be explicit. So much of the TDD “proof” doesn’t handle the difference between writing the tests before and writing them after. I personally suspect that writing effective unit tests is the key to improvements in quality and that TDD itself has little or no value-add. Unfortunately, nobody is interested in testing that specific hypothesis. I really wish that they would, if only for the sake of those of us like me who find it so much harder, mentally speaking, to write the tests before…

TOP