The Problems with Unit Testing Frameworks

Unit testing is essential in order to ensure that your code does what it’s supposed to do.

But, as much as Typemock is a firm believer in unit testing, well, let’s be honest — it’s not fun.

As our CEO recently put it in Dr. Dobbs, “Unit testing is like staying healthy,” argues Lopian. “Staying healthy requires best practices such as eating right and working out. Similarly, development teams need the right practices in order to innovate faster. Just as it’s hard to start working out, many find it’s hard to unit test and thus stop — despite its well-known benefits”

But unit testing frameworks have many problems.

Some of them include:

1. Lack of Automation – Yes, you can write manual tests and do everything manually. You can also code in Notepad or another text editor. For some, that’s fine – witness the text editor wars in Linux – but others prefer a robust IDE and toolkit. Automatic unit testing saves time, allows for more robust testing,  and allows for tests to be run repeatedly. But even many automatic unit testing frameworks, like xUnit, or mocking-only frameworks don’t let you test much of your code unless you go 100% by the book with excellent design. Most of us simply don’t have code that can be adequately tested by even the most robust open-source frameworks, leaving us (and our business or product) vulnerable to bugs and technical debt.

2. Focus on test instead of code – The goal is working code, delivered on time. But, too often we forget this, focusing on our tests. But, this allows a disconnect between our test and our code. When a test fails, what does that mean for my code? When it passes, how do I translate that into working software?

Frequently, while we know that a test failed, we don’t always know where the error is. This makes it harder to achieve our goal – getting code out in time that just works.

3. Takes too much time – In our high-stakes development environment, frequently understaffed and overloaded, every second counts. Sometimes, we’re also tempted to shortcut now and save time up front – of course we’ll pay for that later but later is abstract when your boss is breathing down your throat. And, yes, even when you have the luxury of doing TDD by the book on greenfield code, it still takes time up front (of course it will save you time overall, but when the release is due, try telling that to your boss).

Currently, most unit testing frameworks run your entire code every time you make a minor change. If you’ made a minor change, you still need to rebuild your code from scratch. That’s time that you just don’t have when your code is overdue. Each change takes time and seconds turn into minutes and hours.

Frameworks need autocomplete – shaving time – and to just run the code that changed.

4. Weak mocking – Require TDD and good design first – can’t deal with legacy code – Simple greenfield code is a nice fantasy but it’s not most projects. What is needed is a framework that can mock complex legacy code, privates, statics, etc. Most frameworks don’t offer powerful mocking or the ability to test legacy code, new code, or even unwritten code.

5. Coverage – Too many frameworks don’t let you know what’s covered and what’s not. Yes, as Uncle Bob said, coverage is not the most important or only metric and 100% code coverage may not always be the ideal.

But, too often, we don’t know what code is covered and what code is insecure.

 

Leave a comment – what are some things you find lacking with unit testing frameworks. Feel free to tweet as well using hashtag #tddfail

 

It’s time for a change.

Luckily, change is coming.

Stay Tuned TV

Stay tuned.

Can’t wait? Discover unit testing tools and practices this Wednesday and enter the raffle to win an Isolator license.

Want more webinars? Sign up!

  • Bob Lauer


    When a test fails, what does that mean for my code? When it passes, how do I translate that into working software?” 

    Duh. TDD is pretty clear about this: When it fails then it is a new test or you have broken something. When it passes , you are done.

  • Kenneth A

    Takes too much time?? What? You have clearly not understood what unit testing is and what it accomplish. If You do it correct – then it actually SAVES time. You don’t have to spend time debugging stuff that was written ages ago. You can actually implement functions that the customer wants and implement them well within the deadline specified by You….
    You should write test and code at the same time! One line of test code, then one line of production code and then back and forth…. Then You can not wander too far away from working code. 
    But of course – it will take a little bit longer in the beginning before You have learnt how to write good tests parallell with production code – but You will gain that time back in the end. And You will keep Your customers happy by being able to deliver WORKING code on time. 

    • Ddd

      You can never know whether your unit tests saved you time or not. You can’t do the project with them and without them.

      Any unit test that never reveals a bug is wasted effort. You can’t know that ahead of time, so you treat unit tests like life insurance and put them in and hope you never need them. The problem is that people are not discerning enough about what they cover and what they don’t so you spent a bunch of effort trying to get really high test coverage that you don’t really need.Software existed before automated unit tests and I don’t think quality has gone up significantly since their invention. I’m not saying they’re bad, in fact I use them myself. I’m just saying that people go too far with them, use them as a crutch, and they do end up wasting time.

  • http://twitter.com/QATestingTools QATestingTools

    great opening…but didn’t sow the webiner

    will be happy to write more about your solutions….
    please visit me at http://www.qatestingtools.com

  • http://qualitycoding.org/ Jon Reid

    “It’s not fun”? Au contraire, unit testing gives me a heady rush. And with a safety net in place, I can refactor code to my heart’s content.

  • Manish Shivhare

    Hi,

    As it is said, practically testing cannot be expected to catch every error in the program: it is impossible to evaluate every execution path in all but the most trivial programs. The same is true for unit testing. Additionally, unit testing by definition only tests the functionality of the units themselves. Therefore, it will not catch integration errors or broader system-level errors (such as functions performed across multiple units, or non-functional test areas such as performance). Unit testing should be done in conjunction with other software testing activities. Like all forms of software testing, unit tests can only show the presence of errors; they cannot show the absence of errors. 

    Regards,
    Manish

TOP