Parameterized Tests and RowTests

Mel Grubb talks about parameterized tests and why use it at all. He also raises the question whether having the ability to run parameterized tests can rule out usage of a framework. Personally, I think there are better ways to choose unit tests framework, but that’s not what this post about.

First, what are parameterized tests?

Regular tests, in NUnit, MSTest or other frameworks, are void methods that take no arguments. Parameterized tests are tests that certain frameworks (like Gallio) can run several times, each with a different set of input parameters and expected result.  MbUnit uses the attribute RowTest, because each row contained a set of input parameters and a result, like a list. (And if you’re familiar with Fit/Fitnesse, the table metaphor may remind you of this as well).

With different parameters, I can run the same code, and not need to copy/paste it. This looks like a great productivity boost. But like Mel said, you’re losing some readability of the tests. If you came back to the code 2 weeks, months or years and look at the code, you wouldn’t know what differentiates the 2nd from the 3rd row. They all look the same.

So what Mel suggests is doing some refactoring, and increasing the readability by naming the tests accordingly, which I agree whole heartedly with.

But I’ll go a step further. Which test would you like to debug, if it broke? The one with the descriptive name that tests a single scenario, or the one with the laundry list of arguments? I’d go with the first.

One more thing: Now you can use the parameter in the test itself. Instead of arg1, you can now send to the tested method arg1*2. This starts to smell, you’re actually putting logic into the test. So, especially for beginners, or other people who would like to cut corners, I wouldn’t recommend using it.

Let’s say I have a component that does authentication. You give it a name and password, and it gives you back a yes/no result. For all the specific criteria I’d write specific and named tests: name is null, password is longer than 6 characters, password contains alphanumeric characters. Basically, if I can characterize it in words, I’ll write a unit test for that.

Now if I have a batch of hundreds of collected names and passwords, that I just want to make sure that pass authentication, I might want to write a parameterized tests, or just loop through all the list and assert the result. That’s in addition to the other tests I’ve already written.

By the way, if one of those breaks, I’d write a specific test for that as well. I’d recognize the case of failure, and characterize it. So the next time I come across, a case like that, I can debug more quickly.

Parameterized tests are nice. But do yourself a favor, for unit tests stick to the basics.

Unit Tests Myth Busting

Arjan Einbu spills the beans on his experience with unit testing, busting one myth after the other. Like the “I don’t have time” excuse, if you’ve used any of these excuses, better find a new one. Scratch that, write a test.

And about the one where management doesn’t buy into it? Go guerrilla and do it behind their backs. You might not get the pat on the back, but they’ll be busy blaming all the other developers for breaking the app, while you’ll be gloating about your perfect code.

Eric Shupps on SharePoint, TDD and Unit Testing

A while ago, Eric Shupps wrote about TDD and SharePoint development. His follow up post is really a good read, because rather concentrating on the technique (TDD or Test After), he’s concentrating on why we do unit tests in the first place: Making better code, for less bugs and shortening development cycles.

And I like this quote:

"In fact, I would argue that, due to the complexity of the SharePoint object model, the various undocumented and often erratic behaviors, and the level of skill required to effectively troubleshoot and optimize such code, Unit testing has more value in a SharePoint context than in any other.”

Yes, you can replace SharePoint with your favorite technology and it’s still works…

Solving Problems with Isolator – Windows Service Interception

At the same day I posted about different usages of Isolator in the real world, that are not just regular faking, Travis Illig contacted me about an experiment he’s doing, which obviously succeeded.

Travis, a Typemock MVP, used Isolator to change the behavior of a single method inside 3rd party code, running in a Windows service that was giving him pain. The pain was removed by allowing Isolator to run inside the production code, intercept the offending call, and redirect it to another implementation. Now, how cool is that?

I’ve discussed with Travis other ways to intercept and modify calls. The first one Travis suggested is using a Swap.AllInstances of the offending type and then using a WhenCalled-DoInstead combo, to intercept the call and execute another piece of code instead.

The second option is redirecting the calls to another object, as Travis solution does. In this case, you redirect calls from one object to another. Only methods that have the same signature get redirected. All others are executed on the original object.

The final option is using CThru. With CThru you specify all the method calls you want to intercept, and then implement an event handler that is invoked when the method is intercepted.

For all these to work, you need the Isolator engine to run, and that’s why the environment variables are set. Basically, you can run TMockRunner and do the same thing, although it does get a little heavy when handling Windows services.

Well done Travis! Another reminder that innovation comes from everywhere – when people use a tool differently than what it was intended for, great things can happen.

Videos from Typemocks Unit Testing open discussion Event at #NDC09

Below you can see videos from Typemock’s Unit Testing and Beer open discussion event at NDC09. The first 2 videos are of Roy Osherove and Carl Franklin from .NET Rocks singing (a huge thanks to both of them!), the other 2 are of the open discussion.
We want to thank again all the developers who participated in our event, and we hope to see you again in future events.




Roy Osherove Intro words + Carl Franklin from .NET Rocks singing:

Roy Osherove Singing:

Open Spaces discussion - 1:

Open Spaces discussion - 2:

52% of developers say: Unit testing is the answer for the EU software liability bill

According to a survey held by Typemock, more than half of the developers believe that unit testing can help companies avoid law suits if the new EU software liability bill will pass.

Typemock, the Unit testing tools provider, held a survey asking .NET developers if they think that unit testing can help companies avoid law suits if the new EU software liability bill, proposed by Commissioners Viviane Reding and Meglena Kuneva, will pass. 52% of them said that it can.

Typemock asked developers this Online, in their blog, and offline, in Typemock’s Unit Testing Open Microphone event at the 2009 Norwegian Developers Conference (NDC). In this event developers took turns in voicing their opinions on Unit testing and the EU software liability bill. While the majority of them said that they shouldn't be held liable for their code, they believe that if this bill passes, unit testing will help companies to be confident in their product and avoid future law suits.

Unit testing was one of the central topics at the NDC conference, with keynote speaker Robert C. Martin (Uncle Bob) declaring that unit testing and TDD are a must for professional software developers, a whole day dedicated to unit testing sessions, and as mentioned, Typemock’s Unit testing event.

“We believe that companies should always strive to release products with the highest quality, that produce real value for the customer, and that unit testing is the most effective way to start making that happen” Says Roy Osherove, Senior Developer at Typemock, and the author of the book the Art of Unit testing.

Unit testing is an upcoming software development method in which developers write tests for small parts of their code, and check them individually (in ‘isolation’), and not as part of the whole application. This is done in the first development stage, while coding the application and not in the QA (quality assurance) stage like most tests. Unit testing lowers the ‘Bug Fix Time’ and ‘Bug fix Cost’, and frees up the developers’ time, so that they can create new features instead of fixing bugs.

Typemock develops solutions for .NET unit testing, giving developers the power to easily perform unit testing by making them easy to write and automate. Typemock’s unit testing tools have enabled companies to release their products with higher quality and fewer bugs, while saving both time and money.

Testing without SmtpClient

This is pretty awesome. here’s another way of really unit testing smtpClient code, without having a server running:

 

I’m using Isolator to capture and redirect calls from the next instance of SmtpClient, capturing the parameters to a method call on it, and then asserting on the passed in object.

Working with Racer, Part II

This week I played with Racer and Gallio, the current incarnation of mbUnit. Did you know that Gallio comes with Typemock Isolator extensions?

Anyway, Gallio has some thread manipulation classes, like TaskContainer and ThreadTask, as well as others. I tried to find deadlocks around those classes. No dice. I’m not discouraged though. I’m just going to need a more thread-complicated project where a deadlock is waiting to be found.

But until then, here are things I find during my exercises, useful ones, I think.

First, when using MSTest, which I do, you need to turn off deployment in the solution’s test configuration file. There’s a small bug that causes a weird exception when you run with the default deployment on.

Next, running with Isolator. Yes, Typemock tools do work together. When you have both installed, and you want to run Racer, you should disable Isolator and enable Racer. This is the default mode, since Racer runs the regular code paths.

However, there come times where you need dependency isolation. In my tests, I found that the code I want to Racer to test calls system files dependencies. In another case, the function got an abstract class as an argument. In both, those didn’t have any impact on what I was looking for, namely thread manipulation methods. So I faked them with Isolator.

In order to make that work, you need to disable from the Typemock menu and enable Isolator. Then you need to link Isolator with Racer, using Isolator’s Configuration program. And that’s it, you can now use Isolator’s APIs to your heart’s content. This can make the process of running much shorter, because as we know dependencies can be heavy on the processing side, while not important for the purpose of the test.

So far I haven’t found a worthy opponent. And by worthy I mean a complex enough multi-threaded project that might actually cause deadlocks. What I’ve encountered until now are simple locks. What do you suggest I try next?

Avoiding Fragile Tests

This post came from reading a question on our forum, as well as reading one on TDD experience. It’s about fragile tests, and the cost of maintenance.

We’re talking about refactoring. And not just for making the code prettier and maintainable. We refactor and then our tests break. Well, sometimes they do, depending on how we wrote them. But if we change functionality, yes they will have to change.

It becomes worse when our tests know too much about the inner logic of our components. If component A uses B in some way, and you change that, it maybe that integration tests for A are still passing, but the ones where you fake B break.

What should we do? Are we doomed? Is this our fate? Is there no way out?

Alas, there is no simple answer. First, remember that the cost of maintaining your tests is largely outweighed by having these tests in place at all. Having no tests at all costs a lot more than having to maintain a few tests. So if someone tries this on you, show them the door.

The second thing is to come to terms with the fact that test code is just that - code, and therefore is subject to the lifetime changes such as production code is. If you are using Isolator or any mocking framework out there, you’re bound to hit this sometime.

But you can minimize the risk of creating fragile tests. Here are a few suggestions:

• Verify less. The Verify APIs (or Mocks in general) tie your test code to internal functionality. These have much more impact on breaking tests on changes than WhenCalled APIs (or Stubs). Use them only when you need to.
• Write focused tests. The more focused you test, meaning testing a small portion of the code, you’ll be less affected when you change other code.
• Minimize usage of  NonPublic. This makes a lot of sense, although obviously it’s there because you need it. Still, try. 
• Use less WhenCalled statements. Less statements means less knowledge of tests of the tested code. Here’s Isolator biggest advantage when you use recursive fakes. Since you’re changing the behavior of an entire tree of objects (without specifying which objects) in a single line, it minimizes the risk of a broken tests.
• Use proper tools, especially refactoring tools. Resharper, CodeRush and others make changes much safe then by doing them manually. Like I always say: use the proper tools for the right job.

What’s you learned-the-harsh-way lessons for avoiding fragile tests?

The “I don’t have time to write a test” Excuse

I like this one. I usually hear it in a much larger organizational context, but hey one developer is a micro-cosmos of the entire org, right?

Alk talks about the cost saving of writing a test in terms of just running and debugging an app for a bug. We always talk about how writing tests impacts development now, but saves so much during the entire lifecycle of the application. But, writing a unit test to fix a bug is the one case where the developer sees in his own eyes how much time gets wasted when he’s debugging. It’s time saved for him, not for “everyone”.

When developers see they don’t need the entire app just to tests a single method, it’s one of those “Aha” moments we all love. If you want to convince a developer that unit testing can help him, not only the entire project, show him how a writing a simple test can save him a lot of quality time with the debugger. And then say: “Oh this little tool? It’s called Typemock Isolator. Saves me hours, and it can help you do that too”. 

Unit Testing 101 Presentation

This is a nice one by Yaron Naveh. Sure, it uses Moq and not Isolator, but what the hell.

The funny thing is that currently I’m showing demos of faking DateTime.Now for testability, and suddenly wrapping it up just for testability (like in this presentation) seems weird.

New Tutorial-Unit Testing ASP.NET MVC applications with Typemock's AAA Syntax

A new unit testing tutorial was uploaded to the article section on our site.
This is a comprehensive tutorial on Unit Testing ASP.NET MVC applications with Typemock Isolator.
A big thanks goes to Soon Hui Ngu , for writing this.
Check out Soon Hui’s blog for more .NET development info.

Pictures from Typemocks #NDC09 - Unit testing and Beer - evening

We would like to thank everyone who participated in Typemock’s – Unit testing and Beer – event on NDC. A big thanks also goes out to Roy Osherove, Carl Franklin (host of .NET Rocks!) and the good people of NDC especially Rune and Kjersti.

We got great feedback from you guys, both for the Unit testing and software liability discussions and for Roy Osherove’s and Carl Franklin’s song.

We have several videos from this event (including Roy’s and Carl’s song), they will be uploaded next week as we still need to edit them.

In the meanwhile have a look at the pictures below.

-

Problem Solving with Isolator

I’ve written before how Richard Fennell is using CThru to fake a SharePoint workflow. But he’s not the only one using Isolator to solve problems.

Take for example, Apostolis Bekiaros. He’s writing an MVC app, and wants to write more readable tests. Sure, he can do this the hard way, since MVC allows it. But his code with Isolator is much more elegant. By the way, he has a new open-source project called eXpand, which is tested with Isolator.

On the other end of the globe, Francis Cheung, uses Isolator in integration tests, by redirecting calls from the SharePoint’s server to object you construct locally. I tend to say it a lot lately: The same tools today are available both unit testing and integration testing (MSTest or NUnit, Isolator) and you can use both for different purposes in different ways. And what Francis did is really cool.

Well done guys!

Back from the NDC

For those of you that weren’t there, Typemock had an open-space event alongside the NDC geek-beer evening – we were talking about the latest EU bill proposal that’ll make developers liable for their code (yes, you can be sued for bugs in your code). We’ve got tons of material which still needs to be tidied up – so in the meantime, here’s a teaser: