Friday, May 23, 2008

Running Typemock Isolator Tests using TeamCity Nunit Runner

Wendy just sent me over an email telling me how she was able to use TeamCity's Nunit test runner to run Unit tests that use Typemock. the "problem" was that for typemock to run, it's profiler needs to be active as part of the unit runner process.

here's what she did:

"figured it out – I used the environmental variables setting for the build and set the values from mocking_on:

Cor_Enable_Profiling=0x1

COR_PROFILER={B146457E-9AED-4624-B1E5-968D274416EC}

I’m so clever :P"

Indeed you are! Thanks for the heads up, Wendy.

the Mocking_on she refers to is the mocking_on.bat that comes as part of the Typemock Isolator installation.

Wednesday, May 21, 2008

Automated Unit Tests in my life - Part 3

Part 3. Work or play?

The World before

Let’s think about our working cycle. I am not sure about you but… here how my working style looked before:

1) Wrote the code, which is a solution for requirements

2) Wrote a tester (Exe, or GUI)

3) Checked that it works. Mostly I checked expectations visually: using debugger or printing the results to a file (Text or XML) and looking at the file.

Let’s see which problems could appeared:

You write code to solute some requirement. Then you check it. Sometimes you have to write more code, which breaks your old test, so you write a new test. This is the reason that you will want first to finish writing all the code and then to write tests at the end. And I will tell you from experience - this is bad, really bad! Coding like this is problematic: the code becomes bombastic - it will take you more thinking and time to write it, you should remember many details, if you went out in the middle, takes long time to remember all the details, and this is the reason we write some not needed, I call it not "clean" code.

2) Tests run took a long time because I had to look at each test result visually and it took lot of time (sometimes had to look at big files). Of course we don’t want to run “old” tests when new are added.

3) Mostly sometimes we forget to write all tests, especially when the code is done after tests are written. Even if we write tests, we not always run all of them because it takes long time – to run each test and look visually at all tests results.

There are also situations when unit tests were not made. In order to save time people made integration tests, or in better case “unit tests” that check number of dependent components together. It never ends well, I can tell.
When you work test driven it’s all different… What is different ?

Work is a game

After few weeks of using automated unit tests and test driven, I made my “own” system of writing a code.
And the work became to look like a game. Here are the rules of the game:

1) Write down requirements logic – all detailed requirements (on paper or document)

· Try to write a question for each requirement in the code – this will be the name of your test function.

· Run current test. Of course the test will fail – sure there is no code yet.

· Now write a minimal code in order to pass the test.

· Run current test and make it pass (by adding required code and etc). Test passes when expected result equals to test result.

· Write the different expectations for current function, so copy – paste same code, just change input and expected result values. Think when your specific test can fail. So each requirement will have different test for different situation

Now add new function (or logic) test. Each time when you add a new test – run all tests. This will show you that if you have added some new logic you did not break old one. Sometimes when we add a new logic we break old one. When new requirements are implemented, logic is changed, so expected result is affected and an assertion Assert.AreEqual(actual, expected); fails.

After practicing this I found in internet a nice diagram that shows similar flow.
They call it Test Driven Development. And here is a TDD (Test Driven Development) cycle:

I hope we are agreed that this is a big change in coding cycle. A different way of thinking and different way of writing the code. It is not so easy to change the way of thinking when we are used to other things. And it was not as easy for me at the beginning.

But in our life we always learn something new, change things, our tastes change.
We become better, our code quality becomes higher – I call it self-development or evolution.




Sunday, May 18, 2008

Automated Unit Tests in my life - Part 2

Part 2. First experience

A day later I began to like it… And you know why?
Here are the first things I liked about automated unit tests and the Test Driven:
First of all – I realized that writing tests and code using automated unit tests did not take more time then my “regular” way of working at that time.

Now before writing a new function (method) in class I write test that calls that function. I have to think about how client will call my function, about possible arguments and possible returned values – this saves time !!!
Why else does it save time?
a) I do not have to write “*.exe” tester. (or some Gui tester)
b) I do not have to look at test results visually, I can see at once if test is passed (green colors)
Automated means – you write expectations and results and expected results are compared automatically. This is a part of an assertion code
Assert.AreEqual(actual, expected);
If expected is equal to actual, the test is passed (green), if not – test is failed (red colour). Assert is done automatically in runtime and you do not have to look at each test result visually.

Think about 10 tests that run in a second and you can say at once if they passed or failed.

How long it takes to run each of them and look at the result vusually (sometimes using debugger) ???
c) It is easier and quicker to write a test that uses files (text or XML)
d) I did not have to write same initialization or tear down code (objects creation or termination, initialization or etc...) to number of tests, because Test Initialize and Test Tear Down is called automatically with each test for test project (file).
Automated – helped me to find bugs in my code and produce better quality code in the short time
a) I wrote function, test passes, i wrote more code to function, wrote one more test. The new test passes, but it appeared that I broke the last test with new logic. I knew it in a minute - when I ran 2 tests automatically - figured out that have “broken” one of tests by adding code.

First test was RED !!!

So I had to go back and fix code. I realized now that I can come quicker to a higher quality code.

2) Visual view at all tests helps to understand if there are more situations and scenarios to be tested
On the second day when beginning to work with my code, I looked at the tests and could “see the implemented logic”. Each test name was a question. When you see the tests - you can easily understand if you missed some tests:
Later I figured out that my work became to look like a game. I the next part I will
write about this cool experience...



Saturday, May 17, 2008

Swapping calls between classes without dependency injection

Eli Lopian, Typemock CEO and awesome Coder, just created a nice little API wrapper around Typemock Isolator that would allow a very simple and readable "Swapping" effect between classes. It would allow you to write code like this:

    [Test]

       public void SwapStatic_CallsFake()

       {

           Swap.StaticCalls<TypeA, TypeB>();

           Assert.AreEqual(2, TypeA.StaticReturnOne());

           Swap.Rollback();

       }

 

       [TestMethod]

       public void SwapInstance_CallsFake()

       {

           Swap.NextNew<OriginalClass, FakeClass>();

           OriginalClass swappedInstance = new OriginalClass();

           Assert.AreEqual(2, swappedInstance.ReturnOne());

       }

 

The first test uses the "Swap" class to replace static calls and redirect them to your own class with static methods. The second one is more advanced and will mock all instance calls made on the next instance that will be created of that type. so the last line on the second test will actually call a method against the FakeClass type.

pretty cool and very readable. download the code and binaries from his post.

Why ValidateArgsOnVerify Became Obsolete

There's one breaking change in Isolator v4.2.4. A small one really. The ValidateArgsOnVerify flag is now obsolete, and Isolator will throw an exception both when calling the method and on the Verify method.

Well, for those who know about this flag this sentence may seem reasonable, but for the rest of us, let's run through what was, what will be and why.

The context is checking arguments. Let's look at this example:

[TestMethod]
public void CheckArguments_SendAnotherName_ExpectException()
{
using (RecordExpectations recorder = new RecordExpectations())
{
ProductFactory.HasProduct("Name");
recorder.Return(true).CheckArguments();
}
ProductFactory.HasProduct("AnotherName");
MockManager.Verify();
}

In the recording block, apart from setting the expectation and return value, we make sure that the parameter to the HasProduct is "Name". As you can see from the code, we send "AnotherName", which is a mismatch, and therefore throws a VerifyException.

The exception is thrown when the HasProduct method is called. So we don't even get to the MockManager.Verify line. This was a limitation some of our customers could not live with, and they asked us to allow to delay the exception throwing until Verify was called. We graciously accepted this, and so the ValidateArgsOnVerify was born.

Using this flag is an either/or only: the default (which is false) throws on the method itself, and when set to true, we'll throw on Verify. Usually that's enough. Now let's change our example a bit:

[TestMethod]
public void CheckArguments_SendAnotherName_ExpectException()
{
using (RecordExpectations recorder = new RecordExpectations())
{
ProductFactory.HasProduct("Name");
recorder.Return(true).CheckArguments();
}
ProductFactory.ReadProductFromFile();
MockManager.Verify();
}

The expectations are the same, but we're calling the ReadProductFromFile method, which looks like this:

public void ReadProductFromFile(string productName)
{
try
{
string value = ReadFromFile();
ProductFactory.HasProduct(value);
}
catch
{
Console.WriteLine("Logging The Error");
}
}


Wrapping I/O operations with try/catch blocks is common practice. So what will happen when no I/O problem occurs, but a VerifyException is thrown because of argument mismatch? It gets swallowed. And since we're throwing the verification only ONCE, we now have the case of the missing exception. And as we know, there's no police around when you need them.

Well, not anymore. From now on, we'll be throwing exceptions right and left. Actually, both on the method call AND the Verify method. There will be no escape. And that's why the ValidateArgsOnVerify is no longer necessary.

Another case where the truth comes out eventually. Because, as we all know, the truth is out there.

Thursday, May 15, 2008

Twittering

Some of us are on Twitter, but I'll let others expose themselves. (They're very good at that, too).

Here's me. You can twit me "Hello" and "How about that post?" and "Where's this park where you took this picture?" (St. Patrick's Green, Dublin).

See you at the twitter-sphere.

Typemock Isolator v4.2.4 Released

Yes, another one! Can you believe it?

So this version is a compilation of bug fixes we've made in the last couple of months. The full list appears below, but I'd like to mention a couple here.

The first is the infamous "Invalid Operation Exception". This was a tricky one. It took a while to reproduce in our dark basement. Actually, most of the time was spent on reproducing the problem. After it surfaced, the road to the solution was short. (But isn't that the case with all these problems?)

The second one is the LINQ problem. Remember the chain of posts awhile ago? That's the origin. And the courage/fear post? Also. Long story.

Internally, we called it the "Chain inside Chain". The reason is that LINQ queries are parsed into long method calls with expressions and expression tree. And this goes into our chain interpretation mechanism. So now, complex LINQ queries are mocked correctly.

Here's the complete bug fix list:

  • “Invalid Operation Exception” when working with NHybernate and XML serialization was fixed.
  • Complex LINQ queries are now mocked correctly.
  • Manual assembly loading does not cause an exception. AssemblyResolve event fires correctly.
  • RepeatAlways now works correctly following WhenArgumentsMatch.
  • Returning values of types that inherit from generic type now work correctly.
  • GetMocks return the correct mocks for abstract classes and interfaces.
  • Following undeploy Typemock Isolator remains registered.
  • Methods up to 30 parameters can now be mocked.

Before you run off to get your copy, I'd like to take a minute to thank our customers. First, for reporting their problem. Then helping with the investigations. And lastly, having the patience to wait until we solve them.

So go get it now. All the cool kids are playing with it.

Wednesday, May 14, 2008

Automated Unit tests in my life - beginning

Introduction

This is an article of a software developer, who decided to share some knowledge and experience in automated unit tests. I think this one can help in everyday work of people like us.
This is not a pure technical article. It is a real story and comes to help producing better quality code. I will try to give some nice tips and show black and white sides of my automated unit tests experience.
About the name of an article ... You might ask: Hey, man – isn’t it too much? You could name it “Unit Tests in my code, work, or etc – but “in my life”? You will find the answer on that question by reading this material, which consists of several parts. And here they are:
· How did it begin – how, when and why did I begin using automated unit tests
· My first experience – Good, bad, why do I need it
· The New style of work. How could Programming be like playing a game.
· Programming without automated unit tests and Test Driven Development - differences
· Self confidence in your work
· Automated unit test weakness and how to prevent them
· How can we mock without affecting the code

Part 1. How did it begin?
Before writing about automated unit tests technical details, I will write you about my experience with the automated unit tests… A little bit about my programmer history:
After graduating University in 1997 (Applied Mathematics), life took me to work as algorithms and software developer in some complicated big project with lots of mathematical calculations and logical stuff, where I earned lots of programmer skills. The project was one those, where all the developers had to make lots of tests (the situation where quality was more important than time). All the team tested all code by writing unit tests and integration tests. This is why the quality of my code was good. At the time I did not know anything about automated unit tests. All the tests were written manually…

So when the automatic unit tests came into my life? All began with my work in some software medical company. After about a year of my work in the company my boss suggested me to “try” write unit tests automatically, using “Nunit” and more then that – write tests before code (test driven development).
I was shocked and disappointed… First of my thoughts were:
”Why me? Is my code not good enough? I have a good testing experience with my working and testing style. Why should I change the way of working? More than that - why should I write the tests before code? And changes now - when I have so much work?” It felt like some entering to my privacy – way of programming, way of life, thinks I was used to… But as you can understand- I agreed, because am writing this one. After few hours of practicing I still did not understand why automated unit test and test driven it so good? In the next part you will see why first experience began to change my opinion…

Funny first experience: The naughty CD ROM build breaker

I wanted to share a funny story that happened to us in the office today: I'm a new developer for the Typemock team, and after running all unit tests locally, going through peer code review etc. I happily committed my first changes to our source repository.

Once our continuous integration system caught up with the changes it went on to start the build process, and then everything broke: the build process got stuck for over an hour and the tests would not run.

Naturally I got that nasty feeling at the bottom of my stomach: it's my 4th day at the company and I killed the build server. After briefly imaging my conversation with Lior on the following morning I grabbed Menahem to help and we started investigating: as the build process worked perfectly on my machine, we performed a rollback on the build server a tried to build again. No luck. Aside for the immediate relieve I felt (as it was not my code breaking the build) we were still left with a broken build.

After investigating further, we found the culprit: a C++ project in our codebase was taking x10 time to build than usual. Other C++ projects compiled just fine, as well the .NET projects. We (the entire dev team now) speculated around different reasons for the C++ compiler to misbehave, but found out nothing. Finally, we got around to manually going over the project configuration and comparing it to other projects and when we nailed it: it turns out the cpp project configuration had additional include directories pointing to "d:\software\something". This is probably left over from way back - all the company's computers now have a single c:\ drive and d:\ is a CD ROM. This means that each time the precompiler encountered a #include directive (a couple of dosen times) it went to the CD ROM drive to find the files! this action took 1-2 minutes for each #include, bogging down the build process and ultimately causing it to fail. After removing these include directories the build process came back to normal.

The only mistery we had left is what happened? these unnecessary include libs were in the configuration for a while, and only today it broke the build... It turns out that someone installed something on the build server yesterday and left a CD in the drive. In the past the precompiler looked for the include libs, found out they were on an unavailable drive and gracefully ignored them. Because we had a CD in the drive this time, it caused the CD to spin and seek the files that were not there, taking a minute before the build process recovered each time...

Friday, May 9, 2008

NUnit (but not only) Best Practices

Scott White blogged on NUnit best practices. He has some interesting points that apply not only to NUnit, but unit testing in general. I think that the most important point, however, is not in his list, but rather in his first sentence:

"Programming is not just an art, not just a science, but a discipline."

How true.

Thursday, May 8, 2008

Mocking Performance

Aaron Jensen compared the performance of different mocking frameworks. Interesting numbers. And I'd also like to see numbers on MSTest, both VS2005 and VS2008.

I'd like to comment on the performance diffs between TD.Net and NUnit console, as well as other test runners. Different runners have different ways to run tests (I know, hard to believe). Some run them in different threads, and some in AppDomains, that may account for some differences. Because of these differences we also may see other phenomena, like discovering or hiding dependencies between tests.

If anyone has more numbers, I'm very interested in them. Last year we solved a performance hit bug, and like to keep getting better, if possible.

Tuesday, May 6, 2008

Ivonna lets you write ASP.NET Unit Tests

Looking for a way to write unit and integration style tests against ASP.NET Pages and WebForms? Ivonna is a new project that aims to give an alternative to things you'd usually do in stuff like Watin or Watir or NUnitASP, using the help of Typemock underneath to isolate the WebForms runtime from the real HTTP context:

"Ivonna is an Asp.Net testing tool that allows writing unit tests for your WebForms applications. What makes it different is that Ivonna runs in the same process as the test runner.
Unlike other testing tools, Ivonna is not a "client-side" tool. It doesn't just test the HTML output (although such a possibility exists), it creates and runs an Asp.Net request and allows you to examine the intrinsic objects, such as the Page object. This opens many new possibilities, such as

  • Examining properties of page controls and other objects, such as the HttpContext object.
  • Mocking external dependencies in order to isolate the page behavior.
  • Injecting setup code and assertions into the page's lifecycle event handlers.

Ivonna is being developed in partnership with TypeMock and runs as an add-on to the TypeMock Isolator framework.

Sunday, May 4, 2008

Free Typemock Isolator license for Microsoft MVPs

If you're a Microsoft MVP, you're entitled to your own personal Typemock Isolator , Enterprise Edition License for 1 year. Just send an email to MVP at Typemock.com and we'll set you up:

  • Send in a link to your current MVP profile (you have to be in an existing MVP status)
  • Once you've used it for a while, we ask that you write one thing you *didn't* like about it on your site\blog if you have one so that we can make the product better.