Monday, April 28, 2008

Fear, Courage and Tests

During our end-of-iteration meeting today, we discussed a task I'm working on, say, for the 3rd or 4th iteration now. Originally I paired on this task, then worked alone, then paired again with another person. This was not done continuously, obviously. (By the way, I've managed to progress much better with a pair). And the task is complex. Luckily we have lots of tests to see if things get broken when we try to insert a new component in. And the last couple of times I went back there was to make sure all of the tests pass.

So how come we're not finished yet?

Could it be fear?

Lior suggested that as an option today. Fear of fiddling with the code too much. "But what are your tests for?" you may say and you would be correct. Tests supply a safety net, and a confidence builder. Which is great, because without this feedback, we wouldn't know where we stand.

But let's look at it another way: most of time spent on this task is fixing a couple of tests, watch them pass, and then watch others fail. It is very exhausting mentally as time goes by. We see progress, but convergence towards stability is slow. And I admit to losing confidence in changing code, just because it may break tests that are now passing.

Tests should give us courage. We need to remember that they are not there to cripple our progress, but help us move forward.

And Lior's courage is going to be tested next, since he's my next (and, hopefully, last ) pair for the task.

Tuesday, April 22, 2008

Can Num3rs Lie?

In a series of posts Oren (Ayende) and Sasha have debated the need to design for performance. Both parties make some very compelling arguments and its real interesting reading material. Since I still need to sort my thoughts on the matter (which will be presented in a future post) here's just a teaser.

Oren has rose to the challenge, and in a series of experiments have tested the performance difference between the two design approaches. Here's one of his experiment result

And the results are:

  • IList<T> - 5789 milliseconds
  • List<string>, NullList - 3941 millseconds

Note that this is significantly higher than the previous results, because no we run the results two hundred million times.

Individual method call here costs:

  • IList<T> - 0.0000289489
  • List<string>, NullList - 0.0000197096

We now have a major difference between the two calls. When we had a single impl of the interface, the difference between the two was: 0.00000406 milliseconds.

With multiple implementations, the difference between the two methods is: 0.0000092393 milliseconds. Much higher than before, and still less than a microsecond.

which Oren concludes that although there is a performance difference, the actual time difference even in this close to "worse" case is so small so there is not much point to consider it.

is there?

lets try a different interpretation:

Reading the numbers I can also say that using the List<String> is 31% (0.0000092393/0.0000289489) faster then using the IList<String>.

Question - How much time and money do you think that Intel&AMD will invest in the next couple of years to achieve a CPU which runs 31% faster then current technology? (actually I don't know either, but I sure wouldn't mind putting my hands on a small part of it).

Point is, that in cases where performance is an issue (which is not true for all cases), I would most definitely go down the design road which offer a 31% speedup.

Monday, April 21, 2008

3rd Typemock Basic Video - Mocking Database Access

We've just published our 3rd video on our site. It covers a topic we've been asked many times - "we have a data access layer which we want to mock. What do we do?"

Well, grab a cup and watch the movie.

And do let us know what you think about these movies. What would you like to see more, what less, and who's the best narrator.

Wednesday, April 16, 2008

Can I Mock Web Services?

With Typemock Isolator, sure you can!

This came up in the forums. So here's the explanation again, but with better graphics.

A web service is a great example for mocking dependencies. You have code that accesses the cloud. What if you're working offline? Or, someone else is working on the service, and has not published it yet, while you're stuck with the client code and no way to test it? Mocking to the rescue.

imageWhen you consume a web service, a class is generated for the service. The trick to find this proxy code is (thanks Roy!) to click "Show all files" when standing on the web reference. And you get something like this:



See the Reference.cs? That's where the proxy code is created.
So let's say we have the original wizard generated service. Here's the code:

public class Service1 : System.Web.Services.WebService
{
[WebMethod]
public string HelloWorld()
{
return "Hello World";
}
}
For this web method, the generated proxy looks like this:
[Bunch of attributes removed]
public string HelloWorld()
{
object[] results = this.Invoke("HelloWorld", new object[0]);
return ((string)(results[0]));
}

Now that we have type and method we can reference, we can mock it. Here's an example using reflective mocks:

[TestMethod]
public void ReflectiveMocksTestForWebService()
{
Mock mockService = MockManager.Mock<Service1>();
mockService.ExpectAndReturn("HelloWorld", "SomethingElse");
Service1 myService = new Service1();
Assert.AreEqual("SomethingElse", myService.HelloWorld());
}
And here's one using Natural mocks:
[TestMethod]
public void NaturalMocksTestForWebService()
{
using (RecordExpectations rec = RecorderManager.StartRecording())
{
Service1 mockService = new Service1();
rec.ExpectAndReturn(mockService.HelloWorld(), "SomethingElse");
}
Service1 myService = new Service1();
Assert.AreEqual("SomethingElse", myService.HelloWorld());
}
There you go: simple yet powerful.

A Great Quick-Start Guide for Isolator

Our friend Travis Illig just posted a primer on Typemock Isolator.
Great and thorough read.

Thanks Travis!

Sunday, April 13, 2008

The perils of Obfuscating your code

Suppose you have an enum type.

    internal enum SomeEnum       
{
SomeVeryDescriptiveName = 5
}

For some reason you want to use the name of the enum field as a string and not its value.
you can write:

Console
.WriteLine(SomeEnum.SomeVeryDescriptiveName);

The output will be:
"SomeVeryDescriptiveName"
This is exactly what you wanted and every thing works fine.

But when you build the release version it doesn't work.
So you debug it and of course under debug it works fine again.
Then you pull the heavy guns like the giants who walked on this earth long time ago taught you. Logging.
And than you find it:
When we build in release mode the assembly is obfuscated. This means that all internal and private names are mangled.
Since the enum is internal SomeEnum.SomeVeryDescriptiveName will become something like:
z.k and the output will be:
"k"
Thou shall not use strings!

Saturday, April 12, 2008

Tracer Inside

A taste of things to come: The Tracer window running inside Visual Studio.


(Click image to enlarge).

Thursday, April 10, 2008

Argument Checking

One common use of a mocking framework is to check interaction between classes, Specifically checking correct arguments passing between calls.

Like any other framework Typemock Isolator also contain such an ability:

using (RecordExpectations recorder = RecorderManager.StartRecording())
{
MyClass mock = new MyClass();
mock.DoStuff(5);
recorder.CheckArguments();
}

However when doing so its important to note that the verification is done during the actual call time. When later on in the test when someone calls the DoStuff method and passes arguments they are checked immediately and the Isolator will fail the test by throwing an exception on the call.

this approach which is used by most framework can be confusing when happening inside a try-catch block.

for example the following code will not fail as expected:

try
{
MyClass target = new MyClass();
target.DoStuff(4);
}
catch
{
//some error handling code
}

The exception thrown by the Isolator is caught, and in most cases not re-thrown and the test does not see it.

The trick to solve such an issue is to defer the argument checking to the final verification stage (the call to MockManager.Verify) by doing:


MockManager.ValidateArgsOnVerify = true;

That being said we are working on removing the need for the user to specify this. We hope to incorporate this in our next release.

Tuesday, April 1, 2008

Defaults

One of the more paining decisions we are faced during development of a product is deciding on default behaviors. Whether it is specific values we need to consider or more generic logic inside our product. again and again I find that choosing defaults are always harder and more time consuming then we expected.

At least from my experience the reason for this is quite simple. Choosing a default is the type of decision that we always try (and told) to avoid:

1. You will always be wrong.

By default ;) there will always be users that will need the exact opposite . And if that's not enough this will probably be the only feedback that will be arriving (after all only these encounter issues with the defaults)

2. Its almost never based on real knowledge.

At the time of choice we don't really know what the users need. Gathering this kind information is very time consuming and costly, to the point that I've yet to see it happen. At the end we make this decision based on gut feeling or educated guesses at best.

3. Its a hard to change decision.

Changing the defaults will always affect part of the user base. More likely since the first decision was not made using a coin, it will affect a substantial part of them. At the end this means that changing the default will always be substantial.

In short we end with a decision that is always based on partial knowledge (at best) we will always make it wrong (for some users) and it will cost a lot to change it.

i.e. a decision to avoid at all cost (or at least postpone to the far future).

But the real question is how to approach these?

I'm not really sure. What I try to do is:

1) First decide. A decision must be made, and I find it easier to make it after I remind myself that it cant be avoided.

2) Stick with it -  as much as possible, to the point that it become an absurd not to change it. this is also easier done if I constantly remind myself that for every request to change ti there are ten more that will come once it is changed.

3) Be consistent - and this I think is the key point here.  Defaults are constantly challenged. if the default behavior is consistent it will be much easier to explain and defend. if however it is not consistent it will be so much harder to do so and for some reason that will be the exact path the users will go with.

and ill end up with a question:

What practical approaches do you use to decide on the default?

Mocking Fields - Why it was implemented

One of the new features in our new release 4.2 is the ability to mock fields. Since we have published this new feature we were asked several times why this was added.

Actually, there are several answers to this, but in this post I would like to focus on the least obvious one.
Part of our process here in Typemock (which we try to make as agile as possible) is periodically going over all our users feedback and looking for common cases that we can and solve by adding new functionality. When doing so for the 4.20 version we saw a relative large number of cases in which users tests failed when accessing fields on objects which were mocked.
here's an example (simplified of course):

public class UsedClass
{
public int ReturnFive()
{
return 5;
}
}
public class UnderTest
{
private UsedClass _usedInstance;

public UnderTest()
{
_usedInstance = new UsedClass();
}

public void Initialize()
{
//do some complex stuff we wish to mock in our tests
}

public int Calculate(int num)
{
Initialize();
return _usedInstance.ReturnFive() * num;
}
}

[TestMethod]
[VerifyMocks]
public void FailingTest()
{
using (RecordExpectations rec = new RecordExpectations())
{
UnderTest mock = new UnderTest();
mock.Initialize();
}
UnderTest target = new UnderTest();
int actual = target.Calculate(6);
Assert.AreEqual(30, actual);
}


Running this example with previous versions of the Isolator resulted in a NullReferanceError being thrown from target.Calculate(6). It is thrown because by default the Isolator mocks calls for constructors on mocked objects. Therefore the actual initialization of the _usedInstance is skipped during the test, leaving it as null when calling the Calculate method.

Surprisingly, this kind of defect is quite hard to locate especially if the call to the fields is hidden inside several layers of code.

So how does this relates to Field Mocking?

When using the Natural Mocks syntax, we automatically attach mock controllers to all instance fields of mocked instances (and for all static fields on mocked classes) created inside the recording block. This gives us the ability to detect that calls are issues on fields which has no real value and in these cases we throw the following exception:

TypeMock.VerifyException: TypeMock Verification: The call to UsedClass.ReturnFive() was made on an uninitialized field.

Hopefully this exception will be much more helpful and will save frustrating debugging time when actually encountering this.

However, this has its price. Values of fields of mocked instances are no longer equal to null, which does affect application logic if somewhere inside the code fields are compared to null, (during lazy loading for example).

However, we do feel that this price is minor and most people will not encounter this ill affect but just in case we added the ability to change this default behavior and instruct the recorder to ignore fields this can be done by setting the RecordExpectations.DefaultBehavior.AutomaticFieldMocking to false.

The other main reason for adding this feature is that it is useful.

There are cases in which people wants to mock a fields without the need to implement some sort of bypassing mechanisms (singleton mocking is a good example for it).