Can Num3rs Lie?

In a series of posts Oren (Ayende) and Sasha have debated the need to design for performance. Both parties make some very compelling arguments and its real interesting reading material. Since I still need to sort my thoughts on the matter (which will be presented in a future post) here’s just a teaser.

Oren has rose to the challenge, and in a series of experiments have tested the performance difference between the two design approaches. Here’s one of his experiment result

And the results are:

  • IList<T> – 5789 milliseconds
  • List<string>, NullList – 3941 millseconds

Note that this is significantly higher than the previous results, because no we run the results two hundred million times.

Individual method call here costs:

  • IList<T> – 0.0000289489
  • List<string>, NullList – 0.0000197096

We now have a major difference between the two calls. When we had a single impl of the interface, the difference between the two was: 0.00000406 milliseconds.

With multiple implementations, the difference between the two methods is: 0.0000092393 milliseconds. Much higher than before, and still less than a microsecond.

which Oren concludes that although there is a performance difference, the actual time difference even in this close to “worse” case is so small so there is not much point to consider it.

is there?

lets try a different interpretation:

Reading the numbers I can also say that using the List<String> is 31% (0.0000092393/0.0000289489) faster then using the IList<String>.

Question – How much time and money do you think that Intel&AMD will invest in the next couple of years to achieve a CPU which runs 31% faster then current technology? (actually I don’t know either, but I sure wouldn’t mind putting my hands on a small part of it).

Point is, that in cases where performance is an issue (which is not true for all cases), I would most definitely go down the design road which offer a 31% speedup.

  • Ayende Rahien

    You would have a point if the 31% was in a hot spot.
    In 99.9999999% of the cases, it is not a bottle neck, ignore it.
    Premature optimizatio, etc

  • Thomas Eyde

    I agree with Ayende. I also think it’s easy to forget our context while we think about how to optimize. 31% sounds impressing, but if there were no performance issues to start with, or if the improvement is neglible, then I argue the effort is wasted.

    And don’t forget we need developer performance, as well. Optimized code has a certain smell to it, it tends to be hard to understand. That hurts our performance.

  • Thomas Eyde

    I agree with Ayende. I also think it’s easy to forget our context while we think about how to optimize. 31% sounds impressing, but if there were no performance issues to start with, or if the improvement is neglible, then I argue the effort is wasted.

    And don’t forget we need developer performance, as well. Optimized code has a certain smell to it, it tends to be hard to understand. That hurts our performance.

  • Eli Lopian

    Thomas,
    I agree that premature optimization is the wrong way to go, and that while writing code – optimization can lead to wrong solution.

    In the context of this discussion, it is not really optimization that we are talking about, but the opposite – it is degrading the system on purpose. Actually a developer has to spend more of his precious time to make the code unoptimized.

    So the ‘developer performance’ is actually degraded by the ‘unoptimized’ code.

    Having direct calls leads to both faster code and easier to understand code, Using Interfaces and Virtual Methods leads to both slower code and more abstractions making it HARDER to understand.

    This is what Sasha is trying to convey, why are you making the code slower and harder to understand?

    So you can argue that it is not that slow (although 31% is quite a lot – It means that the code takes 45% longer to run) but you loose developer performance too
    Why? What for? Just to be cool and say that you use IoC.

    Or is this to support tools that don’t have the ability to test the code in any other way then to degrade the performance of your application

  • Lior Friedman

    Aynede:
    you are right. However if you are going to do the same all over the code, eventually you will hit Hot spots.
    and the questions is: what would be the better approach, write and improve/fix or prevent the problem in the first place?

  • Lior Friedman

    Thomas: the entire discussion is made under the assumption that performace is an issue. (it is also stated in the last paragraph of my post)

    And as Eli said, do you think that injecting interfaces all over the code does help us to understand the code?

TOP