Of Lies, Damned Lies, and Benchmarks

It’s no secret that we like statistics. Human beings like being able to compare two things, apples to apples. We like the ability to say “this thing is better because…” and being able to back it up with a seemingly solid fact, rather than just our opinion. We build charts to show how things stack up side-by-side, and we even build software to make those charts, which are then compared using the charts, and on we go.

But benchmarks, for all their decision-making aid, fail under the best of circumstances for one simple reason: they’re not real life.

Never more is this true than in Joe Stagner’s blog post on whether Windows or Linux, and PHP or ASP was faster. It should really come as no surprise that a Microsoft employee would write an article that would conclude that Windows was faster, and that ASP was faster (I’m pretty sure if he’d found the other way and published it that Ballmer would have canned him).

Benchmarks come loaded with all sorts of problems. It doesn’t matter if it’s Microsoft doing them or Apple doing them; they don’t mimic real-world conditions, and any number of factors affect how the benchmarks are rendered (e.g. other processes running, configuration, installed modules, CLI or web, how the interpreter reads the code, etc).

Some of the problems I found with his benchmarks: there’s no discussion of configuration. Did he use the defaults? What extensions did he install alongside the core installation? The code he used isn’t published. Update: It appears that he did include his code, though I don’t know when the links were added. They were not in the original draft of the article. How was the server configured? How does the engine render each loop? ASP.NET is closed-source; that’s to say, there’s no way for us to know if the interpreter sees an empty loop or function and decides to bail rather than actually execute it. What version of PHP are you using? Why isn’t there a Ubuntu test for PHP 5.3 (I actually know the answer to that question)?

I don’t think that Mr. Stagner is trying to insult anyone or assert that ASP is 2900% faster than PHP at every task. I do think that the benchmarks provided show that ASP is faster at the given tasks and that’s it.

So, if you need to iterate over 20 million empty for loops, ASP.NET is your language.

Update: After a conversation with Cal Evans and a read about ASP.NET it appears that ASP.NET comes with a built-in byte code cache that cannot be diabled. However, Mr. Stagner conducted his tests WITHOUT APC installed on PHP. While APC is not part of the PHP core (and bytecode caching appears to be part of the ASP.NET core), PHP without APC is a poor configuration choice.

Brandon Savage is the author of Mastering Object Oriented PHP and Practical Design Patterns in PHP

Duane Gran (@duanegran) wrote at 8/11/2009 8:46 am:

These types of benchmarks are dissapointing at best and most often misleading. In the real world things get complicated. As a PHP advocate (and critic when warranted) I can point to high profile sites like facebook and wikipedia as evidence that PHP scales. My guess is that their configuration and architecture evolved to support their current load, but it is encouraging to see how many of the top traffic grossing sites out there are powered by PHP.

Whether you use PHP, ASP, RoR or Pascal to build your site (well, maybe not Pascal) it won’t be slow because the language is at fault. That is a cheap shot. If it is slow it is because the developer/architect didn’t anticipate the next stage to scale up.

If we could benchmark developers that would be interesting… and useful.

Andrey wrote at 8/11/2009 9:00 am:

I have seen the MySQL testing code, which shows that ext/mysql is slower on Windows, 5.3. I think all the example code, for PHP, is shown.

Brandon Savage (@brandonsavage) wrote at 8/11/2009 9:02 am:

The example code that Joe used for his benchmark is published? If it is I missed it; you have a link?

Herman Radtke (@hermanradtke) wrote at 8/11/2009 10:07 am:

He updated the post with links. They are at the bottom of the post, right before the comments.

Sam Shull wrote at 8/11/2009 10:54 am:

I hope that someone uses his own code to contradict his findings, since he didn’t provide enough details to exactly replicate his experiment. :)

Steve Clay (@mrclay_org) wrote at 8/11/2009 10:55 am:

Lesson: PHP without a byte-code cache on a desktop OS is not ideal. Good thing no one does this.

Brandon Savage (@brandonsavage) wrote at 8/11/2009 10:57 am:

Sadly people do this, Steve. It’s not recommended, though.

Samuel Folkes (@SamuelFolkes) wrote at 8/11/2009 12:31 pm:

Brandon, I’m very glad you wrote this article. That ASP/PHP comparison stinks from the bottom up for a number of reasons in cluding but not limited to the fact that these tests were carried out in an environment that did not mimic a real world production evironment closely enough. Of course, Hardly anyone from Microsoft has given PHP the time of day before now. I wholeheartedly believe that Mr. Stagner’s blog post is an indication that like linux, PHP is now a huge concern for Microsoft as formidable competitor. It is no longer prudent for them to ignore the strides that PHP as a viable and in my opinion preferable alternative to ASP has taken. I think we should expect more of these tainted assertions to surface in the near future. I say we use Facebook and Wikipedia as our benchmarks. Nothing makes a statement like some good old action.

P.S. Nice work revealing that ASP.NET has a built in bytecode cache. That’s something I’m almost sure Mr. Stagner was aware of but failed to mention in his article.

Brandon Savage (@brandonsavage) wrote at 8/11/2009 12:40 pm:

Samuel, I think that Joe’s intent was good and that while he may have known that ASP.NET has a built-in bytecode cache, I doubt he was trying to pull a once-over on anyone.

Microsoft has invested considerably in the last few years to get PHP to run better on the Windows platform, and I applaud their effort. Even if developers don’t use ASP.NET to build applications, getting them to use Windows still means a license. And I know that Microsoft is interested in remaining relevant, which they have to support PHP well in order to do.

I work in a shop that uses a number of different languages, and I’ve learned that languages are tools that are used for a specific purpose. You don’t use a hammer to dig a ditch, and likewise you don’t use PHP to build desktop applications, for example.

And though I make a dig at ASP.NET at the end of my article, I’m willing to admit that I’d use it if the situation called for it. To interface with Microsoft database tools, I don’t think anything beats it. I want to make sure the debate remains civil; if our product is really better we can win the debate by proving it.

Samuel Folkes (@SamuelFolkes) wrote at 8/11/2009 1:05 pm:

Brandon, it is not his intent that I doubt. It the pertinence of the tests and the environment in which they were conducted. PHP on Windows means a license but ASP.NET on Windows means 2 (or 3 if you count the inevitable inclusion of SQL Server). I will with no hesitation declare that I am a PHP advocate. I use PHP loving its strengths and accepting its weaknesses and limitations. As you stated in your article, as humans we love comparisons. We love numbers and statistics. However if one can truly accept the limitations of their chosen development tool then the numbers and the statistics should become almost irrelevant. I must admit, I am very interested in seeing the numbers (produced from fair, balanced tests of course), but that interest is purely academic. If we all adhere to the tried tested and proven concept of using the right tool for the job, in the end, the outcome of this debate will (should) have very little consequence.

Joe Stagner (@MisfitGeek) wrote at 8/11/2009 2:50 pm:

Samuel,

Don’t worry. Brandon is a tool and if I said eating dog shit was bad for you – he’d argue with me about it ! :)

-Joe

Joe Stagner (@MisfitGeek) wrote at 8/11/2009 3:00 pm:

Just kidding Brandon :)

PLEASE email me your suggestions as this was only the first of many perf tests I want to do and I – I have been a PHP developer longer than I’ve been a .NET developer and I’m not interested “engineered” results.

If PHP on Windows is not ACTUALLY faster, and if ASP.NET is not ACTUALLY faster, I want to make it that way.

-Joe

Joe Stagner (@MisfitGeek) wrote at 8/11/2009 3:02 pm:

And btw – the code was published at the same time as the orrigional article.

Brandon Savage (@brandonsavage) wrote at 8/11/2009 3:51 pm:

Now I know. ;-) Thanks for clearing that up.

Joe Stagner (@MisfitGeek) wrote at 8/11/2009 3:59 pm:

Brandon – as I’m researching op code cahcing for my next set of tests. Is it really true that theONLY way to get APC for Windows / PHP 5.2 is to buildit from source ?

tsb (@bergheim) wrote at 8/12/2009 7:55 am:

@Duane
I’m sorry, how are ORDERS OF MAGNITUDE not important, just because “evidently it will still work, as shown by Facebook and Wikipedia”?

The article isn’t about scalability, it is about raw performance. And php is s-l-o-w, that is no secret (however, I have no idea about how acurate these benchmarks are). Really, C approaches 3 orders of magnitude that of PHP for many things.

This “throw more servers at it”-mentality really is not a good engineering-standpoint at all. And caching the hell out of your website because the language is horribly slow does not count in this context.

@Joe: Use APC or xcache (http://xcache.lighttpd.net/) or something similar, also throw in Mono on Linux :)

Duane Gran (@duanegran) wrote at 8/12/2009 8:52 am:

@tsb
Sorry if I implied that raw speed wasn’t important. My main point was that most people who experience speed issues look for failure in the application stack instead of in the application code. You do however bring up a good point that scalability and performance are different matters. My bad.

Thomas Bergheim (@bergheim) wrote at 8/12/2009 9:27 am:

@Duane
Ah yes. Sometimes the stack can be partly blamed though :) For instance, there are a good number of the gazillion web-frameworks that bring in so much “bling” that it really takes up a lot in terms of bootstrapping (of course it was your own fault for using it in the first place!) (case in point: Zend Framework handled 29 res/sec, opcached of course, on my computer, while django doing the same handled almost 200! – they provide different things of course so this is a framework comparison).

However usually most of the time spent in webapps these days ends up in SQL-statements so I agree that it is not as important as these “synthetic” benchmarks seem to imply.

Still, I like knowing in which ballpark various languages lie when doing various primitive operations, that might just be me though..

artur ejsmont wrote at 8/12/2009 10:05 am:

LOL

1. Never believe benchmarks with agenda.

2. if your benchmark is 100 times faster on different machine you probably screwed up the setup ;- ) or …. you get errors that return instantly and you are not checking for that ;-)

3. shame he did all this stupid loop-the-loop instead of something more interesing like how much does it take to search the xml or parse it or do some more complex things. Same thing with massive inclusions etc.

PHP is cool and all benchmarks lie.

Anyone knows some extensive benchmark suite for PHP let me know :- )
I really dream of a nice benchmarking suite for PHP not only to compare platforms but for internal validation and configuration tweaks.

Art

Dave wrote at 8/12/2009 9:51 pm:

Brandon, thanks for responding to Stagner’s “drive-by comparison” in a productive manner. As Cal Evans wrote the other day —

“The PHP community has matured to the point where we don’t feel the need to be Zealots. We can respect other languages as good at what they do, even if the developers in other communities don’t offer the same respect back.”

Dave

Copyright © 2023 by Brandon Savage. All rights reserved.