Of Lies, Damned Lies, And Benchmarks – Redux!

Upon request of the PHP community, Joe Stagner conducted additional tests of PHP, using both PHP 5.3 and APC. In his article he also revisits the questions raised by myself and Andi Gutmans about the fairness of his tests.

The new tests reveal that APC doesn’t do much to improve the performance of his applications. This wasn’t a significant surprise to me; after reviewing his code I realized that the script wouldn’t be called again and thus APC wouldn’t provide a benefit. And though some of his tests (MySQL, file system tests, and Postgres) are dependent on external factors (like how well MySQL performs on Linux versus Windows), I think they paint a fair picture. Joe’s tests were fair.

The only thing left to evaluate is the engines that power the core of PHP versus ASP.NET.

One thing that no one brought up is that in his tests, the “for” loop performed more poorly than the “while” and, in some cases, “do-while”. Wait, what? Check out his test results and you’ll see that in each and every test, “for” is outperformed by “while”, and in many cases, even outperformed by “do-while”.

If you ask Sara Golemon, she’ll tell you that shouldn’t be. But it is. PHP’s nature as an open-source project means that contributions can be made by lots of different people, and sometimes you’re going to have performance suffer for it. But almost always you get a better product.

For those who are upset or incensed by Joe’s test results, I offer you this challenge: make PHP better. PHP is only as good as the people who contribute to it. Those who contribute to the core are hard-working volunteers. No one gets paid to write core PHP components or extensions. It’s all volunteerism. Those who work on it now are class acts who should be rewarded. And they should be aided, too.

We can’t examine the ASP.NET core to determine why it’s faster (and I wouldn’t be surprised if there were certain shortcuts in the way Windows and IIS handles requests), but we can examine the core of PHP and find ways to boost performance. And that’s the one thing about PHP that makes it different than ASP.NET: if you find a bug, you can patch it yourself. You don’t have to wait for the next release, and you don’t have to get some Microsoft engineer to care about it. You, yourself, can fix it.

The difference in PHP is you. Go make it better.

Brandon Savage is the author of Mastering Object Oriented PHP and Practical Design Patterns in PHP

Gaetano (@gggeek) wrote at 8/13/2009 1:11 pm:

I did not yet read the update on the tests, but a quick glance at the php source code was revealing enough to tell me that adding an opcode cache would be of little use: the test stress execution of empty loops N times in a row, in a single page.

While it is formally correct, that metric is of absolutely zero interest to any web developer. The overall time spent by an application executing the branches will be about 1/1000th,1/100th of a second out of a 2,3 secs total time in any real application.

Adding an opcode cache for a single page run might even give you poorer results, since if you omit a cache-warmup pass you will now include in the response the time spent stocking the opcode list to cache before execution.

A more interesting metric would be using AB, Jmeter or some other tool that measures pages-per-second, in sets where 1,10,100 parallel clients are used to pound on the server.

Janier Davila (@janierdavila) wrote at 8/13/2009 5:43 pm:

Very encouraging words: “The difference in PHP is you.” Good job.

I personally see both technologies as tools. It doesn’t hurt knowing both, like a carpenter would know how to use a hammer or any other tool. That is my own karma, my own statement to make up my mind. And…I love them both and I use them both :). Programming languages don’t make good programmers…

Anyways, good article! Keep it up!

Luis Morales wrote at 8/14/2009 7:14 am:

I’m following this subject too. My humble suggestion of what someone in the php community should do: a C++ compiled dll extension MVC framework to php. The actual frameworks are killing php; making it slower, abstracting it and even changing its default behavior. Agree?

Brandon Savage (@brandonsavage) wrote at 8/14/2009 7:21 am:

I think that PHP frameworks have their place. Certainly due to PHP’s “share nothing” structure and the fact that everything is garbage collected at the end of a request we have some performance issues with intense frameworks; this can’t be avoided easily. Still, I think that frameworks are good, and writing libraries of common functionality is the next evolution in any language.

I’d like to see frameworks make use of caching, placing objects into memory and some improvement in the request handling abilities. I think there’s a lot that can be done for PHP to resolve this performance issue.

Luis Morales wrote at 8/14/2009 7:42 am:

@Brandon

I agree with you. Libraries of common functionality are good. But I think the current frameworks have too many inheritance and includes in their trees. I like the DRY principle, but we could hard code some things for sake of performance. Did you see Rasmus presentation at Drupalcon on the subject of php frameworks?

Luis Morales wrote at 8/14/2009 10:40 pm:

What I said at http://www.misfitgeek.com:

Luis Morales wrote on Friday, August 14 2009

There are lots of things to consider, besides benchmark results. My comparison includes:

1- Platforms supporting the technology:

– ASP.NET: Microsoft Server and Apache (Mono Castle)

– PHP: All known commercial and community platforms.

2- Corporations Backing or Contributing to the technology:

– ASP.NET: Microsoft

– PHP: Microsoft, Zend, IBM, Oracle, Sun, Google, etc.

3- How intuitive is the technology:

– ASP.NET: Good

– PHP: Excellent

4- How flexible is the technology:

– ASP.NET: Debatable.

– PHP: Flexible by nature, not debatable, almost an absolute fact.

5- Performance:

– ASP.NET: Excellent on Windows Server. The Mono / Castle project is still working on performance.

– PHP: Good on most platforms. Results may vary depending on a lot of settings out of the box or included by extensions. But given to an expert on PHP and Apache, can be as fast as ASP.NET on a Windows Server.

6- How expensive would be implementing the technology on a large scale:

– ASP.NET: ask MySpace and Monster (very expensive but works good out of the box = good for Fortune 500).

– PHP: ask Facebook, Flickr and others (considerably cheap and there are no license issues = good for startups.)

Scott Mitchell wrote at 8/15/2009 12:50 pm:

Just one small correction in your post. You said: “We can’t examine the ASP.NET core to determine why it’s faster.”

That’s not true. You can view the .NET Framework code, by using disassemblers like Reflector. Also, Microsoft has made most of the .NET Framework (including the ASP.NET bits in System.Web) shared source.

See: http://referencesource.microsoft.com/netframework.aspx

Granted, Microsoft won’t accept patches from the community, but you can certainly examine how ASP.NET’s internals work.

Thanks

Manuel Lemos wrote at 8/17/2009 3:38 am:

Brandon, PHP/Zend engine can be better if you use an opcode optimizer. There are several optimizer extensions from Zend and other developers for PHP but for obvious business reasons Zend guys kept such extensions out of the main PHP distributions.

So it is not the PHP community’s fault that PHP official distribution is not better.

Andrei Rinea (@AndreiR23) wrote at 8/30/2009 4:47 am:

I must back up Scott Mitchel’s comment in that we (and you!) have a free tool, called reflector which you can download here :

http://www.red-gate.com/products/reflector/

since like 2001 and you can dissamble *ANY* .NET compiled code. Including the framework itself. It is a powerful, easy-to-use and day-to-day helpful decompiler which I suggest you should use it to inspect “why [ASP.NET] it’s faster”.

Copyright © 2023 by Brandon Savage. All rights reserved.