PHP Benchmarking Mythbusters

Tags: doctrine2, outlet

Posted over 2 years ago by romanb

First of, this blog post sucks. I thought I would never write such a senseless apples and oranges comparison with artificial and meaningless benchmarks, but I was just a bit astonished by the results that I would like to share.

I use object-relational mapping tools in many different languages, from Java to C# to PHP. One of the many supposedly lightweight alternatives in PHP to Doctrine is Outlet. After stumbling upon this comment on a stackoverflow.com post: "Gotta second Outlet. Doctrine is comically bloated - it is WAY too big to be a sensible choice for anything but the lightest of server loads." I thought I take a look at Outlet. This ORM seems to consist of only 9 classes! It can't get any more lightweight right? I assumed it would blow Doctrine out of the water performance-wise in all situations.

So I downloaded Outlet 0.7 and created a simple test database with just 1 table. Then I wrote a small script that bootstraps Outlet, inserts 500 objects into the database and reads them out afterwards (Yes, it's stupid, just like most other artificial benchmarks).

Environment is PHP 5.3.0 with APC.

<?php

echo PHP_EOL . (memory_get_usage() / 1024) . ' KB ' . PHP_EOL;

include 'config.php';

set_include_path(get_include_path() . PATH_SEPARATOR . __DIR__ . '/outlet');

require 'Outlet.php';

ini_set('error_reporting', E_ALL);

class Bug
{
    public $ID;
    public $Title;
    public $Description;
}

$config = array(
  'connection' => array(
    'dsn'      => 'mysql:host=localhost;dbname=outletbenchmark',
    'username' => $mysqlUsername,
    'password' => $mysqlPassword,
    'dialect' => 'mysql'
  ),
  'classes' => array(
    'Bug' => array(
      'table' => 'bugs',
      'props' => array(
        'ID'        => array('ID', 'int', array('pk' => true, 'autoIncrement' => true)),
        'Title'     => array('Title', 'varchar'),
        'Description' => array('Description', 'varchar')
      )
    )
  )
);
Outlet::init($config);
$outlet = Outlet::getInstance();
$outlet->createProxies();

$s = microtime(true);
$outlet->getConnection()->beginTransaction();
for ($i = 0; $i < 500; ++$i) {
    $bug = new Bug;
    $bug->Title = 'This is a test bug';
    $bug->Description = 'Hey there!';

    $outlet->save($bug);
}
$outlet->getConnection()->commit();

$e = microtime(true);
echo "\nInsert:" . ($e - $s) . "\n";

$outlet->clearCache();

$s = microtime(true);
$bugs = $outlet->select('Bug');
$e = microtime(true);

$outlet->clearCache();

echo "\nQuery:" . ($e - $s) . "\n";

echo "\n" . (memory_get_usage() / 1024) . ' KB ' . PHP_EOL;

First off, ini_set('error_reporting', E_ALL); was necessary to silence the following E_STRICT warnings coming from Outlet:

Strict Standards: Non-static method OutletMapper::get() should not be called statically,
assuming $this from incompatible context in /Users/robo/dev/php/outlet/outlet-0.7/classes
/outlet/Outlet.php on line 184

Strict Standards: Only variables should be passed by reference in /Users/robo/dev
/php/outlet/outlet-0.7/classes/outlet/OutletMapper.php on line 546

Does not really look good (and does not speak for Outlet very well), but anyway.

I did the same for Doctrine 2.0, without setting up a metadata or query cache.

<?php

echo PHP_EOL . (memory_get_usage() / 1024) . ' KB ' . PHP_EOL;

include 'config.php';

require 'doctrine/Doctrine/Common/IsolatedClassLoader.php';

/**
 * @Entity
 * @Table(name="bugs")
 */
class Bug
{
    /** @Id @Column(type="integer") @GeneratedValue(strategy="AUTO") */
    public $ID;
    /** @Column(type="string") */
    public $Title;
    /** @Column(type="string") */
    public $Description;
}

$classLoader = new \Doctrine\Common\IsolatedClassLoader('Doctrine');
$classLoader->setBasePath(__DIR__ . '/doctrine');
$classLoader->register();

$config = new \Doctrine\ORM\Configuration;

$config->setProxyDir(__DIR__ . '/Proxies');
$config->setProxyNamespace('Proxies');
$connectionOptions = array(
    'driver' => 'pdo_mysql',
    'user' => $mysqlUsername,
    'password' => $mysqlPassword,
    'host' => 'localhost',
    'dbname' => 'doctrine2benchmark'
);

$em = \Doctrine\ORM\EntityManager::create($connectionOptions, $config);

$s = microtime(true);
for ($i = 0; $i < 500; ++$i) {
    $bug = new Bug;
    $bug->Title = 'BBug';
    $bug->Description = 'Hello there!';

    $em->persist($bug);
}
$em->flush();

$e = microtime(true);
echo "\nInsert:" . ($e - $s) . "\n";

$em->clear();

$s = microtime(true);
$bugs = $em->getRepository('Bug')->findAll();
$e = microtime(true);

$em->clear();

echo "\nQuery:" . ($e - $s) . "\n";

echo "\n" . (memory_get_usage() / 1024) . ' KB ' . PHP_EOL;

Here are my results.

1st Run

Measurement Outlet Doctrine
Insert Time 0.23142600059509 0.11601996421814
Query Time 0.070523977279663 0.025638818740845
Used Memory 644.5546875 KB 1061.83203125 KB

No, I did not swap the numbers, I promise. You see that the D2 version uses about 400KB more memory but the result of the timings are quite surprising. Being curious I ran both scripts several times which means the query section has to hydrate 500 objects more for each run.

2nd Run

1st refresh (1000 objects)

Measurement Outlet Doctrine
Insert Time 0.26595592498779 0.11661005020142
Query Time 0.14437794685364 0.052286863327026
Used Memory 875.0703125 KB 1313.15625 KB

3rd Run

2nd refresh (1500 objects)

Measurement Outlet Doctrine
Insert Time 0.2314441204071 0.11621117591858
Query Time 0.21359491348267 0.079329013824463
Used Memory 1139.5859375 KB 1541.59375 KB

Did you expect these results? After all Doctrine is so bloated, right? (Doctrine 2 full package ~250 classes) and Outlet is so lightweight (~9 classes)?

Bottom line:

  • The number of classes barely means anything. (Its probably a good criterion if you're short on disk space).
  • "Lightweight" is a buzzword and meaningless without a reference point.
  • Don't judge a library by its size and certainly dont try to draw conclusions from the size to the performance, or worse to the scalability. It just doesnt work.
  • Artificial benchmarks suck.
  • To all the folks hunting for everything lightweight and micro-benchmarking all day long: You're wasting your time (Just like I did with this stupid benchmark...).
  • Don't trust artificial benchmarks (Not even this one).

PS: This is no post against Outlet, so if any Outlet guys or fans are reading this, please don't feel offended. Since I dont know Outlet well I'm sure I did a lot of things wrong but thats really not important here. I am just making a stance against all the ridiculously stupid artificial benchmarks out there that try to make people believe Doctrine is slow and bloated. This post shows I can make it look the other way around easily. That just shows how meaningless these comparisons are.

All the code used to run these benchmarks can be downloaded from here. It is a zip archive containing all the code you need to run the benchmarks yourself.


Comments (22) [ add comment ]

Great Post Posted by Tommy Smith about over 2 years ago.

I am right there with you Brother! I hear the arguments against number of classes and they remind me of the arguments against OOP. Don't use OOP - it is slower than procedural. Silly, Silly, Silly. So, any time you hear people making these types of arguments - FIND OUT FOR YOURSELF! Do your own research! Way to go romanb!

I liked it ... Posted by b00giZm about over 2 years ago.

... when you said you're not offending the Outlet though there's a big yellow box with something like, HEY LOOK AT THOSE STUPID PHP ERRORS THOSE SUCKERS PRODUCE!!! ^^

It's okay dude, if I was you I'd defend my work in a similar way ;) You can be proud, D2 will be a pretty awesome release. You know it, we know it, so don't give a sh*t for those guys ;)

Meten=weten Posted by drm / Gerard about over 2 years ago.

We Dutch have a saying "Meten is weten", which translates as "Measure it and you know it". It's good practice anyway, and I don't know why it isn't done more often to bust many more myths. So if anyone feels offended in any way, the only correct way to respond is with other benchmark proof ;)

Keep up the good work, btw, Doctrine kicks ass.

Keep with Doctrine Posted by Pete Hatton about over 2 years ago.

Interesting benchmark, that Doctrine uses a little extra memory, but I respect that Doctrine 2.0 is still in Alpha stages.

What would be interesting to see with the 5.3 style class loader that Doctrine 2.0 uses, is how many classes Doctrine had to load in those benchmarks. Let's face it, it won't be all of them.

Also I had a quick look at the Outlet zip file. Does it handle YAML to Model generation, Database to Model generation and migrations? (Plus other stuff that I've missed from Doctrine's feature list.) Outlet is missing a big chunk of functionality that Doctrine has out of the box.

Keep up the good work guys!

Outlet Posted by Giorgio Sironi about over 2 years ago.

I see some posts on Outlet this week and I already thought that it has a semplicistic approach... Doctrine 2 has multiple points where you can insert caching, so performance won't be a problem for me. "The number of classes barely means anything. (Its probably a good criterion if you're short on disk space)" ...hilarious :) With autoloading obviously you don't expect to load all those different classes at the same time.

@pete @giorgio Posted by romanb about over 2 years ago.

Of course only a small amount of the D2 classes are (auto)loaded in this test, still more than Outlet has though. Thats one reason for the higher mem-usage. When you disable APC the mem-difference gets bigger (by about 1MB). This is memory used up by PHP itself for lexing, parsing and producing bytecode. With APC the opcodes for the requested/used classes in a request are merely copied from shared memory to local memory of the process, afaik. This is not only faster but saves a good amount of memory also.

However if you run this test without APC or you run any larger PHP application without APC (or another opcode cache) and then benchmark it its like shooting yourself in the foot before timing your 100m sprint and pretending your foot is OK and no reason for the slowdown. In my eyes, at least. When crossing the border from a php script to a php application you dont really want all your code to be compiled again and again and again...

The higher mem-usage was expected by myself, Doctrine stores and uses more stuff in order to do more but the timings were surprising. It highlights again that you dont need much code in order to produce something that is "slow". A single bad algorithm, careless use of expensive functions in some critical places etc. and you lost 10 times the performance you gained through countless hours of previous artificial benchmarking and micro-optimizing.

Performance problem? Profile -> Fix the bottleneck(s) -> Continue work. Its not that hard.

cache ? Posted by desfrenes about over 2 years ago.

"Doctrine 2 has multiple points where you can insert caching, so performance won't be a problem for me."

you may still have to invalidate cache which is another problem.

without APC? Posted by Welja about over 2 years ago.

Could you post the same test without APC? I know you said it's like running a race with busted feet, but it would be nice to see them together with the APC enabled numbers.

Still, nice to see that Doctrine 2 will rock!

php Posted by mr. glue about over 2 years ago.

I think that PHP is not very well suited for building large frameworks with large number of classes (I know that people still build them: Zend, Symfony, Doctrine, etc.). This is because of its shared nothing execution model. If Doctrine was build as a PHP extension, it would work so much better. PHP is all about gluing those extensions together. And its usually better if you can achieve the same with less glue. Simplicity is a key to success. Features aren't. The product is not complete when there isn't anything to add, but when there isn't anything to remove.

re: mr. glue Posted by coding horror about over 2 years ago.

If your language doesn't allow you to approach your problem in the nicest possible way, then you should abandon that language.

That having said, I see no problem with Zend, Doctrine and the like. A large number of classes is a good sign. 1. You won't use all classes at once. It's silly that even now people still needs to find the splautoloader. 2. In order for a framework to be usable it needs to be extremely modular. The SRP principle needs to be taken seriously. 3. Simplicity will break your project. If you mean procedural clutter like in phpmyadmin and so many other failing php projects. 4. You need APC or similar, true, but this will be shipped by default in php 6.

@mr. glue Posted by romanb about over 2 years ago.

"I think that PHP is not very well suited for building large frameworks with large number of classes ... This is because of its shared nothing execution model... If Doctrine was build as a PHP extension, it would work so much better."

Its more like "shared nothing except the database" and thats where you get the real problems when scaling out, because its not as easy with databases as it is with php servers.

And I disagree with your statement. You're implying that large libraries in php are slow due to their large number of classes. This is simply not true, especially not with an opcode cache where the opcodes of all classes are held in shared memory. And it doesnt require much memory. You can cache thousands of classes with the default configuration of just 32MB of shared memory.

You need to back up your claim.

It is no secret that php is not a very fast language itself either (compared to statically typed, JIT-compiled languages like Java, compared to other scripting languages it does fine) but still the most common reasons for slow php programs are slow code (not the slow language) and too much database/network access, not the number of classes. And these 2 factors are the same, no matter what programming language you use.

Simplicity... well, yea I like simplicity, too. And I agree that its equally important to leave things out as it is to get things in. It just often doesnt line up with real-world requirements. If a customer demands a feature, you cant say "no, sorry, I want to keep it simple."

There are certain requirements you want to achieve with a program/library/software and these requirements are not always simple, mostly they're complex.

Many libraries use "simplicity" as an excuse for being inable to provide the functionality also.

just ask yourself Posted by mr. glue about over 2 years ago.

Would you rather have doctrine as extension or as PHP code? I'm pretty sure that almost everyone wants to have it as an extension rather than huge collection of PHP classes. Why do you use JSON extension instead of several PHP based JSON libraries? Why do you use PDO instead of writing the low level code in PHP? The reason is that they are much better when they are written as an extension. And that's why PHP is a glue. PHP really should have some default ORM implementation as an extension. That would be interesting, and I bet that I'm not the only one who thinks like this.

And what do you think that Rasmus said when he wrote about No Framework Framework: http://toys.lerdorf.com/archives/38-The-no-framework-PHP-MVC-framework.html. And it was really somewhat against building huge PHP class libraries.

@mr. glue Posted by romanb about over 2 years ago.

I use these extensions because they are available. If they were not available I would use corresponding php classes, however, and would have no problem with that.

I dont even want to start to imagine what a huge amount of C code Doctrine would be as an extension. Certainly far beyond what I would be willing to maintain.

As far as Rasmus is concerned, sorry but I do not value his opinion on this matter because I dont think he is a good library/framework/API developer/designer. Proof is php itself. I am aware of his position on frameworks/libraries and I dont share it because I think its an unsustainable position if you want to move forward and not stand still.

Most people can't afford and dont have the time to reinvent the wheel over and over again just because of being afraid of using other people's code.

@mr. glue Posted by romanb about over 2 years ago.

Also, luckily Rasmus is by far not the only one on the php project (otherwise we would still be at php 3, I guess) and a lot of the other people support and embrace a rich library and framework ecosystem. And if you ask me, thats how it is supposed to be. A programming language should always embrace a rich library/framework ecosystem among the users of the language and not work against it.

true-true Posted by coding horror about over 2 years ago.

@mr. glue You are very right that it would be tremendesly nice if the complete doctrine project was available as a native C-extension. Of course. But this is not going to happen since developing in C is often slower than in doctrine. Besides extensions are required to have a stable interface. If you want to do some BC breaks, you need to distribute doctrine-2.3.1, doctrine-2.6.0 etc. Shared hosters are already horrible by having no pdo-mysql enabled etc, let alone giving you the right doctrine extension version.

As far as Rasmus is concerned, sorry but I do not value his opinion on this matter because I dont think he is a good library/framework/API developer/designer. Proof is php itself. I am aware of his position on frameworks/libraries and I dont share it because I think its an unsustainable position if you want to move forward and not stand still.

<<<

Spot on. I hate to say it, but php is damn ugly in many respects.

Outlet performance Posted by Alvaro Carrasco about over 2 years ago.

Hi, I'm the main developer of Outlet. I want to point out that the outlet performance issues and strict warnings referenced in the post were corrected recently by v1.0rc1. I really don't think that the performance and memory usage of either was bad to begin with. The point that you can get good performance with a lot classes is well taken. I am also glad that doctrine is finally moving in the direction of a transparent orm with version 2.0, that was one of my original goals when I started Outlet. For those that are very defensive about these things, keep in mind that we are just developers giving our time away in the hopes that our code will be useful to others. I have a lot of respect for the developers of Doctrine and any other open source project.

@Alvaro Carrasco Posted by romanb about over 2 years ago.

Hi Alvaro,

thanks for chiming in. I'm glad you didnt take offense from this post as it was not meant as such. Its just an anti-artificial-benchmark post :-).

Great to hear that the issues mentioned here have been resolved.

All the best for your project!

Roman

goods Posted by deed about about 1 year ago.

We Dutch have a saying "Meten is weten", which translates as "Measure it and you know it". It's good practice anyway, and I don't know why it isn't done more often to bust many more myths. So if anyone feels offended in any way, the only correct way to respond is with other benchmark proof ;)

... Posted by uh about about 1 year ago.

What about comparing Outlet 0.7 (stable) with Doctrine 1.2 (stable) on php 5.2, or Outlet 1.0 (release candidate) with Doctrine 2.0 (alpha) on php 5.3? I believe that would be more interesting and informative.

... Posted by uh about about 1 year ago.

What about comparing Outlet 0.7 (stable) with Doctrine 1.2 (stable) on php 5.2, or Outlet 1.0 (release candidate) with Doctrine 2.0 (alpha) on php 5.3? I believe that would be more interesting and informative.

........ Posted by uh about about 1 year ago.

What about comparing Outlet 0.7 (stable) with Doctrine 1.2 (stable) on php 5.2, or Outlet 1.0 (release candidate) with Doctrine 2.0 (alpha) on php 5.3? I believe that would be more interesting and informative.

........ Posted by uh about about 1 year ago.

Sorry for the multiple posts. I get a blank page when I post.

Create Comment