Write your own ORM on top of Doctrine2

Posted about 1 month ago by beberlei

The Doctrine ActiveEntity Extension is just an experiment, nothing that will be developed much further from the Doctrine Dev Team. It is only a show-case for what is possible with Doctrine2. Please feel free to take the code and develop it further.

Did you feel the urge to write your own Object-Relational Mapper after reading Martin Fowlers PoEAA? I am guilty to have tried implementing two different ORMs on my own, both now safely dumped into the trash.

In isolation each ORM pattern is easy to describe, understand and even implement. However the combination of a large set of patterns into a single implementation introduces a lot of hard to solve complexity in your code. Even simple Object-Relational-Mappers require a lot of patterns to become useful: Metadata Mapping, Identity Map, Foreign Key Mapping, Association Table Mapping and Query Object. Implementations with more features at least need the UnitOfWork and probably many more, for example handling inheritance, locking, value objects and such.

Doctrine2 already solves a lot of the head aching problems in a consistent approach. We have been working on this project for almost 2 years now, with all the experience we gained implementing Doctrine 1. Additionally we make use of well-understood concepts from other ORM implementations across various languages.

We as developers think that Doctrine2 responsibilities are very well separated such that you can exchange larger parts of the Doctrine2 core without having to re-implement everything. So if you ever feel inspired to implement your own ORM, we would be happy to offer you Doctrine2 as a foundation to build upon.

There are examples of other ORMs that have taken the re-use instead of re-implement road. For example the Groovy Grails ORM is an ActiveRecord implementation on top of the popular Hibernate Java ORM. Since Groovy is a java-virtual-machine language it can safely use the Hibernate ORM as a dependent library.

This article will describe some possible extensions and show where you can hook into the Doctrine2 core to implement your own ORM. The article will be very code focused and also comes with a Github project where all the code and some tests are hosted.

Doctrine2 and ActiveRecord

Doctrine2 is implementing the DataMapper pattern, however many programmers think ActiveRecord is better for various reasons. For me data-mappers are superiour to ActiveRecord, however I do understand why ActiveRecord is so popular: Its very easy to get started and do cool stuff with it! If you want Doctrine2 to be ActiveRecord you can have it. Actually it is very easy to turn it into a powerful ActiveRecord implementation, keeping all the powerful features such as DQL.

Some while ago Jonathan already released his approach, called the "ActiveEntity" extension. Its a single abstract php class that your entities have to implement, the code is still in our SVN repository. However a more recent version of this code is available as a project on Github. I won't support this experiment any further, I hope somebody picks it up and starts maintaining it.

With Jonathans old code, to allow active record entities, you have to bootstrap the ActiveEntity by passing a static EntityManager:

<?php

\DoctrineExtensions\ActiveEntity::setEntityManager($em);

Now say we have a User Entity (using Jonathans old ActiveEntity):

<?php

namespace Entities;

use DoctrineExtensions\ActiveEntity;

/** @Entity */
class User extends ActiveEntity
{
    /** @Id @GeneratedValue @column(type="integer") */
    private $id;

    /** @Column(type="string") */
    private $name;
}

With PHP 5.3 late-static binding functionality we can now access the EntityRepository, a finder object for entities using a Ruby on Rails'ish notation:

<?php

$user = User::find($id);
$users = User::findBy(array("name" => "beberlei"));
$beberlei = User::findOneBy(array("name" => "beberlei"));

The code to allow this functionality is very simple:

<?php

public static function __callStatic($method, $arguments)
{
    return call_user_func_array(array(self::$_em->getRepository(get_called_class()), $method), $arguments);
}

There are also some additional methods on the ActiveEntity class that use magic __get and __set and __call methods to access the private properties of an Entity (such as the User id and name shown above). Additionally you can call save() or remove() on any instance.

For starters this offers a great ActiveRecord implementation with all the powerful features that Doctrine2 offers, such as DQL and UnitOfWork. However we can still go much further:

  • Eliminate the need to define ActiveEntity properties by metadata mapping inference
  • Adding your own powerful Metadata Mapping Layer
  • Add a Doctrine 1.2 behaviour system using the PHP 5.3.99DEV Traits functionalitiy
  • Add validation to properties of an ActiveEntity

Lets begin with a simple introduction to the Doctrine Metadata Model to explain how this is all possible.

Doctrine2 Metadata Model

You probably already saw that Doctrine2 offers many different metadata configuration mechanisms: Annotations, YAML, XML and plain PHP. Any one of this implementations will transform into an instance of Doctrine\ORM\ClassMetadata which is then cached for subsequent web requests. The ClassMetadataFactory is responsible for creating and managing those metadata instances.

Doctrine2 uses the ClassMetadata instance internally for all runtime access to your entities metadata, which means that you have to extend this class such that it works exactly the same from the outside.

If you wanted to extend the inner workings of Doctrine2, this is indeed the way to go. First extend the EntityManager to replace the ClassMetadataFactory used. This piece of code is the only hackish workaround, everything else is rather nice :-)

<?php

namespace DoctrineExtensions\ActiveEntity;

use DoctrineExtensions\ActiveEntity\Mapping\ClassMetadataFactory;

class ActiveEntityManager extends \Doctrine\ORM\EntityManager
{
    protected function __construct(Connection $conn, Configuration $config, EventManager $eventManager)
    {
        parent::__construct($conn, $config, $eventManager);

        $metadataFactory = new ActiveClassMetadataFactory($this);
        $metadataFactory->setCacheDriver($this->getConfiguration()->getMetadataCacheImpl());

        // now this is the only hack required to get it work:
        $reflProperty = new \ReflectionProperty('Doctrine\ORM\EntityManager', 'metadataFactory');
        $reflProperty->setAccessible(true);
        $reflProperty->setValue($this, $metadataFactory);
    }

    public static function create($conn, Configuration $config, EventManager $eventManager = null)
    {
        // ... copy paste from EntityManager::create()

        return new ActiveEntityManager($conn, $config, $conn->getEventManager());
    }
}

And both the ClassMetadataFactory and ClassMetadata:

<?php

namespace DoctrineExtensions\ActiveEntity\Mapping;

class ActiveClassMetadataFactory extends \Doctrine\ORM\Mapping\ClassMetadataFactory
{
    protected function _newClassMetadataInstance($className)
    {
        return new ActiveClassMetadata($className);
    }
}

class ActiveClassMetadata extends \Doctrine\ORM\Mapping\ClassMetadata
{
}

This is the foundation of your own Doctrine2-based ORM. We will see in the next section how we can use this.

Exchange Doctrine2 Reflection for Array-based Field Storage

Doctrine2 uses reflection to access the current values of an entity. This is necessary, because Doctrine2 is a Data Mapper that enforces clean separation between entities and persistence. If we extend it to be an ActiveRecord implementation this separation is not wanted anymore and we can opt for a new approach, using the get()/set() methods on our ActiveEntities.

Defining the properties "id" and "name" will then not be necessary anymore, they will all be saved in an array hash-map called "_data" inside the ActiveEntity. We cannot use annotations for metadata anymore, however the XML or YAML drivers would still work smoothly.

To get started we have to modify our ActiveClassMetadata a bit to exchange the contents of reflClass and reflFields with our own classes. Looking at the ClassMetadata code and doing some project wide searches I found out about all the necessary changes. To replace the ReflectionClass we only need to exchange getProperty and keep the rest. To exchange ReflectionProperty we only have to overwrite setAccessible(), getValue() and setValue().

<?php

namespace DoctrineExtensions\ActiveEntity\Reflection;

class ActiveEntityReflectionClass extends \ReflectionClass
{
    public function getProperty($name)
    {
        return new ActiveEntityPropertyReflection($this->name, $name);
    }
}

class ActiveEntityReflectionProperty
{
    public $name = null;
    public $class = null;

    public function __construct($class, $name)
    {
        $this->class = $class;
        $this->name = $name;
    }

    public function setAccessible($flag) {}

    public function setValue($entity = null, $value = null)
    {
        $entity->set($this->name, $value);
    }

    public function getValue($entity = null)
    {
        return $entity->get($this->name);
    }
}

This is about enough to exchange reflection transformation against a simple ActiveRecord get/set approach. Now we need to replace the all the instantiations of ReflectionClass relevant for runtime mapping with our implementation:

<?php

namespace DoctrineExtensions\ActiveEntity\Mapping;

use DoctrineExtensions\ActiveEntity\Reflection\ActiveEntityReflectionClass;
use DoctrineExtensions\ActiveEntity\Reflection\ActiveEntityReflectionProperty;

class ActiveClassMetadata extends \Doctrine\ORM\Mapping\ClassMetadata
{
    public function __construct($entityName)
    {
        parent::__construct($entityName);
        $this->reflClass = new ActiveEntityReflectionClass($entityName);
        $this->namespace = $this->reflClass->getNamespaceName();
        $this->table['name'] = $this->reflClass->getShortName();
    }

    /**
     * Restores some state that can not be serialized/unserialized.
     *
     * @return void
     */
    public function __wakeup()
    {
        // lots of code here, see the Github Repository
    }
}

Again, this is enough and our ActiveEntity Mapping now works. We can heavily modify the ActiveEntity now to loose the requirement to specify properties for the defined metadata. We can rewrite the User entity to be:

<?php

namespace Entities;

use DoctrineExtensions\ActiveEntity\ActiveEntity;

class User extends ActiveEntity
{
}

Using an XML or YAML Mapping is already enough for this ActiveEntity to work out of the box.

Implementing your own Metadata Mapping Driver

In the spirit of Doctrine 1.* or GORM there should be a PHP based metadata mapping driver now and actually Doctrine2 ships with one already:

<?php

$config = new \Doctrine\ORM\Configuration();
$config->setMetadataDriverImpl(new \Doctrine\ORM\Mapping\Driver\StaticPHPDriver());
// ...

This allows to specify the metadata within the User class:

<?php

namespace Entities;

use DoctrineExtensions\ActiveEntity\ActiveEntity;
use DoctrineExtensions\ActiveEntity\Mapping\ActiveClassMetadata;

class User extends ActiveEntity
{
    static public function loadMetadata(ActiveClassMetadata $cm)
    {
        // work with $cm here!
    }
}

You could extend that Static PHP Driver even more for the next section. We could add additional metadata information, such as names of behaviours to extend or validators or anything else.

Using Traits for Behaviours

We want to add a simple "Timestampable" behaviour now, hooking into the loadClassMetadata event as described in the documentation:

Now this is untested code, as i don't have a PHP-5.3.99-DEV version compiled at this machine.

The following trait can be used by our User entity:

<?php

namespace DoctrineExtensions\ActiveEntity\Behaviour;

trait Timestampable
{
    public function created()
    {
        return $this->get('created');
    }

    public function updated()
    {
        return $this->get('updated');
    }

    /** will be a prePersist lifecycle hook */
    public function setCreated()
    {
        return $this->set('created', new \DateTime("now"));
    }

    /** will be a preUpdate lifecycle hook */
    public function setUpdated()
    {
        return $this->set('updated', new \DateTime("now"));
    }
}

class User extends ActiveEntity use Timestampable
{

}

We now need an Event that modifies the ActiveClassMetadata as required:

<?php

namespace DoctrineExtensions\ActiveEntity\Behaviour;

use Doctrine\ORM\Event\LoadClassMetadataEventArgs;

class TimestampableEvent
{
    public function loadClassMetadata(LoadClassMetadataEventArgs $eventArgs)
    {
        $classMetadata = $eventArgs->getClassMetadata();
        $traits = $classMetadata->reflClass->getTraitNames();
        if (!in_array("DoctrineExtensions\ActiveEntity\Behaviour\Timestampable", $traits)) {
            return;
        }

        $classMetadata->mapField(array(
            'type' => 'datetime',
            'fieldName' => 'created',
        ));
        $classMetadata->mapField(array(
            'type' => 'datetime',
            'fieldName' => 'updated',
        ));
        $classMetadata->addLifecycleCallback("prePersist", "setCreated");
        $classMetadata->addLifecycleCallback("prePersist", "setUpdated");
        $classMetadata->addLifecycleCallback("preUpdate", "setUpdated");
    }
}

You can now register this behaviour with your Entity Manager and just the usage of the trait Timestampable adds two additional fields and updates them accordingly.

Again, the trait code is untested. It should work, but I cannot guarantee! :)

Conclusion

What are you waiting for? This article showed a very deep modification of the Doctrine2 core to turn it into Active Record. The changes required some understanding of the inner workings of Doctrine2, however not many changes were required in the end.

See the code on GitHub!


Comments (17) [ add comment ]

Custom ClassMetadataFactory Posted by sognat about about 1 month ago.

One question.

Why don't you allow to simply inject custom ClassMetadataFactory into Configuration/EntityManager?

@sognat Posted by beberlei about about 1 month ago.

Because this is currently the only, not very well-thought out, use-case for extending the ClassMetadataFactory.

We are not yet in the position to make it easy for people to extend Doctrine classes, nor do we want people to be able in some cases. Remember we are also in Beta only, we don't want people to use possible extension points that might get dropped or reworked later.

Unless the code stabilizies much more, we won't turn variables protected or make them configurable for only very slim use-cases.

However such code examples allow us to evaluate our decisions and maybe adjust them accordingly.

Really great! Posted by Markus about about 1 month ago.

That's really another great extension. Now, with this tutorial, is anyone working on a kind of ACL (maybe using zend_acl) for Doctrine 2? I would try to do it, but I don't really know much about zend_acl.

acl Posted by Lukas about about 1 month ago.

an ACL extension would be awesome indeed

@Lukas @Markus Posted by beberlei about about 1 month ago.

How would an ACL extension work?

I doubt it requires such an extension to be build on top, but can be more generic in the context of Doctrine2 alone. Can you describe some use-cases?

ACL Posted by Michael about about 1 month ago.

Hey there. Yeah, I'm waiting for an ACL extension since I started coding with doc2.0

The following possibilities are in my mind: Privilegues of 1. a Role on an entire Ressource (e.g. Article Admin, full access to all articles) 2. a Role to specific attributes of an entire Ressource 2.1 to value fields (e.g. read access to all articles (from 1.) but write access only to body_text) 2.2 to related entitis 2.2.1 Rights on the relation (add, remove related objects, e.g. in a forum users might have permission to add a new thread) 2.2.2 Rights on related objects (delete, edit, e.g. forum users can create new threads, forum moderators can edit and delete threads) 2.2.3 pass Rights on relation to all sub-related objects of type X (Forum moderator has rights on threads and must also have rights on posts, polls and attachments related to this thread)

It should be possible to filter Repository Requests by ACL Privilegues. If a user has no read access to a ressource and it is filtered in the application, this could break pagination (only 19 elements listed instead of 20) If a Moderator wants to move a thread, there is no need to show him forums he doesn't have write access in.

What do you think of this?

re acl Posted by jwage about about 1 month ago.

Maybe someone can write up something for the ACL on http://wiki.doctrine-project.org/dashboard.action so we can collaborate together?

not a mapped superclass Posted by gordonslondon about about 1 month ago.

looks like really good stuff !

i have this error: Class DoctrineExtensions\ActiveEntity\ActiveEntity is not a valid entity or mapped super class.

any help? thanks

@gordonslondon Posted by beberlei about about 1 month ago.

that is not very helpful, doing what does this happen and please post the complete strack trace.

re not a mapped superclass Posted by gordonslondon about about 1 month ago.

i use the annotation driver, the error comes when i try MyEntity::find($id)

stack trace:

at Doctrine/ORM/Mapping/MappingException.php line 137

at Doctrine\ORM\Mapping\MappingException::classIsNotAValidEntityOrMappedSuperClass('DoctrineExtensions\ActiveEntity\ActiveEntity')

in Doctrine/ORM/Mapping/Driver/DriverChain.php line 113

at Doctrine\ORM\Mapping\Driver\DriverChain->isTransient('DoctrineExtensions\ActiveEntity\ActiveEntity')

in Doctrine/ORM/Mapping/ClassMetadataFactory.php line 202

at Doctrine\ORM\Mapping\ClassMetadataFactory->_getParentClasses('MyEntity')

in Doctrine/ORM/Mapping/ClassMetadataFactory.php line 224

at Doctrine\ORM\Mapping\ClassMetadataFactory->_loadMetadata('MyEntity')

in Doctrine/ORM/Mapping/ClassMetadataFactory.php line 148

at Doctrine\ORM\Mapping\ClassMetadataFactory->getMetadataFor('MyEntity')

in Doctrine/ORM/EntityManager.php line 235

at Doctrine\ORM\EntityManager->getClassMetadata('MyEntity')

in Doctrine/ORM/EntityManager.php line 510

at Doctrine\ORM\EntityManager->getRepository('MyEntity')

in DoctrineExtensions/ActiveEntity/ActiveEntity.php line 268

at DoctrineExtensions\ActiveEntity\ActiveEntity::__callStatic('find', array(1)) in n/a line n/a

at MyEntity::find(1)

in MyEntityController.php line 33

annotations dont work Posted by beberlei about about 1 month ago.

ActiveEntity does not work with Annotations, you have to use XML or YAML. The blog post also states so.

@beberlei Posted by gordonslondon about about 1 month ago.

ok thanks for your fast reply, but i don't see the point, why can't i use the annotation driver ? i was thinking that all mapping types were converted in a unique format

@annotations Posted by beberlei about about 1 month ago.

Because you cannot use the property annotations with magic properties that don't exist. It may work, if you state all your properties explicitly, however I am not sure. Still I don't know why this happens.

Your problem is actually a bug in the DriverChain mapping driver. I opened a ticket.

re: acl Posted by Lukas about about 1 month ago.

the idea would be that i could easily specify how permissions are to be set when storing a model for the first time (for example the model instance would be owned by the creator), there should also be an API to modify the access permissions (this might in some cases be part of the workflow process, aka when sending it to an editor one would make owned also by the a certain group). then when doing fetches for that model, the current user's permissions would be added as a filter on the fetch.

as there would need to be different strategies: user ownership (one or many), group ownership (one or many) and of course you would also want to eventually support roles. but the idea is that all the required joins etc are all done automatically by telling the given D2 session the PK of the user doing the fetch.

for caching there might also need to be some smart solution to ensure that you could still sensibly do some caching.

Incredibly good stuff Posted by ornicar about about 1 month ago.

Thanks a lot for the work. Doctrine2 is awesome!

I can't wait to use the trait PHP feature for behaviours. How will Doctrine2 behaviours add methods to Entity classes, until traits are stable? Using a __call() method in the Entity class?

@ornicar Posted by beberlei about about 1 month ago.

This is just an experimental extension to show what is possible with Doctrine2.

This will not be developed any further from the Doctrine2 core team unless one of the developers feels like it. I for myself have enough projects and can't take another one :-)

You are free (and we encourage you!) to use this code as basis to extend Doctrine2.

@beberlei Posted by Colin about about 1 month ago.

For the record, I too would like to be able to inject a custom ClassMetadataFactory rather than have to use reflection. Using reflection is overriding protected properties, at least a setClassMetadataFactory() would be a stable API. My intention is to have the metadata built by merging multiple xml/yaml files from different application modules, which doesn't play well with reflection.

Create Comment