Whats the status here? Do we have any?

Since we moved to Symfony Console I don't think this is needed anymore.
The purpose of this ticket was actually to test our own CLI support, which was dropped.

I'm closing the ticket due to this. Reopen if you have any other comment.

I think we do need some basic functional tests of our Commands, they have been subject to many bugs in the past becaues they are not tested.

Fixed another fatal error in the command due to missing namespace dependency. We need tests for all the commands, there have been dozens of issues on these things so far.

This commit shows a simple approach on how testing is easily possible for symfony commands:

http://github.com/doctrine/doctrine2/commit/51e6681934a7cf4448b85c5670c04045f66c6056

Can we expect some more tests for beta4 or is it unlikely that you find the time? Should we move this further back or does someone else want to step in?

There is another approach for this using several subqueries to build an IN clause, the Paginator extension supports this: http://github.com/beberlei/DoctrineExtensions

I rather go the extension approach than changing the DQL for this feature.

I just saw your second example, that is rather cool though and gets +1 from me.

I had the same idea for "not initialized proxies", i.e.

$em->getUnitOfWork()->initializeProxies('Customer');

Second example is a duplicate of DDC-734

From reading the issue i know what the bug is, indeed this sucks.

@Jon: Any more information coming?

@Benjamin: Can you summarize the essence of the issue shortly?

@Roman: The UnitOfWork (may) still be pushed as a listener into that entity, and still recieve noticies of update. Which may throw notices because the oid hashes are removed everywhere. Additionally you cant serialize the thing because you still got the UoW inside there.

I don't have anymore information currently. The issue was relayed to me. I will try and find some more information and report back.

There is no way to "fix" this issue, i am turning it into a feature request. There needs to be a "postDetach" event that is triggered where the developer can detach the change notification objects.

Updating fix version

not a bug, its a feature (though a dumb one). Improvement in a next version.

Note: If the fist code block is always performed before the last 2 blocks then there is no issue here, just a need to make it more clear in Doc that this is possible but that you should not rely custom implementation as PersistentCollection will be injected when loaded from db.

When this will be fixed?

I think this is related to http://groups.google.com/group/doctrine-user/browse_thread/thread/7e1cfa9c4c99af31

Sounds logic, each driver would expect NULL or data (wrapped specifically for the driver used)

Requirements for batching of eager loads:

1. Since we are using an IN() query for this we can only support this feature for entities that have a single column primary key.
2. If we want to support composite keys we need to build it as WHERE ( (id1 = ? AND id2 = ?) OR (id1 = ? AND id2 = ?)) but this is currently not possible with the way how internally the $criteria array is used.

Next item to think about: What if an exception or event breaks the flow and "triggerEagerLoads()" is never called?

Descheduled to 2.x

Updating fix version

Updating fix version

Removed from master, as i dont like the api at all

Updating fix version

Updating fix version

Is there any fix for this? i have the same problem.

It makes a perfect sense here, I wish it was possible, it would give us a room for even more optimization. Any input on the issue from the developers?

Idea:

Develop a datastructure of sorts that allows saving information about skipping tables and columns therein when reverse engeneering.

This is not possible unless you take advantage of Topological Sorting to map class dependencies like we do inside of UnitOfWork AFTER creating the ClassMetadata.

The necessity of having this is mandatory because we can never skip classes that have associations to other ones though FK.
You may try that, but it doesn't compensate the effort. I'd rather mark this bug as won't fix, but I'm leaving for you do that. =)

Updating fix version

Has to be pushed back as immutable date time cannot be implemented in userland that well.

Updating fix version

Updating fix version

It occurs even if the value is not HIDDEN.

@Christian this sounds like a completly different error, please explain why you think this belongs here or open a new ticket.

Benjamin: Because I get exactly the same error message regardless of using HIDDEN. So HIDDEN doesn't seem to be liable here.

Can you paste your DQL and SQL?

Code:

$queryBuilder
	->select('myEntity, 123456789 AS distance')
	->where('distance <= 10')
;

DQL:

SELECT myEntity, 123456789 AS distance FROM MyCompany\MyBundle\Entity\MyEntity myEntity WHERE distance <= 10

SQL:

SELECT s0_.id AS id0, s0_.a AS a1, s0_.b AS b2, s0_.c AS c3, s0_.d AS d4, s0_.e AS e5, s0_.f AS f6, s0_.g AS g7, s0_.h AS h8, s0_.i AS i9, s0_.j AS j10, s0_.k AS k11, s0_.l AS l12, s0_.m AS m13, s0_.n AS n14, s0_.o AS o15, 123456789 AS sclr16, s0_.p AS p17 FROM myEntity s0_ WHERE sclr16 <= 10

That is expected behavior, ANSI SQL defines SELECT to be evaluated AFTER WHERE.

SELECT 1234 AS foo FROM `test` HAVING foo = 1234

DQL has a HAVING clause as well, not sure it works without the group by. Please try.

So we might just close this issue then?

Its not a bug, just the error message is supposed to be improved (if possible) in a cheap way.

Existing PR at https://github.com/doctrine/doctrine2/pull/361

Example:

$dql = "SELECT new CustomerAddressView(c.id, c.name, a.id, a.street, a.number, a.city, a.code)
             FROM Customer c INNER JOIN c.address a WHERE c.id = ?1";

This supersedes DDC-1819.

1. One additional property in ResultSetMapping => $viewModelClass?
2. Changes to Parser (new ... syntax)
3. Changes to sQL Walker?
4. Changes to Hydration (Only object hydration!)

Roman, I'd like to do this one as I have currently a use case for this. Do you have any idea of how to do this? What I'm wondering is whether it is possible to implement this without user intervention. (This would simply mean "store the entities as they were added"). But this would need another column in DB that we'd have to add within oneToMany / manyToMany relationships, but in this case one could save a serialized array holding "entityId => position" key / value pairs.

Afterwards, one could easily rebuild / reorder the collection via $collection->set($entity, $order[$entity->identifier]);

If you've got another thought about this, please don't hesitate to point me into the right direction!

this won't be implemented until 2.1, since its a pretty complex feature.

Changes are probably required in:

1. CollectionPersister - Add a new collection persister that takes the position into account
2. SchemaTool - Add a 'col_position' column to either the many-to-many or the one-to-many tables.
3. EntityPersister - Use and extend current order-by support to make the sorting happen

You can implement this already though with some performance hit in update scenarios. If you use the ORDER BY support and implement an API around your entity that abstracts those changes and always sets a "position" field on the many entity that is supposed to be sorted.

I don't think we necessarily need a new collection persister. Simply adjusting the ManyToManyPersister to be able to deal with it might be sufficient.

For OneToMany, that is always persisted from the "many" side, thus there is no collection persister, we would need to adjust the normal persisters.

They key element for the user should be a new annotation (or corresponding xml/yaml element) @OrderColumn. By default the order should not be persistent, only when an @OrderColumn annotation is present. The name of the order column can have a default, i.e. "position". Thus this enhancement of persisting the order should be fully backwards compatible.

On another note, the getInsertDiff/getDeleteDiff methods of PersistentCollection should already be "ready" for this. That is, when an element in the collection changed only its position, this is already tracked as a change. However the ManyToManyPersister issues no "UPDATE" queries, it simply deletes and inserts. A position change may be more effectively persisted with an UPDATE.

From a mailinglist entry, required check/changepoints:

1. ClassMetadata of Many-To-Many associations have to be extended to publish the required datastructure to the ORM.
2. All Metadata Mapping Drivers have to be extended
3. Persisters\ManyToManyCollectionPersister has to be extended to save the key in the many to many table if desired by the user.
4. Schema-Tool has to be extended to create the additional column.
5. PersistentCollection has to be extended so that lazy loading of collections with additional key works.
6. Array- and ObjectHydrator have to be extended to allow fetch join of collections with key column.
7. Discuss wheather to support this for One-To-Many also with the key-column on the many side. This is much more tricky internally though.

Push back to 2.x, we will have support for DDC-250 first and for this at a later release.

Hi there,
I'm looking for this feature.

Benjamin Eberlei said that : "You can implement this already", but I don't understand the "how to".

Also,
The problem should be solve if RDBMS had a "natural" order. An order based on item position inside table.

To get this feature without any change on Doctrine, I have remplace the PK defined by the target & mapped field identifier. The new PK is a new field with type "integer" and with auto-increment enable.
In this configuration, Doctrine use the "natural" order of the RDBMS. And I can change order of my item inside Collection and persist it.

It's an very bad solution, but It work before an official support.

Waiting for advices, and solutions,

Thomas.

Answering to Benjamin Eberlei on the "7. Discuss wheather to support this for One-To-Many also with the key-column on the many side. This is much more tricky internally though.".

I think that for One-To-Many relations, if user want to store the collection order, Doctrine can store the One-To-Many as Many-To-Many with a "model" limitation.
In that case, if storing order collection for Many-To-Many work, it should work for One-To-Many.

What do you think about it ?

I think that it must be possible to have two keys ordering : the order isn't obligatory reversible.

For exemple with user and group :

• You can order groups for one user : with preference by exemple, or importance.
• And with a different order, users for a group : rank by example.

And maybe more, if you decide to add multi-order : an user show group by his rank in it, if his rank is identical, the order is make by love preference, and after by the importance given by the user (not necessary a number, if we imagine filter on them).

So a default order can be choice with parametized fields and could be :

 
@ManyToMany(targetEntity="Group")
...
@JoinFields ( rank: { type: int} , preference:{type:int}, importance:{type: string, length: 40} )
@OrderByJoinFields({"rank" = "ASC", "preference"="ASC", "importance"="ASC" } )

In this case the order must be optional and would be clean if another order appears in the same scope (DQL...). And manytomany became virtual entities act as other entities except they don't appears permetting in the same time a better conception.

So if the solution take in DDC-181 will become the only solution. This would a good idea to document this. Because, this seems to me a very important point.

My last point is even an unique ordering field created in the join table will be a big and usefull improvement.

Thank a lot for your beautiful work.

In my point of view, a collection can be order in a single way only.
If you want to add more than one order between User & Group, it's a new collection, a new relation.
Like :

• User.memberOf() : Group[]
• Group.members() : User[]
• Group.importantMembers() : User[]

And it's your role to keep a consistency between members & importantMembers array.

Because ManyToMany join table is the reflection of a state of an ArrayCollection. It's not a usefull feature to be able to store all of the state of an ArrayCollection, even the order of this Array. It's just a normal feature that is really missing

Thomas.

I don't think:

If you have three collection, you duplicate one relation 3 times and it's easy in consequence to lost the data integrity and unicity.

By example :

• Thomas have rank 10 in Admin
• Thomas think the admin group has importance noted 3 on all of his groups.
• If a responsable of admin group decide to delete Thomas from it.
• Thomas, in his ordered list of groups, think always to be in group admin.

So in my idea, the many to many relation isn't just an array collection, but should be an virtual entity. In UML or in Merise method this is a common problem to have a parametized relation. I think an orm should just implement this.

Hum,
I agree with you.. In a SQL Schema, it's a good choice to add many fields in a ManyToMany join table to description "order".

I just want to add a piece of Doctrine ORM Document :

"When working with collections, keep in mind that a Collection is essentially an ordered map (just like a PHP array). That is why the remove operation accepts an index/key. removeElement is a separate method that has O ( n) complexity using array_search, where n is the size of the map."

Hi there,
After several discussions. on IRC, I have changed my point of view.

Doctrine Documentation says : "When working with collections, keep in mind that a Collection is essentially an ordered map (just like a PHP array)".
So, I think that Doctrine have to be able to store or not the order of a Collection. By adding a new field on the Joined table to store the position of each elements.

But I not agree with @Nicolas. Because in his case, he's talking about Association Class : http://etutorials.org/Programming/UML/Chapter+6.+Class+Diagrams+Advanced+Concepts/Association+Class/
Because he's talking of a business logic, he's talking of a dedicated Entity class.

What do you think about it ?

Thomas;

Any news ?

Hi, any news on this?

If I may add any info to this feature request, maybe something like JPA/Hibernate could be a good start?

See http://docs.oracle.com/javaee/6/api/javax/persistence/OrderColumn.html or http://docs.jboss.org/hibernate/core/3.6/reference/en-US/html/collections.html#collections-indexed

The idea in Hibernate is that you persist the order of the list in an extra column. This column is not a field of the entity however.

Questions

• I suppose link and unlinked entities would then handled by UnitOfwork commit also?
• Since the collection is not initialized, one does not know upfront if the action will be successful, what happens if:
  • an entity is linked with a collection, although they are already connected.
  • an entity is unlinked from a collection it is not in.

Regarding the naming, i like link/unlink.

What do you mean by "handled by UnitOfWork commit" ? Whether the SQL is "scheduled" or executed immediately? Interesting question.
Scheduling would probably be better but also more difficult.

As far as usage is concerned, I currently imagine it as follows:

// EntityManager#link($sourceObj, $field, $targetObj)
$user = $em->getReference($userId); // $userId probably from request parameters
$address = $em->getReference($addressId); // $addressId probably from request parameters
$em->link($user, 'addresses', $address);

"What happens if: an entity is linked with a collection, although they are already connected."

Probably an SQL error which results in an exception from the driver. Depends on the database constraints though.

"What happens if: an entity is unlinked from a collection it is not in"

Probably nothing, at least not from the SQL side. An exception could be thrown from Doctrine itself if the update affected 0 rows.

Thanks for these initial questions. Thats definitely food for thought. Keep it coming.

Pushed back.

If this is to be included in 2.0 it needs to happen for RC1. However, it is not clear yet whether it will be done in time.

How would you solve this Roman? I thought of adding a query hint so that the postLoad inside unit of work is not triggered and gathering all the entities that have a post load event in an array inside the object hydrator, then iterating it after taking the snapshots of all collections inside hydrateAll

@Benjamin: Not sure what you would use the query hint for but in general that is the approach I had in mind, yes. You can't get around iterating over the entities after the actual hydration.

The query hint would do something like:

        //TODO: These should be invoked later, after hydration, because associations may not yet be loaded here.
        if (isset($class->lifecycleCallbacks[Events::postLoad]) && !isset($hints['hydrationPostLoad'])) {
            $class->invokeLifecycleCallbacks(Events::postLoad, $entity);
        }
        if ($this->evm->hasListeners(Events::postLoad) && !isset($hints['hydrationPostLoad'])) {
            $this->evm->dispatchEvent(Events::postLoad, new LifecycleEventArgs($entity, $this->em));
        }

another way would be to move that code out of UoW::createEntity completly and have the persisters call it when they use that method.

Leaving that code in UoW does not make sense to me, if it is moved, it needs to be moved completely. Why do you think the persisters should do it? Initially I thought collecting the affected entities during hydration and then when hydration is done iterating over them and triggering the postLoad events.

Yes but postLoad has to be triggered for non Hydrated entities (i.e. Persister) also

Moved back

Hi Gyus, I need access to associations in postLoad or similar event, and my idea is to dispatch new event after full initialisation of object, what do You think about it? If I can help please let me know It's important for me.

Now in my PostLoad access to associations is work fine. Why this issue is still in Unresolved status?

What do you (Roman and Benjamin) think about adding postHydrate event which would be called within ObjectHydrator::hydrateAllData() on every entity collected during hydration? I could prepare a patch for this. I personally think this would be better than adding a hint that changes behaviour of postLoad event.

In this implementation Schema-Tool would generate a table:

elementTable (entity_id-1, ..., entity_id-n, key, value) and using the Platform Type Generation of keyType and valueType

Column Names should be Change-able also since there could be people who name their primary keys "key" and "value" o_O

Ordering could be implemented on top of this using the @OrderColumn JPA implementation by adding another column to the table with a numeric order that will be "order by"'d on select time.

Pushed back

@Roman A possible fix for this in my opinion is another map in UnitOfWork $mergedEntities = array(); and a patch like this:

diff --git a/lib/Doctrine/ORM/UnitOfWork.php b/lib/Doctrine/ORM/UnitOfWork.php
index 242d84b..1d0d8b3 100644
--- a/lib/Doctrine/ORM/UnitOfWork.php
+++ b/lib/Doctrine/ORM/UnitOfWork.php
@@ -1340,6 +1340,10 @@ class UnitOfWork implements PropertyChangedListener
             return; // Prevent infinite recursion
         }
 
+        if (isset($this->mergedEntities[$oid])) {
+            return $this->mergedEntities[$oid];
+        }
+
         $visited[$oid] = $entity; // mark visited
 
         $class = $this->em->getClassMetadata(get_class($entity));
@@ -1468,6 +1472,8 @@ class UnitOfWork implements PropertyChangedListener
 
         $this->cascadeMerge($entity, $managedCopy, $visited);
 
+        $this->mergedEntities[$oid] = $managedCopy;
+
         return $managedCopy;
     }

I have tested this patch with my application and it fixes the problem in all my relevant test cases apart from one. The test case that's failing is one that persists a bi-directional many to many relationship, so the associations interweave with each other (if you know what I mean).

I wonder if perhaps doMerge need to continue cascading even if it finds an item in $this->mergedEntities

This is the Flextrine code that fails - it results in no entries in movie_artist. This might also be related to DDC-758?

m1 = new Movie();
m1.title = "Movie 1";

m2 = new Movie();
m2.title = "Movie 2";

a1 = new Artist();
a1.name = "Artist 1";

a2 = new Artist();
a2.name = "Artist 2";

m1.artists.addItem(a1); a1.movies.addItem(m1);
m1.artists.addItem(a2); a2.movies.addItem(m1);

m2.artists.addItem(a1); a1.movies.addItem(m2);
m2.artists.addItem(a2); a2.movies.addItem(m2);

// These translate to cascade merges on the server
em.persist(m1);
em.persist(m2);
em.persist(a1);
em.persist(a2);

// Now flush
em.flush();

P.S. This test passes if I translate em.persist() to $em->persist() (not cascading) on the server instead of translating it to a cascade merge; not sure if that helps

I'd really like to avoid introducing an additional instance variable just to solve this issue but I did not find the time yet to really look into it.

Does someone have a unit test for this already and can attach it to the issue?

Rescheduling for RC1.

Here is a functional test case containing three tests:

testMultiMerge tests basic merging of two new entities, checking that only a single entity ends up in the database. This passes with Benjamin's patch.

testMultiCascadeMerge tests the more complex case of merging a OneToMany association. This also passes with Benjamin's patch.

testManyToManyPersistByReachability tests the ManyToMany case described above and this fails with Benjamin's patch, probably because doMerge doesn't cascade down entities that it has already merged and some ManyToMany associations are being ignored. Its a bit hard to be certain what is causing this as even without Benjamin's patch this test would fail due to DDC-758.

@Roman i thought about this issue, its not possible without that additional map of merged entities. There is no way we can get that information from other sources.

Problem is rather that the use-case probably only applies in mass-merging scenarios and client-server serialization.

Added another failing test case - adding the same entity from different ends of a many to many bi-directional association to check that there isn't an integrity constraint violation caused by Doctrine trying to add the same row twice.

Attached a patch for this issue.

can you comment why all the additionall stuff is necessary compared to my patch?

It fixes the two additional test cases - testManyToManyPersistByReachability and testManyToManyDuplicatePersistByReachability.

testManyToManyPersistByReachability was failing with your original patch because there are ManyToMany cases where an entity may have already been merged, but its still necessary to add it to an association and continue to cascade. Running the following with the original patch will miss out some of the associations.

$m1 = new Movie();
$m1->title = "Movie 1";

$m2 = new Movie();
$m2->title = "Movie 2";

$a1 = new Artist();
$a1->name = "Artist 1";

$a2 = new Artist();
$a2->name = "Artist 2";

$m1->artists->add($a1); $a1->movies->add($m1);
$m1->artists->add($a2); $a2->movies->add($m1);
$m2->artists->add($a1); $a1->movies->add($m2);
$m2->artists->add($a2); $a2->movies->add($m2);

$em->merge($a1);
$em->merge($a2);
$em->flush();

The other change in my patch is to protect against this case. It ensures that the following code doesn't add the same entity twice to a collection.

$em->merge($m1);
$em->merge($m2);
$em->merge($a2);
$em->merge($a2);
$em->flush();

I am not sure if the issue here is rather multiple calls to merge that contain different parts of the same object-graph.

There should be a very simple fix for this, call ->clear() after each merge.

I am not sure if this patch drags us into a blackhole of issues with merging.

Calling ->clear() and ->flush() after each merge is a workaround for the simple case, but unless I am misunderstanding I don't think its a solution for cases where the merging is happening automatically in cascadeMerge. I've actually encountered this issue in another project and scenario to do with creating REST APIs and merging JSON objects into entities, and applying the patch fixed it so a) I think this issue might be a more common that we first thought and b) the patch basically seems to work (plus it doesn't introduce any failing cases in the existing test suite). I can actually still find one edge case to do with cascading merging interlinked many to many associations that this doesn't fix, but I was planning to open that as a new ticket after this My feeling is that the current merge already has issues and this definitely improves it.

It cannot happen inside a single merge, single merges use the $visited to avoid infinite recursions, each entity can only be merged once inside a single merge operation.

Added a note into the documentation about using EntityManager#clear between merging of entities which share subgraphs and cascade merge.

Handling this issue in UnitOfwork will be declared an improvement, not a bug anymore and be scheduled for later releases. The required changes to the core are to dangerous and big.

Where in the docs is that?

Just to summarize, the equivalent operation to having multiple merges and a single flush is to call merge followed by flush each time, with the whole thing surrounded by a transaction? Does this have a big impact on performance?

Ben - even given the decision not to implement this (and I do understand your thinking, as it is a major change), is there any reason not to implement the bit that ensures that the same entity isn't added to a collection twice during a merge? I can't think of a situation where this should be allowed, and I have a use case where I get 'DUPLICATE KEY' errors if this isn't there.

Please see attached patch.

What bit of that huge patch is that? Can you extract it into another ticket if thats possible?

I added it to "Working with Objects" and the descripton of Merge. Its not yet live on the site.

Using this current workaround has a performance impact, since more SELECT statements have to be issued against the database.

Apologies for not being clear - only the 3rd patch (multipleaddmerge.diff) is relevant to the 'DUPLICATE KEY' error I am now talking about, but I'll put it in a nother ticket if you prefer.

please add a new ticket, patch looks good.

Created as DDC-875

This very minimal patch removes the use of these temporary files:

--- library/Doctrine/ORM/Proxy/ProxyFactory.php	(revision 2)
+++ library/Doctrine/ORM/Proxy/ProxyFactory.php	(working copy)
@@ -78,9 +78,8 @@
         $fqn = $this->_proxyNamespace . '\\' . $proxyClassName;
 
         if ($this->_autoGenerate && ! class_exists($fqn, false)) {
-            $fileName = $this->_proxyDir . DIRECTORY_SEPARATOR . $proxyClassName . '.php';
-            $this->_generateProxyClass($this->_em->getClassMetadata($className), $proxyClassName, $fileName, self::$_proxyClassTemplate);
-            require $fileName;
+            $file = $this->_generateProxyClass($this->_em->getClassMetadata($className), $proxyClassName, null, self::$_proxyClassTemplate);
+            eval('?>'.$file);
         }
 
         if ( ! $this->_em->getMetadataFactory()->hasMetadataFor($fqn)) {
@@ -144,6 +143,9 @@
 
         $file = str_replace($placeholders, $replacements, $file);
 
+        if ($fileName === null)
+            return $file;
+
         file_put_contents($fileName, $file);
     }
 

The proxy dir is used for the "doctrine orm:generate-proxies" command in the case of "autogenerate= false", so you need to define it anyways.

You have to use proxies, the option is not for Proxy yes/no. If you have autogenerate=false and doctrine requires a proxy for a use case but can't find it you will get a fatal error.

I just saw the eval() keyword, ieeks It could maybe be a convenience option for those that don't want to use proxy direcotiries, however i am not sure.

eval() is no different than writing code to a file and using require().

When using runtime-generated proxies, there are no benefits (that I know of) from writing them to a file. The disadvantages are:

• slower because of write disk access
• has problems with high concurrency, unless special care is taken (DDC-716)
• potentially has permission problems if code is executed by different users (e.g. nobody for a daemon and www-data for http)
• requires setup of a writable directory

This is a nicer patch, which makes _generateProxyClass() return a string, just like other _generate* methods:

--- library/Doctrine/ORM/Proxy/ProxyFactory.php	(revision 2)
+++ library/Doctrine/ORM/Proxy/ProxyFactory.php	(working copy)
@@ -78,9 +78,8 @@
         $fqn = $this->_proxyNamespace . '\\' . $proxyClassName;
 
         if ($this->_autoGenerate && ! class_exists($fqn, false)) {
-            $fileName = $this->_proxyDir . DIRECTORY_SEPARATOR . $proxyClassName . '.php';
-            $this->_generateProxyClass($this->_em->getClassMetadata($className), $proxyClassName, $fileName, self::$_proxyClassTemplate);
-            require $fileName;
+            $code = $this->_generateProxyClass($this->_em->getClassMetadata($className), $proxyClassName);
+            eval($code);
         }
 
         if ( ! $this->_em->getMetadataFactory()->hasMetadataFor($fqn)) {
@@ -107,19 +106,19 @@
         foreach ($classes as $class) {
             $proxyClassName = str_replace('\\', '', $class->name) . 'Proxy';
             $proxyFileName = $proxyDir . $proxyClassName . '.php';
-            $this->_generateProxyClass($class, $proxyClassName, $proxyFileName, self::$_proxyClassTemplate);
+            $code = $this->_generateProxyClass($class, $proxyClassName);
+            file_put_contents($proxyFileName, "<?php\n" . $code);
         }
     }
 
     /**
      * Generates a proxy class file.
      *
-     * @param $class
-     * @param $originalClassName
+     * @param ClassMetadata $class
      * @param $proxyClassName
-     * @param $file The path of the file to write to.
+     * @return string The code of the generated methods.
      */
-    private function _generateProxyClass($class, $proxyClassName, $fileName, $file)
+    private function _generateProxyClass(ClassMetadata $class, $proxyClassName)
     {
         $methods = $this->_generateMethods($class);
         $sleepImpl = $this->_generateSleep($class);
@@ -142,9 +141,9 @@
             $methods, $sleepImpl
         );
 
-        $file = str_replace($placeholders, $replacements, $file);
+        $file = str_replace($placeholders, $replacements, self::$_proxyClassTemplate);
 
-        file_put_contents($fileName, $file);
+        return $file;
     }
 
     /**
@@ -244,8 +243,7 @@
 
     /** Proxy class code template */
     private static $_proxyClassTemplate =
-'<?php
-
+'
 namespace <namespace>;
 
 /**

Scheduled usage of eval() for 2.1, if the following conditions exist:

1. Autogenerate is set to TRUE
2. No Proxy Directory is configured.

Great, this is even better. This way you can have 1) autogenerated in ram-only, 2) autogenerated in files and 3) pregenerated.

And the minimal amount of config needed to get up and running is reduced, which is always nice.

you should know though, eval is dead slow. It generates the necessary proxies on EACH request and that cannot be cached in APC.

It's no slower than current autogeneration (file_put_contents+require). TBH, I don't know why anyone would want to use that over eval(), but I don't mind it being there.

Pre-generation is, of course, a different thing. Seems like a valid tradeoff to offer: build/a bit of config/better perfomance vs. no build/no config/potentially slower.

Yes, that is because file_put_contents + require is a development only strategy. The manual clearly states that autogenerate has to be false in production.

Using eval() instead of producing and requiring the file in the case of enabled auto-generation of proxy classes sounds like a good improvement for 2.1 to make proxies more transparent during deveopment and for anyone for whom performance is no issue.

I'm increasing the priority as I think it is easy to implement for 2.1 and a good enhancement.

A note on why having the proxies written to a file can be useful even with autogenerate being on: it makes it really easy to check the proxy code being generated. I use that a lot currently.

The solution suggested, giving three possibilities is cool, though.

If this is to be implemented for 2.0, it needs to happen for RC1, therefore rescheduling to RC1. Feel free to reschedule to 2.x if necessary.

Only oracle supports lock timeouts and no other vendor seems to plan to support it. I move to 2.x, but i guess this would rather be an issue of user extension.

Ubuntu 10.04 + Oracle 11g Entreprise + PHP 5.3.2 + Doctrine2 Git (up-to-date)

This is the continuation of http://www.doctrine-project.org/jira/browse/DDC-616. Only the schema is different.

just for understanding this scenario:

Is this a One-To-One relation and the TABLE_TEST2 "inherits" the primary key from its parent TABLE_TEST1?

If yes, this construct is not yet supported by Doctrine 2, we still need to include an ID-Generator that supports this kind of schema.

Change for a more understandable use case. Note that it's not my real use case and that I work on legacy database on which I can't change the structure.

updated the issue topic to get a better grasp of what needs to be done here.

I have the same error with Mysql whit the same condition.

More details on the work to be done:

The relevant code is in Doctrine/ORM/Mapping/Driver/DatabaseDriver.php only.

The idea is currently many-to-many tables are detected by checking that the table has foreign keys on all the primary key columns (no additional columns!)

Now with the 2.1 feature of foreign key/primary key entities this is not necessarily true anymore. You can have the primary keys being foreign keys BUT have additional columns that are not part of the primary key. This has to be detected.

If a foreign key-primary-key entity is found that has additional columns a ClassMetadata has to be created and the associations have to be created with the "id" => true flag in mapManyToOne().

For what it's worth, I'm getting this error when I have a PK that is a single column and not a FK.

PRIMARY KEY (`id`),
UNIQUE KEY `cycle_station_id` (`cycle`,`station_id`),
KEY `station_id_idx` (`station_id`),
KEY `readings` (`readings`),
KEY `source` (`source`),
KEY `temperature_min_max` (`temperature_max`,`temperature_min`),
KEY `station_id_cycle` (`station_id`,`cycle`,`updated_at`),
CONSTRAINT `compiled_1_station_id_stations_id` FOREIGN KEY (`station_id`) REFERENCES `stations` (`id`),
CONSTRAINT `compiled_1_station_id_stations_id_1` FOREIGN KEY (`station_id`) REFERENCES `stations` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=160833690 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci

Might need to be pushed back to a 2.0.x / 2.x.x bugfix release. Not clear yet.

Doctrine_Record defines a replace() method.

In the MySQL Doctrine implementation, however, it is not the same as INSERT .. ON DUPLICATE KEY UPDATE. The replace() method implemented in Doctrine_Connection_Mysql uses the REPLACE INTO syntax, which is a DELETE and then INSERT when the key exists. This is fine, except for tables that use auto-increment fields. The delete-then-insert operation yields a new auto-incremented value, whereas INSERT .. ON DUPLICTATE KEY UPDATE would not.

MySQL (and SQLite) REPLACE is a no go. It causes way too much disc I/O and worse yet totally screws up the on disk data structures because of the deleting.

Scheduled for 2.2

Evaluating this makes me sad, except MySQL support for this is rather non-existant, and the oracle merge is aiming at batch operations.

Should this be done with

1. Select first, then insert
2. Catch and evaluate exception then update

I am leaning towards 1.

Updating fix version

formated snippets nicely

I need this feature too.

But I would suggest using the same annotation used by JPA

@Embeddable

+1

You should also take into consideration different storage strategies of ValueObjects.

Martin Fowler points out - in „PoEAA" - two approaches: Embedded Value (which is the one presented above) and Serialized LOB .
Both have their pros and cons, that's why Doctrine2 should give developers choice of selecting the fittest solution.

Of course technically we can similate a serialized LOB with a new Doctrine 2 type.

I don't like that idea - Its so not generic.

VO as a pattern is important building block of domain model, which clearly indicates that VO as a feature of Doctrine2 should be tailor-made.

To anyone of dev-team reading this issue: without VOs Doctrine is not yet DDD-ready, please hurry

Serialized LOB is not very useful IMHO and has lots of problems (many mentioned in PoEEA already).

@Alan: I appreciate your nice reminder and I'm sure you mean it in a friendly way, but please keep in mind that noone is paid to work on this project. It all happens in free/spare time and the current state of the project already consumed at least 1 1/2 years spending many hours weekly on this project from me alone. Not to speak of the others.

Thus, there is no point in demanding something or telling us to hurry. The best way to get a feature in is to provide a (good) patch that we find worth including.

I started to add notes to this issue to collect all the things that need to be done for this feature.

In the meantime, its not too hard/ugly to get a half-way decent embedded value yourself:

/** @Entity @HasLifecycleCallbacks */
class Foo {
    // annotations not shown
    private $id;
    private $embedded;
    private $value1; // never reveal to public
    private $value2; // never reveal to public
    private $value3; // never reveal to public

   public function getEmbedded() {
       return $this->embedded;
   }

   public function setEmbedded($embedded) {
       $this->embedded = $embedded;
   }
   
   /** @PrePersist @PreUpdate */
   function _destructEmbedded() {
       // destruct $embedded into $value1, $value2, $value3
   }

   /** @PostLoad */
   function _constructEmbedded() {
      // construct $embedded from $value1, $value2, $value3 
   }
}

Several variations of this are possible, also with an external event listener instead of callbacks but in that case you might need to use reflection to get at the values.

I want to share my thoughts on possible VOs collections implementations.

1. As it was mentioned earlier serialized (C)LOB is one solution. Implementation of storing/retrieving object graphs alone is quite simple, but it's complex in terms of SELECTs with conditions.
Composing SQL condition would result in some nasty constructions e.g. vo_collection_column LIKE '%foo%bar%' which output format would depend on serialization target (CSV, XML, YAML, PHP serialized objects etc.). Also in most cases it would be impossible to obtain eligible result.

I'm not taking Regexp or XPath operators into consideration as only few RDBMS support them.

2. The second solution is to break VOs graph into separate related table... or tables if we consider that VO can contain another VO(s). It's not so fast as serialized LOB but more flexible and it utilize power of RDMS,
But there is one catch: Doctrine2 must preserve nature of VO. To make it happen during Entities persisting - if any change in dependant VOs graph has been made - all associated VOs rows in database should be deleted and the new/changed VOs graph should be inserted in their place.
I know it could be inefficient while dealing with large object graphs, yet faster than comparing VOs one-by-one.

In conclusion:
serialized LOB is extremely fast in CRUD-like operations on aggregates, however very search unfriendly.
Separate ValueObjects tables are better where serialized LOB lacks, but slower in exploitation.

I can't tell which approach is superior, because each of them is valid under different circumstances.

Hope this helps.

@Alan: I appreciate your nice reminder and I'm sure you mean it in a friendly way [...]

Of course I do.

It would be easy to implement value objects in userland using the XML capabilities of many RDBMS:

1. Implement an Xpath function on the Dql Parser
2. Implement a User-Defined Type for each value object that handles the translation from and to XML.

The second point can be heavily optimized when value objects are immutable with an own identiy map of value types inside the Type flyweight instance.

I more or less suggested something similar above.

ah, my bad - i must have overseen this

+1

This would be awesome.

Don't forget (especially with regard to SLOBs) that values might in turn contain references to Entities.

Example: An "Order" might be an @Entity and might have a field (an array) of OrderLineItems as value. Each OrderLineItem might e. g. carry quantity or disconunt and references a Product (@Entity).

So even if you don't need the traversal from Product to all the Orders it is contained in, serializing the OrderLineItems needs a way to "cut off" the object graph at the transition towards the Product but must place some kind of referral there so that upon unserialization (of the OrderLineItem list, that is, during Order load) the Product references in every OrderLineItem are at least initialized with proxies again.

Don't know whether/how referential integrity (OrderLineItems <-> Products) would make sense or could be implemented here.

Pushed back to 2.x, this feature is probably the largest feature request we have and we'd rather focus on small improvements for 2.1

Several thinks to consider/not to oversee here:

1) There are value objects with identity. I know that is not DDD-conform but only at first sight. It means they are technically entities but are treated like VOs.
Common examples are Zipcode or country. As they have identity (e.g. Zipcode: de-40723) they are entities but are created and interchanged like normal VOs.
On the google DDD-List they were often referenced aS Lookup Entities.

2) In virtually all (business) cases a collection of VO is an Entity. How else could you reference (add or remove) single elements of that list?
There are exceptions here like a undefinded number of VOs in a collection, but in that case you can only add or remove a quantity of it.
As a true collection (say 3 addresses for a client = Entity ClientAdresses) you would have to give them some kind of identity, even if it is only having
a sequential number in that collection.

@Matthias: OrderLineItems is an example of actually being an Entity.

Hi guys. I face this in my own way. Hope you won't wake up your neighbours with loud laugh.

Every @Entity extends my BaseEntity object which provide kind of wrap for value with ValueBase object. So when want to get/set value from entity you call $entity->getData() where you won't get value "data" but wrapping ValueBase for value "data". Then you can get bare value by getValue(). Name of value class is in annotation and would be child of ValueBase.

There's also parent class Base for EntityBase and ValueBase. In my case class Base is something like HTML element. So in the end you can use $entity->renderHtml() or $value->renderHtml() no matter if you're rendering value or @Entity. There's more features like validation, filtering and hydration value/entity from HTML forms, but it's extra.

Implementation:

 
 /* @MappedSuperclass */
abstract class Base {
  /* there're methods like _getParent(), _getPropertyName(), etc. used in code behind */
}

 
abstract class ValueBase extends Base { 
   public function getValue() {
        return $this->_getParent()->{$this->_getPropertyName()};
   }
   
   public function setValue($value) {
        $this->_getParent()->{$this->_getPropertyName()} = $value;
   }
}

 
/** @MappedSuperclass */
abstract class EntityBase extends Base {
    public function __call($name, $arguments) {
        /* get property object */
        $pattern = '/^get(.*)$/u';
        preg_match($pattern, $name, $matches);
        if (isset($matches[1])) {
            $propertyName = lcfirst($matches[1]);
            return $this->get($propertyName);
        }

        /* set entity */
        $pattern = '/^set(.*)$/u';
        preg_match($pattern, $name, $matches);
        if (isset($matches[1])) {
            $propertyName = lcfirst($matches[1]);
            return $this->set($propertyName, $arguments[0]);
        }
    }

    public function get($propertyName) {
	    $property = $this->_getElementProperty($propertyName);

	    if ($property == null)
		throw new Exception(sprintf("There isn't property like '%s'.", $propertyName));

	    /* for collections and entities */
	    if ($property["type"] == "collection" || $property["type"] == "entity") {
		$element = $this->{$propertyName};
		if ($element != null) {
		    $element->_setParent($this);
		    $element->_setPropertyName($propertyName);
		} elseif ($property["type"] == "entity") {
		    $element = new $property["class"];
		    $element->_setParent($this);
		    $element->_setPropertyName($propertyName);
		    $element->_setNullEntity();
		    $this->{$propertyName} = $element;
		}
		return $element;
	    }
	    else {
	    /* for values */
		if (!isset($this->_loadedEntities[$propertyName])) {
		    $this->_loadedEntities[$propertyName] = new $property["class"]($this, $propertyName);
		}
		return $this->_loadedEntities[$propertyName];
	    }
    }

    public function set($propertyName, $value) {
        $property = $this->_getElementProperty($propertyName);

        if ($property == null)
            throw new Exception(sprintf("There isn't property like '%s'.", $propertyName));

        /* for collections and entities */
        if ($property["type"] == "collection" || $property["type"] == "entity") {
            $this->{$propertyName} = $value;
        }
        /* for values */ else {
            throw new Exception(sprintf("Can't call set on value property '%s'.", $propertyName));
        }

        return $this;
    }
}

Note that there's something i call "NullEntity". Instead of getting bare "null" you'll get @Entity child of EntityBase, where is set property nullEntity. Then there's posibility to work with null entity (for example renderHtml with empty inputs).

It would be nice, if this is support by Doctrine natively, because i have some performace problems with my implementation. If it's interest in my whole code i can send you. But of course there's some security holes so i'll send it privetely. Thanks for understand and for Doctrine of course.

Note that Roman's workaround presented here does not work.

   /** @PrePersist @PreUpdate */
   function _destructEmbedded() {
       // destruct $embedded into $value1, $value2, $value3
   }

Doctrine tracks changes and does not perform updates when no changes are found. $embedded is not mapped, so it's not tracked and won't be taken into account by Doctrine when updating. Therefore, if $embedded is the only value that was changed, the PreUpdate event won't be triggered.

The easiest thing to do is to simply destruct the VO on every mutation:

   public function setEmbedded($embedded) {
       $this->embedded = $embedded;
       $this->_destructEmbedded();
   }

The downside is that you need to remember to call the method in every setter, but apart from that, there are no side effects, it always works and it's just one line of code

_constructEmbedded() keeps working as is, postLoad will always be triggered.

Hi,

This feature would be awesome !
VOs are really essential in a good domain design.

If you plan to implement this, please remember that you can have nested VOs.
Take the design for a Booking process for instance, you would have a DateRange object embedding two DateTime objects (in the simplest case).

I have no doubts that you've already took this in consideration, but i prefer pointing this out, just in case

work has been started, https://github.com/doctrine/doctrine2/pull/265

Does the new "complex sql types" feature help here - I mean, could that be used to map a value object to more than one column in the database?

@Benjamin Eberlei The request seems to be closed in the link you provided! Does that mean that this feature won't be implemented?!

songoko songowan no, it just probably wasn't the correct way of implementing this

I'm curious if any effort is currently being put into this. I would really love to have this feature available.

Daniel Pitts this is being developed in DDC-2374 ( https://github.com/doctrine/doctrine2/pull/634 )

During development, I tried to have the out-of-the-box ORM layer handle as much of the queries as possible, essentially I used the Repository functions a lot:

For example, having a specific Repository extend the Doctrine EntityRepository and do something like:

 
    public function findByName($name)
    {
        $criteria = array('_name' => $name);
        return parent::findBy($criteria);
    }

Now all functionality is developed, I am optimizing performance and I find myself having to refer my Repository to my DAO layer which uses DQL, so I can enable the DQL Result Cache...

 
    public function findByName($name)
    {
       //Use the DAO so we can enable DQL ResultSet caching
        return $this->_getDao()->loadByName($name);
    }

It would be nice to be able to configure 'DQL Result Cache = on' on Repository level as well...

Ubuntu, PHP 5.3.2, MySQL

Verified, i just don't understand why you are using a one-to-one relation and then "deactivate" the database constraint for this. You could easily use Many-To-One

You are right. It's not a bug, it's feature.

This might still be a good improvement to allow the flexibility, but its not a bug. Updating to "Minor Improvmenet for 2.x"

Debian LAMP - PHP5.3 - Apache 2

Updating fix version

Also, documentation for post* methods is broken at the website:

"Changes in here are not relevant to the persistence in the database, but you can use this events to"

It cuts off in mid-sentence.

RDohms, this paragraph was already sorted out.

The actual ticket is still valid here.

Updating fix version

The problem is, full support for soft-delete throughout the system is not feasible and very fragile. Simple soft-delete through skipping the delete operation is the easiest part. Then you will probably want to modify all DQL queries so that they adhere to it automatically and then there will always be still queries that do NOT go through DQL, like even internal lazy-load queries or native queries or others, which would need to be modified also.

To sum it up, implementing soft-delete "inside" doctrine is absolutely not worth the effort and imho a bad idea and I'm certainly not willing to make lots of adjustments to the core that have a negative impact on performance just to make this soft-delete possible.

I really recommend handling "soft" deletes yourself, the normal way, by simply abstracting entity retrieval and persistence through a DAO/repository layer. As a nice side-effect you get less magic and it still works when you swap out doctrine for another persistence provider.

I am willing to add support for skipping deletes and maybe some other operations through events but I'm not willing to go any further, as explained above.

Rescheduled for 2.1. Might be 2.x.

Reschedule for 2.x

I think this relates to the usage of Custom Persisters

Rescheduling for beta3.

Pushing back to beta4.

Moved to 2.1 due to lack of time for any larger new features for 2.0.

implemented this in a feature branch for now, it really doesnt touch any other runtime code so maybe we can still merge this before RC1

http://github.com/doctrine/doctrine2/tree/OverridePersisters

Is this forgotten? you should merge it since it does not affect any other parts of ORM, this is a great feature

This has not been forgotten, but the Persister is due for a heavy refactoring for 2.2 probably, when we will make it use the SQL Query object that we are working on.

So I cannot merge this, because the API will probably break big time.

Does that mean we will not see this feature before 2.2?

Yes, that is correct. I dont want to add it as experimental/undocumented feature because people will take it for granted and make us responsible for possible bc breaks.

I will update the target version accordingly.

Sorry for disappointing you, but this feature is fundamentally important at the core of the library. That means we have to get it right and not rush into it.

Just as I thought that first you will want to make a query builder object for all persisters. since now they use plain sql. Thanks for all your work on this

I might be mistaken, but AFAICS mentioned Persister heavy refactoring did not made through to 2.2 version. Is there any plan to have it in 2.3 or at any later stage?

@Adam I refactored all Persisters optimizing their code, but I could not complete the move from SQL string generation to Doctrine\DBAL\Query.
We missed it, yes. I may reschedule for 2.3

Why is it still missing in 2.3? I would require this for an extension that uses its own overridden entity persister and using a custom persister is the solution that you guys recomend for not overriding the entity manager.

Any change seeing this soon? I really need this for a security feature.

What is making this so hard? just adding an setEntityPersister($entityName, $object) should do the trick.
I don't need any fancy stuff, just a way to limit the fields in the SELECT list.

Edit: OK, I'm shot I CAN NOT overwrite the entity manager as the UnitOfWork is private!
Got any other idea?

Any chance you could add this quickly? I need this feature urgently to complete an extension using a custom persister. Thanks in advance.

Any news?
Why query hint? What about temporary switching like fetch mode changing via query object?

Any news on this?

I think this is a must have feature. Thanks for all your work.