irrelevant

Hi

Could you please attach your entities or a failing test case ?

Cheers

sure

LineRelated.php :

class LineRelated
{
    /**
     * @ORM\Id
     * @ORM\Column(type="integer")
     * @ORM\GeneratedValue(strategy="AUTO")
     */
     protected $id;

    /**
     * @ORM\ManyToOne(targetEntity="Line", inversedBy="line_related")
     * @ORM\JoinColumn(name="line_id", referencedColumnName="id",nullable=false)
     */
     protected $line;
     
    /**
     * @ORM\Column(name="line_id_related", type="integer")
     */
    protected $line_related;

    /**
     * @ORM\Column(type="smallint",nullable=false)
     */
     protected $count = 0;

###### get/set etc....... #######

Line.php

class Line
{
    /**
     * @ORM\Id
     * @ORM\Column(type="integer")
     * @ORM\GeneratedValue(strategy="AUTO")
     */
    protected $id;
########## blablabla #############

my action:

    public function right_line_relatedAction($line = null, $title='相关线路')
    {

        $lines = $l->getByUser($user, array());
//anyway,$lines is an array,It has several elements,each element is an instance of LineEntity.

        $related = $this->getDoctrine()->getRepository('MyWebBundle:LineRelated')->findBy(array('line' => $lines), array('count' => 'DESC'), 20);
//this findBy function generate the sql which is slow.

        return $related;
    }

Hi,

How did you get this query string ?

Repository#findBy does not quote the values, It uses PDO:bindParam.
so the expected query string should be someting like :

WHERE t0.line_id IN (?, ? ,?) 

I tried to reproduce but in my tests the generated Query binds the parameters as "PDO::PARAM_INT".

I have added a test case.
Could you please can try to change it and make fails.

Cheers

reproduced :

SELECT t0.id AS id1, t0.line_id_related AS line_id_related2, t0.count AS count3, t0.line_id AS line_id4 FROM line_related t0 WHERE t0.line_id IN ('6059', 4851, '6068', 126352, '6060', '1000000') ORDER BY t0.count DESC LIMIT 20
Parameters: [['6059', 4851, '6068', 126352, '6060', '1000000']] 
[Hide runnable query]
Time: 234.53 ms [   Explain query ]

let me have a look on what's going on

interesting. I've dump(using ladybug_dump) the $lines,and I found out that when the element is a Proxies Object(Object(Proxies_GC_\My\WebBundle\Entity\Line)),then the id of that Object will be with quoted,when the elememt is an Real Entity,then It will be without quote.

for example,in my last comment, the parameters is [['6059', 4851, '6068', 126352, '6060', '1000000']]
the result of dumping $lines is :

array(6)
[0]: object(Proxies_CG_\Zuo\WebBundle\Entity\Line)
[1]: object(Zuo\WebBundle\Entity\Line)
[2]: object(Proxies_CG_\Zuo\WebBundle\Entity\Line)
[3]: object(Zuo\WebBundle\Entity\Line)
[4]: object(Proxies_CG_\Zuo\WebBundle\Entity\Line)
[5]: object(Proxies_CG_\Zuo\WebBundle\Entity\Line)

tell me if you need more information. thanks

This may be because

$_identifier

in proxies ( https://github.com/doctrine/doctrine2/blob/42e83a2716d19eada4f1cd49ece77d5f5229a239/lib/Doctrine/ORM/Proxy/ProxyFactory.php#L383 ) is not necessarily composed by integers. This could be fixed with DCOM-96. I'll add the tests to my development branch and will ping you back tomorrow

thanks

I see what is going on here... But this should not be a problem anyway, since they're bound anyway as "PDO::PARAM_INT", as Fabio B. Silva told you.

That's only a problem with the logger showing them as string. PDO will handle the conversion before the value hits the DB as far as I know.

I can understand your point,but what I don't really get is that the execute time of sql is very long,that explained the quote should be in the sql,not like what you said,that's only a problem with the logger.

scourgen can you profile the difference directly in CLI? What about checking the bound parameter type? Are those values bound as INTs in your case?

@ocramius I wish I could, but I was using doctrine2 with symfony2,So It looks like It will takes some time to simulating all environment and settings that could allow me to reproduced the problem.

but anyway,I will have a try and tell you what happen when I found something.

scourgen ok, awaiting your reply then

I've spent some time on playing with native doctrine2. It took me awhile to setup everything. but I just don't get that how to retrive data with its Proxy ojbect(for example Proxies_CG_\My\WebBundle\Entity\Line).

I mean the result of

$this->_em->getRepository("something")->findxxx()

always return an array of real object. I can't reproduced the situation(#comment-19186) that happens on symfony2+doctrine2.

anyway,I can make sure the problem is real exist,Because the execute time of that slow sql from the tool bar of symfony2 is same as I executed it at mysql cli. If the sql shows up on log with quote but running at mysql without quote,the execute time won't be same(actually It will be much more faster,in my case,20x times,from 2xxms to 10ms).

scourgen you can use

$em->getReference($className, $identifier)

(identifier being a key=>value array) to force proxies.

Give it a try

looks like I reproduced it.

    public function testIssue()
    {   
         $no_used=   $this->_em->getRepository(__NAMESPACE__. '\DDC2214Line')->findOneById(1);
        $lines=array(
            //$this->_em->getRepository(__NAMESPACE__. '\DDC2214Line')->findOneById(1),
            $this->_em->getReference(__NAMESPACE__. '\DDC2214Line',1),
            $this->_em->getReference(__NAMESPACE__. '\DDC2214Line','2'),
            $this->_em->getReference(__NAMESPACE__. '\DDC2214Line',3),
        );  
        $logger  = $this->_em->getConnection()->getConfiguration()->getSQLLogger();
        $ids     = array_map(function($r){
            return $r->id;
        }, $this->relatedList);

        //$related = $this->_em->getRepository(__NAMESPACE__ . '\DDC2214LineRelated')->findBy(array('line' => $lines), array('count' => 'DESC'), 20);
        $related = $this->_em->createQuery('select lr from '.__NAMESPACE__ . '\DDC2214LineRelated lr where lr.id in (:ids)')->setParameter('ids',$lines)->getResult();
            
        $query   = end($logger->queries);
//\Doctrine\Common\Util\Debug::dump($query['params']);

        $this->assertCount(3, $related);
        $this->assertEquals($ids, $query['params'][0]);
        $this->assertEquals(\Doctrine\DBAL\Connection::PARAM_INT_ARRAY, $query['types'][0]);
    }   
}

I use MySql Query log to see what's really happen in database(http://dev.mysql.com/doc/refman/5.5/en/query-log.html)

this is the log from table mysql.general_log

2013-01-08 12:23:44	[root] @ localhost [127.0.0.1]	59	0	Connect	root@localhost on doctrine_tests
2013-01-08 12:23:44	root[root] @ localhost [127.0.0.1]	59	0	Query	CREATE TABLE DDC2214Line (id INT AUTO_INCREMENT NOT NULL, PRIMARY KEY(id)) DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci ENGINE = InnoDB
2013-01-08 12:23:44	root[root] @ localhost [127.0.0.1]	59	0	Query	CREATE TABLE DDC2214LineRelated (id INT AUTO_INCREMENT NOT NULL, line_id INT NOT NULL, count SMALLINT NOT NULL, line_id_related INT NOT NULL, INDEX IDX_D31307994D7B7542 (line_id), PRIMARY KEY(id)) DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci ENGINE = InnoDB
2013-01-08 12:23:44	root[root] @ localhost [127.0.0.1]	59	0	Query	ALTER TABLE DDC2214LineRelated ADD CONSTRAINT FK_D31307994D7B7542 FOREIGN KEY (line_id) REFERENCES DDC2214Line (id)
2013-01-08 12:23:44	root[root] @ localhost [127.0.0.1]	59	0	Query	START TRANSACTION
2013-01-08 12:23:44	root[root] @ localhost [127.0.0.1]	59	0	Query	INSERT INTO DDC2214Line (id) VALUES (null)
2013-01-08 12:23:44	root[root] @ localhost [127.0.0.1]	59	0	Query	INSERT INTO DDC2214Line (id) VALUES (null)
2013-01-08 12:23:44	root[root] @ localhost [127.0.0.1]	59	0	Query	INSERT INTO DDC2214Line (id) VALUES (null)
2013-01-08 12:23:44	root[root] @ localhost [127.0.0.1]	59	0	Query	INSERT INTO DDC2214LineRelated (count, line_id_related, line_id) VALUES (1, 1, 1)
2013-01-08 12:23:44	root[root] @ localhost [127.0.0.1]	59	0	Query	INSERT INTO DDC2214LineRelated (count, line_id_related, line_id) VALUES (2, 2, 2)
2013-01-08 12:23:44	root[root] @ localhost [127.0.0.1]	59	0	Query	INSERT INTO DDC2214LineRelated (count, line_id_related, line_id) VALUES (3, 3, 3)
2013-01-08 12:23:44	root[root] @ localhost [127.0.0.1]	59	0	Query	COMMIT
2013-01-08 12:23:44	root[root] @ localhost [127.0.0.1]	59	0	Query	SELECT t0.id AS id1 FROM DDC2214Line t0 WHERE t0.id = 1 LIMIT 1
2013-01-08 12:23:44	root[root] @ localhost [127.0.0.1]	59	0	Query	SELECT d0_.id AS id0, d0_.count AS count1, d0_.line_id_related AS line_id_related2, d0_.line_id AS line_id3 FROM DDC2214LineRelated d0_ WHERE d0_.id IN (1, '2', 3)
2013-01-08 12:23:44	root[root] @ localhost [127.0.0.1]	59	0	Quit	

you can see,in database level,the second parameter of last query but two has quote ( (1, '2', 3) )

A related Github Pull-Request [GH-247] was opened
https://github.com/doctrine/common/pull/247

A related Github Pull-Request [GH-247] was closed
https://github.com/doctrine/common/pull/247

My pull request (https://github.com/doctrine/doctrine2/pull/548) contains an implementation of the method. Note that no exception is thrown if you query for the state of a non-existing filter - in such a case, false is returned as for disabled filters.

Debian Wheezy, Mysql 5.1, Apache2, PHP 5.4

A little up... for inspiration from JPA

http://en.wikibooks.org/wiki/Java_Persistence/OneToMany#Undirectional_OneToMany.2C_No_Inverse_ManyToOne.2C_No_Join_Table_.28JPA_2.0_ONLY.29

Ok this is much more complicated to allow then i thought. The problem is that the QueryComponents are passed by value, as an array, not by reference. That prevents changing them because this change wouldn't be visible in the output walker.

I can add a method to allow this in the OutputWalker for now, but generally this requires a bigger refactoring on the Query Components.

Added setQueryComponent() in SQL Walker to allow modification in output walker.

I'm afraid that this doesn't solve the initial problem at all. I'll try to describe it in more details to show what I mean. Suppose we have two doctrine extensions each of which contain its own tree walker. Each of these tree walkers need to modify AST and add new component to it (joined with some component already existing in the query). The first problem is that each tree walker has its own queryComponents array which is not passed between them, although they not necessary need to use queryComponents - they could use only AST. The second, bigger problem is that the Parser class does not know anything about modifications of queryComponents in tree walkers and cannot pass modified version to the OutputWalker. The goal of submitting this issue was to allow adding new components to the query in tree walkers which is not achievable by your fix. I think it may be the first step in the right direction. Maybe TreeWalkerAdapter should have public method getQueryComponents() which would be used by the Parser to pass modified queryComponents between different tree walkers and finally to the OutputWalker ? This would not break backward compatibility and solve this issue. What do you think about it?

I've tried to implement the solution mentioned in previous comment but it's also not so clean and easy as I thought. Each tree walker (including TreeWalkerChain) would have to implement getQueryComponents() and setQueryComponent($alias, array $component) methods. The same with SqlWalker, so the TreeWalker interface should have these methods, which would break BC in some way (walkers that do not inherit from SqlWalker or TreeWalkerAdapter will fail to compile). So maybe my first solution (PR #464) is not so bad for now? In the future queryComponents could be replaced by a special object or could be passed by a reference to allow modifications.

Marked as improvement as its not a bug.

A solution might probably implement an object holding all the QueryComponent, implementing ArrayAccess. So that way the state can be shared.

Just hit this while developing an ast walker... Will look into it too since I need it more than soon.

As a VERY UGLY workaround, I used a static variable and a custom sql walker in combination with my AST walker.

namespace Comcom\Versioning\ORM\Query;


use Doctrine\ORM\Query\SqlWalker;

class WorkaroundSqlWalker extends SqlWalker
{
    public function __construct($query, $parserResult, array $queryComponents)
    {
        parent::__construct($query, $parserResult, $queryComponents);

        foreach (VersionWalker::$additionalAliases as $alias => $value) {
            $this->setQueryComponent($alias, $value);
        }
    }
}

Existing PR at https://github.com/doctrine/doctrine2/pull/361

Not a bug

Hi all!
Maybe if this task is hard, you could do a simplier variant, "FROM Class1 a1 JOIN a1,a2 WITH a2 INSTANCE OF Class2"?
Doctrine 2.3 supports it, but in fact, it does not work. I can't use Class2 fields in query, and Doctrine ignores the INSTANCE OF operator when building a native queries.
I am working with system where I have many inherited classes with Class Table Inheritance. When I do a JOIN query, it generates native sql query more than 1KB(?!) length, and MySQL freezes for more than 7 minutes (?!) on it. It must join only tables for Class2, but it joins all my 20 tables inherited of my base class
I don't know where to send bugreport
It will be very good if I could manually select a joined class to make Doctrine do a better queries.
Sorry for my english, I am from Moscow.

onFluish is not the start of a transaction. It has nothing to do with this.

Is a third event needed? Or is this to be marked as "won't fix"?

Maybe onBeginTransaction, onCommit and onRollback.

However since you can start transactions manually using $em->beginTransaction(), the Flush events are somehwat independent of transactions anyways.

Well, user can start transaction anytime, but the fact is that if we think ORM we do not know nothing about the database. we just persist and flush objects.

Yes I think these would be very useful, from how I see it, if you use event listeners, is:

loadClassMetadata: you can apply extra mapping

onFlush: you can modify entity changesets, or persist recalculate new ones, without triggering the database, since it is not used to begin the database modifications yet.

onBeginTransaction: could use the database modifications keeping in sync the entity changesets. the thing about this event is that usually in behavioral way atomic updates are required. for example nestedset tree sync lft rgt columns, sortable sync the sort index, materialized path, all these requires atomic updates, and the best place is the start of transaction.

onCommit: could be useful to execute right before commit, finalizing database modifications could be done.

onRollback: this one is really something, since if you go far, there might be something like files uploaded during the entity processing, and you may want to remove them if transaction fails.

This situation was barely documented here: http://www.doctrine-project.org/jira/browse/DDC-1443

We need a better Transaction API that completely fixes the computation of changesets and also allow more fine grained control over Entities and their corresponding information.

I'd postpone this one until 3.0.

(I have no Link Privileges, but this one #DDC-1379 is a duplicate with more extent info.)

First, you are using internal API therefore you are on your own anyways.

This is marked as improvment now, the functionality works, it may just be inefficient.

Hi,

I'm marking issue as invalid because you're conceptually wrong.
What you're trying to do is telling that a collection of new entities is holded by a collection of Persistent entities.
The reference internally of PersistentCollection to ArrayCollection means a lot here.

Correct code would be you to regenerate the collection (a new ArrayCollection) and just assign it to setTournamentLocations($newCollection);

Does this explanation is enough for you?

Cheers,

Hi Guilherme,

If I do this:

$locations = new ArrayCollection();

foreach ($form->getValue('tournamentLocations') as $id) {
    $locations->add($em->getReference('Tournaments_Model_TournamentLocation', $id));
}

$bracket->setTournamentLocations($locations);

... then all the records are deleted, before adding the new records. This is inefficient and causes extra, unnecessary write operations.

Can't Doctrine perform diffs when persisting the collection, so that only the necessary deletes and inserts are executed?

We could add it, but I don't think it worth the effort.
Main problem with this one is that we use C-level binary comparison to get the diff. That's what you entities/hash pointers are different.
We would have to write our own diff-comparator for both collections, which would probably slowdown the entire Doctrine.

I'd rather consider that it's not possible to be done at the moment, but I need much more investigation for that. This will be something that I'll probably only do when I look at this issue again with a lot of time (which is really hard to happen).

If you have some spare time, feel free to make some attempts.
Just don't forget to enable performance tests in Doctrine Unit Test suite.