[DDC-3680] [GH-1377] Failing test case for broken paginator case Created: 08/Apr/15  Updated: 09/Apr/15  Resolved: 09/Apr/15

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: ORM
Affects Version/s: 2.5
Fix Version/s: 2.5.1
Security Level: All

Type: Bug Priority: Major
Reporter: Doctrine Bot Assignee: Bill Schaller
Resolution: Fixed Votes: 0
Labels: limitsubqueryoutputwalker, mysql, paginator


 Description   

This issue is created automatically through a Github pull request on behalf of fprochazka:

Url: https://github.com/doctrine/doctrine2/pull/1377

Message:

http://www.doctrine-project.org/jira/browse/DDC-3679



 Comments   
Comment by Doctrine Bot [ 08/Apr/15 ]

A related Github Pull-Request [GH-1377] was assigned:
https://github.com/doctrine/doctrine2/pull/1377

Comment by Doctrine Bot [ 08/Apr/15 ]

A related Github Pull-Request [GH-1377] was labeled:
https://github.com/doctrine/doctrine2/pull/1377

Comment by Doctrine Bot [ 09/Apr/15 ]

A related Github Pull-Request [GH-1377] was merged:
https://github.com/doctrine/doctrine2/pull/1377





[DDC-3650] [GH-1357] Drop useless execution bit Created: 01/Apr/15  Updated: 02/Apr/15  Resolved: 02/Apr/15

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: Git Master
Fix Version/s: 2.5
Security Level: All

Type: Bug Priority: Major
Reporter: Doctrine Bot Assignee: Marco Pivetta
Resolution: Fixed Votes: 0
Labels: file-mode, paginator


 Description   

This issue is created automatically through a Github pull request on behalf of DavidPrevot:

Url: https://github.com/doctrine/doctrine2/pull/1357

Message:



 Comments   
Comment by Doctrine Bot [ 02/Apr/15 ]

A related Github Pull-Request [GH-1357] was assigned:
https://github.com/doctrine/doctrine2/pull/1357

Comment by Doctrine Bot [ 02/Apr/15 ]

A related Github Pull-Request [GH-1357] was labeled:
https://github.com/doctrine/doctrine2/pull/1357

Comment by Doctrine Bot [ 02/Apr/15 ]

A related Github Pull-Request [GH-1357] was merged:
https://github.com/doctrine/doctrine2/pull/1357





[DDC-3646] Do not select unused columns in inner queries of Paginator Created: 31/Mar/15  Updated: 14/Apr/15

Status: Open
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: 2.4.7
Fix Version/s: None
Security Level: All

Type: Improvement Priority: Minor
Reporter: Malte Wunsch Assignee: Steve Müller
Resolution: Unresolved Votes: 1
Labels: paginator
Environment:

In my case MySQL 5.1, other versions or DBMS may optimise that by themselves



 Description   

When a {{Paginator count()}}s, it fires a query like this:

SELECT COUNT(*) AS dctrn_count FROM (SELECT DISTINCT id0 FROM (%originalQuery%) dctrn_result) dctrn_table

an getIterator() does something similar for joined collections to get the ids in the first place:

SELECT DISTINCT id0, lastmod11 FROM (%originalQuery%) dctrn_result ORDER BY lastmod11 DESC LIMIT 10 OFFSET 0

My problem is that both the inner queries can be too heavy for a regular temporary table.

At first sight I noticed several LEFT JOIN s that could be stripped from the inner query without changing the result of the outer one. But that might have been a specially easy case, a general optimisation of the JOIN clauses might be harder. I think http://www.doctrine-project.org/jira/browse/DDC-2381 deals with that.

On second thought we noticed some ugly BLOB columns as part of the projection, which unhealthily bloated the temporary table. Ultimately, the BLOBs are needed for the paginated view, but they are neither needed for counting nor for getting the paginated ids.

Hence, I propose to remove unused columns from the projection in the inner queries. "Unused" means not being part of the SELECT or ORDER BY clause in the outer query (and maybe not part of GROUP BY, HAVING... - haven't checked on this yet). The key could be the $innerSql in the LimitSubqueryOutputWalker->walkSelectStatement(), but before investigating further, I'd like to hear from you what you think.



 Comments   
Comment by Marco Pivetta [ 31/Mar/15 ]

Please review https://github.com/doctrine/doctrine2/pull/1353, as it might solve this issue

Comment by Malte Wunsch [ 31/Mar/15 ]

Thanks for your quick reply!

To be honest, I don't fully get the PR. At least it's goals look very different to me. Anyway, in LimitSubqueryOutputWalker the $innerSQL is now being generated in getInnerSQL(). In there, some hiddenAliasResultVariable are set to false (and are later restored) so that the parent SQLWalker can reference selected fields better. This might be a requirement for implementing my idea, but as far as I understand, there are no columns removed from the select clause so far?

Comment by Doctrine Bot [ 31/Mar/15 ]

A related Github Pull-Request [GH-1353] was closed:
https://github.com/doctrine/doctrine2/pull/1353

Comment by Marco Pivetta [ 06/Apr/15 ]

Malte Wunsch the PR is actually related as it affects the selected fields.

Consider checking again against 2.5.0, as we indeed fixed many paginator issues, and this issue needs to be re-validated.

Comment by Malte Wunsch [ 07/Apr/15 ]

Great! I'll be glad to check. It might take some time, as we currently only have PHP 5.3 at our disposal (please don't tell anybody), and Doctrine 2 ORM 2.5.0 requires at least PHP 5.4. Fortunately, this is a very good trigger to put some pressure on the PHP upgrade process. I'll keep my eye on it.

Comment by Malte Wunsch [ 10/Apr/15 ]

Marco Pivetta Unfortunately, Doctrine 2.5 had no effect on my inner original query. It still looks like this:

SELECT 
  COUNT(*) AS dctrn_count 
FROM 
  (
    SELECT 
      DISTINCT id0 
    FROM 
      (
        SELECT 
          [all fields from t0_],
          [all fields from t1_],
          [all fields from t2_],
          [all fields from t3_]
        FROM 
          table1 t0_ 
          LEFT JOINs...
        WHERE 
          ...
        ORDER BY 
          ...
      ) dctrn_result
  ) dctrn_table

Just to be clear, my point is with the most inner query. If I'm correct it is generated in LimitSubqueryOutputWalker->getInnerSql(). In the special case with the query above, one could simplify it by dropping the order clause, removing the left joins and selecting only the id0 field. I have no plan for inner joins and other more complex optimisations, but I guess that removing the unused fields from the select clause might be in reach.

Comment by Matthias Pigulla [ 10/Apr/15 ]

When removing fields from the SELECT clause, we need to keep those that are referenced from GROUP BY or HAVING clauses (when ORDER BY is dropped).

Comment by Matthias Pigulla [ 14/Apr/15 ]

http://www.doctrine-project.org/jira/browse/DDC-2381 seems to be related or identical.





[DDC-3645] [GH-1353] Paginator fixes take3 Created: 27/Mar/15  Updated: 31/Mar/15  Resolved: 31/Mar/15

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: Git Master
Fix Version/s: 2.5
Security Level: All

Type: Bug Priority: Major
Reporter: Doctrine Bot Assignee: Marco Pivetta
Resolution: Fixed Votes: 0
Labels: limitsubqueryoutputwalker, paginator

Issue Links:
Duplicate
is duplicated by DDC-3637 [GH-1347] problem with LimitSubqueryO... Resolved
is duplicated by DDC-3642 [GH-1351] Failing test cases regardin... Resolved

 Description   

This issue is created automatically through a Github pull request on behalf of zeroedin-bill:

Url: https://github.com/doctrine/doctrine2/pull/1353

Message:

This PR fixes issues reported in #1347 and #1351.

  • Ordering a query by a column from an association which uses SINGLE_TABLE inheritance no longer throws an error.
  • Limiting a query which is ordered by fields from a joined -To-Many association now works as expected.

This patch converts LimitSubqueryOutputWalker to use the ROW_NUMBER window function for queries on platforms that support it. Other platforms use a modified version of the previous method. The modification is to not select the ORDER BY columns in the wrapping query. The reason those columns were selected like that in the first place is that ORDER BY on columns not in the select list is unsupported on many platforms. MySQL and SQLite are fine with it, and don't support ROW_NUMBER.

The ROW_NUMBER solution is actually much simpler.

LimitSubqueryOutputWalker SQL generation tests for PG and Oracle have been changed.

*There are 2 failing expectedException tests right now.* The LimitSubqueryWalker needs to be modified to throw an exception when it detects a query that does the following things together:

  • Joins a -To-Many association
  • Orders by a column from the associated entity
  • Limits the result

As with the previous patch, this depends on changes to SQLServerPlatform2008 that have not yet been merged. SQLServerPlatform2008 in DBAL/master currently mangles the generated queries when trying to apply LIMIT/OFFSET.



 Comments   
Comment by Doctrine Bot [ 31/Mar/15 ]

A related Github Pull-Request [GH-1353] was assigned:
https://github.com/doctrine/doctrine2/pull/1353

Comment by Doctrine Bot [ 31/Mar/15 ]

A related Github Pull-Request [GH-1353] was labeled:
https://github.com/doctrine/doctrine2/pull/1353

Comment by Doctrine Bot [ 31/Mar/15 ]

A related Github Pull-Request [GH-1353] was closed:
https://github.com/doctrine/doctrine2/pull/1353





[DDC-3642] [GH-1351] Failing test cases regarding to #1325 #1337 Created: 27/Mar/15  Updated: 31/Mar/15  Resolved: 31/Mar/15

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: Git Master
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: Doctrine Bot Assignee: Marco Pivetta
Resolution: Duplicate Votes: 0
Labels: paginator

Issue Links:
Duplicate
duplicates DDC-3645 [GH-1353] Paginator fixes take3 Resolved

 Description   

This issue is created automatically through a Github pull request on behalf of vaheshadunts:

Url: https://github.com/doctrine/doctrine2/pull/1351

Message:



 Comments   
Comment by Doctrine Bot [ 30/Mar/15 ]

A related Github Pull-Request [GH-1351] was closed:
https://github.com/doctrine/doctrine2/pull/1351

Comment by Doctrine Bot [ 31/Mar/15 ]

A related Github Pull-Request [GH-1351] was assigned:
https://github.com/doctrine/doctrine2/pull/1351

Comment by Doctrine Bot [ 31/Mar/15 ]

A related Github Pull-Request [GH-1351] was labeled:
https://github.com/doctrine/doctrine2/pull/1351

Comment by Marco Pivetta [ 31/Mar/15 ]

Merged into DDC-3645 (will be solved there)





[DDC-3637] [GH-1347] problem with LimitSubqueryOutputWalker when use InheritanceType Created: 25/Mar/15  Updated: 31/Mar/15  Resolved: 31/Mar/15

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: Git Master
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: Doctrine Bot Assignee: Marco Pivetta
Resolution: Duplicate Votes: 0
Labels: paginator

Issue Links:
Duplicate
duplicates DDC-3645 [GH-1353] Paginator fixes take3 Resolved

 Description   

This issue is created automatically through a Github pull request on behalf of alexander-orabey:

Url: https://github.com/doctrine/doctrine2/pull/1347

Message:

When we use InheritanceType ("SINGLE_TABLE") catch exception that field in joined class does not exist in base class.



 Comments   
Comment by Doctrine Bot [ 25/Mar/15 ]

A related Github Pull-Request [GH-1347] was labeled:
https://github.com/doctrine/doctrine2/pull/1347

Comment by Doctrine Bot [ 31/Mar/15 ]

A related Github Pull-Request [GH-1347] was closed:
https://github.com/doctrine/doctrine2/pull/1347

Comment by Marco Pivetta [ 31/Mar/15 ]

Merged into DDC-3645 ( PR #1353 )





[DDC-3629] [GH-1342] Paginator functional tests Created: 19/Mar/15  Updated: 24/Mar/15  Resolved: 24/Mar/15

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: None
Fix Version/s: 2.5
Security Level: All

Type: Bug Priority: Major
Reporter: Doctrine Bot Assignee: Marco Pivetta
Resolution: Fixed Votes: 0
Labels: distinct, mssql, oracle, orderBy, orderby, paginator, postgresql

Issue Links:
Dependency
depends on DDC-3623 [GH-1337] Paginator OrderBy fix take 2 Resolved
is required for DDC-3606 [GH-1325] fixed PostgreSQL and Oracle... Resolved

 Description   

This issue is created automatically through a Github pull request on behalf of zeroedin-bill:

Url: https://github.com/doctrine/doctrine2/pull/1342

Message:

PaginationTest has been expanded and now covers ordering and limiting rather thoroughly.

4 tests fail on PostgreSQL and MySQL, 14 tests fail on SQL Server, and I didn't try any other DBMS.



 Comments   
Comment by Doctrine Bot [ 22/Mar/15 ]

A related Github Pull-Request [GH-1342] was labeled:
https://github.com/doctrine/doctrine2/pull/1342





[DDC-3623] [GH-1337] Paginator OrderBy fix take 2 Created: 17/Mar/15  Updated: 24/Mar/15  Resolved: 24/Mar/15

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: None
Fix Version/s: 2.5
Security Level: All

Type: Bug Priority: Major
Reporter: Doctrine Bot Assignee: Marco Pivetta
Resolution: Fixed Votes: 0
Labels: distinct, mssql, oracle, orderBy, paginator, postgresql

Issue Links:
Dependency
is required for DDC-3606 [GH-1325] fixed PostgreSQL and Oracle... Resolved
is required for DDC-3629 [GH-1342] Paginator functional tests Resolved

 Description   

This issue is created automatically through a Github pull request on behalf of zeroedin-bill:

Url: https://github.com/doctrine/doctrine2/pull/1337

Message:

This PR fixes issues created in #1220, reported in #1267. The original PR was reverted in #1283.

The issue was that in queries ordered by joined association columns, the joined column aliases did not exist in the outer query created in LimitSubqueryOutputWalker.

This PR adds functionality to LimitSubqueryOutputWalker which finds any such columns referenced in the OrderBy clause, and adds them to the SelectStatement prior to SQL generation.



 Comments   
Comment by Doctrine Bot [ 17/Mar/15 ]

A related Github Pull-Request [GH-1337] was labeled:
https://github.com/doctrine/doctrine2/pull/1337

Comment by Doctrine Bot [ 17/Mar/15 ]

A related Github Pull-Request [GH-1337] was labeled:
https://github.com/doctrine/doctrine2/pull/1337

Comment by Doctrine Bot [ 24/Mar/15 ]

A related Github Pull-Request [GH-1337] was assigned:
https://github.com/doctrine/doctrine2/pull/1337





[DDC-3606] [GH-1325] fixed PostgreSQL and Oracle pagination issues Created: 09/Mar/15  Updated: 24/Mar/15  Resolved: 24/Mar/15

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: None
Fix Version/s: 2.5
Security Level: All

Type: Bug Priority: Major
Reporter: Doctrine Bot Assignee: Marco Pivetta
Resolution: Fixed Votes: 1
Labels: distinct, mssql, orderBy, paginator, postgresql

Issue Links:
Dependency
depends on DDC-3623 [GH-1337] Paginator OrderBy fix take 2 Resolved
depends on DDC-3629 [GH-1342] Paginator functional tests Resolved

 Description   

This issue is created automatically through a Github pull request on behalf of vaheshadunts:

Url: https://github.com/doctrine/doctrine2/pull/1325

Message:

Pagination with ordering on 1:m and m:m relations, was not working
properly because of selecting the ordered fields in main select
statement with distinction (e.g. SELECT DISTINCT e0_.id, p1_.name from
... ) in this case we've received less rows than we've required in
query. So I've modified the subquery generation part.
In case of PostgreSQL and Oracle added the row_number in the subquery
over order by statement, then the main select is grouped by id and
selected min of row number, also ordering by rownumber asc, because they
are on right order already (e.g. select e0_.id, min(rownum) as rownum
from ..... order by rownum).
In case of MySQL, the subselect result with ids are in right order so
there is no need to select that fields(this fixes the same issue too)
In other cases I haven't tested because of that leaved the same.



 Comments   
Comment by Vahe Shadunts [ 09/Mar/15 ]

Hi Benjamin Eberlei, please let me know if I need to change something, I've used regular expression to change the doctrine's generated select statement, if there is a better way please let me know.

The code I've modified
https://github.com/vaheshadunts/doctrine2/blob/DDC-1958/lib/Doctrine/ORM/Tools/Pagination/LimitSubqueryOutputWalker.php#L221

Comment by Vahe Shadunts [ 09/Mar/15 ]

Also please let me know should I have to modify the test assertions of the queries which are have changed by my modifications? Or I have to skip the continuous integration ?

Comment by Doctrine Bot [ 16/Mar/15 ]

A related Github Pull-Request [GH-1325] was labeled:
https://github.com/doctrine/doctrine2/pull/1325

Comment by Doctrine Bot [ 16/Mar/15 ]

A related Github Pull-Request [GH-1325] was labeled:
https://github.com/doctrine/doctrine2/pull/1325

Comment by Doctrine Bot [ 24/Mar/15 ]

A related Github Pull-Request [GH-1325] was assigned:
https://github.com/doctrine/doctrine2/pull/1325

Comment by Doctrine Bot [ 24/Mar/15 ]

A related Github Pull-Request [GH-1325] was unlabeled:
https://github.com/doctrine/doctrine2/pull/1325





[DDC-3598] Paginator incorrect ordering Created: 03/Mar/15  Updated: 03/Mar/15

Status: Open
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: 2.4.7
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: Kris Assignee: Steve Müller
Resolution: Unresolved Votes: 0
Labels: paginator


 Description   

When ordering by multiple fields, with the order by fields not contained in the original select statement the paginator outputs in an incorrect order.
I found the issue is in the LimitSubqueryOutputWalker in the preserveSqlOrdering function. This is the query it generates

SELECT DISTINCT id1, date_added3 FROM (SELECT t0_.description AS description0, t0_.id AS id1, t0_.date AS date2, t0_.date_added AS date_added3 FROM Ticket t0_ INNER JOIN user u1_ ON t0_.assigned_user_id = u1_.id INNER JOIN user u2_ ON t0_.created_user_id = u2_.id WHERE t0_.project_id = ? ORDER BY u2_.first_name DESC, t0_.date_added DESC) dctrn_result ORDER BY date_added3 DESC

So it appears that it ignores any order by field that is not included in the inner select statement.






[DDC-3575] Paginator's CountOutputWalker keeps the ORDER BY in the subquery for all non-MSSQL platforms Created: 20/Feb/15  Updated: 20/Feb/15

Status: Open
Project: Doctrine 2 - ORM
Component/s: ORM
Affects Version/s: None
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: John Flatness Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: paginator

Issue Links:
Reference
relates to DDC-3519 [GH-1267] Failing test for an ORDER B... Resolved
relates to DDC-3538 [GH-1283] #1267 - order by broken in ... Resolved

 Description   

The Paginator's CountOutputWalker leaves the ORDER BY clause in the subquery that's executed to get the total count.

The original (and current) version of that code included the following comment:

Note that the ORDER BY clause is not removed. Many SQL implementations (e.g. MySQL) are able to cache subqueries. By keeping the ORDER BY clause intact, the limitSubQuery that will most likely be executed next can be read from the native SQL cache.

My understanding is that MySQL does not, in fact, cache subqueries in the query cache, so cached results are not available between executions of different queries. The current MySQL manual still states that "Queries that are a subquery of an outer query" are not cached.

This pull request was merged in 2013 to address errors with mssql and paginating ordered queries. Midway through the discussion on that issue, it was determined that it would be better to just remove the ORDER BY clause for all platforms, not just MS SQL. However, that decision was reversed when it came time to merge the PR.

Does the comment on that method accurately describe the rationale for retaining the ORDER BY clause for the count query? If so, am I simply wrong about how MySQL operates, or does the comment refer to some other platform that benefits from this pattern?



 Comments   
Comment by Marco Pivetta [ 20/Feb/15 ]

See https://github.com/doctrine/doctrine2/pull/1283

This change was reverted because MSSQL is pretty much a niche, and fixing bugs for it is really making things complicated for all the other storages.

Comment by John Flatness [ 20/Feb/15 ]

Thanks for the reply, Marco.

I think this is getting conflated with the other Paginator-related ORDER BY changes and reversions that have happened breaking and fixing the actual sorting behavior. The pull you linked looks like its relating to the LimitSubqueryOutputWalker, not the CountOutputWalker.

The thing I'm talking about is still there, and does specifically address only a MSSQL bug: https://github.com/doctrine/doctrine2/blob/master/lib/Doctrine/ORM/Tools/Pagination/CountOutputWalker.php#L82

The particular part of the discussion on PR #572 I was referring to above is here: https://github.com/doctrine/doctrine2/pull/572#discussion_r3154962

My concern is that the rationale for keeping the ORDER BY intact for all the other platforms (only when running the count query, not the "real" one) is a performance-related one, and that it's a flawed rationale. (Or, at least, that the comment about it is.)





[DDC-3574] the Paginator does not support arbitrary join should be back ported to 2.4 Created: 19/Feb/15  Updated: 24/Feb/15  Resolved: 19/Feb/15

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: Git Master, 2.3.4, 2.4.1
Fix Version/s: None
Security Level: All

Type: Improvement Priority: Major
Reporter: Sagar Assignee: Marco Pivetta
Resolution: Won't Fix Votes: 0
Labels: paginator


 Description   

Using the following query with the paginator fails:

SELECT u
FROM User u
JOIN Message m WITH m.author = u
WHERE m.status = 'draft' -- a condition justifying the join

The CountWalker and the CountOutputWalker both throw a exception saying "Cannot count query which selects two FROM components, cannot make distinction"

This message is wrong (I can make distinction here. Only User is selected in the result set) and confusing (I spent some time finding the second FROM before figuring it was related to the use of an arbitrary join instead of an association join).



 Comments   
Comment by Sagar [ 19/Feb/15 ]

Though it is fixed in 2.5 "the Paginator does not support arbitrary join should be back ported to 2.4"

Comment by Marco Pivetta [ 19/Feb/15 ]

This is a new feature that won't be backported into stable versions.

Comment by Sagar [ 24/Feb/15 ]

I really don't understand How this is new feature? It is missing part of Pagination Component.

Comment by Marco Pivetta [ 24/Feb/15 ]

It was limitation that was refactored out. Since it's an improvement, it won't land in a patch release.





[DDC-3538] [GH-1283] #1267 - order by broken in pagination logic (reverts #1220) Created: 24/Jan/15  Updated: 20/Feb/15  Resolved: 24/Jan/15

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: Git Master
Fix Version/s: 2.5
Security Level: All

Type: Bug Priority: Critical
Reporter: Doctrine Bot Assignee: Marco Pivetta
Resolution: Fixed Votes: 0
Labels: limitsubqueryoutputwalker, orderBy, orderby, paginator, subquery

Issue Links:
Duplicate
is duplicated by DDC-3519 [GH-1267] Failing test for an ORDER B... Resolved
Reference
is referenced by DDC-3575 Paginator's CountOutputWalker keeps t... Open

 Description   

This issue is created automatically through a Github pull request on behalf of Ocramius:

Url: https://github.com/doctrine/doctrine2/pull/1283

Message:

This PR reverts #1220 and merges #1264



 Comments   
Comment by Doctrine Bot [ 24/Jan/15 ]

A related Github Pull-Request [GH-1283] was labeled:
https://github.com/doctrine/doctrine2/pull/1283

Comment by Doctrine Bot [ 24/Jan/15 ]

A related Github Pull-Request [GH-1283] was merged:
https://github.com/doctrine/doctrine2/pull/1283





[DDC-3519] [GH-1267] Failing test for an ORDER BY that is INNER JOINed in a subquery Created: 19/Jan/15  Updated: 20/Feb/15  Resolved: 24/Jan/15

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: Git Master
Fix Version/s: None
Security Level: All

Type: Bug Priority: Critical
Reporter: Doctrine Bot Assignee: Marco Pivetta
Resolution: Duplicate Votes: 0
Labels: limit, limitsubqueryoutputwalker, orderBy, orderby, paginator, subquery

Issue Links:
Duplicate
duplicates DDC-3538 [GH-1283] #1267 - order by broken in ... Resolved
Reference
is referenced by DDC-3575 Paginator's CountOutputWalker keeps t... Open

 Description   

This issue is created automatically through a Github pull request on behalf of austinsmorris:

Url: https://github.com/doctrine/doctrine2/pull/1267

Message:

A failing test case to demonstrate a problem with #1220.

cc: @Ocramius @zeroedin-bill, @mvrhov, @stof, @brianium, @mrkrstphr



 Comments   
Comment by Doctrine Bot [ 24/Jan/15 ]

A related Github Pull-Request [GH-1267] was labeled:
https://github.com/doctrine/doctrine2/pull/1267

Comment by Doctrine Bot [ 24/Jan/15 ]

A related Github Pull-Request [GH-1267] was assigned:
https://github.com/doctrine/doctrine2/pull/1267

Comment by Doctrine Bot [ 24/Jan/15 ]

A related Github Pull-Request [GH-1267] was closed:
https://github.com/doctrine/doctrine2/pull/1267





[DDC-3514] LimitSubqueryOutputWalker should not duplicate orderBy clauses Created: 17/Jan/15  Updated: 17/Jan/15  Resolved: 17/Jan/15

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: DQL, Tools
Affects Version/s: Git Master, 2.4.7
Fix Version/s: 2.5
Security Level: All

Type: Improvement Priority: Major
Reporter: Marco Pivetta Assignee: Marco Pivetta
Resolution: Fixed Votes: 0
Labels: limitsubqueryoutputwalker, orderBy, orderby, paginator, performance, subquery


 Description   

This issue is just tracking https://github.com/doctrine/doctrine2/pull/1220






[DDC-3435] [GH-1211] DDC-3434 - paginator ignores `HIDDEN` fields in `ORDER BY` query Created: 05/Dec/14  Updated: 05/Dec/14  Resolved: 05/Dec/14

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: DQL, Tools
Affects Version/s: 2.4.4, 2.4.5, 2.4.6
Fix Version/s: 2.5, 2.4.7
Security Level: All

Type: Bug Priority: Major
Reporter: Doctrine Bot Assignee: Marco Pivetta
Resolution: Fixed Votes: 0
Labels: dql, limitsubqueryoutputwalker, paginator

Issue Links:
Dependency
is required for DDC-3434 LimitSubqueryOutputWalker does not re... Resolved

 Description   

This issue is created automatically through a Github pull request on behalf of Ocramius:

Url: https://github.com/doctrine/doctrine2/pull/1211

Message:

See DDC-3434 ( http://www.doctrine-project.org/jira/browse/DDC-3434 )



 Comments   
Comment by Doctrine Bot [ 05/Dec/14 ]

A related Github Pull-Request [GH-1211] was closed:
https://github.com/doctrine/doctrine2/pull/1211

Comment by Doctrine Bot [ 05/Dec/14 ]

A related Github Pull-Request [GH-1211] was assigned:
https://github.com/doctrine/doctrine2/pull/1211





[DDC-3434] LimitSubqueryOutputWalker does not retain correct ORDER BY expression fields when dealing with HIDDEN sort fields Created: 05/Dec/14  Updated: 05/Dec/14  Resolved: 05/Dec/14

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: DQL, Tools
Affects Version/s: 2.4.4, 2.4.5, 2.4.6
Fix Version/s: 2.5, 2.4.7
Security Level: All

Type: Bug Priority: Major
Reporter: Marco Pivetta Assignee: Marco Pivetta
Resolution: Fixed Votes: 0
Labels: dql, limitsubqueryoutputwalker, paginator

Issue Links:
Dependency
depends on DDC-3435 [GH-1211] DDC-3434 - paginator ignore... Resolved
Reference
relates to DDC-3336 Undefined property: Doctrine\ORM\Quer... Resolved
relates to DDC-1958 pager produces wrong results on postg... Resolved

 Description   

As per discussion in DDC-3336, the Doctrine\ORM\Tools\Pagination\LimitSubqueryOutputWalker does not retain the correct fields in the ORDER BY condition when one of the selected fields is marked as HIDDEN.

As an example, take a DQL query like:

SELECT a, a.name AS HIDDEN ord FROM Doctrine\Tests\ORM\Tools\Pagination\Author a ORDER BY ord DESC

This will result in an SQL query like:

SELECT DISTINCT id_0 FROM (SELECT a0_.id AS id_0, a0_.name AS name_1, a0_.name AS name_2 FROM Author a0_ ORDER BY name_2 DESC) dctrn_result

Removing the HIDDEN modifier will cause the query to produce the correct SQL:

SELECT DISTINCT id_0, name_2 FROM (SELECT a0_.id AS id_0, a0_.name AS name_1, a0_.name AS name_2 FROM Author a0_ ORDER BY name_2 DESC) dctrn_result ORDER BY name_2 DESC


 Comments   
Comment by Doctrine Bot [ 05/Dec/14 ]

A related Github Pull-Request [GH-1210] was closed:
https://github.com/doctrine/doctrine2/pull/1210





[DDC-3433] [GH-1210] DDC-3336 - undefined property with paginator walker and scalar expression in ORDER BY clause Created: 05/Dec/14  Updated: 05/Dec/14  Resolved: 05/Dec/14

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: DQL, Tools
Affects Version/s: 2.4.4, 2.4.5, 2.4.6
Fix Version/s: 2.5, 2.4.7
Security Level: All

Type: Bug Priority: Major
Reporter: Doctrine Bot Assignee: Marco Pivetta
Resolution: Fixed Votes: 0
Labels: ast, dql, limitsubqueryoutputwalker, paginator

Issue Links:
Dependency
is required for DDC-3336 Undefined property: Doctrine\ORM\Quer... Resolved

 Description   

This issue is created automatically through a Github pull request on behalf of Ocramius:

Url: https://github.com/doctrine/doctrine2/pull/1210

Message:

Ping @glen-84

Fixes DDC-3336 - http://www.doctrine-project.org/jira/browse/DDC-3336



 Comments   
Comment by Doctrine Bot [ 05/Dec/14 ]

A related Github Pull-Request [GH-1210] was closed:
https://github.com/doctrine/doctrine2/pull/1210





[DDC-3404] [GH-1188] Fixed counting exception Created: 20/Nov/14  Updated: 28/Nov/14  Resolved: 27/Nov/14

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: DQL, Tools
Affects Version/s: 2.4.6
Fix Version/s: 2.5, 2.4.7
Security Level: All

Type: Bug Priority: Major
Reporter: Doctrine Bot Assignee: Marco Pivetta
Resolution: Fixed Votes: 0
Labels: count, paginator, parameters


 Description   

This issue is created automatically through a Github pull request on behalf of merixstudio:

Url: https://github.com/doctrine/doctrine2/pull/1188

Message:

Fixed "Invalid parameter number: number of bound variables does not match number of tokens " exception during execution count on Query where select part of query contains :parameters.



 Comments   
Comment by Doctrine Bot [ 27/Nov/14 ]

A related Github Pull-Request [GH-1188] was assigned:
https://github.com/doctrine/doctrine2/pull/1188

Comment by Doctrine Bot [ 27/Nov/14 ]

A related Github Pull-Request [GH-1188] was closed:
https://github.com/doctrine/doctrine2/pull/1188





[DDC-3336] Undefined property: Doctrine\ORM\Query\AST\SimpleArithmeticExpression::$field Created: 04/Oct/14  Updated: 05/Dec/14  Resolved: 05/Dec/14

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: DQL, Tools
Affects Version/s: 2.4.4, 2.4.5, 2.4.6
Fix Version/s: 2.5, 2.4.7
Security Level: All

Type: Bug Priority: Critical
Reporter: Glen Ainscow Assignee: Marco Pivetta
Resolution: Fixed Votes: 0
Labels: ast, dql, limitsubqueryoutputwalker, paginator

Issue Links:
Dependency
depends on DDC-3433 [GH-1210] DDC-3336 - undefined proper... Resolved
Reference
relates to DDC-1958 pager produces wrong results on postg... Resolved
is referenced by DDC-3434 LimitSubqueryOutputWalker does not re... Resolved

 Description   

I upgraded from 2.4.1 to 2.4.4, and now I'm seeing this notice here & there:

Undefined property: Doctrine\ORM\Query\AST\SimpleArithmeticExpression::$field in /*/vendor/doctrine/orm/lib/Doctrine/ORM/Tools/Pagination/LimitSubqueryOutputWalker.php on line 200

This seems to be in a method with the description "Generates new SQL for Postgresql or Oracle if necessary." ... but we're on MySQL (Percona).



 Comments   
Comment by Glen Ainscow [ 05/Oct/14 ]

Changed this to critical. It's also screwing up ordering, for example:

SELECT tp, p, CASE WHEN tp.memberTo IS NULL THEN 1 ELSE 0 END AS HIDDEN ord FROM Tournaments_Model_TeamPlayer tp INNER JOIN tp.player p LEFT JOIN p.user u WHERE tp.team = ?1 ORDER BY ord DESC, tp.memberFrom ASC

The paginator doesn't order the data correctly unless I remove the 2nd order by (tp.memberFrom ASC). The generated SQL from the QB/DQL is correct, so the paginator must be messing something up.

See: http://www.doctrine-project.org/jira/browse/DDC-1958?focusedCommentId=24010&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-24010

Comment by Marco Pivetta [ 13/Oct/14 ]

Need a test case in order to verify the issue

Comment by Glen Ainscow [ 26/Oct/14 ]

Okay, try this:

        $qb->from('Team', 't')
              ->select('t')
              ->orderBy('(1-1000)*1', 'DESC')
              ->getQuery()
              ->getResult();

This is simplified, the actual query works with field values, like this:

        ->orderBy('(((t.eloRating-1000)*t.reliability) + 1000)', 'DESC')

This causes the first issue.

Comment by Glen Ainscow [ 26/Oct/14 ]

The second issue is explained by Guilherme Santos.

Comment by Marco Pivetta [ 05/Dec/14 ]

The first comment goes to a new issue.

Comment by Marco Pivetta [ 05/Dec/14 ]

Moved that issue to DDC-3434

Comment by Doctrine Bot [ 05/Dec/14 ]

A related Github Pull-Request [GH-1211] was closed:
https://github.com/doctrine/doctrine2/pull/1211

Comment by Doctrine Bot [ 05/Dec/14 ]

A related Github Pull-Request [GH-1211] was assigned:
https://github.com/doctrine/doctrine2/pull/1211





[DDC-3330] Bad Pagination - rows with sorted collection Created: 29/Sep/14  Updated: 16/Jun/15  Resolved: 16/Jun/15

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: ORM
Affects Version/s: Git Master
Fix Version/s: 2.5.1

Type: Bug Priority: Minor
Reporter: Thomas Lallement Assignee: Bill Schaller
Resolution: Fixed Votes: 0
Labels: paginator, tests
Environment:

Ubuntu 12.04, PHP 5.5.3


Attachments: File DDC3330Test.php    

 Description   

I use the Doctrine Paginator to be able to have a correct pagination with collection.
I followed the documentation here:
http://doctrine-orm.readthedocs.org/en/latest/tutorials/pagination.html

It works well in most cases but there is a problem when ordering on a property of the entity + on an other property of the collection.

See the failing unit test I joined to this ticket.



 Comments   
Comment by Bill Schaller [ 16/Jun/15 ]

Closed PR via manual merge





[DDC-3056] Return value mismatch between code under HHVM and Zend Created: 28/Mar/14  Updated: 18/Aug/14  Resolved: 18/Aug/14

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: ORM
Affects Version/s: Git Master
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Andy hunt Assignee: Marco Pivetta
Resolution: Invalid Votes: 0
Labels: hhvm, orm, paginator
Environment:

Two environments:
LAMP stack with PHP 5.4.25 on Ubuntu 12.04
HHVM 3.0.0.-dev (rel) ob Ubuntu 12.04



 Description   

The following code produces differing results under Zend and HHVM runtimes.

// $all::build uses the query builder to select all entities of a type
/** @var \Doctrine\ORM\Query $query **/
$query = $all->build($qb);
$query->setMaxResults($pageSize)->setFirstResult($start);

$paginator = new Paginator($query);
$results = array_values((array)$paginator->getIterator());

Under Zend, $results is a 1-dimensional array containing N elements:
[1, 2, 3].

Under HHVM, $results is a 2-dimensional array containing a single array, containing N elements:
[ [1,3,3] ]



 Comments   
Comment by Christophe Coevoet [ 28/Mar/14 ]

I suggest reporting it to the HHVM team as a bug

Comment by Marco Pivetta [ 28/Mar/14 ]

Also: why are you using an array cast and not iterator_to_array?

Comment by Christophe Coevoet [ 28/Mar/14 ]

@Marco this should be equivalent. Casting a Traversable to array should traverse it. If HHVM does not do it, it is a bug.

Comment by Marco Pivetta [ 28/Mar/14 ]

Christophe Coevoet not really: http://3v4l.org/Z3t4t

Comment by Marco Pivetta [ 18/Aug/14 ]

Resolving as invalid: this seems to be a misuse in my opinion.





[DDC-2954] Paginator loses items Created: 05/Feb/14  Updated: 12/Feb/14

Status: Awaiting Feedback
Project: Doctrine 2 - ORM
Component/s: ORM
Affects Version/s: 2.3.4, 2.4.1
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: Mariusz Jaskółka Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: Pagination, Paginator
Environment:

Linux and Windows, PHP 5, Oracle(OCI8)


Attachments: File LimitSubqueryOutputWalker.php     File LimitSubqueryOutputWalker_bugfix.php    

 Description   

Sometimes when when I use Paginator (Doctrine\ORM\Tools\Pagination\Paginator)

  • with query contains orderBy
  • with $fetchJoinCollection = true
  • lot of joins with toMany associations

there are too few items in result (no, it is not the end of list). There two situations:
1. I have four items on page 1 and two items on page 2 (pageLimit is 5). It is no so bad
2. I have four items on page 1 and there is no page 2 (while there should be 5 all items). This is very bad because I lose one item.

------------------------------------------------------------------
EDIT:
In function Paginator::getIterator there is variable $ids. In my situation it contains five numbers [34,26,34,15,12]. There is duplicated value 34 but ids of top-level entities should be distinct (as far as we do not use CROSS JOIN, or maybe I am wrong).

The Paginator::count function works correctly, it does not count duplicated values twice.

Statement that gets $ids is like following:
SELECT a.* FROM (SELECT DISTINCT ID2, BEGINTIME70 FROM (...) dctrn_result ORDER BY BEGINTIME70 DESC) a WHERE ROWNUM <= 5

------------------------------------------------------------------
EDIT 2 - Bugfix description:
The result items of the query is not unique because of
"SELECT DISTINCT col_with_id, order_by_column (...) ORDER BY order_by_column".
If we had items like following:
(1,A)
(1,B)
(1,B)
(2,C)

After DISTINCT operation the result would be:
(1,A)
(1,B)
(2,C)

But we want to have unique firs column, not pairs. That's why we should do "ORDER BY" before "DISTINCT" - not in the same time.
Please confirm if the solution is correct.



 Comments   
Comment by Marco Pivetta [ 05/Feb/14 ]

Mariusz Jaskółka this needs more details

Comment by Mariusz Jaskółka [ 06/Feb/14 ]

OK, I will try to find out where the problem is.

Comment by Mariusz Jaskółka [ 06/Feb/14 ]

I have edited description, maybe additional information will help.

Comment by Mariusz Jaskółka [ 07/Feb/14 ]

I send the bugfix in attachment. I will describe it soon.

Comment by Mariusz Jaskółka [ 07/Feb/14 ]

Bugfix version 2 - previously I did not notice that oracle loses order after DISTINCT operation. Thus I used GROUP BY.

Comment by Mariusz Jaskółka [ 11/Feb/14 ]

I do not know if I can change status of this issue from "Awaiting Feedback". I can not see such option





[DDC-2890] Paginator generates invalid sql for some dql with setUseOutputWalkers(false) and $fetchJoinCollection = true Created: 07/Jan/14  Updated: 17/Apr/14  Resolved: 17/Apr/14

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: 2.4.1
Fix Version/s: 2.5
Security Level: All

Type: Bug Priority: Major
Reporter: Jiri Kavalik Assignee: Guilherme Blanco
Resolution: Fixed Votes: 0
Labels: paginator
Environment:

ubuntu 12.04, ZF2 2.2.5, mysql 5.5.34



 Description   

We use doctrine paginator in zf2 for list pagination.

We tried to disable UseOutputWalkers because of performance gain - for some entities expected table size is in millions and we are paginating simple lists with some inner joins - but with UseOutputWalkers(false) and fetchJoinCollection=true (default) we get exception for queries ordering by referenced entity id.

Examples:

  • OK - DQL:
    SELECT Transaction FROM Transaction\Entity\Transaction Transaction ORDER BY Transaction.balance asc
    

    SQL:

    SELECT t0_.id AS id0, t0_.value AS value1, t0_.balance AS balance2, t0_.created_on AS created_on3, t0_.type_id AS type_id4, t0_.canceled_id AS canceled_id5, t0_.canceling_id AS canceling_id6, t0_.wallet_id AS wallet_id7 FROM transaction t0_ ORDER BY t0_.balance ASC
    

    Paginator SQL:

    SELECT count(DISTINCT t0_.id) AS sclr0 FROM transaction t0_
    SELECT DISTINCT t0_.id AS id0, t0_.balance AS balance1 FROM transaction t0_ ORDER BY t0_.balance ASC LIMIT 10 OFFSET 0
    SELECT t0_.id AS id0, t0_.value AS value1, t0_.balance AS balance2, t0_.created_on AS created_on3, t0_.type_id AS type_id4, t0_.canceled_id AS canceled_id5, t0_.canceling_id AS canceling_id6, t0_.wallet_id AS wallet_id7 FROM transaction t0_ WHERE t0_.id IN (?) ORDER BY t0_.balance ASC
    
  • Exception - Error producing an iterator - DQL:
    SELECT Transaction FROM Transaction\Entity\Transaction Transaction ORDER BY Transaction.type asc
    

    SQL:

    SELECT t0_.id AS id0, t0_.value AS value1, t0_.balance AS balance2, t0_.created_on AS created_on3, t0_.type_id AS type_id4, t0_.canceled_id AS canceled_id5, t0_.canceling_id AS canceling_id6, t0_.wallet_id AS wallet_id7 FROM transaction t0_ ORDER BY t0_.type_id ASC
    

    Paginator SQL with error:

    SELECT count(DISTINCT t0_.id) AS sclr0 FROM transaction t0_
    SELECT DISTINCT t0_.id AS id0, t0_. AS _1 FROM transaction t0_ ORDER BY t0_.type_id ASC LIMIT 10 OFFSET 0
    SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'AS _1 FROM transaction t0_ ORDER BY t0_.type_id ASC LIMIT 10 OFFSET 0' at line 1
    

    Same query with $fetchJoinCollection = false - OK - paginator SQL:

    SELECT count(DISTINCT t0_.id) AS sclr0 FROM transaction t0_
    SELECT t0_.id AS id0, t0_.value AS value1, t0_.balance AS balance2, t0_.created_on AS created_on3, t0_.type_id AS type_id4, t0_.canceled_id AS canceled_id5, t0_.canceling_id AS canceling_id6, t0_.wallet_id AS wallet_id7 FROM transaction t0_ ORDER BY t0_.type_id ASC LIMIT 10 OFFSET 0
    
  • using setUseOutputWalkers(true) generates most robust queries but count is really slow for 200k+ tables


 Comments   
Comment by Guilherme Blanco [ 17/Apr/14 ]

As of https://github.com/doctrine/doctrine2/commit/be1cc14a9c8641774d614f788103cef4a5373bb1 issue is now fixed.





[DDC-2794] the Paginator does not support arbitrary join Created: 14/Nov/13  Updated: 19/Feb/15  Resolved: 22/Sep/14

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: Git Master, 2.3.4, 2.4.1
Fix Version/s: 2.5
Security Level: All

Type: Improvement Priority: Major
Reporter: Christophe Coevoet Assignee: Marco Pivetta
Resolution: Fixed Votes: 3
Labels: paginator

Issue Links:
Dependency
depends on DDC-3233 [GH-1092] Arbitrary Join count walker... Resolved

 Description   

Using the following query with the paginator fails:

SELECT u
FROM User u
JOIN Message m WITH m.author = u
WHERE m.status = 'draft' -- a condition justifying the join

The CountWalker and the CountOutputWalker both throw a exception saying "Cannot count query which selects two FROM components, cannot make distinction"

This message is wrong (I can make distinction here. Only User is selected in the result set) and confusing (I spent some time finding the second FROM before figuring it was related to the use of an arbitrary join instead of an association join).



 Comments   
Comment by Pierre-Antoine Pinel [ 23/Dec/13 ]

Hi,

I have the same issue.

My query is : SELECT entity FROM W\XBundle\Entity\X entity LEFT JOIN Z\YBundle\Entity\Y y WITH y.x = entity LEFT JOIN y.v v WHERE v.name LIKE :like

Is there a way to fix this issue ? I really need this to work with the Paginator.

Best,

Comment by Christophe Coevoet [ 23/Dec/13 ]

Why changing the status to "Awaiting feedback" ? It is not waiting for a feedback of the reporter to check the fix. It is an open ticket which does not have a fix yet

Comment by Glen Ainscow [ 30/Sep/14 ]

Is this going to be back-ported to 2.4.x?

Comment by Marco Pivetta [ 01/Oct/14 ]

No, this is actually a new supported feature: I'm changing the issue type.

Comment by Sagar [ 19/Feb/15 ]

This issue should be back-ported to 2.4.x as this seems not just new supported feature but should be present in 2.4 as improvements or fix.





[DDC-2752] Item order lost using paginator Created: 20/Oct/13  Updated: 13/Dec/13  Resolved: 13/Dec/13

Status: Closed
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: None
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: Kristof Mariën Assignee: Benjamin Eberlei
Resolution: Won't Fix Votes: 0
Labels: paginator


 Description   

We are using the doctrine pagination tool for faster queries in paginated results.

But doctrine is now creating the following query:

SELECT DISTINCT id0 FROM (
SELECT c0_.id AS id0, c0_.number AS number1
FROM cudi.stock_orders_items c0_
INNER JOIN cudi.stock_orders c1_ ON c0_.order_id = c1_.id
WHERE c1_.date_ordered IS NOT NULL AND c1_.date_ordered > $1 AND c1_.date_ordered < $2
ORDER BY c1_.date_ordered DESC
) dctrn_result LIMIT 25 OFFSET 50

But due to the distinct, the order is lost and the results are not sorted by date_ordered anymore. How can this be solved?

Thanks



 Comments   
Comment by Marco Pivetta [ 20/Oct/13 ]

Kristof Coomans is the order lost already in this result or in the `WHERE IN()` subsequent query?

Comment by Kristof Mariën [ 20/Oct/13 ]

I was digging in the paginator code and this is the query used to select the ids in \Doctrine\ORM\Tools\Pagination\Paginator (line 173)

The subquery from above, outputs the results in the right order. But after applying the distinct the order is messed up. I think this problem needs to be solved by the preserveSqlOrdering function in LimitSubqueryOutputWalker. But this function doesn't add a column to sort on...

Comment by Benjamin Eberlei [ 26/Oct/13 ]

Kristof Mariën Since you have the example already, so you mean the ORDER BY has to be part of the subquery and ALSO added to the outer query? That actually makes sense.

Comment by Kristof Mariën [ 28/Oct/13 ]

That would solve the problem for me.

Comment by Kristof Mariën [ 20/Nov/13 ]

Any progress on this issue?

Comment by Kristof Mariën [ 04/Dec/13 ]

I have found another query with the same problem. The common thing they have is that I want to order on a column of a joined entity.

Comment by Kristof Mariën [ 06/Dec/13 ]

After digging in the code I found that I just had to add the joined entity to my select statement. This issue can be closed





pager produces wrong results on postgresql (DDC-1958)

[DDC-2729] Same bug affects SQLServer2008Platform Created: 07/Oct/13  Updated: 07/Oct/13

Status: Open
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: 2.4
Fix Version/s: None
Security Level: All

Type: Sub-task Priority: Major
Reporter: Rafi Adnan Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: paginator
Environment:

SQL Server 10.50.1617
unixODBC 2.2.14
PHP 5.4.19



 Description   

If the class is patched with the following:

if ($this->platform instanceof SQLServer2008Platform)

{ $this->preserveSqlOrdering($AST, $sqlIdentifier, $innerSql, $sql); }

It fixes the issue.






[DDC-2381] Pagination query can be simplified when simple joins are applied Created: 31/Mar/13  Updated: 14/Apr/15

Status: Open
Project: Doctrine 2 - ORM
Component/s: ORM
Affects Version/s: 2.3, 2.4
Fix Version/s: None
Security Level: All

Type: Improvement Priority: Minor
Reporter: Sergey Gerdel Assignee: Marco Pivetta
Resolution: Unresolved Votes: 3
Labels: paginator

Attachments: HTML File EXPLAIN.htmL     HTML File EXPLAIN1.html     HTML File EXPLAIN2.html     HTML File EXPLAIN3.htm     HTML File EXPLAIN4.htm    

 Description   

Hi.
In mysql db table i have > 200,000 items.
I use native doctrine pagination for paging the items list.
But generated query that gets ids for items list in paging works more then 150 sec on my workstation

SELECT DISTINCT id0 FROM (SELECT m0_.id AS id0, m0_.title AS title1, m0_.text AS text2, m0_.price AS price3, m0_.originalPrice AS originalPrice4, m0_.condition_type AS condition_type5, m0_.image_1 AS image_16, m0_.image_2 AS image_27, m0_.image_3 AS image_38, m0_.image_4 AS image_49, m0_.image_5 AS image_510, m0_.video AS video11, m0_.contact_email AS contact_email12, m0_.contact_name AS contact_name13, m0_.contact_phone AS contact_phone14, m0_.contact_type AS contact_type15, m0_.published AS published16, m0_.type AS type17, m0_.status AS status18, m0_.highlight AS highlight19, m0_.urgent AS urgent20, m0_.topads AS topads21, m0_.period AS period22, m0_.hits AS hits23, m0_.ip AS ip24, m0_.created_at AS created_at25, m0_.updated_at AS updated_at26 FROM milla_message m0_ INNER JOIN milla_currency m1_ ON m0_.currency_id = m1_.id INNER JOIN milla_category m2_ ON m0_.category_id = m2_.id INNER JOIN milla_region m3_ ON m0_.region_id = m3_.id INNER JOIN milla_city m4_ ON m0_.city_id = m4_.id WHERE m0_.status = 1 ORDER BY m0_.published DESC) dctrn_result LIMIT 20 OFFSET 0

source code
https://github.com/doctrine/doctrine2/blob/master/lib/Doctrine/ORM/Tools/Pagination/LimitSubqueryOutputWalker.php#L141

why SELECT DISTINCT %s FROM (%s) dctrn_result ???
why not SELECT DISTINCT m0_.id AS id0 FROM milla_message m0_ WHERE m0_.status = 1 ORDER BY m0_.published DESC LIMIT 20 OFFSET 0



 Comments   
Comment by Marco Pivetta [ 31/Mar/13 ]

Not a blocker

Comment by Marco Pivetta [ 31/Mar/13 ]

What's the result of `EXPLAIN` on a query without the subquery?

Comment by Sergey Gerdel [ 31/Mar/13 ]

explain without the subquery

Comment by Marco Pivetta [ 31/Mar/13 ]

Sergey Gerdel that's not the same query.

Comment by Marco Pivetta [ 31/Mar/13 ]

Sergey Gerdel this is still using

Using index; Using temporary; Using filesort

Check your indexes

Comment by Sergey Gerdel [ 31/Mar/13 ]

Not in the index problem

SELECT DISTINCT id0 FROM (SELECT m0_.id AS id0, m0_.title AS title1, m0_.text AS text2, m0_.price AS price3, m0_.originalPrice AS originalPrice4, m0_.condition_type AS condition_type5, m0_.image_1 AS image_16, m0_.image_2 AS image_27, m0_.image_3 AS image_38, m0_.image_4 AS image_49, m0_.image_5 AS image_510, m0_.video AS video11, m0_.contact_email AS contact_email12, m0_.contact_name AS contact_name13, m0_.contact_phone AS contact_phone14, m0_.contact_type AS contact_type15, m0_.published AS published16, m0_.type AS type17, m0_.status AS status18, m0_.highlight AS highlight19, m0_.urgent AS urgent20, m0_.topads AS topads21, m0_.period AS period22, m0_.hits AS hits23, m0_.ip AS ip24, m0_.created_at AS created_at25, m0_.updated_at AS updated_at26 FROM milla_message m0_ WHERE m0_.status = 1 ORDER BY m0_.published DESC) dctrn_result LIMIT 20 OFFSET 0

Time: 104.614s explain 3

SELECT DISTINCT m0_.id AS id0 FROM milla_message m0_ WHERE m0_.status = 1 ORDER BY m0_.published DESC LIMIT 20 OFFSET 0;

Time: 0.001s explain 4

Comment by Marco Pivetta [ 01/Apr/13 ]

Sergey Gerdel the ORM cannot simplify a complex query that way. There may be a conditional on one of the joined results, or generally usage of one of the joined results.

Things that could be optimized here are:

  • Removal of the `ORDER BY` clause when grouping (check ORM master, I think somebody already did that)
  • Trying to simplify the query by doing some serious hacking on the AST.

The problem I see here is that the chance to spawn random bugs because of the optimization is very high, and you'd have to rewrite `walkSelectStatement`

Comment by Marco Pivetta [ 01/Apr/13 ]

Marking as improvement

Comment by Sergey Gerdel [ 07/Apr/13 ]

Minor?
i have 100 sec for this query.
200k items are selected for temporary table. wtf?

OK. Programmers may be mistaken in parser
expect ORDER BY m0_.published DESC LIMIT 20 OFFSET 0) dctrn_result
Time: 0.001s

reality ORDER BY m0_.published DESC) dctrn_result LIMIT 20 OFFSET 0

Comment by Marco Pivetta [ 07/Apr/13 ]

Sergey Gerdel this problem does not introduce security issues and can be worked around by you while using your own pagination logic. It does not stop you from doing anything, that's why it's minor.

Comment by Sergey Gerdel [ 08/Apr/13 ]

ok)
i have already created my own paginator.
at last
please see how to fix this problem
https://github.com/Sergic/doctrine2/commit/2733c815387273d3bd199a68acb717e0cbc8ccfe

Comment by Tom Pryor [ 11/Oct/13 ]

I've also run into this problem which makes Doctrine's Paginator useless for large datasets. The actual query takes 0.002ms but the SELECT DISTINCT query doctrine executes takes over 30s because MySQL creates a temporary table with 200k+ records.

You don't need to remove the joined tables from the paginator query (I have conditions on the joined tables anyway), this has a negligible impact performance, but rather it is caused by SELECT DISTINCT and ORDER BY which no index configuration can solve. Rather, perhaps a flag could be added to the paginator to indicate my query does not fetch join any has many collections (i.e each row returned will be unique) negating the need for the SELECT DISTINCT. The pagination would only then need to perform the original query with LIMIT and OFFSET applied along with a separate COUNT query on the primary key, both of which are very fast as they'd use the indexes setup for the the original query.

Comment by Christophe Coevoet [ 11/Oct/13 ]

This flag already exists for the select query. See the second argument of the constructor.

For the count query, you should call $paginator->setUseOutputWalkers(false) to make it use a DQL AST walker instead of the SQL Output walker (the AST walker does not support counting on queries using HAVING which is why it is not selected by default)

Comment by Robert (Jamie) Munro [ 29/Nov/13 ]

I think this is more important than "minor", as I've experienced this when upgrading from 2.2. My site became unusably slow.

I can't easily work around it because I am using Symfony bundles that use this, I am not using this directly. None of the workarounds mentioned so far seem to have helped.

Comment by Matthias Pigulla [ 14/Apr/15 ]

http://www.doctrine-project.org/jira/browse/DDC-3646 seems to be related or identical.





[DDC-2283] Paginator with orderBy in joined data retrieve bad result Created: 07/Feb/13  Updated: 26/Feb/13

Status: Open
Project: Doctrine 2 - ORM
Component/s: ORM
Affects Version/s: 2.3.2
Fix Version/s: None

Type: Improvement Priority: Minor
Reporter: Jean-Philippe THEVENOUX Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: paginator


 Description   

entity A have many entity B

If DQL is something like "select A, B from A join B order by A.field1, B.field2"
Then Paginator retrieve different Id by a query like :
"select distinct a.id, a.field1, b.field2 from ( ....) order by a.field1 ASC , b.field2 ASC limit 15 offset 0"

so, if a entity A have 20 entity B (and these sub-entity have all a different b.field2) then there's only 1 A retrieved






[DDC-2236] SUM(..) with Pagination gives incorrect result Created: 11/Jan/13  Updated: 10/Feb/13

Status: In Progress
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: 2.2.3
Fix Version/s: None

Type: Documentation Priority: Minor
Reporter: Oleg Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 1
Labels: paginator
Environment:

Linux



 Description   

https://github.com/whiteoctober/Pagerfanta/issues/69

<?php
$query = $em->getRepository('M\E\Q')
->createQueryBuilder('q')
->select('q', 'SUM(q.price) AS amount')
->where('q.id IN(19, 20, 22)')
->groupBy('q.customer')
;

$pager = new Pagerfanta(new DoctrineORMAdapter($query));
$pager->setMaxPerPage(30);
$pager->setCurrentPage($request->query->get('page', 1));

$result = $pager->getCurrentPageResults();
print_r($result[0]['amount']); // 156.71 - Incorrect

$result = $query->getQuery()->getResult();
print_r($result[0]['amount']); // 553.47
?>

Sql for the above:

SELECT DISTINCT id0 FROM (SELECT q0_.id AS id0, SUM(q0_.price) AS sclr36 FROM Q q0_ WHERE q0_.id IN (19, 20, 22) GROUP BY q0_.customer_id) dctrn_result LIMIT 30 OFFSET 0
SELECT q0_.id AS id0, SUM(q0_.price) AS sclr36 FROM Q q0_ WHERE q0_.id IN (19, 20, 22) AND q0_.id IN ('19') GROUP BY q0_.customer_id
SELECT q0_.id AS id21, SUM(q0_.price) AS sclr36 FROM Q q0_ WHERE q0_.id IN (19, 20, 22) GROUP BY q0_.customer_id

Sql with fetchJoin = false (new DoctrineORMAdapter($query, false))

SELECT q0_.id AS id0, SUM(q0_.price) AS sclr36 FROM Quote q0_ WHERE q0_.id IN (19, 20, 22) GROUP BY q0_.customer_id LIMIT 30 OFFSET 0
SELECT q0_.id AS id0, SUM(q0_.price) AS sclr36 FROM Quote q0_ WHERE q0_.id IN (19, 20, 22) GROUP BY q0_.customer_id



 Comments   
Comment by Alexander [ 09/Feb/13 ]

Can you also test this with doctrine >= 2.3? The pagination code changed quite a lot.

Comment by Oleg [ 10/Feb/13 ]

Looks like no change

composer.json:
"doctrine/orm": "2.3.*",

php composer.phar update
Loading composer repositories with package information
Updating dependencies

  • Installing doctrine/common (2.3.0)
    Loading from cache
  • Installing doctrine/dbal (2.3.2)
    Loading from cache

then cleared cache but result is same
Here's the code

 
$query = $this->getDoctrine()->getEntityManager()->getRepository('MyBundle:Invoice')
  ->createQueryBuilder('q')
  ->select('q', 'SUM(q.amount) AS amount')
  ->groupBy('q.customer')
;
 
95 Connect	root@localhost on **
95 Query	SELECT DISTINCT id0 FROM (SELECT i0_.id AS id0, i0_.invoice_num AS invoice_num1, i0_.date AS date2, i0_.amount AS amount3, i0_.vat_amount AS vat_amount4, i0_.amount_paid AS amount_paid5, i0_.md5 AS md56, i0_.is_exported AS is_exported7, i0_.created AS created8, SUM(i0_.amount) AS sclr9 FROM Invoice i0_ GROUP BY i0_.customer_id) dctrn_result LIMIT 30 OFFSET 0
95 Query	SELECT i0_.id AS id0, i0_.invoice_num AS invoice_num1, i0_.date AS date2, i0_.amount AS amount3, i0_.vat_amount AS vat_amount4, i0_.amount_paid AS amount_paid5, i0_.md5 AS md56, i0_.is_exported AS is_exported7, i0_.created AS created8, SUM(i0_.amount) AS sclr9, i0_.customer_id AS customer_id10 FROM Invoice i0_ WHERE i0_.id IN ('2') GROUP BY i0_.customer_id
95 Query	SELECT i0_.id AS id0, i0_.invoice_num AS invoice_num1, i0_.date AS date2, i0_.amount AS amount3, i0_.vat_amount AS vat_amount4, i0_.amount_paid AS amount_paid5, i0_.md5 AS md56, i0_.is_exported AS is_exported7, i0_.created AS created8, SUM(i0_.amount) AS sclr9, i0_.customer_id AS customer_id10 FROM Invoice i0_ GROUP BY i0_.customer_id
130210 16:08:25	   95 Quit	

But I understand why that happens, it's due to group by and pagination nature.
The first query returns only one row with id "2", second query should be actually "..WHERE i0_.id IN ('2', '3', '4')"

If I do

$pager = new Pagerfanta(new DoctrineORMAdapter($query, false));

I get this sql

SELECT i0_.id AS id0, i0_.invoice_num AS invoice_num1, i0_.date AS date2, i0_.amount AS amount3, i0_.vat_amount AS vat_amount4, i0_.amount_paid AS amount_paid5, i0_.md5 AS md56, i0_.is_exported AS is_exported7, i0_.created AS created8, SUM(i0_.amount) AS sclr9, i0_.customer_id AS customer_id10 FROM Invoice i0_ LIMIT 30 OFFSET 0

I think it should be noted somewhere that if you do groupBy you should set fetchJoin to false?

Comment by Marco Pivetta [ 10/Feb/13 ]

Updating to Documentation issue.





[DDC-2213] Paginator does not work with composite primary key entity Created: 25/Dec/12  Updated: 15/Aug/13

Status: Open
Project: Doctrine 2 - ORM
Component/s: ORM, Tools
Affects Version/s: 2.3.1
Fix Version/s: None
Security Level: All

Type: New Feature Priority: Major
Reporter: Stanislav Anisimov Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 5
Labels: composed, key, paginator
Environment:

php 5.4



 Description   

Paginator does not work with composed primary key.

"Single id is not allowed on composite primary key in entity" exception is thrown here
https://github.com/doctrine/doctrine2/blob/master/lib/Doctrine/ORM/Tools/Pagination/WhereInWalker.php#L90

Only first column values are fetched while retrieving primary keys here
https://github.com/doctrine/doctrine2/blob/master/lib/Doctrine/ORM/Tools/Pagination/Paginator.php#L173



 Comments   
Comment by Marco Pivetta [ 23/Jan/13 ]

Limitation was confused by issue reporter and considered bug

Comment by Austin Morris [ 15/Aug/13 ]

I'd like to add some additional information because the title and description are misleading.

Paginator does work with composite primary key entities. You just cannot use the Paginator when your query does a fetch join with a collection (a one-to-many or many-to-many association). See the documentation for a brief description:
http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/tutorials/pagination.html

In order to use Paginator with a composite primary key entity, you need to instantiate the Paginator with the $fetchJoinCollection flag set to false (it defaults to true). Now, pagination will skip the WhereInWalker which throws an exception when using a composite primary key. Fetch joins with entities (one-to-one or many-to-one) will still work. You can even use a regular join with a collection.

The only "problem" is when your Paginator query calls for a fetch join to a collection. The work around for this is to use a regular join as mentioned above. The drawback is that your paginated entities will not be hydrated with the collection. The collection will have to be lazy-loaded when called upon.





[DDC-1958] pager produces wrong results on postgresql Created: 30/Jul/12  Updated: 05/Dec/14  Resolved: 12/Nov/12

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: 2.3
Fix Version/s: 2.3.1
Security Level: All

Type: Bug Priority: Major
Reporter: Miha Vrhovnik Assignee: Benjamin Eberlei
Resolution: Fixed Votes: 1
Labels: paginator
Environment:
  • Postgres 9.1, 9.2
  • PHP 5.4

Issue Links:
Reference
is referenced by DDC-3336 Undefined property: Doctrine\ORM\Quer... Resolved
is referenced by DDC-3434 LimitSubqueryOutputWalker does not re... Resolved
Sub-Tasks:
Key
Summary
Type
Status
Assignee
DDC-2593 Same bug occurs in MariaDB 5.5 Sub-task Resolved Marco Pivetta  
DDC-2729 Same bug affects SQLServer2008Platform Sub-task Open Benjamin Eberlei  

 Description   

The query build by pager to get the subset of PKs to fetch produces wrong results on potgresql (and probably any database), that conforms to the SQL standard. The standard says, that if you wish to have the results in specific order, then you have to specify that by using an ORDER BY clause. If such a clause is not present the database can return the results in whatever order it sees fit.

Testcase fixtures:

CREATE TABLE test (
    id integer,
    name text
);

INSERT INTO test VALUES (1, 'c');
INSERT INTO test VALUES (2, 'a');
INSERT INTO test VALUES (3, 'e');
INSERT INTO test VALUES (4, 'b');
INSERT INTO test VALUES (5, 'd');
INSERT INTO test VALUES (6, 'a');
INSERT INTO test VALUES (7, 'g');
INSERT INTO test VALUES (8, 'h');
INSERT INTO test VALUES (9, 'e');
INSERT INTO test VALUES (10, 'j');

Passing f.e.

$qb = $this->repository
    ->createQueryBuilder('t')
    ->select('t')
    ->setFirstResult(0)
    ->setMaxResults(5)
    ->addOrderBy('t.name', 'ASC')

to pager produces SQL like this modified for readability

SELECT DISTINCT id FROM (
    SELECT id, name FROM test ORDER BY name
  ) dctrn_result
  LIMIT 5 OFFSET 0

Now there is nothing wrong with this modified query per se, but there is no ORDER BY clause in the outer query so according to the standard the DB can choose whatever order it seems fit. Now mysql chooses the same order, but postgresql does not and it's probably not the only DB doing so.

If you are interested in the results, this is the output I'm seeing:

  • postgresql: 8,4,1,5,3
  • mysql : 2,6,4,1,5

I and my coworker came to the standard compliant solution it was also tested on the dataset above on both postgresql and mysql and it produced equal results. We have found only one corner case this won't work and IMHO that can't be fixed. The problem is when you do a sort on a field from a table that is in 1:n relation to the main table.. e.g tables posts and tags, where one post can have a multiple tags and you want your results sorted by a tag.

Recipe for a correct query is:

  • remember the ORDER BY fields from original query and then remove them
  • wrap the original query with a DISTINCT query, but add the fields from ORDER BY to the SELECT part of that query and add the whole ORDER BY to the end of it, also add the PK to the order by clause, and add the LIMIT clause
  • wrap the resulting query into another query and select just the id.

so if I take the example from above the SQL should look like this:

SELECT id FROM (
  SELECT DISTINCT id, name FROM (
    SELECT id, name FROM test
  ) dctrn_result_inner
  ORDER BY name, id LIMIT 5 OFFSET 0
) dctrn_result


 Comments   
Comment by Jean-Philippe THEVENOUX [ 08/Nov/12 ]

I reproduce same problem with Postgres 7.4, Doctrine 2.3 whereas with doctrine 2.2, all is fine
Hope there'll a fix in next doctrine version

Comment by Raymond Kolbe [ 09/Apr/13 ]

http://www.doctrine-project.org/jira/browse/DDC-1800 This relates.

I just published a PR for an Oracle fix, but your solution appears to work for Oracle as well (issue is the same).

Comment by Bojidar Hristov [ 06/Aug/13 ]

Same bug occurs in MariaDB 5.5.

I commented out check for PostgreSQL and it works fine. Can you fix it for MariaDB too? Thanks

Comment by Guilherme Santos [ 21/Aug/14 ]

I make a ORDER BY with a HIDDEN field and this still happen to me! Like this:
$qb->addSelect('CASE WHEN p.description LIKE :description THEN 0 ELSE 1 END HIDDEN relevance')->addOrderBy('relevance');

This order by is ignored causing the same error!

In this line https://github.com/doctrine/doctrine2/blob/master/lib/Doctrine/ORM/Query/SqlWalker.php#L1310
you don't add to ResultSetMapping so when you verifying in function preserveSqlOrdering the field doesn't exists!

Comment by Liam O'Boyle [ 16/Sep/14 ]

I've just been bitten by the "corner case" described above, "the problem is when you do a sort on a field from a table that is in 1:n relation to the main table.. e.g tables posts and tags, where one post can have a multiple tags and you want your results sorted by a tag.".

This is a pretty significant bug, as the end result is that data that should come back from the query doesn't. While there probably isn't a good universal workaround, the MySQL behaviour before this was already correct because the outer query was returning the ids in the same order as the internal query (even though it isn't required to by the standard). Is it possible to avoid having this apply to MySQL so that it doesn't introduce an additional bug in an attempt to fix an issue that doesn't apply to that platform anyway?

Comment by Miha Vrhovnik [ 16/Sep/14 ]

@Liam As you can se above the same applies to mariadb and if you look at the issues on the githubs doctrine project page you'll see that there is the same problem with newer mysql releases. AS I've written above. this corner case cannot be solved.

Comment by Liam O'Boyle [ 16/Sep/14 ]

Thanks Miha. I couldn't find this on the github page so didn't realise that it was affecting some newer MySQL releases (it didn't seem to affect mine, 5.5). If that's the case, then as you point out it can't even be fixed for MySQL.

Perhaps the lack of support could be more explicit instead? If you attempt to use the paginator with two FROM tables then a RuntimeException is thrown, if we did the same when the ORDER BY conditions applied to tables joined via a 1:m relationship then at least users would know that things were going wrong rather than getting strangely unpredictable results.





[DDC-1935] Not all identifier properties can be found in the ResultSetMapping: id Created: 19/Jul/12  Updated: 29/Jul/12  Resolved: 29/Jul/12

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: Git Master
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: dquintard Assignee: Benjamin Eberlei
Resolution: Duplicate Votes: 0
Labels: paginator


 Description   

$qb1 = $em->createQueryBuilder()->select('c.id');
...
$query1 = $qb1->getQuery();

$doctrinePaginator = new \Doctrine\ORM\Tools\Pagination\Paginator($query1);
$doctrinePaginator->count(); => generate error

But works fine with $query2 !

$qb2 = $em->createQueryBuilder()->select('c');
...
$query2 = $qb2->getQuery();

$doctrinePaginator = new \Doctrine\ORM\Tools\Pagination\Paginator($query2);
$doctrinePaginator->count(); => no error



 Comments   
Comment by dquintard [ 20/Jul/12 ]

It seems that this issue is the same that:
http://www.doctrine-project.org/jira/browse/DDC-1927

Comment by Benjamin Eberlei [ 29/Jul/12 ]

Duplicate of DDC-1927





[DDC-1927] Pagination of a SELECT of specific fields results in a RuntimeException Created: 16/Jul/12  Updated: 12/Oct/14  Resolved: 25/Nov/12

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: ORM
Affects Version/s: Git Master
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: Zacharias Luiten Assignee: Benjamin Eberlei
Resolution: Can't Fix Votes: 0
Labels: paginator


 Description   

When paginating a DQL string which selects specific fields it results in the following error: PHP Fatal error: Uncaught exception 'RuntimeException' with message 'Not all identifier properties can be found in the ResultSetMapping: id'

NOT working: 'SELECT c.id, c.number FROM Application\Entity\Course c'
WORKING: 'SELECT c FROM Application\Entity\Course c'
WORKING: 'SELECT c, c.id, c.number FROM Application\Entity\Course c'

Setting hydration mode to scalar results makes no difference.

Gist to example code and stack trace: https://gist.github.com/d5cd6d0b0ac28e722dd7



 Comments   
Comment by Benjamin Eberlei [ 29/Jul/12 ]

First results: This is very complicated to support, the pagination was designed for entity results. I have to check this when I have more time.

Comment by dquintard [ 08/Aug/12 ]

Hi Benjamin,
I can change my code to select entity results (even for hundreds/thousands results).
But for now Pagination is not usable with ResultSetMapping::addScalarResult

Comment by Matt Pinkston [ 13/Nov/12 ]

The reason this error occurs is because the Paginator creates its own ResultSetMapping and relies on the SqlWalker to configure it (see Doctrine\ORM\Query\SqlWalker::walkSelectExpression).

Doctrine will interpret each field in the first example (SELECT c.id, c.number...) as a PathExpression and add it to the result set mapping as a scalar result. This makes it impossible for the paginator to reliably know which field can be considered an identifier.

A quick fix might be to re-write the query to use PartialObjectExpression: SELECT partial c.

{id, number}

...

(edited after a re-read and realization that a custom ResultSetMapping wouldn't cut it)

Comment by Benjamin Eberlei [ 25/Nov/12 ]

Allowing generic+complex pagination on scalar results is impossible for us, closing as can't fix.

Just use LIMITs yourself here or as suggested partial objects.

Comment by Stefano [ 18/Apr/13 ]

imho this pagination feature is quite useless if we are forced to fetch the complete Entity. Take for example a big table with a lot of data: extracting all the infos will take a lot of time... There should be a way to support the first query type

Comment by Busta Rhymes [ 10/Sep/13 ]

Did i miss something about Doctrine 2 ? What is the point if we can't use pagination? Selecting only several/needed fields is as standard practice as brushing your teeth every day. And pagination doesn't work when i select only specific fields (it throws 'Not all identifier properties can be found in the ResultSetMapping: id' error). I also thought that `$query->setFirstResult(20);` and `$query->setMaxResults(100);` works fine with joins but it turns out it doesn't work. It looks too complicated to integrate such common tool as pagination (I'm switching from CakePHP based ORM which works just fine with any kind of joins etc, no problems with pagination so far, but we are switching to Symfony 2 which goes with Doctrine 2 and actually i was recommended to use Doctrine 2, but i clearly see solid problems atm :/)

Comment by Zacharias Luiten [ 10/Sep/13 ]

@Busta Rhymes - Use partial entities (only the data of the fields you specify is fetched and set in the entity). As pagination is most of the time used just for displaying data, one is fine with that.

Comment by Thomas Godar [ 23/Oct/13 ]

I was pretty disappointed with this resolution. I fail to see how the tasks of a paginator fail to work with Query object foo yet work with Query object bar. They are the same objects. I would think a query is a query for the intent and purpose of the paginator, get a count, set limit and offset, give me back the page of results. All the more so if that Query has HydrationMode set to an array.

What does "generic+complex pagination" even mean?

Comment by Marco Pivetta [ 23/Oct/13 ]

As discussed in private, the problem is that doctrine needs a selected root entity in a DQL query in order to paginate the results.

That's a current limitation that cannot be worked around.

What could be done is giving a "meaning" to scalars being selected, so that they are upgraded to (for example) identifiers of a particular root entity.

That is still not worth it given the amount of bugs and increase in LOC that spawns from it.

Comment by Steve Todorov [ 12/Oct/14 ]

For anybody who might be experiencing this issue, a possible workaround might be:

$paginator = new Paginator($query);
$paginator->setUseOutputWalkers(false);





[DDC-1918] Paginator gives strange results in the end of the resultset Created: 10/Jul/12  Updated: 29/Aug/12  Resolved: 29/Aug/12

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: DQL
Affects Version/s: Git Master
Fix Version/s: Git Master
Security Level: All

Type: Bug Priority: Major
Reporter: Alessandro Tagliapietra Assignee: Benjamin Eberlei
Resolution: Fixed Votes: 0
Labels: paginator
Environment:

Mac OSX MAMP (apache + php 5.3.6 + mysql 5)



 Description   

Hello,
i've an image and a tag entities, image has a one to many with tags. I was using limit and offset and it was giving 7 images instead of 9 (limit is set to 9) due one image has 3 tags. So i started using paginator to resolve the join-fetch issue. It worked fine at the beginning of the resultset, the problem is at the end (i have 29 images), when i increase the offset.
These are the last 3 calls:

  • offset 27, distinct query is:

SELECT DISTINCT i0_.id AS id0, i0_.creation_time AS creation_time1 FROM Image i0_ LEFT JOIN Tag t1_ ON i0_.id = t1_.image WHERE i0_.owner = 1 ORDER BY i0_.creation_time DESC LIMIT 9 OFFSET 27

it returns 2 images, correct, so the final query is

SELECT i0_.id AS id0, i0_.name AS name1, i0_.status AS status2, i0_.last_processing AS last_processing3, i0_.creation_time AS creation_time4, i0_.height AS height5, i0_.width AS width6, i0_.layers AS layers7, i0_.foldersize AS foldersize8, i0_.sourcesize AS sourcesize9, i0_.title AS title10, i0_.description AS description11, i0_.originalUrl AS originalUrl12, i0_.private AS private13, i0_.watermark AS watermark14, i0_.favorite AS favorite15, t1_.id AS id16, t1_.tag AS tag17, i0_.owner AS owner18, t1_.image AS image19 FROM Image i0_ LEFT JOIN Tag t1_ ON i0_.id = t1_.image WHERE i0_.owner = 1 AND i0_.id IN (?, ?) ORDER BY i0_.creation_time DESC

with parameters the 2 images id, still correct.

  • offset 29, distinct query is:

SELECT DISTINCT i0_.id AS id0, i0_.creation_time AS creation_time1 FROM Image i0_ LEFT JOIN Tag t1_ ON i0_.id = t1_.image WHERE i0_.owner = 1 ORDER BY i0_.creation_time DESC LIMIT 9 OFFSET 29

which doesn't get any result, but it still goes on querying without id filter:

SELECT i0_.id AS id0, i0_.name AS name1, i0_.status AS status2, i0_.last_processing AS last_processing3, i0_.creation_time AS creation_time4, i0_.height AS height5, i0_.width AS width6, i0_.layers AS layers7, i0_.foldersize AS foldersize8, i0_.sourcesize AS sourcesize9, i0_.title AS title10, i0_.description AS description11, i0_.originalUrl AS originalUrl12, i0_.private AS private13, i0_.watermark AS watermark14, i0_.favorite AS favorite15, t1_.id AS id16, t1_.tag AS tag17, i0_.owner AS owner18, t1_.image AS image19 FROM Image i0_ LEFT JOIN Tag t1_ ON i0_.id = t1_.image WHERE i0_.owner = 1 ORDER BY i0_.creation_time DESC LIMIT 9 OFFSET 29

and it gets the last 2 images maybe due the fetch join like as i'm not using paginator.



 Comments   
Comment by Alessandro Tagliapietra [ 15/Aug/12 ]

No updates?

Comment by Benjamin Eberlei [ 15/Aug/12 ]

Sorry I am swamped with work at the moment and other important projects, I couldnt look into it yet.

Comment by Alessandro Tagliapietra [ 27/Aug/12 ]

Commenting the if this part:

Paginator.php
            // don't do this for an empty id array
            //if (count($ids) > 0) {
                $namespace = WhereInWalker::PAGINATOR_ID_ALIAS;

                $whereInQuery->setHint(Query::HINT_CUSTOM_TREE_WALKERS, array('Doctrine\ORM\Tools\Pagination\WhereInWalker'));
                $whereInQuery->setHint(WhereInWalker::HINT_PAGINATOR_ID_COUNT, count($ids));
                $whereInQuery->setFirstResult(null)->setMaxResults(null);
                foreach ($ids as $i => $id) {
                    $i++;
                    $whereInQuery->setParameter("{$namespace}_{$i}", $id);
                }
            //}

make it works, for me is better, because if you don't have results to show as the result of the select distinct is empty you shouldn't have any result at all. Because if you don't add the "WHERE IN" query you get the results joined with no difference between using or not the paginator.

Comment by Benjamin Eberlei [ 29/Aug/12 ]

This is weird, an empty IN() normally produces an error. We could add a non matching condition instead, like 0 = 1.

Comment by Alessandro Tagliapietra [ 29/Aug/12 ]

Not really, in my case (mysql) it creates the following query:

SELECT i0_.id AS id0, i0_.name AS name1, i0_.status AS status2, i0_.last_processing AS last_processing3, i0_.creation_time AS creation_time4, i0_.height AS height5, i0_.width AS width6, i0_.layers AS layers7, i0_.foldersize AS foldersize8, i0_.sourcesize AS sourcesize9, i0_.title AS title10, i0_.description AS description11, i0_.originalUrl AS originalUrl12, i0_.private AS private13, i0_.watermark AS watermark14, i0_.favorite AS favorite15, t1_.id AS id16, t1_.tag AS tag17, i0_.owner AS owner18, t1_.image AS image19 FROM Image i0_ LEFT JOIN Tag t1_ ON i0_.id = t1_.image WHERE i0_.owner = 1 AND i0_.id IS NULL ORDER BY i0_.creation_time DESC

so it sets "AND i0_.id IS NULL" which solves the problem, the best solution for me is to directly return an empty set if the DISTINCT gives no results.

Comment by Alessandro Tagliapietra [ 29/Aug/12 ]

Solved with commit bc2476f342ceda98559b87a314806581cf969796





Generated at Wed Jul 29 15:55:10 UTC 2015 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.