[DDC-1958] pager produces wrong results on postgresql Created: 30/Jul/12  Updated: 06/Aug/13  Resolved: 12/Nov/12

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: Tools
Affects Version/s: 2.3
Fix Version/s: 2.3.1
Security Level: All

Type: Bug Priority: Major
Reporter: Miha Vrhovnik Assignee: Benjamin Eberlei
Resolution: Fixed Votes: 1
Labels: paginator
Environment:
  • Postgres 9.1, 9.2
  • PHP 5.4

Sub-Tasks:
Key
Summary
Type
Status
Assignee
DDC-2593 Same bug occurs in MariaDB 5.5 Sub-task Resolved Marco Pivetta  
DDC-2729 Same bug affects SQLServer2008Platform Sub-task Open Benjamin Eberlei  

 Description   

The query build by pager to get the subset of PKs to fetch produces wrong results on potgresql (and probably any database), that conforms to the SQL standard. The standard says, that if you wish to have the results in specific order, then you have to specify that by using an ORDER BY clause. If such a clause is not present the database can return the results in whatever order it sees fit.

Testcase fixtures:

CREATE TABLE test (
    id integer,
    name text
);

INSERT INTO test VALUES (1, 'c');
INSERT INTO test VALUES (2, 'a');
INSERT INTO test VALUES (3, 'e');
INSERT INTO test VALUES (4, 'b');
INSERT INTO test VALUES (5, 'd');
INSERT INTO test VALUES (6, 'a');
INSERT INTO test VALUES (7, 'g');
INSERT INTO test VALUES (8, 'h');
INSERT INTO test VALUES (9, 'e');
INSERT INTO test VALUES (10, 'j');

Passing f.e.

$qb = $this->repository
    ->createQueryBuilder('t')
    ->select('t')
    ->setFirstResult(0)
    ->setMaxResults(5)
    ->addOrderBy('t.name', 'ASC')

to pager produces SQL like this modified for readability

SELECT DISTINCT id FROM (
    SELECT id, name FROM test ORDER BY name
  ) dctrn_result
  LIMIT 5 OFFSET 0

Now there is nothing wrong with this modified query per se, but there is no ORDER BY clause in the outer query so according to the standard the DB can choose whatever order it seems fit. Now mysql chooses the same order, but postgresql does not and it's probably not the only DB doing so.

If you are interested in the results, this is the output I'm seeing:

  • postgresql: 8,4,1,5,3
  • mysql : 2,6,4,1,5

I and my coworker came to the standard compliant solution it was also tested on the dataset above on both postgresql and mysql and it produced equal results. We have found only one corner case this won't work and IMHO that can't be fixed. The problem is when you do a sort on a field from a table that is in 1:n relation to the main table.. e.g tables posts and tags, where one post can have a multiple tags and you want your results sorted by a tag.

Recipe for a correct query is:

  • remember the ORDER BY fields from original query and then remove them
  • wrap the original query with a DISTINCT query, but add the fields from ORDER BY to the SELECT part of that query and add the whole ORDER BY to the end of it, also add the PK to the order by clause, and add the LIMIT clause
  • wrap the resulting query into another query and select just the id.

so if I take the example from above the SQL should look like this:

SELECT id FROM (
  SELECT DISTINCT id, name FROM (
    SELECT id, name FROM test
  ) dctrn_result_inner
  ORDER BY name, id LIMIT 5 OFFSET 0
) dctrn_result


 Comments   
Comment by Jean-Philippe THEVENOUX [ 08/Nov/12 ]

I reproduce same problem with Postgres 7.4, Doctrine 2.3 whereas with doctrine 2.2, all is fine
Hope there'll a fix in next doctrine version

Comment by Raymond Kolbe [ 09/Apr/13 ]

http://www.doctrine-project.org/jira/browse/DDC-1800 This relates.

I just published a PR for an Oracle fix, but your solution appears to work for Oracle as well (issue is the same).

Comment by Bojidar Hristov [ 06/Aug/13 ]

Same bug occurs in MariaDB 5.5.

I commented out check for PostgreSQL and it works fine. Can you fix it for MariaDB too? Thanks

Generated at Sat Aug 02 06:52:55 UTC 2014 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.