[DBAL-1081] Paginator - Query Limit for SQL Server - SqlServerPlatform.php Created: 16/Dec/14  Updated: 16/Jun/15

Status: Open
Project: Doctrine DBAL
Component/s: None
Affects Version/s: 2.5
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Maciej Grajcarek Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: dbal, dql, orderBy, sqlserver
Environment:

Windows



 Description   

Hi!
I have a problem with Query results limit when ordering by SUM of a field.

My query looks like this:

            SELECT ypk.name keyword, SUM(ypk.value) total
            FROM YpBundle:YellowPageKeyword ypk
              JOIN ypk.yellowpage yp
              JOIN yp.listing l
            WHERE l.customer = :customerId
              AND ypk.originDate >= :dateFrom
              AND ypk.originDate <= :dateTo
            GROUP BY ypk.nameCrc, ypk.name
            ORDER BY SUM(ypk.value) DESC

First I was catching following error:
message: "[Syntax Error] line 0, col 395: Error: Expected known function, got 'SUM'"
class: Doctrine\ORM\Query\QueryException

It only accourse if SUM is used in ORDER BY clause.

I have registered new class Sum which extends FunctionNode.

Now, query is build and executed but it has an error:

'SELECT * FROM 
     (SELECT y0_.name AS name_0, SUM(y0_.value) AS sclr_1, ROW_NUMBER() OVER (ORDER BY value) DESC) AS doctrine_rownum FROM yellowpage_keywords y0_ INNER JOIN yellowpages y1_ ON y0_.yellowpage_id = y1_.id INNER JOIN listings l2_ ON y1_.listing_id = l2_.id WHERE l2_.customer_id = ? AND y0_.origin_date >= ? AND y0_.origin_date <= ? GROUP BY y0_.name_crc, y0_.name) AS doctrine_tbl WHERE doctrine_rownum BETWEEN 1 AND 10 ORDER BY doctrine_rownum' with params ["111", "2013-01-01 00:00:00.000000", "2014-12-31 23:59:59.000000"]

The line :

ROW_NUMBER() OVER (ORDER BY value) DESC)

should look like

ROW_NUMBER() OVER (ORDER BY SUM(y0_.value) DESC)

In doModifyLimitQuery method I have modified:

                $pattern = sprintf('/%s\.%s\s+(?:AS\s+)?([^,\s)]+)/i', $column['table'], $column['column']);

to:

                $pattern = sprintf('/%s\.%s\s+(?:AS\s+)?([^,\s)]+)/i', str_replace('(', '\(', $column['table']), str_replace(')', '\)', $column['column']));

Now preg_match founds matching strings and OVER part of query is build correctly.

I checked other issues about this problem (which are marked as already fixed) and I have no idea why it's not working for me.

Thanks in advance!



 Comments   
Comment by Marco Pivetta [ 17/Dec/14 ]

Seems like a bit of information is missing: Is this issue related to the paginator API or not?

Comment by Maciej Grajcarek [ 17/Dec/14 ]

First of all - thank you for formatting my issue.
Secondly yes - issue is directly connected with paginator API.

Here is a code which should help you to replicate the problem:

            SELECT ypk.name keyword, SUM(ypk.value) total
            FROM YpBundle:YellowPageKeyword ypk
              JOIN ypk.yellowpage yp
              JOIN yp.listing l
            WHERE l.customer = :customerId
              AND ypk.originDate >= :dateFrom
              AND ypk.originDate <= :dateTo
            GROUP BY ypk.nameCrc, ypk.name
            ORDER BY SUM(ypk.value) DESC

        $dql = $this->getEntityManager()->createQuery($query);
        $dql
            ->setParameter('customerId', $customerId)
            ->setParameter('dateFrom', $dateFrom)
            ->setParameter('dateTo', $dateTo);
        $dql->setMaxResults(10);    

        $keywords = $dql->getArrayResult(); 
Comment by Marco Pivetta [ 17/Dec/14 ]

That doesn't involve the paginator, just DQL.

SUM() and computed values are not supported in the ORDER BY clause: you have to select them first. Try:

            SELECT ypk.name keyword, SUM(ypk.value) total
            FROM YpBundle:YellowPageKeyword ypk
              JOIN ypk.yellowpage yp
              JOIN yp.listing l
            WHERE l.customer = :customerId
              AND ypk.originDate >= :dateFrom
              AND ypk.originDate <= :dateTo
            GROUP BY ypk.nameCrc, ypk.name
            ORDER BY total DESC
Comment by Maciej Grajcarek [ 18/Dec/14 ]

If I will do it, it will result in a following query:

SELECT *
FROM (
	SELECT y0_.name AS name_0, SUM(y0_.value) AS sclr_1, ROW_NUMBER() OVER (ORDER BY sclr_1 DESC) AS doctrine_rownum 
	FROM yellowpage_keywords y0_
	INNER JOIN yellowpages y1_ ON y0_.yellowpage_id = y1_.id
	INNER JOIN listings l2_ ON y1_.listing_id = l2_.id
	WHERE l2_.customer_id = 111
		AND y0_.origin_date >= '2014-01-01'
		AND y0_.origin_date <= '2014-12-01'
	GROUP BY y0_.name_crc, y0_.name
) AS doctrine_tbl
WHERE doctrine_rownum BETWEEN 1 AND 10
ORDER BY doctrine_rownum

It's an incorrect query for an SQL Server. Take a look on this part:

ROW_NUMBER() OVER (ORDER BY sclr_1 DESC) AS doctrine_rownum

SQL Server do not support aliasing in OVER clause.

It should look like this, to work:

ROW_NUMBER() OVER (ORDER BY SUM(y0_.value) DESC) AS doctrine_rownum

It looks like this is the copy of this issue: http://www.doctrine-project.org/jira/browse/DBAL-788
I'm on doctrine version 2.5 which has patch from that issue included, but I have no idea why it's not working for me.

Comment by Steve Müller [ 24/Dec/14 ]

Maciej Grajcarek looks like it is an issue with your SUM function implementation. If you change your DQL to use ORDER BY COUNT(ypk.value) instead of ORDER BY SUM(ypk.value), does it work then? If so, there is something wrong with your SUM function and therefore not an issue with DBAL.

Comment by Steve Müller [ 24/Dec/14 ]

well forget about it I think I am wrong here.

Comment by Maciej Grajcarek [ 29/Dec/14 ]

Hi,
I will try to use COUNT as soon as I will be able to do that.

But I already tried other aggregation functions before and the result was exactly the same.

Comment by Luca Cerretani [ 16/Jun/15 ]

I get a similar error using this sql:

$sql = "SELECT table_one.id, table_one.number, table_two.name " . 
           "FROM table_one " .
	   "LEFT JOIN table_two	ON table_two.table_one_id = table_one.id " .
           "ORDER BY table_one.id DESC";

using the doModifyLimitQuery i get the uncorrect sql

SELECT * FROM (SELECT table_one.id, table_one.number, table_two.name, ROW_NUMBER() OVER (ORDER BY id DESC) AS doctrine_rownum  FROM table_one LEFT JOIN table_two ON table_two.table_one_id = table_one.id) AS doctrine_tbl WHERE doctrine_rownum BETWEEN 1 AND 1 ORDER BY doctrine_rownum

it should be

SELECT * FROM (SELECT table_one.id, table_one.number, table_two.name, ROW_NUMBER() OVER (ORDER BY table_one.id DESC) AS doctrine_rownum  FROM table_one LEFT JOIN table_two ON table_two.table_one_id = table_one.id) AS doctrine_tbl WHERE doctrine_rownum BETWEEN 1 AND 1 ORDER BY doctrine_rownum




[DBAL-1070] AzureSQL specificities are not taken into account Created: 11/Dec/14  Updated: 11/Dec/14

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.5
Fix Version/s: 2.4.3

Type: Bug Priority: Major
Reporter: Nicolas Séverin Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: azure, dbal, sqlserver
Environment:

Azure website using an AzureSQL database (based on SQL Server 2012).



 Description   

In Azure SQL, table ‘sys.extended_properties’ does not exist.
But SQLAzurePlatform inherits from class SQLServerPlatform, which uses it.
The code in question is here /Doctrine/DBAL/Platforms/SQLServerPlatform.php#L845.

Modifications to this portion of code were made to handle comments on columns differently in doctrine/dbal 2.5, but it used to work in 2.4.3.



 Comments   
Comment by Marco Pivetta [ 11/Dec/14 ]

Reverted to priority Major





[DBAL-1058] It seems that MSSQL syntax was changed Created: 05/Dec/14  Updated: 29/Jan/15  Resolved: 12/Jan/15

Status: Resolved
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.4, 2.5
Fix Version/s: 2.4.4, 2.6, 2.5.1

Type: Bug Priority: Blocker
Reporter: man4red Assignee: Marco Pivetta
Resolution: Fixed Votes: 0
Labels: dbal, sqlserver

Issue Links:
Reference
is referenced by DBAL-1060 [GH-736] [DBAL-1058] Fix database and... Resolved
is referenced by DBAL-1061 [GH-737] [DBAL-1058] [2.4] Fix databa... Resolved

 Description   

I'm using dblib, MSSQL (2012).
So, problem is here:

doctrine-module orm:schema-tool:update --dump-sql

Doctrine\DBAL\Driver\PDOException: SQLSTATE[HY000]: General error: 20018 Invalid object name 'SYS.SCHEMAS'. [20018] (severity 16) [SELECT name FROM SYS.SCHEMAS WHERE name NOT IN('guest', 'INFORMATION_SCHEMA', 'sys')] in /var/www/domains/internal.dc.hayas.ru/data/partners.zf2/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php on line 106

So it seems, that problems is here:

Doctrine\DBAL\Platforms\SQLServerPlatform.php
At Line 1036

    public function getListNamespacesSQL()
    {
        return "SELECT name FROM SYS.SCHEMAS WHERE name NOT IN('guest', 'INFORMATION_SCHEMA', 'sys')";
    }

SQL Server >= 2005 uses sys.schemas (lowercase)

Maybe need to add to SQLServer2005Platform.php

SELECT name FROM sys.schemas ...

and also at line 1028 SQLServerPlatform.php

    public function getListDatabasesSQL()
    {
        return 'SELECT * FROM SYS.DATABASES';
    }

add to SQLServer2005Platform.php

    public function getListDatabasesSQL()
    {
        return 'SELECT * FROM sys.databases';
    }


 Comments   
Comment by Steve Müller [ 05/Dec/14 ]

man4red thanks for reporting. I'll have a look at it this evening. Weird that the functional tests pass though in my setup :S

Comment by Marco Pivetta [ 05/Dec/14 ]

Steve Müller please note that he is using dblib, which (afaik) we do not officially support.

Comment by man4red [ 05/Dec/14 ]

I've checked by direct query to SQL via SQL Management Studio.
Got multiple servers with a diffirent versions.

Here some test

QUERY:

 SELECT name FROM SYS.SCHEMAS WHERE name NOT IN('guest', 'INFORMATION_SCHEMA', 'sys') 

9.0.5069 (SQL Server 2005 Service Pack 4) PASS
10.50.4000.0 (2008 R2 SP2) FAIL
11.0.5058 (SQL Server 2012) FAIL

QUERY:

SELECT name FROM sys.schemas WHERE name NOT IN('guest', 'INFORMATION_SCHEMA', 'sys')

9.0.5069 (SQL Server 2005 Service Pack 4) PASS
10.50.4000.0 (2008 R2 SP2) PASS
11.0.5058 (SQL Server 2012) PASS

I've tested on 5 servers 11.0.5058 (SQL Server 2012).
QUERY:

SELECT name FROM SYS.SCHEMAS WHERE name NOT IN('guest', 'INFORMATION_SCHEMA', 'sys')

Failed on each of them

Other tests:

QUERY:

SELECT * FROM SYS.DATABASES

9.0.5069 (SQL Server 2005 Service Pack 4) PASS
10.50.4000.0 (2008 R2 SP2) PASS
11.0.5058 (SQL Server 2012) PASS

QUERY:

SELECT * FROM sys.databases

9.0.5069 (SQL Server 2005 Service Pack 4) PASS
10.50.4000.0 (2008 R2 SP2) PASS
11.0.5058 (SQL Server 2012) PASS

by the way - is it neccessary to query * from SYS.DATABASES ?

Doctrine\DBAL\Platforms\SQLServerPlatform.php

Line 1030

    public function getListDatabasesSQL()
    {
        return 'SELECT * FROM SYS.DATABASES';
    }

Maybe need to query only names? (name field)
Just asking

Comment by man4red [ 05/Dec/14 ]

According to tests I've added next code to SQLServer2008Platform.php

    /**
     * {@inheritDoc}
     */
    public function getListNamespacesSQL()
    {
        return "SELECT name FROM sys.schemas WHERE name NOT IN('guest', 'INFORMATION_SCHEMA', 'sys')";
    }

And modified my ZF2 application doctrine config config/autoload/doctrine.local.php (platform added):

return array(
    'doctrine' => array(
        'connection' => array(
            'orm_default' => array(
                'driverClass' => 'class to work with dblib',
                'params' => array(
                    'host' => 'hostname',
                    'port' => 1433,
                    'user' => 'user',
                    'password' => 'pass',
                    'dbname' => 'database',
                    'platform' => new Doctrine\DBAL\Platforms\SQLServer2012Platform()
                )
            )
        )
    )
);

Now I've got no issues with MSSQL 2012
I hope my fix was correct

Comment by Steve Müller [ 05/Dec/14 ]

Patch provided: https://github.com/doctrine/dbal/pull/736

Comment by Doctrine Bot [ 05/Dec/14 ]

A related Github Pull-Request [GH-736] was assigned:
https://github.com/doctrine/dbal/pull/736

Comment by man4red [ 05/Dec/14 ]

Dear friends,

I'm new here, and I don't know how all this works here, but can you help me?
As always when one bug fixed - another two produced

Now I've got another problem.
ZendDeveloperTool throws Exception

Uncaught exception 'PDOException' with message 'You cannot serialize or unserialize PDO instances'

of course because of my

    'platform' => new Doctrine\DBAL\Platforms\SQLServer2012Platform()

ok... my mistake

let's fix it in ZF2 way

    'platform' => 'Doctrine\DBAL\Platforms\SQLServer2012Platform'

Now we got another exception:

Doctrine\DBAL\DBALException: Invalid 'platform' option specified, need to give an instance of \Doctrine\DBAL\Platforms\AbstractPlatform.

let's look to doctrine\dbal\lib\Doctrine\DBAL\Connection.php Line: 387

    private function detectDatabasePlatform()
    {
        ...
        } elseif ($this->_params['platform'] instanceof Platforms\AbstractPlatform) {
            $this->platform = $this->_params['platform'];
        } else {
            throw DBALException::invalidPlatformSpecified();
        }
        ...
    }

So my question is

Can we implemet a feature and change this

    private function detectDatabasePlatform()
    {
        if ( ! isset($this->_params['platform'])) {
            $version = $this->getDatabasePlatformVersion();

            if (null !== $version) {
                $this->platform = $this->_driver->createDatabasePlatformForVersion($version);
            } else {
                $this->platform = $this->_driver->getDatabasePlatform();
            }
        } elseif ($this->_params['platform'] instanceof Platforms\AbstractPlatform) {
            $this->platform = $this->_params['platform'];
        } else {
            throw DBALException::invalidPlatformSpecified();
        }

        $this->platform->setEventManager($this->_eventManager);
    }

to this (or similar)

    private function detectDatabasePlatform()
    {
        if (! isset($this->_params['platform'])) {
            $version = $this->getDatabasePlatformVersion();

            if (null !== $version) {
                $this->platform = $this->_driver->createDatabasePlatformForVersion($version);
            } else {
                $this->platform = $this->_driver->getDatabasePlatform();
            }
        } elseif ($this->_params['platform'] instanceof Platforms\AbstractPlatform) {
            $this->platform = $this->_params['platform'];
        } elseif (is_subclass_of($this->_params['platform'], 'Doctrine\DBAL\Platforms\AbstractPlatform')) {
            $this->platform = new $this->_params['platform']();
        } else {
            throw DBALException::invalidPlatformSpecified();
        }

        $this->platform->setEventManager($this->_eventManager);
    }

or this problem is only mine and I need to fix it by my self and to write some forks/mods etc?

Thx for your help anyway

Comment by Marco Pivetta [ 05/Dec/14 ]

man4red that seems to be related with DBAL-1057 - I'll mark this issue as resolved.

Comment by man4red [ 05/Dec/14 ]

Marco Pivetta, thx! Is there planned some big reworking of this section, am I right?
Am I need to post my last comment to DBAL-1057 thread?

Comment by Marco Pivetta [ 05/Dec/14 ]

man4red this section needs some work for 2.5.1, yes. As for posting to DBAL-1057, please do, but only the bits that may be relevant and that you feel that add up to the discussion without cluttering it.

Comment by Doctrine Bot [ 12/Jan/15 ]

A related Github Pull-Request [GH-736] was merged:
https://github.com/doctrine/dbal/pull/736

Comment by Marco Pivetta [ 12/Jan/15 ]

Fixed in DBAL-1060

Comment by Doctrine Bot [ 23/Jan/15 ]

A related Github Pull-Request [GH-783] was closed:
https://github.com/doctrine/dbal/pull/783

Comment by Doctrine Bot [ 29/Jan/15 ]

A related Github Pull-Request [GH-783] was closed:
https://github.com/doctrine/dbal/pull/783





Generated at Sun Aug 30 03:57:22 EDT 2015 using JIRA 6.4.10#64025-sha1:5b8b74079161cd76a20ab66dda52747ee6701bd6.