[DBAL-1178] Mapping import errors on SQL Server 2000 Created: 19/Mar/15  Updated: 23/Mar/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.5.1
Fix Version/s: 2.5.2
Security Level: All

Type: Bug Priority: Critical
Reporter: Ciro Vargas Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: dbal, mapping, sqlserver
Environment:

Microsoft SQL Server 2000 - 8.00.2066 (Intel X86)
May 11 2012 18:41:14
Copyright (c) 1988-2003 Microsoft Corporation
Standard Edition on Windows NT 5.2 (Build 3790: Service Pack 2)


Attachments: PNG File System tables.png    

 Description   

The Doctrine documentation says:

SQLServerPlatform for version 2000 and above.

And the mapping query on the platform SQLServerPlatform.php is:

    public function getListTableColumnsSQL($table, $database = null)
    {
        return "SELECT    col.name,
                          type.name AS type,
                          col.max_length AS length,
                          ~col.is_nullable AS notnull,
                          def.definition AS [default],
                          col.scale,
                          col.precision,
                          col.is_identity AS autoincrement,
                          col.collation_name AS collation,
                          CAST(prop.value AS NVARCHAR(MAX)) AS comment -- CAST avoids driver error for sql_variant type
                FROM      sys.columns AS col
                JOIN      sys.types AS type
                ON        col.user_type_id = type.user_type_id
                JOIN      sys.objects AS obj
                ON        col.object_id = obj.object_id
                JOIN      sys.schemas AS scm
                ON        obj.schema_id = scm.schema_id
                LEFT JOIN sys.default_constraints def
                ON        col.default_object_id = def.object_id
                AND       col.object_id = def.parent_object_id
                LEFT JOIN sys.extended_properties AS prop
                ON        obj.object_id = prop.major_id
                AND       col.column_id = prop.minor_id
                AND       prop.name = 'MS_Description'
                WHERE     obj.type = 'U'
                AND       " . $this->getTableWhereClause($table, 'scm.name', 'obj.name');
    }

But the system tables in the database is (attached)

There are several errors in queries using tables and fields that do not exist in SQL Server (in all map methods). So I can not map the entities



 Comments   
Comment by Marco Pivetta [ 19/Mar/15 ]

Ciro Vargas we don't support SQLServer 2000: the oldest version we support is SQLServer 2005 as per https://github.com/doctrine/dbal/blob/3b901cd314d1f79a54c93131d634aad507114b34/lib/Doctrine/DBAL/Platforms/SQLServer2005Platform.php

Comment by Benjamin Eberlei [ 19/Mar/15 ]

This doesnt end the world for you, you must extend from the SQL Server platform and change what you need.

Comment by Ciro Vargas [ 19/Mar/15 ]

Marco Pivetta http://doctrine-dbal.readthedocs.org/en/latest/reference/platforms.html in the docs says:

"SQLServerPlatform for version 2000 and above."

The right way is create mapping querys in the version platform and override on the upper, right? The doctrine team are updating the base platform

Comment by Ciro Vargas [ 19/Mar/15 ]

Benjamin Eberlei yes, i can do. Or mapping hardcoded

Comment by Ciro Vargas [ 19/Mar/15 ]

See on https://github.com/doctrine/dbal/blob/2a9e9943f33610bfde4637abeafe00edd201803c/lib/Doctrine/DBAL/Platforms/SQLServerPlatform.php

that's works no fine, but works

the right is update for each database version, no update only for the last.

Down versions of 2012 can be bugged too

Comment by Ciro Vargas [ 23/Mar/15 ]

Solved https://gist.github.com/cirovargas/e5c0bfbc404bb414bb2b





[DBAL-943] dbal db2 platform uses incorrect column modification strategy for clob Created: 20/Jul/14  Updated: 20/Jul/14

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: None
Fix Version/s: None
Security Level: All

Type: Bug Priority: Critical
Reporter: chris rehfeld Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None
Environment:

db2 v10.5 centos 6.5



 Description   

db2 only allows certain column types to be used in an ALTER TABLE ALTER COLUMN myCol ... type statement.

Example entity

Widget2.php
<?php
/**
 * @ORM\Entity
 **/
class Widget
{
    /** @ORM\Id @ORM\Column(type="integer") @ORM\GeneratedValue **/
    protected $id;

    /** @ORM\Column(type="string") **/
    private $str;
}

orm:schema-tool:create produces:
CREATE TABLE Widget2 (id INTEGER GENERATED BY DEFAULT AS IDENTITY NOT NULL, str VARCHAR(255) DEFAULT NULL, PRIMARY KEY(id));

If you then change the column type from string to text via ...

Widget2.php
    /** @ORM\Column(type="text") **/
    private $str;

... and you then run orm:schema-tool:update, it will try to run:

ALTER TABLE WIDGET2 ALTER STR str CLOB(1M) NOT NULL;

The sql syntax is wrong, but that's a different issue I'll address elsewhere.Lets assume it's fixed to be proper syntax:
ALTER TABLE WIDGET2 ALTER COLUMN str SET DATA TYPE CLOB(1M) ALTER COLUMN str SET NOT NULL;

This triggers sql error code -190, sqlstate 42837
http://www-01.ibm.com/support/knowledgecenter/SSEPEK_10.0.0/com.ibm.db2z10.doc.codes/src/tpc/n190.dita

Because the from type => to type isn't valid. This link:
http://www-01.ibm.com/support/knowledgecenter/SSEPGG_9.1.0/com.ibm.db2.udb.admin.doc/doc/r0000888.htm?lang=en
seems to list the valid alterations.

I'm guessing that a different strategy needs to be used; where we drop the old column, and create the new one in such scenarios. This could cause unexpected data loss though.






[DBAL-968] SQL Server modifyLimitQuery broken Created: 11/Aug/14  Updated: 11/Aug/14

Status: Open
Project: Doctrine DBAL
Component/s: Drivers, Platforms
Affects Version/s: 2.5
Fix Version/s: None
Security Level: All

Type: Bug Priority: Critical
Reporter: Bill Schaller Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: paginator, sqlserver
Environment:

SQL Server



 Description   

The recent change to SQLServerPlatform.php @jaylinski:improved sqlserver 'doModifyLimitQuery' select-from pattern broke the ORM Paginator's queries on SQL server.

I investigated, and found that some of the test cases for the SQL Server platform weren't actually correct SQL. Also, there were no test cases that covered what the paginator is doing, so I've written test cases for those. I will open a pull request for this issue.

The modifyLimitQuery method in SQLServerPlatform.php should be fixed to pass the fixed old tests and the new tests.

My concern is that that method is becoming too complex, but that's an issue for another day.



 Comments   
Comment by Marco Pivetta [ 11/Aug/14 ]

I'm gonna cry. Thank you, MSSQL, you make our lives so much "easier"





[DBAL-1057] Connection is not lazy anymore when guessing the platform is necessary Created: 05/Dec/14  Updated: 16/Jan/15

Status: Open
Project: Doctrine DBAL
Component/s: Drivers, Platforms
Affects Version/s: 2.5
Fix Version/s: None
Security Level: All

Type: Bug Priority: Critical
Reporter: Christophe Coevoet Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 13
Labels: None

Issue Links:
Duplicate
is duplicated by DBAL-1067 mysql: selecting db issue Resolved
Reference
is referenced by DDC-3475 Avoid db connection in constructor Open

 Description   

In DBAL 2.5, many driver can rely on different versions of the platform. Unless the version is explicitly provided, the driver will guess it at instantiation time, killing the lazyness of the connection.
This is a critical issue in any context using DI as it means that injecting the connection into anything else will connect to the server.



 Comments   
Comment by Christophe Coevoet [ 05/Dec/14 ]

Actually, the Connection class itself defers the guessing until the first time the platform is accessed. But many places in DBAL and in the ORM are retrieving the platform and storing it in a property of the class at instantiation time to avoid method calls when they need to access the platform. So this might be much harder to fix

Comment by Steve Müller [ 05/Dec/14 ]

Christophe Coevoet Can we analyze the use cases where retrieving the platform is necessary before actually connecting? I only know the custom type registering so far... Maybe we can defer that somehow?

Comment by Christophe Coevoet [ 05/Dec/14 ]

Steve Müller the issue is that many places in DBAL and the ORM are retrieving the connection before they use it. the Connection class itself in DBAL is already deferring the guessing

Comment by Steve Müller [ 05/Dec/14 ]

Christophe Coevoet I know that. The question is WHY do those defered connections need to access Connection::getDatabasePlatform() without connecting? What are the use cases?

Comment by Christophe Coevoet [ 05/Dec/14 ]

Steve Müller The issue is that all those objects are calling $connection->getDatabasePlatform() in their own constructor to store a reference to it for faster access later (no more method calls). This means that *instantiating* the ORM (or some parts of DBAL) triggers the platform guessing, which connects to the DB. This breaks the lazyness and hurts DI contexts (maybe the ORM will not even be used in this process, but it was instantiated because of being a dependency in a complex object graph).

Comment by Christophe Coevoet [ 05/Dec/14 ]

and the issue is precisely that all these parts of Doctrine are *not* deferring the retrieval of the platform.

Comment by Steffen Brem [ 07/Dec/14 ]

This is causing a lot of issues when using CI servers. Where it is very important that those things are lazy, since you do not have the database configured on most applications that build on a CI server.

Comment by Craig Heydenburg [ 01/Jan/15 ]

This is also an issue for the project I am working as I am trying to use Symfony to to generate forms and so on in order to install the project. sForms and the main project's front controller trigger DI events and trigger this error. The workaround mentioned here (https://github.com/doctrine/DoctrineBundle/issues/351#issuecomment-65771528) fixes the problem but this cannot be a long term solution. I look forward to the fix for this issue.

Comment by Alan Hartless [ 07/Jan/15 ]

I have the same issue as Craig. This poses a major hassle for applications that have an installer UI. We have a Symfony based application that gives the user options to configure the database such as driver, table, credentials, etc. Then the application will make a dynamic connection to check database, install data, etc. Eventually Symfony's cache is cleared to use the new credentials post install.

Because of this, the installer fails out of the box since it can't connect to the server.

We can't use the work around as mentioned above because of giving the option to choose what driver to use in the installer.

Thanks,
Alan

Comment by Craig Heydenburg [ 14/Jan/15 ]

Tried an experiment today. As noted in the workaround you can set the server_version: 5.1 (or whatever your version is).

Well, my version is 5.5 so I originally used that. Today I tried 5 and 52 and 1 and they all worked!

so it seems that the actual version doesn't matter (at least when I need them, which is before I've actually set up the DB credentials) as long is there is some value.

Comment by Steve Müller [ 15/Jan/15 ]

Craig Heydenburg this is expected behaviour. You can theoretically specify any value you want, DBAL will try to find the appropriate platform for you by the specified version via version compare.
See the example for PostgreSQL: https://github.com/doctrine/dbal/blob/master/lib/Doctrine/DBAL/Driver/AbstractPostgreSQLDriver.php#L97-L119
As soon as the specified value/version does not match a specific platform version, DBAL will always fallback to the "default" platform (the one that was chosen by default prior 2.5).
Once the serverVersion parameter is set, auto detection of the proper platform is always bypassed.

Comment by Jan Rosier [ 16/Jan/15 ]

Doctrine now also throws a connection exception when it tries to detect the database platform, but can't make a connection to the server. Even if the connection isn't really used.

I think it would be better to do a fallback to the "default" platform in that case and only throw an connection exception if the connection is required for something else than guessing the platform.

For example if you haven't set up the DB credentials yet and only call Doctrine\DBAL\Connection::getDatabasePlatform() you get the "default" platform.





[DBAL-550] Complete/non-overridable event dispatching in DDL methods of AbstractPlatform Created: 22/Jun/13  Updated: 23/Dec/13

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: None
Fix Version/s: None
Security Level: All

Type: Improvement Priority: Major
Reporter: Stéphane Klein Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: migrations, schematool


 Description   

The default implementation provided by the AbstractPlatform class dispatches several events related to schema modifications (create table, change column and so on) and offers the ability to prevent the default behaviour by setting a flag on the dispatched event. However, it appears that some DDL operations aren't covered by this mechanism (foreign keys creation, for example, doesn't seem to be observable). Furthermore, the concrete platforms can override any of these methods, potentially leading to inconsistancies in components that expect those events to be dispatched.

I'm working on a large modular application build on top of Symfony, in which each bundle must have its own migration classes (i.e. migrations on a per bundle basis, not at the application level), possibly with references to other parts of the schema. I used to write those classes manually using the dbal api, but this is tiedous and error-prone, and I'd like to take advantage of the mapping-based schema generation. I found that using the event system and "filtering" ddl operations to keep only the operations related to a given bundle is a clean and easy way to achieve this goal, but the issues I mentionned prevent me to complete the implementation.

I'd be glad to hear your opinion on that topic and, providing you're interested in it, contribute in a way or another to improve the existing code.



 Comments   
Comment by Benjamin Eberlei [ 22/Jun/13 ]

This is not a blocker

Comment by Benjamin Eberlei [ 22/Jun/13 ]

The DBAL event system is not in its best shape and definately could use some improvement. Your easiest way however would be to hook the process of generating the whole schema, and then dropping tables from those two to only contain the ones you are interested in.

Comment by Stéphane Klein [ 23/Jun/13 ]

That's indeed the simplest solution. Thanks a lot for your answer.

Comment by Steve Müller [ 23/Dec/13 ]

Benjamin Eberlei Is there anything to do here? Should the missing events be added or is that not necessary?





[DBAL-477] Just doublequote all schema names and field names in PostgreSQL sql command generation, and the same for MySQL Created: 28/Mar/13  Updated: 24/Dec/13

Status: Open
Project: Doctrine DBAL
Component/s: Platforms, Schema Managers
Affects Version/s: 2.3.2
Fix Version/s: None
Security Level: All

Type: Improvement Priority: Major
Reporter: jos de witte Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: mysql, postgresql
Environment:

Any PostgreSQL environment


Issue Links:
Reference
is referenced by DBAL-96 Make approach towards identifier quot... Open

 Description   

Generation of any SQL command to the database (From entities or migration versions) does not quote all the reserved keywords (For example a fieldname `right`.

Simple fix that always works: double-quote dbname, schemaname and fieldname

e.g "dbsecurity"."userschema"."users" or "tblusers"

MySQL : use the ` sign.

e.g `security`.`users` or `tblusers` (No support for schemas since I last checked some time ago)



 Comments   
Comment by Steve Müller [ 24/Jun/13 ]

If those are reserved keywords, they should be added to the "PostgreSQLKeywords" class and they will be quoted by Doctrine. As far as I can see those keywords mentioned are not present in this class. Maybe there is something missing?





[DBAL-357] Missing way to set types for CAST declaration Created: 04/Oct/12  Updated: 04/Oct/12

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.2, 2.3
Fix Version/s: None
Security Level: All

Type: Improvement Priority: Major
Reporter: Pete Sisson Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None


 Description   

For a query such as

x = CASE WHEN id=$1 THEN CAST($2 AS int)

there doesn't seem to be a way to correctly assign the type across multiple platforms. E.g. Postgres required "int" but mysql just requires "unsigned". Attempting to use "int" here with mysql will fail.

The method Doctrine\DBAL\Platforms method getIntegerTypeDeclarationSQL will return something like "INT unsigned" for sql, which also fails.






[DBAL-642] Generated IDs are not guaranteed to be unique over the table's lifetime in SQLite Created: 27/Oct/13  Updated: 25/Jun/14

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.4
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: flack Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 1
Labels: None


 Description   

In

http://docs.doctrine-project.org/en/2.0.x/reference/basic-mapping.html#identifier-generation-strategies

it says that MySQL and SQLite use AUTO_INCREMENT by default. The generated SQL for creating an ID field with GENERATOR_TYPE_AUTO looks like this (abbreviated
for readability):

CREATE_TABLE_TEST (id INTEGER DEFAULT 0 NOT NULL, PRIMARY KEY(id))

http://www.sqlite.org/faq.html#q1 states:

If you declare a column of a table to be INTEGER PRIMARY KEY, then whenever you insert a NULL into that column of the table, the NULL is automatically converted into an integer which is one greater than the largest value of that column over all other rows in the table, or 1 if the table is empty. [...] Note that the integer key is one greater than the largest key that was in the table just prior to the insert. The new key will be unique over all keys currently in the table, but it might overlap with keys that have been previously deleted from the table.

So in other words, if you remove an entity and then create a new one, the new one will have the same id as the previously deleted one. If you do both operations on the same entitymanager, id references (in proxies f.x.) will suddenly get confused and point to something else (at least that's my current theory..)

The point is: SQLite doesn't act like MySQL as the documentation implies, and IMHO SQLite's current behaviour makes it useless for more complex scenarios. I've found some reference to this problem here:

https://github.com/doctrine/dbal/pull/66

Unfortunately, it doesn't mention any solution. The problem is that you can't override the columnDefinition, because it would have to be "INTEGER PRIMARY KEY AUTOINCREMENT", but then, you get an exception, because the Platform appends ", PRIMARY KEY(id)", so it's defined twice



 Comments   
Comment by Steve Müller [ 25/Nov/13 ]

As far as I understand there is a conflict in SQLite between having an autoincrement primary key and having a composite primary because you can only choose either way. The PR you referenced removed the autoincrement behaviour in favour of having the opportunity to define composite primary keys. So what would you expect to be the solution here?

Comment by flack [ 25/Nov/13 ]

Well, from a user's point of view, it would be nice if the SQLite Platform implementation would make full use of the possibilities of SQLite. That is to say: If someone uses composite primary, they get the behaviour Doctrine has right now, and those that use the simple case (which is recommended all over the documentation), get the behaviour previous to the pull request, i.e. autoincrement that works like in MySQL (which the documentation implies). As far as I understood the discussion in the pull request, the author was looking for a solution to implement this, but then the PR was merged before the issue was solved.

Comment by flack [ 25/Nov/13 ]

I guess what is bothering me is that the current behaviour breaks assumptions that I think many applications using Doctrine make. At least I know that in my own code, I never planned for the possibility of a database primary key being re-used for a different object, especially not during the same request. And like I wrote above, I suspect that Doctrine itself is not totally prepared for that situation either (also mentioned here: https://github.com/doctrine/dbal/pull/66#discussion_r173623). So IMHO the IDENTIY generator strategy for SQLite seems broken, or at least is behaving unexpectedly. The documentation says

AUTO (default): Tells Doctrine to pick the strategy that is preferred by the used database platform. The preferred strategies are IDENTITY for MySQL, SQLite and MsSQL and SEQUENCE for Oracle and PostgreSQL. This strategy provides full portability.

I don't really see how the current behaviour can be said to provide full portability (at least with MySQL, which supposedly uses the same preferred strategy)

Comment by Steve Müller [ 25/Nov/13 ]

Yeah I get your point. But it's always hard and error prone to work around the vendors lack of common features. A possibility COULD be to implement a trigger in columns declared as autoincrement which simulates the behaviour of an auotincrement column on inserts. Oracle uses a similar workaround with a trigger and a sequence to simulate autoincrement columns. But this is just an idea and has to evaluated for usability first.

Comment by Steve Müller [ 25/Nov/13 ]

Yes that's true. The documentation is misleading here. I guess that it was written before the issue came up and then was not updated. Unfortunately SQLite does not support native sequences which would make life a lot easier. But I will keep that in mind and investigate a solution for this.

Comment by Benjamin Eberlei [ 13/Dec/13 ]

I think there is no fix for this, this is just how SQLite works, and we cannot really keep the last ids somewhere. IMHO its a documentation issue.

Comment by flack [ 13/Dec/13 ]

Well, you cannot fix it for cases with multiple id columns (but the Doctrine documentation already suggests that they should be avoided where possible), but for single integer columns (which is the normal case, as suggested by documentation), SQLite provides all the necessary functionality, as long as you create the column with INTEGER PRIMARY KEY AUTOINCREMENT. So IMHO the best solution would be if support for this could be implemented somehow in the SQL Platform driver.

Comment by flack [ 14/Dec/13 ]

Case in point: I implemented exactly this in a Doctrine adapter I'm working on:

https://github.com/flack/midgard-portable/blob/master/src/midgard/portable/storage/subscriber.php#L136

Granted, this is a very ugly workaround that only works because I know all my ID columns are actually called 'id' (and will never change), but I'm fairly sure that a more general solution could be built with reasonable effort.

Comment by Benjamin Eberlei [ 01/Jan/14 ]

flack We removed the Sqlite AUTOINCREMENT for some weird reason. I am inclined to add this again, however I need to find out what the reasons for removing this have been.

Comment by Steve Müller [ 01/Jan/14 ]

Benjamin Eberlei I checked that lately and I came to the same conclusion. The reason why it was removed is to support composite primary keys which was not possible before somehow. We could add autoincrement if only a single column integer primary key is given I think...

Comment by flack [ 25/Jun/14 ]

Just noticed that the link I posted above is broken now. Here's a stable one:

https://github.com/flack/midgard-portable/blob/04d473356d804c7f64e940ef82dd1538c39fccdd/src/storage/subscriber.php#L170

The workaround is a bit less project-specific now, and might serve as the basis for a real solution I guess (basically, only checks for column type and composite keys would need to be added)





[DBAL-970] Implement partial indexes for missing platforms Created: 12/Aug/14  Updated: 12/Aug/14

Status: Open
Project: Doctrine DBAL
Component/s: Platforms, Schema Managers
Affects Version/s: 2.5
Fix Version/s: None
Security Level: All

Type: Improvement Priority: Major
Reporter: Steve Müller Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None


 Description   

As a follow-up to DBAL-900 there are some platforms left that support partial indexes too and need the implementation for that feature.
Currently partial indexes are only implemented for PostgreSQL.
Additionally there should be a functional test covering creation and introspection of partial indexes if possible.






[DBAL-444] OraclePlatform getSequenceNextValSQL not handling case/quoting properly on 11g Created: 10/Feb/13  Updated: 04/Dec/14

Status: In Progress
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.3.2
Fix Version/s: 2.6
Security Level: All

Type: Bug Priority: Major
Reporter: Max Milaney Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: oci8, oracle, sequence
Environment:

PHP version 5.4.11
Oracle 11g Instant Client version 11.2.0.3.0
Oracle Database 11g Enterprise Edition version 11.2.0.3.0 (x64)
OCI8 DBAL driver


Attachments: File example.php    

 Description   

I have an installer script that uses ORM SchemaTool to create the entities in the DB and then populates with basic data using basic EM->persist calls via ORM.

Sequence objects are created, and when using the 10g Instant Client everything worked correctly, however, upon upgrade to latest version of the Instant Client Oracle seems to be expecting consistent case for these schema objects. It appears as if they are being created with a quoted name as they are created in lowercase. OraclePlatform::getSequenceNextValSQL, however, generates "SELECT entity_id_seq.nextval FROM DUAL" and this fails with error "General error: 2289 OCIStmtExecute: ORA-02289: sequence does not exist".

Executing "SELECT "entity_id_seq".nextval FROM DUAL" directly on the DB returns the correct value.

I believe this may also impact the code in http://www.doctrine-project.org/jira/browse/DBAL-278



 Comments   
Comment by Max Milaney [ 10/Mar/13 ]

Hi there,
Wondering if there is any update on this? I'm having to use a workaround in my applications.
Cheers,
Max

Comment by Benjamin Eberlei [ 14/Mar/13 ]

Can you maybe show an entity definition with its sequence mapping?

Comment by Max Milaney [ 17/Mar/13 ]

Here you are mate. Please see attachment.

Comment by Benjamin Eberlei [ 04/Apr/13 ]

I cant seem to find the problem, in DBAL "lib/Doctrine/DBAL/Platforms/OraclePlatform.php" on line 171, the sequence statement is created with $sequence->getQuotedName($platform), but this only works if quoting is requrested for the sequence.

How do you actually create the sequence? Your entity doesnt have @GeneratedValue.

What does the create schema command say with "--dump-sql" flag? Is the SQL quoted?

Comment by Steve Müller [ 03/Jan/14 ]

Max Milaney Can you please test if this still exists in the current master branch? If so, can you please provide the information requested by Benjamin Eberlei ? Otherwise hunting this down is rather hard... Thank you!





[DBAL-1069] datetimetz type needs to be a commented one by default Created: 10/Dec/14  Updated: 10/Dec/14

Status: Open
Project: Doctrine DBAL
Component/s: Platforms, Schema Managers
Affects Version/s: None
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: Christophe Coevoet Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None


 Description   

By default, the AbstractPlatform creates datetimetz fields as datetime field for platforms not defining a specific creation for fields with timezones.
This can be an acceptable fallback if you know that your application always send the value as UTC even when the database can accept any timezone as input thanks to the datetimetz type (database not supporting timezone would still have consistent data if you always send the same timezone).
However it creates an issue for the SchemaTool: the field will obviously be reported as datetime when inspecting the DB, not as datetimetz (since it does not have a datetimetz). So for platforms not supporting datetimetz natively, the special comment should be used to avoid useless updates.
The special Doctrine command should however not be used for platforms supporting the type natively (PostgreSQL, SQLServer2008, Oracle and SqlAnywhere12).

I have an idea about a fix, so I may send a PR for this in the following days.






[DBAL-1070] AzureSQL specificities are not taken into account Created: 11/Dec/14  Updated: 11/Dec/14

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.5
Fix Version/s: 2.4.3

Type: Bug Priority: Major
Reporter: Nicolas Séverin Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: azure, dbal, sqlserver
Environment:

Azure website using an AzureSQL database (based on SQL Server 2012).



 Description   

In Azure SQL, table ‘sys.extended_properties’ does not exist.
But SQLAzurePlatform inherits from class SQLServerPlatform, which uses it.
The code in question is here /Doctrine/DBAL/Platforms/SQLServerPlatform.php#L845.

Modifications to this portion of code were made to handle comments on columns differently in doctrine/dbal 2.5, but it used to work in 2.4.3.



 Comments   
Comment by Marco Pivetta [ 11/Dec/14 ]

Reverted to priority Major





[DBAL-999] Get a Sql Server error on Order By - Symfony2 Created: 08/Oct/14  Updated: 31/Dec/14

Status: Awaiting Feedback
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: None
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: Maël SOURISSEAU Assignee: Marco Pivetta
Resolution: Unresolved Votes: 0
Labels: orderBy, query, sqlserver


 Description   

Using Symfony with Sql Server and from what I've read, it seems that the connection to the database is not stable.

As soon as I use the orderBy method I get an error :

Here's an example :

Unable to find source-code formatter for language: php. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml
  $qStores =
        $this->getManager()
             ->createQueryBuilder()
             ->select('rpdv')
             ->from('MainBundle:PointDeVenteReference', 'rpdv')
             ->andWhere( 'rpdv.partenaireClient = :id_partner ' )
                 ->setParameter( 'id_partner', $this->getUser()->getPartenaire()->getIdPartenaire() )
             ->orderBy( 'rpdv.idPointDeVenteReference' , 'DESC' )
             ->setFirstResult( 0 )
             ->setMaxResults( 30 );

And the error :

An exception has been thrown during the rendering of a template ("An exception occurred while executing
'SELECT DISTINCT TOP 30 id_point_de_vente_reference0
FROM ( SELECT p0_.id_point_de_vente_reference AS id_point_de_vente_reference0,
p0_.reference AS reference1,
p0_.date_derniere_modification AS date_derniere_modification2,
p0_.blocage AS blocage3
FROM point_de_vente_reference p0_
WHERE p0_.id_partenaire_client = ?
ORDER BY p0_.id_point_de_vente_reference DESC ) dctrn_result
ORDER BY id_point_de_vente_reference0 DESC'
with params [2829]:SQLSTATE[42000]:
[Microsoft][SQL Server Native Client 11.0][SQL Server]
The ORDER BY clause is invalid in views, inline functions, derived tables, subqueries, and common table expressions,
unless TOP, OFFSET or FOR XML is also specified.") in MainBundle:Default:store/list.html.twig at line 79.
I tried to change the class SQLServerPlatform with corrections found on the net, without success.

Do you have any idea?

Thx !



 Comments   
Comment by Steve Müller [ 08/Oct/14 ]

Which version of DBAL are you using? A lot of fixes have been applied to SQL Server's LIMIT/OFFSET query rewriting in DBAL during the last months.

Comment by Maël SOURISSEAU [ 08/Oct/14 ]

2.4 for DBAL.

Under my request, I have :

$stores = new Paginator( $qStores, TRUE );

In passing the second parameter to FALSE, I have no error.

Comment by Marco Pivetta [ 19/Oct/14 ]

I'd suggest checking ORM+DBAL latest to see if the issue still exists, as those component have suffered from radical changes in the last few months.

Comment by Maël SOURISSEAU [ 29/Dec/14 ]

I currently still have the anomaly with the following configuration :

"doctrine/doctrine-bundle": "1.3.*@dev"
"doctrine/dbal": "~2.5"
"doctrine/orm": "~2.4"

Comment by Maël SOURISSEAU [ 29/Dec/14 ]

The problem apparently comes from the class SQLServerPlatform (doModifyLimitQuery)

Comment by Marco Pivetta [ 29/Dec/14 ]

Maël SOURISSEAU that kind of information is insufficient: please pick exact versions from your composer.lock instead.

Comment by Maël SOURISSEAU [ 30/Dec/14 ]
Unable to find source-code formatter for language: json. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml
...
{
    "name": "doctrine/dbal",
    "version": "v2.5.0",
    "source": {
        "type": "git",
        "url": "https://github.com/doctrine/dbal.git",
        "reference": "71140662c0a954602e81271667b6e03d9f53ea34"
    },
    "dist": {
        "type": "zip",
        "url": "https://api.github.com/repos/doctrine/dbal/zipball/71140662c0a954602e81271667b6e03d9f53ea34",
        "reference": "71140662c0a954602e81271667b6e03d9f53ea34",
        "shasum": ""
    },
    "require": {
        "doctrine/common": ">=2.4,<2.6-dev",
        "php": ">=5.3.2"
    },
    "require-dev": {
        "phpunit/phpunit": "4.*",
        "symfony/console": "2.*"
    },
    "suggest": {
        "symfony/console": "For helpful console commands such as SQL execution and import of files."
    },
    "bin": [
        "bin/doctrine-dbal"
    ],
    "type": "library",
    "extra": {
        "branch-alias": {
            "dev-master": "2.5.x-dev"
        }
    },
    "autoload": {
        "psr-0": {
            "Doctrine\\DBAL\\": "lib/"
        }
    },
    "notification-url": "https://packagist.org/downloads/",
    "license": [
        "MIT"
    ],
    "authors": [
        {
            "name": "Roman Borschel",
            "email": "roman@code-factory.org"
        },
        {
            "name": "Benjamin Eberlei",
            "email": "kontakt@beberlei.de"
        },
        {
            "name": "Guilherme Blanco",
            "email": "guilhermeblanco@gmail.com"
        },
        {
            "name": "Jonathan Wage",
            "email": "jonwage@gmail.com"
        }
    ],
    "description": "Database Abstraction Layer",
    "homepage": "http://www.doctrine-project.org",
    "keywords": [
        "database",
        "dbal",
        "persistence",
        "queryobject"
    ],
    "time": "2014-12-04 21:57:15"
},
{
    "name": "doctrine/doctrine-bundle",
    "version": "dev-master",
    "source": {
        "type": "git",
        "url": "https://github.com/doctrine/DoctrineBundle.git",
        "reference": "3beb3a780485ab01f86941f4892cd23ef8c39c6b"
    },
    "dist": {
        "type": "zip",
        "url": "https://api.github.com/repos/doctrine/DoctrineBundle/zipball/3beb3a780485ab01f86941f4892cd23ef8c39c6b",
        "reference": "3beb3a780485ab01f86941f4892cd23ef8c39c6b",
        "shasum": ""
    },
    "require": {
        "doctrine/dbal": "~2.3",
        "doctrine/doctrine-cache-bundle": "~1.0",
        "jdorn/sql-formatter": "~1.1",
        "php": ">=5.3.2",
        "symfony/doctrine-bridge": "~2.2",
        "symfony/framework-bundle": "~2.2"
    },
    "require-dev": {
        "doctrine/orm": "~2.3",
        "phpunit/php-code-coverage": "~1.2",
        "phpunit/phpunit": "~3.7",
        "phpunit/phpunit-mock-objects": "~1.2",
        "satooshi/php-coveralls": "~0.6.1",
        "symfony/validator": "~2.2",
        "symfony/yaml": "~2.2",
        "twig/twig": "~1"
    },
    "suggest": {
        "doctrine/orm": "The Doctrine ORM integration is optional in the bundle.",
        "symfony/web-profiler-bundle": "to use the data collector"
    },
    "type": "symfony-bundle",
    "extra": {
        "branch-alias": {
            "dev-master": "1.3.x-dev"
        }
    },
    "autoload": {
        "psr-4": {
            "Doctrine\\Bundle\\DoctrineBundle\\": ""
        }
    },
    "notification-url": "https://packagist.org/downloads/",
    "license": [
        "MIT"
    ],
    "authors": [
        {
            "name": "Symfony Community",
            "homepage": "http://symfony.com/contributors"
        },
        {
            "name": "Benjamin Eberlei",
            "email": "kontakt@beberlei.de"
        },
        {
            "name": "Doctrine Project",
            "homepage": "http://www.doctrine-project.org/"
        },
        {
            "name": "Fabien Potencier",
            "email": "fabien@symfony.com"
        }
    ],
    "description": "Symfony DoctrineBundle",
    "homepage": "http://www.doctrine-project.org",
    "keywords": [
        "database",
        "dbal",
        "orm",
        "persistence"
    ],
    "time": "2014-11-28 08:32:03"
},
{
    "name": "doctrine/orm",
    "version": "v2.4.7",
    "source": {
        "type": "git",
        "url": "https://github.com/doctrine/doctrine2.git",
        "reference": "2bc4ff3cab2ae297bcd05f2e619d42e6a7ca9e68"
    },
    "dist": {
        "type": "zip",
        "url": "https://api.github.com/repos/doctrine/doctrine2/zipball/2bc4ff3cab2ae297bcd05f2e619d42e6a7ca9e68",
        "reference": "2bc4ff3cab2ae297bcd05f2e619d42e6a7ca9e68",
        "shasum": ""
    },
    "require": {
        "doctrine/collections": "~1.1",
        "doctrine/dbal": "~2.4",
        "ext-pdo": "*",
        "php": ">=5.3.2",
        "symfony/console": "~2.0"
    },
    "require-dev": {
        "satooshi/php-coveralls": "dev-master",
        "symfony/yaml": "~2.1"
    },
    "suggest": {
        "symfony/yaml": "If you want to use YAML Metadata Mapping Driver"
    },
    "bin": [
        "bin/doctrine",
        "bin/doctrine.php"
    ],
    "type": "library",
    "extra": {
        "branch-alias": {
            "dev-master": "2.4.x-dev"
        }
    },
    "autoload": {
        "psr-0": {
            "Doctrine\\ORM\\": "lib/"
        }
    },
    "notification-url": "https://packagist.org/downloads/",
    "license": [
        "MIT"
    ],
    "authors": [
        {
            "name": "Roman Borschel",
            "email": "roman@code-factory.org"
        },
        {
            "name": "Benjamin Eberlei",
            "email": "kontakt@beberlei.de"
        },
        {
            "name": "Guilherme Blanco",
            "email": "guilhermeblanco@gmail.com"
        },
        {
            "name": "Jonathan Wage",
            "email": "jonwage@gmail.com"
        }
    ],
    "description": "Object-Relational-Mapper for PHP",
    "homepage": "http://www.doctrine-project.org",
    "keywords": [
        "database",
        "orm"
    ],
    "time": "2014-12-16 13:45:01"
},
...
Comment by Steve Müller [ 31/Dec/14 ]

I think this issue is being dealt with in this PR: https://github.com/doctrine/doctrine2/pull/1220
It's basically an issue with the ORM paginator because it uses the ORDER BY clause in the derived table which is invalid in SQL Server.





[DBAL-1077] Query limit for sql server Created: 16/Dec/14  Updated: 21/Feb/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.4, 2.5
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: yannick LE LAN Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: dbal, regex, sqlserver
Environment:

sql server



 Description   

On branch master: tests/Doctrine/Tests/DBAL/platforms/AbstractSQLServerPlatformTestCase.php
On branch 2.4: tests/Doctrine/Tests/DBAL/platforms/SqlServerPlatformTest.php

public function testModifyLimitQueryFolcoerr()
{

    $sql = $this->_platform->modifyLimitQuery('SELECT son.label AS Name FROM SqlObjectName son WHERE ( SELECT COUNT(eso.identifier) FROM ExtractedSqlObject eso INNER JOIN ProductionDbName pdn ON eso.ref_ProductionDbName_ID = pdn.identifier AND (pdn.label IN (?, ?)) WHERE eso.ref_SqlObjectName_ID = son.identifier) > 0 ORDER BY son.identifier DESC', 1, 0);

    $this->assertEquals('SELECT * FROM (SELECT son.label AS Name, ROW_NUMBER() OVER (ORDER BY son.identifier DESC) AS doctrine_rownum FROM SqlObjectName son WHERE ( SELECT COUNT(eso.identifier) FROM ExtractedSqlObject eso INNER JOIN ProductionDbName pdn ON eso.ref_ProductionDbName_ID = pdn.identifier AND (pdn.label IN (?, ?)) WHERE eso.ref_SqlObjectName_ID = son.identifier) > 0) AS doctrine_tbl WHERE doctrine_rownum BETWEEN 1 AND 1 ORDER BY doctrine_rownum', $sql);

}

Correction proposed:
on both branches: lib/Doctrine/DBAL/Platforms/SQLServerPlatform.php

protected function doModifyLimitQuery(...)
{
    ...

    #ACTUAL: 
    $selectFromPattern = '/^(\s*SELECT\s+(?:(.*)(?![^(]*\))))\sFROM\s/i';

#CORRECT:

    $selectFromPattern = '/^(\s*SELECT\s+(?:(.*)(?![^(]*\))))\sFROM\s/iU';

explanation: "/U" pattern modifier => http://php.net/manual/en/reference.pcre.pattern.modifiers.php
Ungreedy behavior not susceptible to fall on ?! negative lookahead on parenthesis hunger

  1. has impacts on ORM with setMaxResults (which was failing in my case and made me investigate)


 Comments   
Comment by Marco Pivetta [ 16/Dec/14 ]

Can you send a PR for this change instead?

Comment by yannick LE LAN [ 16/Dec/14 ]

I will do so by the end of the week,

Comment by yannick LE LAN [ 21/Feb/15 ]

Pull requests made:

For dbal version 2.4:
https://github.com/doctrine/dbal/pull/803

For dbal master branch:
https://github.com/doctrine/dbal/pull/804

(as branch 2.4 seems to lack an 'order by' the patch had to be slightly different)





[DBAL-1158] Using alias with order by and then applying a limit causes an SQL invalid column name error Created: 02/Mar/15  Updated: 02/Mar/15

Status: Open
Project: Doctrine DBAL
Component/s: Drivers, Platforms
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: Karl Dawson Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: dbal, sqlsrv
Environment:

Windows Server 2012 with SQL Server 2012. Using PHP sqlsrv extension and SQL Native Client 11.



 Description   

I was originally having a problem where aliases were not working with order by/group by. I switched to 2.5.1 and this fixed the issue; however this led to another one. When applying a limit to a query using an alias in conjunction with order by, it generates the following error:

SQLSTATE [42S22, 207]: [Microsoft][SQL Server Native Client 11.0][SQL Server]Invalid column name 'sclr1'

See the following example:

$qb = $this->createQueryBuilder('u');

$select = array(
'u.title',
'SUM(u.seconds) AS total_seconds',
);

$qb->select($select)
->groupBy('u.title')
->orderBy('total_seconds', 'DESC')
->setMaxResults(5);

return $qb->getQuery()->getResult();

This will return the above error, but removing the setMaxResults(5) from the query will allow it to work.






[DBAL-1160] Oracle - UuidGenerator bug Created: 04/Mar/15  Updated: 04/Mar/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: Loïc Lavoie Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: dbal


 Description   

The current implementation of the generator strategy UUID for Oracle does not work with any of the version I have availalble (9-10-12)

When using it, the database raise an error the following error:

ORA-00923: FROM keyword not found where expected

The reason is that the statement return into the file DBAL\Plateform\OraclePlateform is missing the "FROM DUAL" (mandatory in oracle, you can't execute a query without a from).

Once this is fix, another error is raised:

ORA-01465: invalid hex number

The reason is that the getGuidExpression() return a binary result. Unfortunately, into the ORM part, it is not converted back in hex (using raw2hex).

My recommandation would be to simply return the following code in the OraclePlateform file:

    /**
     * {@inheritDoc}
     */
    public function getGuidExpression()
    {
        return 'RAWTOHEX(SYS_GUID()) FROM DUAL';
    }

This solution actually work on every plateform I've tested so far.






[DBAL-40] Transparent table&column names escaping Created: 05/Aug/10  Updated: 28/Mar/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.3, 2.4, 2.4.1
Fix Version/s: None

Type: Improvement Priority: Major
Reporter: Jan Tichý Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 13
Labels: None

Issue Links:
Reference
relates to DBAL-96 Make approach towards identifier quot... Open

 Description   

Hello, I would like to re-open the discussion about automatic transparent escaping of all table/column names sent from DBAL to database. It was already discussed in http://www.doctrine-project.org/jira/browse/DDC-88 without any satisfactory result.

Why do I have to quote any reserved word used in table or column name? Why Doctrine doesn't do this automatically for all table and column
names used in generated SQL queries?

Before you start to explain how complicated it is and what problems you will be faced with, try to look at excellent DIBI database layer - how it acts in this way - it's behaviour is very cool. Unfortunally at the moment the full documentation is in czech only, but here is a brief automatic google-translation to english - http://dibiphp.com/en/quick-start.

My suggestion to Doctrine 2 ORM/DBAL solution is:

1. Developer should never care about any escaping or avoiding any reserved words - it is not his business, the DBAL shoult solve it transparently and safely.

2. So there should be no need and even no possibility to add any quotation chars in @column or @table annotations as well as in DQL queries. ORM layer has nothing to do with escaping, it is all a business of the DBAL layer. Current possibility for manual escaping the names in mentioned annotations is totally wrong and should be discontinued.

3. DBAL should escape ALL table and column names transparently and automatically. There should be ne option to enable or disable the escaping, there is no reason for disabling it.

4. The escaping should be performed just in the final translation of DBAL queries to native SQL query, not earlier. This is the right place to do that.

So what do you think about that?



 Comments   
Comment by Roman S. Borschel [ 05/Aug/10 ]

My point of view (and the reason for the current implementation) is as follows:

  • Using reserved words is bad practice.
  • Quoting everything is like hitting all the SQL with a huge big hammer just to hit the 1% of reserved words (which are again, bad practice), thus overkill.
  • Quoting everything bloats the generated SQL (just to hit the 1% of reserved words which are bad practice to begin with)
  • Quoting everything automatically is like hiding the fact from developers that they use reserved words, thus hiding a bad practice and silently encouraging usage of reserved words in new database schemas. This is not desirable.
  • Quoting reserved words has more effects than simply making the database "accept" that identifier. It affects the case-sensitivity and that in a very inconsistent way across databases and operating systems (See http://www.alberton.info/dbms_identifiers_and_case_sensitivity.html , especially the conclusion). You say there is no reason for disabling it but in fact there are a lot of reasons to do so, so many that it is disabled by default in MDB2 and discouraged to enable it.

So, supporting selective quoting in the name of a (slightly) better interoperability with legacy schemas looked (and still looks) like the best solution for us. The support is limited, explicit, does not require much implementation or overhead and does not unnecessarily bloat the SQL.

There is only one solution for reserved words: not using them. Quoting is a workaround, not a solution and especially not a good one.

ps: I really wish quoting reserved words would not be available in SQL It's not available in most programming languages and noone cares, people just don't use reserved words, because they simply can't.

Comment by Jan Tichý [ 05/Aug/10 ]

Hi Roman, thank you very much for your response! I storngly disagree with most of your points .

There is no doubt that using reserved words is bad practice - FROM THE VIEW OF DATABASE SYSTEM.

But we are discussing about ORM and DBAL. One of the biggest goals of ORM/DBAL is to provide transparent usage of the storage behind the scene. No matter if it is MySQL or PostgreSQL or even maybe something completely diferent.

The ORM/DBAL layer should prevent me from any specifics of particular storage as much as possible. I don't want to remember (and I never should to) that I cannot create entity Order because "order" is reserved word in some weird technology far away from me as ORM programmer.

It is strictly consistent with what you have written above in your PS - "It's not available in most programming languages and noone cares, people just don't use reserved words, because they simply can't" - just consider Doctrine 2 to be another programming language - and there is no real systematic reason in Doctrine 2 itself to prevent developers create entities named "Order".

Here is an analogy - It is the same as if you would say that you cannot use associative arrays in PHP because C-language or Assembler behind PHP doesn't support associative arrays. Yes, they don't support them but it is the responsibility of PHP to provide them. In the same way I don't want to respect this weird limitations of particular RDBMS behind Doctrine 2. This is Doctrine's responsibility to transparently cover the limitation.

Moreover, when list of registered keywords is different from one to the other RDBMS, so the naming of entities is strongly dependent on current database server.

Moreover, when I realize that I have used a registered keyword as lately as an error returns from database engine, not earlier.

I suppose here is probably no risk of SQL injection, but I feel the current Doctrine 2 acting to be "vulnerable" in very similar way, on principle. Simply - you are sending an unescaped piece of SQL query to the database without any warranty what it is. And sometimes it fails, sometimes not. From this view I don't consider overall escaping to be overkill at all, I consider it to be a necessity.

I am strongly convinced that developer working upon DBAL or even ORM layer should never think about such naming limitations and he even shouldn't know anything about reserved words in his particular DBMS.

Now to mentioned problems with case sensitivity. Resulting from the fact that Doctrine 2 entity names are case insensitive I belive that all table definitions and SQL queries comming from Doctrine 2 to database should act as case insensitive too. And that the only practicable way is to normalize (lowercase) all table and column names just on DBAL side before it is passed as SQL query to database.

Jan

Comment by Benjamin Eberlei [ 05/Aug/10 ]

There is actually a very good reason for not quoting. Oracle columns behave differently in their internal structure when escaped.

for example:

/**
  * @column(type="integer")
 */
private $foo;

With quoting it would lead to a column "foo" being lower-cased IN the database and even returned so from resultsets. Without casing it would be a column "FOO". We would essentially need to implement lots of glue code just to get this annoying Oracle feature to work and i think Postgres has the same with lower-cased columns.

Comment by Roman S. Borschel [ 05/Aug/10 ]

@"Hi Roman, thank you very much for your response! I storngly disagree with most of your points"

I guess we can agree to disagree then

@"But we are discussing about ORM and DBAL. One of the biggest goals of ORM/DBAL is to provide transparent usage of the storage behind the scene. No matter if it is MySQL or PostgreSQL or even maybe something completely diferent."

Actually, no, "hiding" the storage completely from the developer is not the goal just as it is not the goal to "hide" SQL. There is an object model on one side and a relational database on the other side. The goal is to provide a mapping between them which is not the same as "hiding" one from the other. In order to create good applications that use ORM technology you need to know both very well, OOP and relational databases. The goal is not to make relational database knowledge "unnecessary". This only results in inefficient use of the databases. The goal is to give people who know both sides equally well a tool to map between the two. Not even "portability" between different relational database vendors is a main goal of an ORM technology, it is just obvious to provide assistance with that as part of the mapping.

@"and there is no real systematic reason in Doctrine 2 itself to prevent developers create entities named "Order".

Noone prevents you from naming domain classes anything you want. Class naming is different from table naming. That the table name defaults to the class name is just that, a default, that can and should be changed if necessary.

@"Moreover, when list of registered keywords is different from one to the other RDBMS, so the naming of entities is strongly dependent on current database server."

Correct, and if you want to create a portable application that works, and will be deployed on, a different set of vendors, you need to have some knowledge of these databases and consider their characteristics. An ORM/DBAL technology does not give you any guarantee for complete and transparent portability between vendors and especially not that it will perform equally well on all of them. The ORM/DBAL technology helps you for the most part in a lot of cases with portability issues but it is no free ticket.

@"I suppose here is probably no risk of SQL injection, but I feel the current Doctrine 2 acting to be "vulnerable" in very similar way, on principle. Simply - you are sending an unescaped piece of SQL query to the database without any warranty what it is. And sometimes it fails, sometimes not. From this view I don't consider overall escaping to be overkill at all, I consider it to be a necessity."

Do not confuse identifier quoting with quoting/escaping of special characters as it is used for security reasons on input. Identifier quoting is absolutely not a necessity, it is a workaround for using otherwise reserved words as schema element names. Speaking of goals, it is neither a "goal" of ORM/DBAL technology to completely remove the possibilities of SQL injections. You can't. It'll always be possible with wrong usage.

@"I am strongly convinced that developer working upon DBAL or even ORM layer should never think about such naming limitations and he even shouldn't know anything about reserved words in his particular DBMS."

And I am strongly convinced that a developer working with a DBAL/ORM should know the underlying databases pretty well.

I think you're really not aware of all the consequences it has across different database vendors to quote every identifier. If not for developers using Doctrine, you cause at least any developer or application pain that does not access the database through Doctrine and is thus feels the full pain of case-sensitivity and mandatory quoting you enforced on the whole schema. Ubiquitious access to the data is actually a strong point of a relational database and it is far from uncommon that the same database is accessed by many parties.

I think the approach taken by DIBI is a bad idea and even worse if there is no way to turn this behavior off. Do they have Oracle or DB2 users? I'm wondering what the sysadmins behind these databases might think if they see this quoting nightmare since to my knowledge this is considered bad practice among them as well.

Yes, we're disagreeing on many points but if you really think identifier quoting is a good idea then you're ignoring a whole lot of prior experience (not only mine).

Comment by Lukas Kahwe [ 05/Aug/10 ]

I was one of the lead developers of MDB2 and we just ran into tons of issues when we overly aggressively did identifier quoting by default. even the option caused lots of headaches. furthermore I agree that the ORM is not about turning an RDBMS into an Object Database, but instead to make a mapping possible. In this vain using reserved words or making all identifiers case sensitive will be a big pain for the people that do work one level lower aka the DBA's. heck even as a developer I frequently work on the DB's command line.

Now as for helping people prevent issues with reserved words. Back then I added some reserved word checking into MDB2_Schema. Obviously its hard to really keep track of all of the different reserved words for all RDBMS. Maybe its possible to work with this guy for this: http://www.petefreitag.com/item/290.cfm This way it could be possible to validate if the names chosen in the models will not cause issues with a certain list of RDBMS.

Comment by Benjamin Eberlei [ 07/Aug/10 ]

Reserved words checking sounds to be a fair compromise!

Comment by Jan Tichý [ 30/Aug/10 ]

Hello, thank you all for your responses.

This helped me understand much about Doctrine 2 basic objectives - especially that it is designed mainly to "make a mapping possible" only, not to be as much as possible transparent layer between database and application. And even if I don't like this conception (because I personally think ORM should provide all such features - like automatic reserved keywords escaping - to make the particular database as transparent as possible), at the same time I fully understand all metioned arguments for doing things in such way. Thank you again.

Comment by Damian Boune [ 17/Jan/11 ]

I would like to state an agreement with the OP.

I understand where there are difficulties in handling reserved words and backtick/quoting, and certainly one should always avoid the use of reserved words in their own schema designs. This is a given when one is able to exert control.

At present I am working on a project in which I am dealing with an outside database where I have no control over the schema, nor am I able to push the remote into making the most sensible changes to their schema. I must live with what they provide.

DBAL presents me with a set of invaluable tools that can not be used as-is, because it lacks the ability to handle quoting when generating schema sql. I'm sure there are some other places where I will find this lacking as well. This is disappointing.

Regardless of what we as developers should do when designing our own schema, we still need to be able to work and play with others who may not follow the same common sense conventions.

Edit:
My temporary quick solution to just "make it work", was to modify AbstractAsset::getQuotedName and force the use of $platform->quoteIdentifier. It did the trick for now until a more suitable solution presents itself.

Comment by Francesco Montefoschi [ 03/Feb/11 ]

"its hard to really keep track of all of the different reserved words for all RDBMS"

That's the main point for me.

Comment by Adrian Rudnik [ 26/Apr/11 ]

@Damian thanks for the hint. I just ran into a similar situation.

Not every project is a startup. I tried to use doctrine2 on a customers database for a small web ui. Well I told them to rename their `iso3166-1` table and `alpha-2` field, then we had a good laugh. We made the mapping possible but i'll remember the one thing i learned: doctrine did not help, guide, prevent or cared at all. It did not even hesitate to spew invalid sql snippets when asked to dump. Its okay for me, but i've expected something more resilient from a DBAL.

Comment by Robert (Jamie) Munro [ 02/Feb/13 ]

What do you mean by "Quoting everything is like hitting all the SQL with a huge big hammer"? Is there a performance hit?

I have always quoted all names when working with PostGres. Not quoting them has always felt like not quoting strings in PHP (e.g. $foo[bar] instead of $foo['bar'] because unless the string is keyword or defined as a constant somewhere, you don't need to (although you will get a "Use of undefined constant" warning). In the early days of PHP, not quoting array keys was common example practise.

Comment by Marco Pivetta [ 02/Feb/13 ]

If you want quoting by default on everything we have a quoting strategy (in ORM) that you can use. I don't think quoting everything by default is a viable solution. Back in `Zend_Db` times this was eating up a lot of performance for no real reason. Users having a clean schema without horrors like columns called `order` or `group` should not be penalized because of users not using valid naming schemes.

Comment by Steve Müller [ 24/Jun/13 ]

Hello, if I understand correctly, the issue of quoting reserved keywords automatically is solved in https://github.com/doctrine/dbal/pull/302. Besides reserved keywords you can still decide quoting or not quoting identifier manually by passing quotes to the identifier or not.

Comment by Arthur Bodera [ 26/Sep/13 ]

It's still broken in 2.4.

PR 302 only selectively fixes indexes, PK and FK, but ALTER and all CRUD will still fail (and schema tool will produce invalid sql).

There is no performance hit, as all operations already hit `DefaultQuoteStrategy`.

Currently you have the following workarounds:

  • selectively add `quoted=true` to table and column names (ugh)
  • replace `DefaultQuoteStrategy` with strategy that quotes all identifiers.

Here is a class you can use: https://gist.github.com/Thinkscape/6713196

Comment by Arthur Bodera [ 26/Sep/13 ]

QuoteStrategies are not used for ALTER queries. This means that using the EagerQuoteStrategy mentioned above won't fix invalid ALTER queries generated by schema tool.

For ALTER to work, we need this merged:

https://github.com/doctrine/dbal/pull/379

Comment by Doctrine Bot [ 26/Sep/13 ]

A related Github Pull-Request [GH-379] was closed:
https://github.com/doctrine/dbal/pull/379

Comment by Sebastien Lavoie [ 28/Mar/15 ]

My 2 cents from DBAL-96:

1. Users should not have to worry about platform-specific quoting when using the query builder or helpers, the DBAL should do that for you.
2. Users should be able to explicitly quote using a standard quote (`), the quote would then be converted to the platform’s quote upon SQL generation, without any case change.
3. DBAL should not needlessly quote, it adds bloat and it has been said that there is a performance hit.
4. DBAL should not change the case without the user’s knowledge.
5. A connection configuration option (normalize_case) could be added:
• uppercase: always convert unquoted identifiers to uppercase
• lowercase: always convert unquoted identifiers to lowercase
• platform: will use the default value for specific platform. For the case of case-sensitive platform, even when unquoted (MySQL on UNIX), do nothing.
• null (default): no normalization
6. Future versions of DBAL could change the default value to platform, but this would greatly reduce the risk of causing BC breaks at the beginning, giving time to test everything.
7. When using Doctrine\DBAL\Connection::query directly, you must do the quoting yourself since the SQL is executed directly.

Comment by Arthur Bodera [ 28/Mar/15 ]

Sebastien, ad 3. that is incorrect. Read the ticket more closely, look at the PR, look inside schema tool and platform classes. There is already a lot of quoting+unquoting being performed in 2.* and a lot of assumptions. Having quoting enabled across the board might actually increase performance in some cases, because there will be less scanning for keywords (see platform classes) and possibly less quoting/unquoting across Schema*.

The problem is, the quoting right now works in some places and in some platforms and is being performed only when schema/schematool/dql needs it, but is being ignored in all other cases. This means that columns like "group" or table names like "platform" will fail randomly depending on platform/rdbms you actually use. It's a nightmare with cross-platform apps and a struggle for single-platform apps, where your tables are named according to domain-rules and happen to overlap with some rdbms.

Quoting identifiers being "a bloat" is similar to saying, that implicit quoting values is a bloat. Although from security standpoint the former is much rarer, it's the same for portability and stability of the DBAL across platforms.





[DBAL-96] Make approach towards identifier quoting consistent Created: 26/Feb/11  Updated: 28/Apr/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms, Schema Managers
Affects Version/s: None
Fix Version/s: 2.6

Type: Improvement Priority: Major
Reporter: Benjamin Eberlei Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 2
Labels: None

Issue Links:
Duplicate
is duplicated by DBAL-120 MySql platform getAlterTableSQL does ... Resolved
Reference
relates to DBAL-45 Add CLI tool that checks for Reserved... Resolved
relates to DBAL-477 Just doublequote all schema names and... Open
is referenced by DBAL-40 Transparent table&column names escaping Open

 Description   
  • Make the use of `` a general approach for explicit quoting of identifiers
  • introduce AbstractPlatform::getRegularSQLIdentifierCase($identifier)
  • Introduce AbstractPlatform::isRegularIdentifier($identifier)
  • Fix Schema Assets not to lower-case, but to check for explicit quoting before.
  • Filter values of identifiers passed to all platform functions when they are used in information schema queries according to `` explicit quoting rules.

Problem: Schema is independent of a vendor, this means we have to pick a behavior, i propose SQL-92

This means:

  • strtoupper() ALL tables, column, index, foreign key names that are not quoted by ``
  • For any Quoted identifiers by `` the case is kept.
  • We can introduce a validator to detect a schema that cannot be implemented with a given vendor platform.

In conjunction with the SQL reserved keywords tickets we can then improve the DatabaseDriver considerably to detect identifier casings



 Comments   
Comment by Steve Müller [ 24/Dec/13 ]

Benjamin Eberlei this is an interesting approach and I like it. But I have some complaints about it.
1. I doubt users will be happy about forced default casing rules (ALL upper or ALL lower). Therefore we should think about adding a simple configuration option in DBAL allowing to override the default casing behaviour to the user's preference.
2. Using a consistent default casing means we ALWAYS have to quote identifiers as otherwise the underlying database could silently change the case again (don't know if this is an issue).
3. Introducing this approach in 2.x branch is a BC break as it breaks users' mixed-case identifier mappings.

For 2.x we should maybe at least make use of Identifier class throughout the platforms where necessary.

Comment by Sebastien Lavoie [ 28/Mar/15 ]

My 2 cents:

1. Users should not have to worry about platform-specific quoting when using the query builder or helpers, the DBAL should do that for you.
2. Users should be able to explicitly quote using a standard quote (`), the quote would then be converted to the platform’s quote upon SQL generation, without any case change.
3. DBAL should not needlessly quote, it adds bloat and it has been said in DBAL-40 that there is a performance hit.
4. DBAL should not change the case without the user’s knowledge.
5. A connection configuration option (normalize_case) could be added:
• uppercase: always convert unquoted identifiers to uppercase
• lowercase: always convert unquoted identifiers to lowercase
• platform: will use the default value for specific platform. For the case of case-sensitive platform, even when unquoted (MySQL on UNIX), do nothing.
• null (default): no normalization
6. Future versions of DBAL could change the default value to platform, but this would greatly reduce the risk of causing BC breaks at the beginning, giving time to test everything.
7. When using Doctrine\DBAL\Connection::query directly, you must do the quoting yourself since the SQL is executed directly.

Comment by Arthur Bodera [ 28/Mar/15 ]

Sebastien, ad 3. that is incorrect. Read the ticket more closely, look at the PR, look inside schema tool and platform classes. There is already a lot of quoting+unquoting being performed in 2.* and a lot of assumptions. Having quoting enabled across the board might actually increase performance in some cases, because there will be less scanning for keywords (see platform classes) and possibly less quoting/unquoting across Schema*.

The problem is, the quoting right now works in some places and in some platforms and is being performed only when schema/schematool/dql needs it, but is being ignored in all other cases. This means that columns like "group" or table names like "platform" will fail randomly depending on platform/rdbms you actually use. It's a nightmare with cross-platform apps and a struggle for single-platform apps, where your tables are named according to domain-rules and happen to overlap with some rdbms.

Quoting identifiers being "a bloat" is similar to saying, that implicit quoting values is a bloat. Although from security standpoint the former is much rarer, it's the same for portability and stability of the DBAL across platforms.

Comment by Andreas Prucha [ 28/Apr/15 ]

IMO the big problem is, that behaviour across the RDBMs may be completely different:

Some preserve case and are case-insenstive if not quoted (the nicest approach)
Some do not preserve case and normalize to lower or uppercase and are case-insensitive
And some do not preserve case and are case-sensitive (the worst)

The biggest issue arises if the DB needs to be used outside the Doctrine-Environment and all identifiers need to be quoted in statements.





[DBAL-1211] Wrapper Class should enforce a Interface not a Subclass Created: 27/Apr/15  Updated: 27/Apr/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.5.1
Fix Version/s: None

Type: Improvement Priority: Major
Reporter: Flavio Botelho Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: dbal


 Description   

We are creating a Wrapper Class to allow the use of the Oracle Proxy User feature. For that I need to Wrapper a Class around DBAL\Connection.
Unfortunely, the wrapper class needs to be a subclass of DBAL\Connection which doesn't make sense, there should exist an Interface and the wrapper class should be forced to implement that interface.

That way I don't need to create methods to call all DBAL\Connection methods thru polymorphism.






[DBAL-819] Schema-tools does not work on multiple Oracle's schemas Created: 21/Feb/14  Updated: 28/May/15

Status: Awaiting Feedback
Project: Doctrine DBAL
Component/s: Platforms, Schema Managers
Affects Version/s: None
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: Antoine Descamps Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 1
Labels: Cli, mysql, oracle, orm, schematool
Environment:

DB: Oracle 11g



 Description   

The schema-tools, used via the CLI, is not able to detect schema's changes when working on multiple schemas.

For instance, the ORM is configured with a "global" Oracle's user, having permissions on every schemas. To specify which entities belong to which schema, I've prefixed the table name with the corresponding schema.

When trying to do the following command:

orm:schema-tool:update --dump-sql

Doctrine returns me the following message:

Nothing to update - your database is already in sync with the current entity metadata.

If, instead of using the "global" user in the Doctrine's configuration, I set the user of the specific schema I'm trying to generate a table on based from an entity, it works.



 Comments   
Comment by Steve Müller [ 21/Feb/14 ]

Moved this to DBAL for now. It seems to be related to schema prefixed table names not being evaluated in the platforms when generating the SQL for reverse engineering the database schema.

Comment by Steve Müller [ 22/Feb/14 ]

Antoine Descamps Okay after having investigated on this in detail and thinking about the possibilities we have to find a solution for this, I came to the following conclusion:
ORM's schema tool and DBAL's schema introspection are designed to be bound to what Doctrine defines as "database" for each platform. This is the scope of the whole operation. In case of Oracle it is the "user". You cannot break out of this scope in any way as the current DBAL implementation is not designed for a "complete" schema introspection and therefore does not allow it at some points. Fully understanding your concern I am afraid we cannot find a reasonable solution for your use case at this point in development (2.x). Furthermore there is a good reason for limiting the schema introspection to a certain layer. If the schema tool would introspect the "complete" schema without regard to the database it is connected to, it would suggest a lot of schema changes from other databases that do not belong to the context of the entity manager / database. This behaviour would even cause a lot more annoyance to users as it is the default use case that users are only interested in one database per entity manager.
Also this problem is completely independant from your mapping definitions. So it doesn't matter whether you prefix your table names or not. When running the "update" operation on the schema tool, it is the online schema introspection part that is preventing the schema tool from behaving as you would wish.
Changing the schema introspection behaviour in DBAL would completely break BC and is at some places in the code not even possible without changing the API.
I am sorry that I have to disappoint you with this conclusion but we I am afraid we cannot do anything about your issue until we start developing 3.x. We might reevaluate your use case their and see what we can do.

Comment by Pavlo Chipak [ 28/May/15 ]

Steve Müller I'm using Symfony2 and tying to do such things on MySQL (one entity manager, one user and multiple databases). I wrote my own realisation of schema:update based on code of original command. I have config parameter with list of schemas (databases) and then inside a command in the loop I reloading EM with selected database. Then, as usual, I'm getting diff of schema. At the end, a have diff (list of queries) for each database and can run it all at one. I know it's bad solution, but may be it can be done something similar and more elegant in the core? Like create white list of schemas parameter for EM. If thats can be done, there will not be a problem like:

If the schema tool would introspect the "complete" schema without regard to the database it is connected to, it would suggest a lot of schema changes from other databases that do not belong to the context of the entity manager / database.

At next, if in EM config are set more then one schema we always need use schema.table for tabe naming in the Comparator and other core module, and in generated queries. As profit, it must be not hard to allow using cross-database FK.





[DBAL-939] schema update doesnt detect boolean type correctly Created: 16/Jul/14  Updated: 14/Jun/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: None
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: chris rehfeld Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None
Environment:

centos 6 64, db2 luw10.5



 Description   

*edited*

Dbal for db2 doesn't seem to be able to differentiate between a smallint and a boolean field on db2. If I make an entity which has a boolean field, it will create a table using type smallint. If I later try to update the schema, it will detect the column in the table as a small int, not a boolean. So, it will produce sql update statements to change the type from smallint to smallint, which is obviously unnecessary. I understand db2 doesn't have a boolean type, but it would be nice if dbal realized that a smallint is essentially already a dbal boolean.

Additionally, the update statement is invalid syntax and fails, although this is probably a separate issue.

Example entity

Widget.php
<?php
/**
 * @ORM\Entity
 **/
class Widget
{
    /** @ORM\Id @ORM\Column(type="integer") @ORM\GeneratedValue **/
    protected $id;

    /** @ORM\Column(type="boolean", nullable=false) **/
    private $isGoat;
}

orm:schema-tool:create produces:
CREATE TABLE Widget (id INTEGER GENERATED BY DEFAULT AS IDENTITY NOT NULL, isGoat SMALLINT NOT NULL, PRIMARY KEY(id));

If you then run orm:schema-tool:update, it will try to run:

ALTER TABLE WIDGET ALTER ISGOAT isGoat SMALLINT NOT NULL; CALL SYSPROC.ADMIN_CMD ('REORG TABLE WIDGET');

but no column alteration is obviosuly needed.

So in summary, smallint to boolean type mapping isn't realized, causing the update command to think it needs to change the type of the column.



 Comments   
Comment by chris rehfeld [ 14/Jun/15 ]

I was thinking of giving this bug an attempt, but I want some advice on how to fix it properly from those who live and breathe this code base.

Additionally, I recently added a custom utcdatetime type , and now I'm having the same issue with that - doctrine reporting that I should alter all my date columns to modify them to a "TIMESTAMP(0)", even though they're already that type. So, I really think a proper solution would look at the actual mapping definitions being used to decide if a schema change should be suggested.

One idea I have is to modify Doctrine\DBAL\Schema\Comparator->diffColumn() so that it doesn't report this as a diff. However, I think this boolean to smallint mapping fact is platform dependent, so there would need to be some way for the Comparator to behave platform specific for this behavior. This sounds ugly to me.

Another idea would be to modify the specific platform objects themselves to take care of filtering out these frivolous differences in their getAlterTableSQL() methods. The platform object seems to know about type mappings, so maybe it would be appropriate to do something here.

I also see some event listeners that look like they might be able to be leveraged to pull this off here.

Any suggestions? Thanks.





[DBAL-1248] WindowsServer / SQLServer modifyLimitQuery does not work with aggregate functions in ORDER BY Created: 16/Jun/15  Updated: 16/Jun/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: Luca Cerretani Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: dbal, sqlserver
Environment:

Windows Server 2008 / SQL Server 2008



 Description   

The `modifyLimitQuery` method does not work with query on multiple lines in Windows Server.
See the example below:

$sql = "SELECT
	table_one.id,
	table_one.number,
	table_two.name
	FROM table_one
	LEFT JOIN table_two
	ON table_two.table_one_id = table_one.id
	ORDER BY table_one.id DESC
";

$sql = $this->em->getConnection()->getDatabasePlatform()->modifyLimitQuery($sql, 1, 0);

Doctrine generates this SQL which is invalid

SELECT * FROM (SELECT table_one.id, table_one.number, table_two.name FROM table_one LEFT JOIN table_two ON table_two.table_one_id = table_one.id) AS doctrine_tbl WHERE doctrine_rownum BETWEEN 1 AND 1 ORDER BY doctrine_rownum

It should be

SELECT * FROM (SELECT table_one.id, table_one.number, table_two.name, ROW_NUMBER() OVER (ORDER BY table_one.id DESC) AS doctrine_rownum  FROM table_one LEFT JOIN table_two ON table_two.table_one_id = table_one.id) AS doctrine_tbl WHERE doctrine_rownum BETWEEN 1 AND 1 ORDER BY doctrine_rownum

If I change the

$selectFromPattern = '/^(\s*SELECT\s+(?:(.*)(?![^(]*\))))\sFROM\s/i';

and use instead

$selectFromPattern = '\sFROM\s/i';

The preg_replace works fine but i get another error in the order by clause.
The doModifyLimitQuery trim out the table name and I get the error "column name id is ambiguous". The uncorrect generated sql is

SELECT * FROM (SELECT table_one.id, table_one.number, table_two.name, ROW_NUMBER() OVER (ORDER BY id DESC) AS doctrine_rownum  FROM table_one LEFT JOIN table_two ON table_two.table_one_id = table_one.id) AS doctrine_tbl WHERE doctrine_rownum BETWEEN 1 AND 1 ORDER BY doctrine_rownum





[DBAL-1253] Sqlite: inconsistent (non-)support of foreign keys Created: 26/Jun/15  Updated: 26/Jun/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms, Schema Managers
Affects Version/s: None
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: Adirelle Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Tested on Ubuntu 14.10 wth standard PHP packages (5.6 & sqlite 3.8.6) and on Debian Jessie with standard PHP packages (5.6 & sqlite 3.8.6).



 Description   

Though the SqlitePlatform announces it doesn't support foreign keys, it generates foreign key constraints in "alter table" statements.

This leads to issues when combined with Doctrine ORM and/or Doctrine Migrations : the shcema manager doesn't create foreign keys on new tables, but it generates SQL statement to add them on existing tables.

  • Calling the schema update command several times in a row on an existing database causes all tables with relations to be recreated (since Sqlite ALTER TABLE statement is very limited).
  • Even if the database is up to date, using the "diff" command of Doctrine Migrations creates a migration the recreate all tables.

I suggest to disable the declaration of foreign keys at all to avoid these discrepancies.

Background:

  • Before Sqlite 3.6.19, declarations of foreign keys in CREATE TABLE were accepted (and retrieved) but ignored.
  • Starting with 3.6.19, there is a compile option to disable the support of the foreign key syntax in CREATE TABLE statement. There is another compile option to disable the triggers, including foreign key ones. These options can be queried using a PRAGMA statement.
  • Starting with 3.6.19, there is a PRAGMA to enforce the respect of foreign keys.
  • in any case, foreign keys are not supported in ALTER TABLE statements.





[DBAL-1271] MySQL MEDIUMINT is mapped as type INT Created: 27/Jul/15  Updated: 27/Jul/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: None
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: Scott Connerly Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None


 Description   

In MySqlPlatform in initializeDoctrineTypeMappings(), MEDIUMINT's are treated just the same as INT.

My particular use case is in the context of Laravel, renaming a column that is built as a MEDIUMINT. It gets rebuilt as an INT, and breaks the foreign keys.






[DBAL-1274] Travis only testing against PostgreSQL 9.4 Created: 30/Jul/15  Updated: 31/Jul/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: All

Type: Bug Priority: Major
Reporter: Shane Archer Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: postgresql, travis


 Description   

There are two issues with the Travis configuration as it relates to testing against various versions of PostgreSQL.

The first is that the script inside .travis.yml is incorrectly single-quoting a conditional, causing it to appear to work but in fact do nothing. This line:

  - if [ '$DB' = 'pgsql' ]; then sudo service postgresql stop; sudo service postgresql start $POSTGRESQL_VERSION; fi

Should be changed to:

  - if [ $DB = 'pgsql' ]; then sudo service postgresql stop; sudo service postgresql start $POSTGRESQL_VERSION; fi

However, making that change raises the larger problem that since Travis is now running via containers, the "sudo" functionality is not possible, making the start/stop command unusable. It produces this:

$ if [ $DB = 'pgsql' ]; then sudo service postgresql stop; sudo service postgresql start $POSTGRESQL_VERSION; fi

sudo: must be setuid root
sudo: must be setuid root

The command "if [ $DB = 'pgsql' ]; then sudo service postgresql stop; sudo service postgresql start $POSTGRESQL_VERSION; fi" failed and exited with 1 during .

Your build has been stopped.

I am not sure if there is a long term strategy around this or not, but I couldn't find an open issue or PR related to it.



 Comments   
Comment by Shane Archer [ 31/Jul/15 ]

I tested this with the change mentioned in the description and the following:

sudo: required

That appeared to solve everything and produced a passing Travis build while actually appearing to load the proper PostgreSQL versions. Whether or not that is a good idea. I do think it would be nice to get Travis working again in this regard, though.





[DBAL-422] Wrong VARCHAR default length in SQLServerPlatform Created: 24/Jan/13  Updated: 27/Dec/13

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.3.2
Fix Version/s: None
Security Level: All

Type: Bug Priority: Minor
Reporter: Steve Müller Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: platform, sqlserver, sqlsrv, varchar


 Description   

In SQLServerPlatform the default length for a VARCHAR declaration is set to "255". But according to the SQLServer documentation from Microsoft the default length is "1", if omitted in the declaration.
See remarks: http://msdn.microsoft.com/en-us/library/ms186939.aspx
Also the default length is hardcoded in the "getVarcharTypeDeclarationSQLSnippet" method which in my opinion should be evaluated through "getVarcharDefaultLength".

I don't exactly know if the current implementation is intended, otherwise it should be fixed. I would then create an PR if desired.



 Comments   
Comment by Steve Müller [ 27/Dec/13 ]

The implementation is inconsistent on other platforms, too and can first be addressed in 3.0 I think. Otherwise this would be a BC break.





[DBAL-858] oracle IN statement with more than 1000 values Created: 11/Jan/13  Updated: 01/Apr/14

Status: Open
Project: Doctrine DBAL
Component/s: Drivers, Platforms
Affects Version/s: 2.2.2
Fix Version/s: None
Security Level: All

Type: Improvement Priority: Minor
Reporter: Marc Drolet Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None


 Description   

If I have a query with a IN statement with more tahn 1000 values I get an sql error.

I've try IN with implode:
select * from test where id IN(' . implode(',', $values) . ')
and I've also try with executeQuery:
select * from test where id IN(:test)
executeQuery($sql, array($values), array(\Doctrine\DBAL\Connection::PARAM_INT_ARRAY))



 Comments   
Comment by Marc Drolet [ 11/Jan/13 ]

Here is the way I've implement the solution on my side: (for oracle)

into Doctrine/DBAL/Statement.php, I've add this method:

/**
     * Binds a parameter value to the statement.
     * This is implemented this way for oracle only. Other drivers are redirected to bindValue method.
     *
     * The value will be bound with to the type provided (that required to be a table type).
     *
     * @param String $name The name or position of the parameter.
     * @param Array $value The value of the parameter.
     * @param String $type The name of the type to use to bind.
     * @return boolean TRUE on success, FALSE on failure.
     */
    public function bindList($name, Array $value, $type)
    {
        if ('oracle' !== $this->platform->getName())
        {
            $this->bindValue($name, $value, $type);
        }
        else
        {
            return $this->stmt->bindList($name, $value, $type);
        }
    }

into Doctrine/DBAL/Driver/Statement.php I've add:

/**
     * @TODO: docs
     */
    function bindList($param, Array $values, $type);

into Doctrine/DBAL/Driver/OCI8/OCI8Statement.php I've add this method:

/**
     * {@inheritdoc}
     */
    public function bindList($param, Array $value, $type)
    {
        if (!($list = oci_new_collection($this->_dbh, $type)))
        {
            //throw new OCI8Exception::fromErrorInfo($this->errorInfo());
        }

        foreach ($value as $entry)
        {
            $list->append($entry);
        }
        if (!oci_bind_by_name($this->_sth, $param, $list, -1, OCI_B_NTY))
        {
            //throw new OCI8Exception::fromErrorInfo($this->errorInfo());
        }
    }

// NOTE: we should probably add the bindList to all driver Statement object.

into your code you can use it this way:

$sql = "
    SELECT *
    FROM test
    WHERE id IN
    (
        SELECT *
        FROM
        (
            CAST (: p_ids AS list_int_type)
        )
    )
";
$stmt = connection->prepare($sql);
$stmt->bindList(': p_ids', $ids, 'list_int_type');
$stmt->execute();
$rs = $stmt->fetchAll(PDO::FETCH_ASSOC);

NOTE:
list_int_type need to be a valid oracle data type. You can create one with the name you want.
example:
you can have 2 type of accepted array of values: integer and string
let's say we create one for string named: list_str_type and one for integer list_int_type

create or replace type list_str_type as table of varchar2(4000);
create or replace type list_int_type as table of number;

Comment by Benjamin Eberlei [ 01/Apr/13 ]

Hey Marc Drolet

thanks for the feedback and the solution, however i would like to have something generic that is working independent of the database driver. This code is very specific.

Can you point me to some documentation why oci collection works with more than 1000 elements and how it works in PHP?

Comment by Marc Drolet [ 02/Apr/13 ]

Hi Benjamin,

The limitation is not from the oci driver, it's an oracle limitation. There are a couple of possible solution/implementation that can be done but the one I've provide is the one that perform better for the test I've done and from what I can found over the blogs I've read.

I can't find the exact documentation of oracle. oracle doc is so poor.
Here is the best description link I can provide that describe some possible implementation.
http://vsadilovskiy.wordpress.com/substituting-a-collection-for-in-list-performance-study/

I don't know if there is similar limitation with other database. With the implementation I've provided, It will be possible to implement the proper solution depending on the database limitation you face otherwise it will execute the generic IN. What's bad, we need to create the type into the database.

NOTE: In my case, I can not perform a sub-query, I get the my collection from a web service call.





[DBAL-175] Table comments in Doctrine\DBAL\Schema\Table Object Created: 06/Oct/11  Updated: 17/Oct/11

Status: Open
Project: Doctrine DBAL
Component/s: Platforms, Schema Managers
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Minor
Reporter: Asmir Mustafic Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 1
Labels: None


 Description   

Should be useful discover the table comments from database schema.

This feature is already available for column comments, but not for table comments






[DBAL-167] Schema comparator doesn't work properly with columnDefinition's Created: 17/Sep/11  Updated: 08/Jan/15

Status: Open
Project: Doctrine DBAL
Component/s: Drivers, Platforms, Schema Managers
Affects Version/s: 2.0.8, 2.1, 2.1.1, 2.1.2
Fix Version/s: None
Security Level: All

Type: Bug Priority: Minor
Reporter: Dmitry Strygin Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 3
Labels: None

Issue Links:
Duplicate
is duplicated by DBAL-1096 schema-tool:update does not understan... Resolved

 Description   

Schema comparator will mostly always return changed properties on columns for entities defined with columnDefinition even they are identical in the DB. This is due to weak low-lever compatibility of SchemaTool#getCreateSchemaSql() and SchemaTool#getSchemaFromMetadata() – the first one doesn't reconstruct columnDefinition, and the other one never supports 'fixed', 'default', cannot determine, whether it is boolean or integer (ex. TINYINT in the DB), etc...

All this results in extremely annoying unnecessary alter-table-change-columns surrounded by dropping and after that re-enabling constrains dependent on those columns.

I mean stuff like this:

symfony2#app/console doctrine:schema:update --dump-sql
...
ALTER TABLE es_hotels DROP FOREIGN KEY FK_527F88EE584598A3F92F3E70;
ALTER TABLE es_hotels DROP FOREIGN KEY FK_527F88EE584598A37A3ABE5D;
ALTER TABLE es_hotels DROP FOREIGN KEY FK_527F88EE584598A3EE551564;
ALTER TABLE es_hotels CHANGE is_active is_active TINYINT(1) NOT NULL DEFAULT '1', CHANGE checksum checksum CHAR(32) DEFAULT NULL;
ALTER TABLE es_hotels ADD CONSTRAINT FK_527F88EE584598A3F92F3E70 FOREIGN KEY (operator_id, country_id) REFERENCES es_countries(operator_id, id) ON DELETE CASCADE;
ALTER TABLE es_hotels ADD CONSTRAINT FK_527F88EE584598A37A3ABE5D FOREIGN KEY (operator_id, resort_id) REFERENCES es_resorts(operator_id, id) ON DELETE CASCADE;
ALTER TABLE es_hotels ADD CONSTRAINT FK_527F88EE584598A3EE551564 FOREIGN KEY (operator_id, subresort_id) REFERENCES es_subresorts(operator_id, id) ON DELETE CASCADE;
...

The simple solution would be to fix schema comparator not to signal any changes on columns with columnDefinition properties.
But would be much and much better to add some code to all *SchemaManager#getPortableTableColumnDefinition so they would reconstuct columnDefinition and they would be matched in the schema comparator.

I can do this



 Comments   
Comment by Roderick Schaefer | We handle IT [ 16/Oct/11 ]

I'm having the same issue on my production webserver, but not on the development webserver. I find that odd. It tries to drop all foreign keys and create them again, although without the CHANGE statement you are referring to, Dmitry.

Comment by Benjamin Eberlei [ 09/Jan/12 ]

This maybe fixable by making a hash out of the column definition and saving it into a database comment.

The Foreign Key problem maybe because of an old MySQL version 5.0.x

Comment by Joe Cai [ 16/Jul/12 ]

@beberlei, sounds good to me. any plan of implementing this?





[DBAL-1117] id names not quoted on create in MySQL Created: 14/Jan/15  Updated: 14/Jan/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: All

Type: Bug Priority: Minor
Reporter: Anders Jenbo Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Ubuntu 14.04, MySQL 5.5, MySQLi driver



 Description   

Having the following line in the xml configurations will cause creation to fail with a sql error because group is a key word.
<id name="group" association-key="true"/>






[DBAL-1048] Function based index Created: 20/Nov/14  Updated: 13/Feb/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms, Schema Managers
Affects Version/s: 2.4.3, 2.3.5
Fix Version/s: None
Security Level: All

Type: New Feature Priority: Minor
Reporter: Grégoire Paris Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: postgresql


 Description   

I would like to create a function-based index, but it does not seem to be possible ATM. The final goal is to be able to have case insensitive string filters with Postgresql without losing performance.

I tried this mapping :

  indexes:
        my_entity_name_index:
            columns: [ LOWER(name) ]

But here is what doctrine answers :

[Doctrine\DBAL\Schema\SchemaException]

There is no column with name 'LOWER(name)' on table 'my_entity'.



 Comments   
Comment by Marco Pivetta [ 20/Nov/14 ]

I don't think that we can provide platform-specific index column name support here.

Comment by Michael Lambert [ 02/Jan/15 ]

To me this problem is caused by the bigger issue of some DBs defaulting to case-sensitive and some to case-insensitive. From my searching I have been unable to find a solution to make doctrine operate the same across different collations/databases.

Perhaps the issue would be better described as:
Doctrine does not provide consistency between case-sensitive and case insensitive collations/databases

IMHO it would be very beneficial if Doctrine implemented some method for consistency.

I'm not sure if this is the right place to go into more detail, so I'll refrain for now.

Comment by Steve Müller [ 02/Jan/15 ]

Could you please clarify a bit on "From my searching I have been unable to find a solution to make doctrine operate the same across different collations/databases."? What exactly is broken in Doctrine? Can you give examples?

Comment by Michael Lambert [ 02/Jan/15 ]

Nothing is broken, it could just be improved. Having said that, I don't think this would be a minor task.

The issue I am having that lead me to this issue is that like in mysql != like in postgres. Mysql like == postgres ilike.

Additionally, postgres requires a citext for case insensitive text, mysql relies on the db/table collation.

The only way to make a case insensitive unique key in postgres (at least as far as I could find) is to use a function based index.

I hope this helps define the scope of the case sensitivity issues that would be somewhat solved by adding a function based index or a method to enforce case sensitivity /insensitivity.

(sorry for any auto correct failures, I'm on a mobile device on a train)

Comment by Grégoire Paris [ 13/Feb/15 ]

When I create the index manually, without resorting to Doctrine, it does not detect it when updating the schema (might be a bug that was fixed in the latest version IIRC), so I guess this is a workaround in the meantime, but a dangerous one because one can't safely upgrade. Any other ideas ?





[DBAL-1200] DB2 Platform doModifyLimitQuery ORDER BY Created: 21/Apr/15  Updated: 21/Apr/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: All

Type: Bug Priority: Minor
Reporter: Mark Gullings Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None
Environment:

IBM i v7r1



 Description   

Implement TODO note in doModifyLimitQuery

DB2Platform.php
    protected function doModifyLimitQuery($query, $limit, $offset = null)
    {
        if ($limit === null && $offset === null) {
            return $query;
        }
        $limit = (int) $limit;
        $offset = (int) (($offset)?:0);
        // Todo OVER() needs ORDER BY data!
        $sql = 'SELECT db22.* FROM (SELECT ROW_NUMBER() OVER() AS DC_ROWNUM, db21.* '.
               'FROM (' . $query . ') db21) db22 WHERE db22.DC_ROWNUM BETWEEN ' . ($offset+1) .' AND ' . ($offset+$limit);
        return $sql;
    }


 Comments   
Comment by Mark Gullings [ 21/Apr/15 ]

solution is in testing locally





[DBAL-1206] Generating Table SQL without indexes is invalid if using AUTO_INCREMENT Created: 24/Apr/15  Updated: 24/Apr/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: Markus Fasselt Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None
Environment:

MySQL (but maybe other database vendors too)



 Description   

Dumping the following table

CREATE TABLE `users` (
    `id` INT AUTO_INCREMENT NOT NULL,
    PRIMARY KEY (`id`)
);

with the following snippet (0 = NoIndexes)

$platform->getCreateTableSQL($table, 0);

results in


CREATE TABLE `users` (
    `id` INT AUTO_INCREMENT NOT NULL,
);

The problem is, that the table contains an AUTO_INCREMENT column which cannot be used without a primary key. But the primary key is skipped, as I skipped all indexes.

As this SQL is invalid, I suggest to skip the AUTO_INCREMENT argument, too, if the indexes are skipped. Alternatively, the Primary Key always has to be included.

What do you think? I can provide a fix, if you agree with me.






[DBAL-1229] Escape underscore when ignoring pg_ schemas in PostgreSqlPlatform Created: 12/May/15  Updated: 12/May/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: All

Type: Bug Priority: Minor
Reporter: Tom Ploskina Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None


 Description   

In PostgreSqlPlatform.php, there are 3 lines (240,264,263) which query for schema by:

LIKE 'pg_%'

The underscore needs to be escaped:

LIKE 'pgBACKSLASH_%'

If you have a schema that starts with pg, for example, "pglims", the _ will not be evaluated and the tables, views and sequences returned will not include your tables. Our company's initials are PG by coincidence so we name our schemas accordingly. I will create a pull request for this issue.






Generated at Tue Aug 04 05:26:39 UTC 2015 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.