[DBAL-31] Move Schema related Creation code from AbstractPlatform to AbstractSchemaManager Created: 04/Jul/10  Updated: 28/Dec/13

Status: Open
Project: Doctrine DBAL
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major
Reporter: Benjamin Eberlei Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Move Schema related Creation code from AbstractPlatform to AbstractSchemaManager



 Comments   
Comment by Benjamin Eberlei [ 08/Aug/10 ]

Scheduled for Beta4

Comment by Roman S. Borschel [ 16/Aug/10 ]

If this task is more complex and requires larger refactorings we can re-evaluate it in a post-2.0 release.

Comment by Steve Müller [ 28/Dec/13 ]

Benjamin Eberlei Do you still remember what is meant by this ticket? Could be solved already...





[DBAL-40] Transparent table&column names escaping Created: 05/Aug/10  Updated: 28/Mar/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.3, 2.4, 2.4.1
Fix Version/s: None

Type: Improvement Priority: Major
Reporter: Jan Tichý Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 13
Labels: None

Issue Links:
Reference
relates to DBAL-96 Make approach towards identifier quot... Open

 Description   

Hello, I would like to re-open the discussion about automatic transparent escaping of all table/column names sent from DBAL to database. It was already discussed in http://www.doctrine-project.org/jira/browse/DDC-88 without any satisfactory result.

Why do I have to quote any reserved word used in table or column name? Why Doctrine doesn't do this automatically for all table and column
names used in generated SQL queries?

Before you start to explain how complicated it is and what problems you will be faced with, try to look at excellent DIBI database layer - how it acts in this way - it's behaviour is very cool. Unfortunally at the moment the full documentation is in czech only, but here is a brief automatic google-translation to english - http://dibiphp.com/en/quick-start.

My suggestion to Doctrine 2 ORM/DBAL solution is:

1. Developer should never care about any escaping or avoiding any reserved words - it is not his business, the DBAL shoult solve it transparently and safely.

2. So there should be no need and even no possibility to add any quotation chars in @column or @table annotations as well as in DQL queries. ORM layer has nothing to do with escaping, it is all a business of the DBAL layer. Current possibility for manual escaping the names in mentioned annotations is totally wrong and should be discontinued.

3. DBAL should escape ALL table and column names transparently and automatically. There should be ne option to enable or disable the escaping, there is no reason for disabling it.

4. The escaping should be performed just in the final translation of DBAL queries to native SQL query, not earlier. This is the right place to do that.

So what do you think about that?



 Comments   
Comment by Roman S. Borschel [ 05/Aug/10 ]

My point of view (and the reason for the current implementation) is as follows:

  • Using reserved words is bad practice.
  • Quoting everything is like hitting all the SQL with a huge big hammer just to hit the 1% of reserved words (which are again, bad practice), thus overkill.
  • Quoting everything bloats the generated SQL (just to hit the 1% of reserved words which are bad practice to begin with)
  • Quoting everything automatically is like hiding the fact from developers that they use reserved words, thus hiding a bad practice and silently encouraging usage of reserved words in new database schemas. This is not desirable.
  • Quoting reserved words has more effects than simply making the database "accept" that identifier. It affects the case-sensitivity and that in a very inconsistent way across databases and operating systems (See http://www.alberton.info/dbms_identifiers_and_case_sensitivity.html , especially the conclusion). You say there is no reason for disabling it but in fact there are a lot of reasons to do so, so many that it is disabled by default in MDB2 and discouraged to enable it.

So, supporting selective quoting in the name of a (slightly) better interoperability with legacy schemas looked (and still looks) like the best solution for us. The support is limited, explicit, does not require much implementation or overhead and does not unnecessarily bloat the SQL.

There is only one solution for reserved words: not using them. Quoting is a workaround, not a solution and especially not a good one.

ps: I really wish quoting reserved words would not be available in SQL It's not available in most programming languages and noone cares, people just don't use reserved words, because they simply can't.

Comment by Jan Tichý [ 05/Aug/10 ]

Hi Roman, thank you very much for your response! I storngly disagree with most of your points .

There is no doubt that using reserved words is bad practice - FROM THE VIEW OF DATABASE SYSTEM.

But we are discussing about ORM and DBAL. One of the biggest goals of ORM/DBAL is to provide transparent usage of the storage behind the scene. No matter if it is MySQL or PostgreSQL or even maybe something completely diferent.

The ORM/DBAL layer should prevent me from any specifics of particular storage as much as possible. I don't want to remember (and I never should to) that I cannot create entity Order because "order" is reserved word in some weird technology far away from me as ORM programmer.

It is strictly consistent with what you have written above in your PS - "It's not available in most programming languages and noone cares, people just don't use reserved words, because they simply can't" - just consider Doctrine 2 to be another programming language - and there is no real systematic reason in Doctrine 2 itself to prevent developers create entities named "Order".

Here is an analogy - It is the same as if you would say that you cannot use associative arrays in PHP because C-language or Assembler behind PHP doesn't support associative arrays. Yes, they don't support them but it is the responsibility of PHP to provide them. In the same way I don't want to respect this weird limitations of particular RDBMS behind Doctrine 2. This is Doctrine's responsibility to transparently cover the limitation.

Moreover, when list of registered keywords is different from one to the other RDBMS, so the naming of entities is strongly dependent on current database server.

Moreover, when I realize that I have used a registered keyword as lately as an error returns from database engine, not earlier.

I suppose here is probably no risk of SQL injection, but I feel the current Doctrine 2 acting to be "vulnerable" in very similar way, on principle. Simply - you are sending an unescaped piece of SQL query to the database without any warranty what it is. And sometimes it fails, sometimes not. From this view I don't consider overall escaping to be overkill at all, I consider it to be a necessity.

I am strongly convinced that developer working upon DBAL or even ORM layer should never think about such naming limitations and he even shouldn't know anything about reserved words in his particular DBMS.

Now to mentioned problems with case sensitivity. Resulting from the fact that Doctrine 2 entity names are case insensitive I belive that all table definitions and SQL queries comming from Doctrine 2 to database should act as case insensitive too. And that the only practicable way is to normalize (lowercase) all table and column names just on DBAL side before it is passed as SQL query to database.

Jan

Comment by Benjamin Eberlei [ 05/Aug/10 ]

There is actually a very good reason for not quoting. Oracle columns behave differently in their internal structure when escaped.

for example:

/**
  * @column(type="integer")
 */
private $foo;

With quoting it would lead to a column "foo" being lower-cased IN the database and even returned so from resultsets. Without casing it would be a column "FOO". We would essentially need to implement lots of glue code just to get this annoying Oracle feature to work and i think Postgres has the same with lower-cased columns.

Comment by Roman S. Borschel [ 05/Aug/10 ]

@"Hi Roman, thank you very much for your response! I storngly disagree with most of your points"

I guess we can agree to disagree then

@"But we are discussing about ORM and DBAL. One of the biggest goals of ORM/DBAL is to provide transparent usage of the storage behind the scene. No matter if it is MySQL or PostgreSQL or even maybe something completely diferent."

Actually, no, "hiding" the storage completely from the developer is not the goal just as it is not the goal to "hide" SQL. There is an object model on one side and a relational database on the other side. The goal is to provide a mapping between them which is not the same as "hiding" one from the other. In order to create good applications that use ORM technology you need to know both very well, OOP and relational databases. The goal is not to make relational database knowledge "unnecessary". This only results in inefficient use of the databases. The goal is to give people who know both sides equally well a tool to map between the two. Not even "portability" between different relational database vendors is a main goal of an ORM technology, it is just obvious to provide assistance with that as part of the mapping.

@"and there is no real systematic reason in Doctrine 2 itself to prevent developers create entities named "Order".

Noone prevents you from naming domain classes anything you want. Class naming is different from table naming. That the table name defaults to the class name is just that, a default, that can and should be changed if necessary.

@"Moreover, when list of registered keywords is different from one to the other RDBMS, so the naming of entities is strongly dependent on current database server."

Correct, and if you want to create a portable application that works, and will be deployed on, a different set of vendors, you need to have some knowledge of these databases and consider their characteristics. An ORM/DBAL technology does not give you any guarantee for complete and transparent portability between vendors and especially not that it will perform equally well on all of them. The ORM/DBAL technology helps you for the most part in a lot of cases with portability issues but it is no free ticket.

@"I suppose here is probably no risk of SQL injection, but I feel the current Doctrine 2 acting to be "vulnerable" in very similar way, on principle. Simply - you are sending an unescaped piece of SQL query to the database without any warranty what it is. And sometimes it fails, sometimes not. From this view I don't consider overall escaping to be overkill at all, I consider it to be a necessity."

Do not confuse identifier quoting with quoting/escaping of special characters as it is used for security reasons on input. Identifier quoting is absolutely not a necessity, it is a workaround for using otherwise reserved words as schema element names. Speaking of goals, it is neither a "goal" of ORM/DBAL technology to completely remove the possibilities of SQL injections. You can't. It'll always be possible with wrong usage.

@"I am strongly convinced that developer working upon DBAL or even ORM layer should never think about such naming limitations and he even shouldn't know anything about reserved words in his particular DBMS."

And I am strongly convinced that a developer working with a DBAL/ORM should know the underlying databases pretty well.

I think you're really not aware of all the consequences it has across different database vendors to quote every identifier. If not for developers using Doctrine, you cause at least any developer or application pain that does not access the database through Doctrine and is thus feels the full pain of case-sensitivity and mandatory quoting you enforced on the whole schema. Ubiquitious access to the data is actually a strong point of a relational database and it is far from uncommon that the same database is accessed by many parties.

I think the approach taken by DIBI is a bad idea and even worse if there is no way to turn this behavior off. Do they have Oracle or DB2 users? I'm wondering what the sysadmins behind these databases might think if they see this quoting nightmare since to my knowledge this is considered bad practice among them as well.

Yes, we're disagreeing on many points but if you really think identifier quoting is a good idea then you're ignoring a whole lot of prior experience (not only mine).

Comment by Lukas Kahwe [ 05/Aug/10 ]

I was one of the lead developers of MDB2 and we just ran into tons of issues when we overly aggressively did identifier quoting by default. even the option caused lots of headaches. furthermore I agree that the ORM is not about turning an RDBMS into an Object Database, but instead to make a mapping possible. In this vain using reserved words or making all identifiers case sensitive will be a big pain for the people that do work one level lower aka the DBA's. heck even as a developer I frequently work on the DB's command line.

Now as for helping people prevent issues with reserved words. Back then I added some reserved word checking into MDB2_Schema. Obviously its hard to really keep track of all of the different reserved words for all RDBMS. Maybe its possible to work with this guy for this: http://www.petefreitag.com/item/290.cfm This way it could be possible to validate if the names chosen in the models will not cause issues with a certain list of RDBMS.

Comment by Benjamin Eberlei [ 07/Aug/10 ]

Reserved words checking sounds to be a fair compromise!

Comment by Jan Tichý [ 30/Aug/10 ]

Hello, thank you all for your responses.

This helped me understand much about Doctrine 2 basic objectives - especially that it is designed mainly to "make a mapping possible" only, not to be as much as possible transparent layer between database and application. And even if I don't like this conception (because I personally think ORM should provide all such features - like automatic reserved keywords escaping - to make the particular database as transparent as possible), at the same time I fully understand all metioned arguments for doing things in such way. Thank you again.

Comment by Damian Boune [ 17/Jan/11 ]

I would like to state an agreement with the OP.

I understand where there are difficulties in handling reserved words and backtick/quoting, and certainly one should always avoid the use of reserved words in their own schema designs. This is a given when one is able to exert control.

At present I am working on a project in which I am dealing with an outside database where I have no control over the schema, nor am I able to push the remote into making the most sensible changes to their schema. I must live with what they provide.

DBAL presents me with a set of invaluable tools that can not be used as-is, because it lacks the ability to handle quoting when generating schema sql. I'm sure there are some other places where I will find this lacking as well. This is disappointing.

Regardless of what we as developers should do when designing our own schema, we still need to be able to work and play with others who may not follow the same common sense conventions.

Edit:
My temporary quick solution to just "make it work", was to modify AbstractAsset::getQuotedName and force the use of $platform->quoteIdentifier. It did the trick for now until a more suitable solution presents itself.

Comment by Francesco Montefoschi [ 03/Feb/11 ]

"its hard to really keep track of all of the different reserved words for all RDBMS"

That's the main point for me.

Comment by Adrian Rudnik [ 26/Apr/11 ]

@Damian thanks for the hint. I just ran into a similar situation.

Not every project is a startup. I tried to use doctrine2 on a customers database for a small web ui. Well I told them to rename their `iso3166-1` table and `alpha-2` field, then we had a good laugh. We made the mapping possible but i'll remember the one thing i learned: doctrine did not help, guide, prevent or cared at all. It did not even hesitate to spew invalid sql snippets when asked to dump. Its okay for me, but i've expected something more resilient from a DBAL.

Comment by Robert (Jamie) Munro [ 02/Feb/13 ]

What do you mean by "Quoting everything is like hitting all the SQL with a huge big hammer"? Is there a performance hit?

I have always quoted all names when working with PostGres. Not quoting them has always felt like not quoting strings in PHP (e.g. $foo[bar] instead of $foo['bar'] because unless the string is keyword or defined as a constant somewhere, you don't need to (although you will get a "Use of undefined constant" warning). In the early days of PHP, not quoting array keys was common example practise.

Comment by Marco Pivetta [ 02/Feb/13 ]

If you want quoting by default on everything we have a quoting strategy (in ORM) that you can use. I don't think quoting everything by default is a viable solution. Back in `Zend_Db` times this was eating up a lot of performance for no real reason. Users having a clean schema without horrors like columns called `order` or `group` should not be penalized because of users not using valid naming schemes.

Comment by Steve Müller [ 24/Jun/13 ]

Hello, if I understand correctly, the issue of quoting reserved keywords automatically is solved in https://github.com/doctrine/dbal/pull/302. Besides reserved keywords you can still decide quoting or not quoting identifier manually by passing quotes to the identifier or not.

Comment by Arthur Bodera [ 26/Sep/13 ]

It's still broken in 2.4.

PR 302 only selectively fixes indexes, PK and FK, but ALTER and all CRUD will still fail (and schema tool will produce invalid sql).

There is no performance hit, as all operations already hit `DefaultQuoteStrategy`.

Currently you have the following workarounds:

  • selectively add `quoted=true` to table and column names (ugh)
  • replace `DefaultQuoteStrategy` with strategy that quotes all identifiers.

Here is a class you can use: https://gist.github.com/Thinkscape/6713196

Comment by Arthur Bodera [ 26/Sep/13 ]

QuoteStrategies are not used for ALTER queries. This means that using the EagerQuoteStrategy mentioned above won't fix invalid ALTER queries generated by schema tool.

For ALTER to work, we need this merged:

https://github.com/doctrine/dbal/pull/379

Comment by Doctrine Bot [ 26/Sep/13 ]

A related Github Pull-Request [GH-379] was closed:
https://github.com/doctrine/dbal/pull/379

Comment by Sebastien Lavoie [ 28/Mar/15 ]

My 2 cents from DBAL-96:

1. Users should not have to worry about platform-specific quoting when using the query builder or helpers, the DBAL should do that for you.
2. Users should be able to explicitly quote using a standard quote (`), the quote would then be converted to the platform’s quote upon SQL generation, without any case change.
3. DBAL should not needlessly quote, it adds bloat and it has been said that there is a performance hit.
4. DBAL should not change the case without the user’s knowledge.
5. A connection configuration option (normalize_case) could be added:
• uppercase: always convert unquoted identifiers to uppercase
• lowercase: always convert unquoted identifiers to lowercase
• platform: will use the default value for specific platform. For the case of case-sensitive platform, even when unquoted (MySQL on UNIX), do nothing.
• null (default): no normalization
6. Future versions of DBAL could change the default value to platform, but this would greatly reduce the risk of causing BC breaks at the beginning, giving time to test everything.
7. When using Doctrine\DBAL\Connection::query directly, you must do the quoting yourself since the SQL is executed directly.

Comment by Arthur Bodera [ 28/Mar/15 ]

Sebastien, ad 3. that is incorrect. Read the ticket more closely, look at the PR, look inside schema tool and platform classes. There is already a lot of quoting+unquoting being performed in 2.* and a lot of assumptions. Having quoting enabled across the board might actually increase performance in some cases, because there will be less scanning for keywords (see platform classes) and possibly less quoting/unquoting across Schema*.

The problem is, the quoting right now works in some places and in some platforms and is being performed only when schema/schematool/dql needs it, but is being ignored in all other cases. This means that columns like "group" or table names like "platform" will fail randomly depending on platform/rdbms you actually use. It's a nightmare with cross-platform apps and a struggle for single-platform apps, where your tables are named according to domain-rules and happen to overlap with some rdbms.

Quoting identifiers being "a bloat" is similar to saying, that implicit quoting values is a bloat. Although from security standpoint the former is much rarer, it's the same for portability and stability of the DBAL across platforms.





[DBAL-57] Handling of Quoted Elements Created: 30/Oct/10  Updated: 24/Jun/13

Status: Open
Project: Doctrine DBAL
Component/s: Schema Managers
Affects Version/s: 2.0.0-BETA4
Fix Version/s: None

Type: Improvement Priority: Major
Reporter: Benjamin Eberlei Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None

Issue Links:
Reference
is referenced by DDC-832 Not quoted class name when updating e... Resolved

 Description   

Handling of Quoted Table, Column (etc) names is not as good as it could be. Any input is currently accepted, however it is not processed further and used in conjunction with the platform.



 Comments   
Comment by Steve Müller [ 24/Jun/13 ]

May this be an issue fixed by https://github.com/doctrine/dbal/pull/302 ?





[DBAL-76] PostgreSQL Platform list* SQL code is in need of serious love Created: 12/Dec/10  Updated: 08/Jun/11

Status: Open
Project: Doctrine DBAL
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major
Reporter: Benjamin Eberlei Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None


 Description   

The Postgres Schema code is very hard to read and inconsistent across the board. Some use pg_class, some pg_tables. Namespaces /Schema are not always properly checked for. There should be a really unified way on how to approach schema query issues.



 Comments   
Comment by Denis [ 08/Jun/11 ]

I'm not sure what this ticket is about exactly, but...

"Namespaces /Schema are not always properly checked for."

Usually, one would not want to specify them and set the search_path instead. Or are you meaning the schema analysis queries used internally? (If so, yes, it kind of sucks in that case, but note that there are a bunch of *_is_visible() methods, e.g. pg_catalog.pg_table_is_visible(rel.oid) which will strip out invisible schemas directly. This may be simpler than injecting schema references all over the place in that it also processes permissions.)

Also, note that PG has a whole bunch of pg_catalog views, which are available in the information_schema.





[DBAL-95] Interbase/Firebird support Created: 26/Feb/11  Updated: 20/May/15

Status: Awaiting Feedback
Project: Doctrine DBAL
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major
Reporter: Benjamin Eberlei Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 6
Labels: None


 Description   

Implemented support for Interbase/Firebird dialects



 Comments   
Comment by Andreas Prucha [ 17/Apr/15 ]

Hi all,

I've create a driver (or two) for Firebird.

The development branch is available at

https://github.com/helicon-os/doctrine-dbal.git DBAL-95-firebird

There are two versions of the driver: One is based on the ibase-api, there other on Firebird PDO.

I'd consider the ibase-version as almost finished, the PDO-Version is rather experimental.

Just send me a comment and maybe we can merge it into the official doctrine dbal (propably just the ibase-version, not the PDO-Version - at least not as long the driver problems are not fixed)

I've tested it with FB 2.5, but not with the upcoming FB 3 or Interbase.

___________________
ibase_firebird

The Ibase-Driver passes the complete Doctrine-Testsuite.

___________________
pdo_firebird

The PDO-Version does not, and it has some limitations which may be related to Firebirds PDO interface itself:

  • BLOBs: Runs into memory leaks if BLOBs are used quite quickly
  • PDO Firebird's transaction handling is quite strange and the reason why it does not pass the testsuite.
  • Nested Transactions (Savepoints): Fails despite Firebird supports them. -
  • Turns a negative signed 32bit integer into a positive unsigned, if the client is running a 64bit-system. I fixed this with a special IntegerType Class in a project, but that's a ugly workaround.
Comment by Andreas Prucha [ 17/Apr/15 ]

One more comment:

I am not sure how to handle one firebird-specificity: Firebird does not preserve the case of identity-names and converts them all-upper, unless they are quoted. The behaviour looks quite similar to Oracle.

Currently I do not normalize names as the Oracle-driver does, which leads to the problem, that they are all-upper in the database anyway, but automatically quoted keywords are lowercase.

Which behaviour would you guys prefere:

Normalize them All-Upper if not quoted (including keywords)? This would allow case-insensitive references in queries, because Firebird handles unquoted names as all-upper.
Quote everything to preserve case: Unfortunately this would require manual quoting in every query.

Personally I do not really like this all-upper-pattern, but having to quote every identifier everywhere looks even more cumbersome, so it's propably the best to follow the behaviour of the oracle driver and assume uppercase if not quoted.

Does any platform have configuration-options? It might be a solution to let the user decide about the naming, e.g.

setNamingConvention(ALL_UPPER | PRESERVE_CASE)

BTW - Is the doctrine-dev mailinglist gone? I wanted to send this there, but it came back with an error.

Andreas

Comment by Andreas Prucha [ 06/May/15 ]

Is it possible to setup a firebird test environment at travis?

Comment by Jürgen [ 20/May/15 ]

Hello Andreas,

good to hear about your work!

As a long time firebird user I can say that for me it is common to use column names case insensitive. So if you normalize all SQL commands to upper case (except quoted names) is the natural usage in firebird for me.

Jürgen





[DBAL-96] Make approach towards identifier quoting consistent Created: 26/Feb/11  Updated: 28/Apr/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms, Schema Managers
Affects Version/s: None
Fix Version/s: 2.6

Type: Improvement Priority: Major
Reporter: Benjamin Eberlei Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 2
Labels: None

Issue Links:
Duplicate
is duplicated by DBAL-120 MySql platform getAlterTableSQL does ... Resolved
Reference
relates to DBAL-45 Add CLI tool that checks for Reserved... Resolved
relates to DBAL-477 Just doublequote all schema names and... Open
is referenced by DBAL-40 Transparent table&column names escaping Open

 Description   
  • Make the use of `` a general approach for explicit quoting of identifiers
  • introduce AbstractPlatform::getRegularSQLIdentifierCase($identifier)
  • Introduce AbstractPlatform::isRegularIdentifier($identifier)
  • Fix Schema Assets not to lower-case, but to check for explicit quoting before.
  • Filter values of identifiers passed to all platform functions when they are used in information schema queries according to `` explicit quoting rules.

Problem: Schema is independent of a vendor, this means we have to pick a behavior, i propose SQL-92

This means:

  • strtoupper() ALL tables, column, index, foreign key names that are not quoted by ``
  • For any Quoted identifiers by `` the case is kept.
  • We can introduce a validator to detect a schema that cannot be implemented with a given vendor platform.

In conjunction with the SQL reserved keywords tickets we can then improve the DatabaseDriver considerably to detect identifier casings



 Comments   
Comment by Steve Müller [ 24/Dec/13 ]

Benjamin Eberlei this is an interesting approach and I like it. But I have some complaints about it.
1. I doubt users will be happy about forced default casing rules (ALL upper or ALL lower). Therefore we should think about adding a simple configuration option in DBAL allowing to override the default casing behaviour to the user's preference.
2. Using a consistent default casing means we ALWAYS have to quote identifiers as otherwise the underlying database could silently change the case again (don't know if this is an issue).
3. Introducing this approach in 2.x branch is a BC break as it breaks users' mixed-case identifier mappings.

For 2.x we should maybe at least make use of Identifier class throughout the platforms where necessary.

Comment by Sebastien Lavoie [ 28/Mar/15 ]

My 2 cents:

1. Users should not have to worry about platform-specific quoting when using the query builder or helpers, the DBAL should do that for you.
2. Users should be able to explicitly quote using a standard quote (`), the quote would then be converted to the platform’s quote upon SQL generation, without any case change.
3. DBAL should not needlessly quote, it adds bloat and it has been said in DBAL-40 that there is a performance hit.
4. DBAL should not change the case without the user’s knowledge.
5. A connection configuration option (normalize_case) could be added:
• uppercase: always convert unquoted identifiers to uppercase
• lowercase: always convert unquoted identifiers to lowercase
• platform: will use the default value for specific platform. For the case of case-sensitive platform, even when unquoted (MySQL on UNIX), do nothing.
• null (default): no normalization
6. Future versions of DBAL could change the default value to platform, but this would greatly reduce the risk of causing BC breaks at the beginning, giving time to test everything.
7. When using Doctrine\DBAL\Connection::query directly, you must do the quoting yourself since the SQL is executed directly.

Comment by Arthur Bodera [ 28/Mar/15 ]

Sebastien, ad 3. that is incorrect. Read the ticket more closely, look at the PR, look inside schema tool and platform classes. There is already a lot of quoting+unquoting being performed in 2.* and a lot of assumptions. Having quoting enabled across the board might actually increase performance in some cases, because there will be less scanning for keywords (see platform classes) and possibly less quoting/unquoting across Schema*.

The problem is, the quoting right now works in some places and in some platforms and is being performed only when schema/schematool/dql needs it, but is being ignored in all other cases. This means that columns like "group" or table names like "platform" will fail randomly depending on platform/rdbms you actually use. It's a nightmare with cross-platform apps and a struggle for single-platform apps, where your tables are named according to domain-rules and happen to overlap with some rdbms.

Quoting identifiers being "a bloat" is similar to saying, that implicit quoting values is a bloat. Although from security standpoint the former is much rarer, it's the same for portability and stability of the DBAL across platforms.

Comment by Andreas Prucha [ 28/Apr/15 ]

IMO the big problem is, that behaviour across the RDBMs may be completely different:

Some preserve case and are case-insenstive if not quoted (the nicest approach)
Some do not preserve case and normalize to lower or uppercase and are case-insensitive
And some do not preserve case and are case-sensitive (the worst)

The biggest issue arises if the DB needs to be used outside the Doctrine-Environment and all identifiers need to be quoted in statements.





[DBAL-124] OCI8 Adapter (convertPositionalToNamedPlaceholders) Statemachine doesn't consider comments and strings Created: 20/May/11  Updated: 28/Jun/11

Status: Open
Project: Doctrine DBAL
Component/s: Drivers
Affects Version/s: 2.0.5
Fix Version/s: None

Type: Improvement Priority: Major
Reporter: Carsten Hetzel Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None


 Description   

SQL statements may contain question marks in strings, inline comments or comment blocks. The current implementation of the method "convertPositionalToNamedPlaceholders()" doesn't consider these implications and falsely replaces them by named bind variables.

Replacement code with example:

<?php
$s = '-- Testkomm?ntar
	select	/* ? *//* ??? */ ?||\'H"al?l"o?\' as "h?""llo" union /* "? Kommentar" \' */
	/* ?" */ select \'/*Hallo\'||to_char( ? ) union select \'--Welt\'
	union--
	select ?';

echo 'In: ' . $s . PHP_EOL;

$bind = 0;
$skip = array( '--' => PHP_EOL, '/*' => '*/', '"' => '"', "'" => "'" );
for( $i = 0; $i < strlen( $s ) /* size of string might change! */; /* yes, no increment here! */ )
{
	// Skipping comments and literals
	foreach( $skip as $begin => $end )
	{
		$matches = substr_compare( $s, $begin, $i, strlen( $begin ) );
		if( $matches !== false && $matches == 0 )
		{
			$pos = strpos( $s, $end, $i+strlen( $begin ) );
			// echo "Found $begin, skipping at $i to $end at $pos" . PHP_EOL;
			if( $pos === false )
			{
				// No more data or illegal statement - anyway: no more replacements!
				// echo "EOD" . PHP_EOL;
				break 2;
			}
			$i = $pos + strlen( $end );
			continue 2; // Ensure we match /*..*//*..*/, '''' or """" - that's why we don't ++$i in the for-loop!
		}
	}
	if( $s[$i] == "?" )
	{
		// Positional to named
		// echo "Replace $bind" . PHP_EOL;
		$r = ':name' . ++$bind;
		$s = substr_replace( $s, $r, $i, 1 );
		$i += strlen( $r );
	}
	++$i;
}

echo 'Out: ' . $s . PHP_EOL;


 Comments   
Comment by Benjamin Eberlei [ 19/Jun/11 ]

This algorithmus is painfully slow. There has to be something better, why do you need to foreach the loop inside the for?

Comment by Carsten Hetzel [ 28/Jun/11 ]

Ok, sent you a PM but I'll put my response here, too:

"Painfully slow" at what circumstances? Benchmarks?

This was meant as an example implementation to show the problems of the current implementation - there's always room for improvement.

And never forget: Premature optimization is the root of all evil! ;-p

You want speed - try this one:

<?php
$sql = '-- Testkomm?ntar
        select  /* ? *//* ??? */ ?||\'H"al?l"o?\' as "h?""llo" union /* "? Kommentar" \' */
        /* ?" */ select \'/*Hallo\'||to_char( ? ) union select \'--Welt\'
        union--
        select ?';

echo 'In: ' . $sql . PHP_EOL;

$map = array();
for( $cnt = 0; $cnt < 1000; ++$cnt )
{
        $bind = 0;
        $s = $sql;

        if( isset( $map[$s] ) )
                continue;

        $strlen = strlen( $s );
        for( $i = 0; $i < $strlen /* size of string might change! */; /* yes, no increment here! */ )
        {
                $c1 = $s[$i];
                $c2 = isset( $s[$i+1] ) ? $s[$i+1] : '';
                if( $c1 == '-' && $c2 == '-')
                {
                        $pos = strpos( $s, PHP_EOL, $i+2 );
                        if( $pos === false )
                        {
                                break 2;
                        }
                        $i = $pos + 2;
                }
                elseif( $c1 == '/' && $c2 == '*' )
                {
                        $pos = strpos( $s, '*/', $i+2 );
                        if( $pos === false )
                        {
                                break 2;
                        }
                        $i = $pos + 2;
                }
                elseif( $c1 == '"' )
                {
                        $pos = strpos( $s, '"', $i+1 );
                        if( $pos === false )
                        {
                                break 2;
                        }
                        $i = $pos + 1;
                }
                elseif( $c1 == "'" )
                {
                        $pos = strpos( $s, "'", $i+1 );
                        if( $pos === false )
                        {
                                break 2;
                        }
                        $i = $pos + 1;
                }
                elseif( $c1 == "?" )
                {
                        // Positional to named
                        // echo "Replace $bind" . PHP_EOL;
                        $r = ':name' . ++$bind;
                        $s = substr_replace( $s, $r, $i, 1 );
                        $rLen = strlen( $r );
                        $i += $rLen;
                        $strlen += $rLen - 1;
                }
                else
                        ++$i;
        }
        $map[$sql] = $s;
}
echo 'Out: ' . $map[$sql] . PHP_EOL;

ATTENTION: Each "break 2;" means, you have an invalid statement! Do, whatever the statement class is supposed to do in this case!

Regards, Carsten





[DBAL-125] Add Informix Support Created: 08/Jun/11  Updated: 11/Jun/11

Status: Open
Project: Doctrine DBAL
Component/s: Drivers
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major
Reporter: Adolfo Flores Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Linux, Unix & Windows



 Description   

Today exist many customers with Informix 11.50 and they will deveoplment with php+simphony, but doctrine will be an alternative inside the framework but it isn't supported.
My customers can't migration ( they are happy with informix) your application and he need a framework as simphony and an abstration of your environment as Doctrine.
There is a possibility to support Informix in Doctrine?
Thanks and regards

Adolfo Flores
Consultant



 Comments   
Comment by Benjamin Eberlei [ 08/Jun/11 ]

There is a possibility to support Informix, this is described here:

http://www.doctrine-project.org/docs/dbal/2.0/en/reference/supporting-other-databases.html

I don't know informix and never worked with it, if you would volunteer to code this i would be enternelly grateful I could help you with the details of implementation.

I think a working prototype can be implemented in 4-6h of time.

Comment by Adolfo Flores [ 08/Jun/11 ]

good idea, I am an expert in informix database and I may need support in doctrine.

Is there any template to follow? I have read the required structure in the url given, it seems feasible.

What next step?

I am installing Informix, php 5.3 on RHES 5.2 All 32-bit

Regards,

Adolfo

Note: .I have no time basis, so that one week would be a reasonable time for this activity

Comment by Benjamin Eberlei [ 11/Jun/11 ]

I added some more details to the chapter, here is the part:

Implementation Steps in Detail
------------------------------

1. Add your driver shortcut to class-name `Doctrine\DBAL\DriverManager`.
2. Make a copy of tests/dbproperties.xml.dev and adjust the values to your driver shortcut and testdatabase.
3. Create three new classes implementing ``\Doctrine\DBAL\Driver\Driver``, ``\Doctrine\DBAL\Driver\Statement``
and ``Doctrine\DBAL\Driver``. You can take a look at the ``Doctrine\DBAL\Driver\OCI8`` driver.
4. You can run the testsuite of your new database driver by calling "cd tests/ && phpunit -c myconfig.xml Doctrine/Tess/AllTests.php"
5. Start implementing AbstractPlatform and AbstractSchemaManager. Other implementations should serve as good example.

Essentially Driver, Statement are the PDO equivalents of "PDO" and "PDOStatement", they even have the exact same API. OCI8 is a good example to see how it works. If Informix works with PDO then its even simpler, take a look at Doctrine\DBAL\Driver\PDO namespace, you don't need to implement the connection and statement then, just the driver taking care of the configuraiton.

The most complicated thing is to get the Platform and SchemaManager to work. Both are responsible for SQL generation for DDL statements for example. It makes sense to run the phpunit testsuite and fix the bugs when occuring until all tests pass.

Also looking at the platform and schema managers of other databases is a very good help





[DBAL-131] Remove Static Types Created: 19/Jun/11  Updated: 04/Dec/14

Status: Open
Project: Doctrine DBAL
Component/s: Drivers
Affects Version/s: None
Fix Version/s: 2.6

Type: Improvement Priority: Major
Reporter: Benjamin Eberlei Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None


 Description   

the static types should be removed and made to instances of types that can differ between platforms.

Following reasons:

1. Some vendors already convert values to PHP types, for example integers. Not converting again could save about 15% wall-time for integers for example.
2. Some vendors require different type code, this is currently handled by expensive if checks.
3. If you want to change a type for one vendor but also use another vendor in the same request, then you have a problem currently.
4. If types were platform aware then we could make registering custom types simpler.






[DBAL-512] Update schema not working on MsSql due to no support for alter identity Created: 06/May/13  Updated: 16/Dec/13

Status: Open
Project: Doctrine DBAL
Component/s: None
Affects Version/s: 2.2.2
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Flip Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 1
Labels: None
Environment:

Symfony 2.1, SQL Server 2008, driver: pdo_sqlsrv



 Description   

When running: php app/console doctrine:schema:update --force

[Doctrine\DBAL\DBALException]
An exception occurred while executing 'ALTER TABLE tableName ALTER COLUMN id INT IDENTITY NOT NULL':

SQLSTATE[42000]: [Microsoft][SQL Server Native Client 11.0][SQL Server]Incorrect syntax near the keyword 'IDENTITY'.

According to this stackoverflow http://stackoverflow.com/a/1049305/1833322 MSSQL does not support this query.



 Comments   
Comment by Steve Müller [ 25/Nov/13 ]

Obviously SQL Server doesn't support this. So what solution would you expect here? IMO such a scenario should be avoided as there is no reasonable way to deal with changing identity columns. I guess most people are not even aware of this. The solutions available are very risky (creating new schema and then migrating data for example) and can take very long to be finished and is extremly error prone. I don't know if we should support one of those workarounds.

Comment by Flip [ 27/Nov/13 ]

The bug is about the SQL statement that does not work.
1. One solution could be to prevent generating this statement and issue a warning instead. (not preferred)

Two other solutions are mentioned in that Stackoverflow post I mentioned, namely:
2. Create a new table with identity & drop the existing table
3. Create a new column with identity & drop the existing column

That is just the way things have to be done in SQL Server (unfortunately), so it wouldn't be a workaround. I think this is a very reasonable way and in fact similar solutions to similar problems have already been implemented into Doctrine. See http://www.doctrine-project.org/blog/doctrine-2-4-released.html "ALTER TABLE support for SQLite (by hason) by creating new tables, moving all the data and then renaming."

Comment by Benjamin Eberlei [ 13/Dec/13 ]

Flip not many people actually use Sqlite in production, its more of a "play database" with Doctrine, wheras SQL Server carries much more weigh. I don't want to implement this kind of workaround, Steve Müller what do you think is safe here?

Comment by Flip [ 13/Dec/13 ]

Not to argue .. but it's not a workaround

Really .. if it is ever implemented it would have to be in a few steps because SQL Server does not accept this one-line command like that. So basically when you don't want this solution it means that will never be implemented.

Talking about safe .. mind that the safety should also be ensured in other area's like 1. don't run --force on a production database 2. make backups. So when regarding the safety it's a very narrow area to take a look at. The operation could be setup in such a way that if the new table/column could not be created then the old one doesn't get deleted. That way you have always the safety like a database transaction.

I guess no point in making a Proof of Concept, because the general outline of the problem is actually not that complicated ..

Thx guys for the input on this issue.

Comment by Steve Müller [ 16/Dec/13 ]

Benjamin Eberlei we could of course use one of the approaches Flip mentioned, but the question is what to do in case something goes wrong during one of the steps required to alter an IDENTITY column. If it is acceptable that there exist "dead" tables/columns in case something goes wrong, we could try implementing that.





[DBAL-1017] Altering a foreign key column is not done properly for MySQL Created: 21/Oct/14  Updated: 21/Oct/14

Status: Open
Project: Doctrine DBAL
Component/s: Schema Managers
Affects Version/s: 2.5, 2.4.3
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Christophe Coevoet Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: mysql, schematool


 Description   

Altering a foreign key column column (for instance to make it not-nullable) or the index of a foreign key does not work on MySQL (to be exact, it does not work on some MySQL setups, but I haven't found the config setting impacting it yet). Making it work requires dropping the foreign key before altering the column/index and readding it after



 Comments   
Comment by Steve Müller [ 21/Oct/14 ]

Christophe Coevoet DBAL-732 related?





[DBAL-1070] AzureSQL specificities are not taken into account Created: 11/Dec/14  Updated: 11/Dec/14

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.5
Fix Version/s: 2.4.3

Type: Bug Priority: Major
Reporter: Nicolas Séverin Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: azure, dbal, sqlserver
Environment:

Azure website using an AzureSQL database (based on SQL Server 2012).



 Description   

In Azure SQL, table ‘sys.extended_properties’ does not exist.
But SQLAzurePlatform inherits from class SQLServerPlatform, which uses it.
The code in question is here /Doctrine/DBAL/Platforms/SQLServerPlatform.php#L845.

Modifications to this portion of code were made to handle comments on columns differently in doctrine/dbal 2.5, but it used to work in 2.4.3.



 Comments   
Comment by Marco Pivetta [ 11/Dec/14 ]

Reverted to priority Major





[DBAL-1081] Paginator - Query Limit for SQL Server - SqlServerPlatform.php Created: 16/Dec/14  Updated: 16/Jun/15

Status: Open
Project: Doctrine DBAL
Component/s: None
Affects Version/s: 2.5
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Maciej Grajcarek Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: dbal, dql, orderBy, sqlserver
Environment:

Windows



 Description   

Hi!
I have a problem with Query results limit when ordering by SUM of a field.

My query looks like this:

            SELECT ypk.name keyword, SUM(ypk.value) total
            FROM YpBundle:YellowPageKeyword ypk
              JOIN ypk.yellowpage yp
              JOIN yp.listing l
            WHERE l.customer = :customerId
              AND ypk.originDate >= :dateFrom
              AND ypk.originDate <= :dateTo
            GROUP BY ypk.nameCrc, ypk.name
            ORDER BY SUM(ypk.value) DESC

First I was catching following error:
message: "[Syntax Error] line 0, col 395: Error: Expected known function, got 'SUM'"
class: Doctrine\ORM\Query\QueryException

It only accourse if SUM is used in ORDER BY clause.

I have registered new class Sum which extends FunctionNode.

Now, query is build and executed but it has an error:

'SELECT * FROM 
     (SELECT y0_.name AS name_0, SUM(y0_.value) AS sclr_1, ROW_NUMBER() OVER (ORDER BY value) DESC) AS doctrine_rownum FROM yellowpage_keywords y0_ INNER JOIN yellowpages y1_ ON y0_.yellowpage_id = y1_.id INNER JOIN listings l2_ ON y1_.listing_id = l2_.id WHERE l2_.customer_id = ? AND y0_.origin_date >= ? AND y0_.origin_date <= ? GROUP BY y0_.name_crc, y0_.name) AS doctrine_tbl WHERE doctrine_rownum BETWEEN 1 AND 10 ORDER BY doctrine_rownum' with params ["111", "2013-01-01 00:00:00.000000", "2014-12-31 23:59:59.000000"]

The line :

ROW_NUMBER() OVER (ORDER BY value) DESC)

should look like

ROW_NUMBER() OVER (ORDER BY SUM(y0_.value) DESC)

In doModifyLimitQuery method I have modified:

                $pattern = sprintf('/%s\.%s\s+(?:AS\s+)?([^,\s)]+)/i', $column['table'], $column['column']);

to:

                $pattern = sprintf('/%s\.%s\s+(?:AS\s+)?([^,\s)]+)/i', str_replace('(', '\(', $column['table']), str_replace(')', '\)', $column['column']));

Now preg_match founds matching strings and OVER part of query is build correctly.

I checked other issues about this problem (which are marked as already fixed) and I have no idea why it's not working for me.

Thanks in advance!



 Comments   
Comment by Marco Pivetta [ 17/Dec/14 ]

Seems like a bit of information is missing: Is this issue related to the paginator API or not?

Comment by Maciej Grajcarek [ 17/Dec/14 ]

First of all - thank you for formatting my issue.
Secondly yes - issue is directly connected with paginator API.

Here is a code which should help you to replicate the problem:

            SELECT ypk.name keyword, SUM(ypk.value) total
            FROM YpBundle:YellowPageKeyword ypk
              JOIN ypk.yellowpage yp
              JOIN yp.listing l
            WHERE l.customer = :customerId
              AND ypk.originDate >= :dateFrom
              AND ypk.originDate <= :dateTo
            GROUP BY ypk.nameCrc, ypk.name
            ORDER BY SUM(ypk.value) DESC

        $dql = $this->getEntityManager()->createQuery($query);
        $dql
            ->setParameter('customerId', $customerId)
            ->setParameter('dateFrom', $dateFrom)
            ->setParameter('dateTo', $dateTo);
        $dql->setMaxResults(10);    

        $keywords = $dql->getArrayResult(); 
Comment by Marco Pivetta [ 17/Dec/14 ]

That doesn't involve the paginator, just DQL.

SUM() and computed values are not supported in the ORDER BY clause: you have to select them first. Try:

            SELECT ypk.name keyword, SUM(ypk.value) total
            FROM YpBundle:YellowPageKeyword ypk
              JOIN ypk.yellowpage yp
              JOIN yp.listing l
            WHERE l.customer = :customerId
              AND ypk.originDate >= :dateFrom
              AND ypk.originDate <= :dateTo
            GROUP BY ypk.nameCrc, ypk.name
            ORDER BY total DESC
Comment by Maciej Grajcarek [ 18/Dec/14 ]

If I will do it, it will result in a following query:

SELECT *
FROM (
	SELECT y0_.name AS name_0, SUM(y0_.value) AS sclr_1, ROW_NUMBER() OVER (ORDER BY sclr_1 DESC) AS doctrine_rownum 
	FROM yellowpage_keywords y0_
	INNER JOIN yellowpages y1_ ON y0_.yellowpage_id = y1_.id
	INNER JOIN listings l2_ ON y1_.listing_id = l2_.id
	WHERE l2_.customer_id = 111
		AND y0_.origin_date >= '2014-01-01'
		AND y0_.origin_date <= '2014-12-01'
	GROUP BY y0_.name_crc, y0_.name
) AS doctrine_tbl
WHERE doctrine_rownum BETWEEN 1 AND 10
ORDER BY doctrine_rownum

It's an incorrect query for an SQL Server. Take a look on this part:

ROW_NUMBER() OVER (ORDER BY sclr_1 DESC) AS doctrine_rownum

SQL Server do not support aliasing in OVER clause.

It should look like this, to work:

ROW_NUMBER() OVER (ORDER BY SUM(y0_.value) DESC) AS doctrine_rownum

It looks like this is the copy of this issue: http://www.doctrine-project.org/jira/browse/DBAL-788
I'm on doctrine version 2.5 which has patch from that issue included, but I have no idea why it's not working for me.

Comment by Steve Müller [ 24/Dec/14 ]

Maciej Grajcarek looks like it is an issue with your SUM function implementation. If you change your DQL to use ORDER BY COUNT(ypk.value) instead of ORDER BY SUM(ypk.value), does it work then? If so, there is something wrong with your SUM function and therefore not an issue with DBAL.

Comment by Steve Müller [ 24/Dec/14 ]

well forget about it I think I am wrong here.

Comment by Maciej Grajcarek [ 29/Dec/14 ]

Hi,
I will try to use COUNT as soon as I will be able to do that.

But I already tried other aggregation functions before and the result was exactly the same.

Comment by Luca Cerretani [ 16/Jun/15 ]

I get a similar error using this sql:

$sql = "SELECT table_one.id, table_one.number, table_two.name " . 
           "FROM table_one " .
	   "LEFT JOIN table_two	ON table_two.table_one_id = table_one.id " .
           "ORDER BY table_one.id DESC";

using the doModifyLimitQuery i get the uncorrect sql

SELECT * FROM (SELECT table_one.id, table_one.number, table_two.name, ROW_NUMBER() OVER (ORDER BY id DESC) AS doctrine_rownum  FROM table_one LEFT JOIN table_two ON table_two.table_one_id = table_one.id) AS doctrine_tbl WHERE doctrine_rownum BETWEEN 1 AND 1 ORDER BY doctrine_rownum

it should be

SELECT * FROM (SELECT table_one.id, table_one.number, table_two.name, ROW_NUMBER() OVER (ORDER BY table_one.id DESC) AS doctrine_rownum  FROM table_one LEFT JOIN table_two ON table_two.table_one_id = table_one.id) AS doctrine_tbl WHERE doctrine_rownum BETWEEN 1 AND 1 ORDER BY doctrine_rownum




[DBAL-1211] Wrapper Class should enforce a Interface not a Subclass Created: 27/Apr/15  Updated: 27/Apr/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: 2.5.1
Fix Version/s: None

Type: Improvement Priority: Major
Reporter: Flavio Botelho Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: dbal


 Description   

We are creating a Wrapper Class to allow the use of the Oracle Proxy User feature. For that I need to Wrapper a Class around DBAL\Connection.
Unfortunely, the wrapper class needs to be a subclass of DBAL\Connection which doesn't make sense, there should exist an Interface and the wrapper class should be forced to implement that interface.

That way I don't need to create methods to call all DBAL\Connection methods thru polymorphism.






[DBAL-1214] MySQL has gone away using ImportCommand Created: 29/Apr/15  Updated: 29/Apr/15

Status: Open
Project: Doctrine DBAL
Component/s: None
Affects Version/s: 2.3.4
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Sylvain Filteau Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Importing a fairly large SQL file (a MySQL dump) fails with a MySQL has gone away error.

When reading the ImportCommand code, I see that this issue is already "fixed" by checking if the connection is an instance of `\Doctrine\DBAL\Driver\PDOConnection`.

The problem is that I don't see how the connection could be an instance of this class since it doesn't extend `\Doctrine\DBAL\Connection`.

If I'm wrong, then I don't know how to configure my connection to extend the PDOConnection class. Thanks for your help on that






[DBAL-1286] Add Cassandra driver Created: 25/Aug/15  Updated: 25/Aug/15

Status: Open
Project: Doctrine DBAL
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major
Reporter: Oleg Andreyev Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None


 Description   

http://cassandra.apache.org/






[DBAL-150] noSQL Shema Management Created: 21/Aug/11  Updated: 21/Aug/11

Status: Open
Project: Doctrine DBAL
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Minor
Reporter: Alexey Shatunov Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None
Environment:

all



 Description   

I have a few ideas to improve DBAL and ORM management for php:
1. Reading if needed ORM schemas definitions from YAML(existing plugin)
2. Then migration it into the one of NoSQL databases(such as MongoDB or Memcache/MemcacheDB)
3. Creating transactional temporary-changes data layer in NoSQL for a current-changes due the webapp launch and running for each run
4. Periodically migrations of a current changes with migration script into RDB(for example by cron)

So we get:
Fast automatic shemas management(Symphony users - I say you "haha" because you have to do some unuseful things like "doctrine:build-schema" etc)
All advantages of RDB (like JOINS) in a data extraction
Full-independence in data changing due to webapp launch(and no transactions are needed) in the current NoSQL data-layer
Low load to the server because after extration we have a lazy return data into the RDB
In feature a simple caching any fast-changed data into NoSQL

What do you thinking about it?






[DBAL-175] Table comments in Doctrine\DBAL\Schema\Table Object Created: 06/Oct/11  Updated: 17/Oct/11

Status: Open
Project: Doctrine DBAL
Component/s: Platforms, Schema Managers
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Minor
Reporter: Asmir Mustafic Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 1
Labels: None


 Description   

Should be useful discover the table comments from database schema.

This feature is already available for column comments, but not for table comments






[DBAL-319] Doctrine\DBAL\Types\Type Created: 12/Aug/12  Updated: 22/Dec/13

Status: Open
Project: Doctrine DBAL
Component/s: None
Affects Version/s: 2.2.2
Fix Version/s: None

Type: Improvement Priority: Minor
Reporter: Till Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None


 Description   

The API could be improved in the next major release:

  • Type::add() instead of addType()
  • Type::isRegistered() instead of hasType() (has sounds weird)

Etc.. I think the 'type' in the methods is redundant since Type is already the object I am dealing with.

I'd also like a remove() method to unregister a type at runtime. Would make testing a little easier and I don't have to check with hasType() etc..



 Comments   
Comment by Marco Pivetta [ 12/Aug/12 ]

I don't think those namings are really important.
There's one major change to do, which is to somehow get rid of the staticness of the `Doctrine\DBAL\Types\Type` class, and it is not a simple task.
It would also be interesting to set the type objects manually, such as a database `password` type that does encryption/decryption based on a given service. That cannot be done without staticness right now, which renders two different connections requiring the same functionality but with different parameters very difficult.

Comment by Till [ 12/Aug/12 ]

I agree on the static. But I'd also like the API to be cleaned up and the remove method.

Comment by Steve Müller [ 22/Dec/13 ]

API cleanup and staticness can be addressed in 3.0 for the first time. Otherwise we cannot keep BC. You can use Doctrine\DBAL\Types\Type::overrideType('someType', null); as a workaround for removing a type from the registry.





[DBAL-761] Driver\ResultStatement::fetchAll() returns empty array on a seemingly valid Driver\PDOStatement object Created: 03/Jan/14  Updated: 08/Jan/14

Status: Open
Project: Doctrine DBAL
Component/s: Drivers
Affects Version/s: 2.4
Fix Version/s: None

Type: Improvement Priority: Minor
Reporter: Dennis Matveyev Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: mysql, orm, query, schematool
Environment:

Windows 7 64 bit, Zend Server/Studio, PHP 5.4.16,
Server: Apache/2.2.22 (Win32) mod_ssl/2.2.22 OpenSSL/0.9.8x



 Description   

I came across a weird issue, where when running:

vendor/bin/doctrine-module orm:schema-tool:update

I would get:

[Doctrine\DBAL\Schema\SchemaException]
There is no column with name 'resource_id' on table 'role_resource'.

But I did have a column with the above name in the above table, so that was a weird message for me. So I traced it all the way to this line of code:

https://github.com/doctrine/dbal/blob/2.4/lib/Doctrine/DBAL/Connection.php#L630

If I remove "->fetchAll()" from that line, I get this object:

object(Doctrine\DBAL\Driver\PDOStatement)#531 (1) {
["queryString"]=> string(332) "SELECT COLUMN_NAME AS Field, COLUMN_TYPE AS Type, IS_NULLABLE AS `Null`, COLUMN_KEY AS `Key`, COLUMN_DEFAULT AS `Default`, EXTRA AS Extra, COLUMN_COMMENT AS Comment, CHARACTER_SET_NAME AS CharacterSet, COLLATION_NAME AS CollactionName FROM information_schema.COLUMNS WHERE TABLE_SCHEMA = 'loginauth' AND TABLE_NAME = 'role_resource'"
}

Which has a valid SQL query that runs fine and shows Field names on my local machine's MySQL prompt. But when I add ->fetchAll() back in that line of code, an empty array is returned, field names are not returned, and a SchemaException is generated. I found this odd and wanted to report it. Whether it is a bug or not, hopefully I can find the cause of this issue.

For loads more info, please see this: http://stackoverflow.com/questions/20907491/doctrine-2-there-is-no-column-with-name-columnname-on-table-table



 Comments   
Comment by Steve Müller [ 03/Jan/14 ]

Dennis Matveyev It seems you are using MySQL. Can you please tell which version you use?

Comment by Steve Müller [ 03/Jan/14 ]

Dennis Matveyev Please also provide whether you use mysqli or PDO_MYSQL.

Comment by Steve Müller [ 03/Jan/14 ]

Dennis Matveyev Sounds dumb, but also are you sure that the vendor/bin/doctrine-module orm:schema-tool:update uses the correct connection parameters and does not by error connect to another database server or something? Because otherwise everything looks good IMO.

Comment by Dennis Matveyev [ 07/Jan/14 ]

Yes, I am using:
Server version: 5.5.23-log MySQL Community Server (GPL)
mysql client Ver 14.14 Distrib 5.5.23, for Win32 (x86)

Doctrine connection string for 'driverClass' is 'Doctrine\DBAL\Driver\PDOMySql\Driver', so it is PDO_MYSQL.

I am connecting to the right database, but your last suggestion is what helped to uncover the issue. When connecting to the database with the very same user/pass, and running the command, here is what I saw:

mysql> select resource_id from role_resource;
ERROR 1142 (42000): SELECT command denied to user 'login'@'localhost' for table 'role_resource'

Running a GRANT command to allow SELECT for this user solved the problem.

To improve debugging of similar issues, I'd see if there is a way to propagate the error from MySQL server to ./doctrine-module,
or
improve the error message of SchemaException, to i.e. "There is no column with name 'action_id' on table 'role_action', or database permissions prevent table access."

Thanks!

Comment by Steve Müller [ 07/Jan/14 ]

Dennis Matveyev Thank you for reporting this and I'm glad I could help. I will mark this as improvement, though. You are right in that the root cause of the error should be propagated to the user instead.

Comment by Steve Müller [ 08/Jan/14 ]

Dennis Matveyev Okay I found the root cause of the problem. See what MySQL states about access to the information_schema table for retrieving metadata about a certain database:

Each MySQL user has the right to access these tables, but can see only the rows in the tables that correspond to objects for which the user has the proper access privileges. In some cases (for example, the ROUTINE_DEFINITION column in the INFORMATION_SCHEMA.ROUTINES table), users who have insufficient privileges will see NULL.

So it returns NULL instead of raising an error and this is why Doctrine is not able to propagate the proper exception in this case. I'm not sure if changing the exception message in the SchemaException class is the proper way of handling this. Also please note that this is not only about columns (where you got stuck at) but can occurr for almost every list*() action called in the SchemaManager I suppose. Furthermore this might be a MySQL specific issue, not sure about that.

The proper solution would be to throw an exception much earlier, to have a decent behaviour in such an edge case. I think we should throw an exception here: https://github.com/doctrine/dbal/blob/master/lib/Doctrine/DBAL/Schema/AbstractSchemaManager.php#L267 because a table cannot live without columns and indicates an error somewhere. But I would like to hear the opinion of Benjamin Eberlei on that issue.

Comment by Steve Müller [ 08/Jan/14 ]

I checked the SQL Server documentation and it seems they behave exactly the same: http://technet.microsoft.com/en-us/library/ms187113.aspx

In SQL Server 2005 and later, the visibility of metadata is limited to securables that a user either owns or on which the user has been granted some permission. For example, the following query returns a row if the user has been granted a permission such as SELECT or INSERT on the table myTable.

However, if the user does not have any permission on myTable, the query returns an empty result set.





[DBAL-812] SchemaTool ignores multi-columns (composite) indexes when creating a foreign key Created: 13/Feb/14  Updated: 15/Feb/14

Status: Open
Project: Doctrine DBAL
Component/s: Schema Managers
Affects Version/s: 2.4.2
Fix Version/s: None

Type: Improvement Priority: Minor
Reporter: Peter Huynh Assignee: Steve Müller
Resolution: Unresolved Votes: 0
Labels: schematool
Environment:

MySQL



 Description   

When using the schema tool to generate the differences between the db and the entities, I notice the following:

ALTER TABLE sales ADD CONSTRAINT FK_36D222EF603EE73 FOREIGN KEY (vid) REFERENCES vendor (id);
CREATE INDEX IDX_36D222EF603EE73 ON sales (vid);

Normally, it doesn't bother me, however I have had an existing composite index consists of (vid, submit). This leads to the redundant index IDX_36D222EF603EE73 and therefore forcing unnecessary overheads.

It would be nice to have a mean to disable this from happening.

The code in question can be found at https://github.com/doctrine/dbal/blob/594e326bd58d1d7af578f0dc3143655b9d119d45/lib/Doctrine/DBAL/Schema/Table.php#L543.

A helpful member from #doctrine IRC also pointed out that it does not check the unique constraints also.

Regards,



 Comments   
Comment by Marco Pivetta [ 13/Feb/14 ]

Just a hint on this: looks like the current logic just checks hashed indexes, ignoring the "uniq" prefixed indexes to avoid duplicates.

Comment by Steve Müller [ 15/Feb/14 ]

Peter Huynh I get the point here. However after having a quick look into this it seems this behaviour exists for a good reason. See here: https://github.com/doctrine/dbal/blob/master/lib/Doctrine/DBAL/Schema/Index.php#L191-L195
The question is how could this reasonable extended to fulfil the needs you explained. I am not entirely sure about this and this peace of code is rather sensitive and critical. I don't want to rush any assumptions here.

Comment by Peter Huynh [ 15/Feb/14 ]

Hi Steve,

I fully understand your reservation.

Thanks for looking into things.

Peter





[DBAL-879] Sequence default value [PGSQL] Created: 29/Apr/14  Updated: 26/Oct/14

Status: Open
Project: Doctrine DBAL
Component/s: Drivers, Schema Managers
Affects Version/s: 2.4.2
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: Mohammad Niknam Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: autoincrement, postgresql, sequence
Environment:

ArchLinux
PostgreSQL 9.3.4



 Description   

Hi
I'm using dbal to generate schmea from database via Schema-Manager. The problem is that my primary field 'id' have default value of 'nextval('test1_id_seq'::regclass)' but when I retrive columns using Doctrine\DBAL\Schema\AbstractSchemaManager::listTableDetails() or Doctrine\DBAL\Schema\Table::getColumns() , default value of the column 'id' is null.
In Doctrine\DBAL\Schema\PostgreSqlSchemaManager::_getPortableTableColumnDefinition() method at line 292 default value replaced with null, I don't know why but I guess It's because Driver compatibility.
Also Doctrine\DBAL\Schema\Sequence has no method to retrieve that table.
So I don't have the default value (pointing at sequence) and I can't find out what Sequence is linked to this table either.



 Comments   
Comment by Steve Müller [ 26/Oct/14 ]

Can you please provide a code example of how you create table + sequence and retrieve it?





[DBAL-1046] Broken link Created: 12/Nov/14  Updated: 02/Dec/14

Status: Open
Project: Doctrine DBAL
Component/s: None
Affects Version/s: 2.3.5
Fix Version/s: None

Type: Task Priority: Minor
Reporter: ted bohus Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None


 Description   

On the website, the link to download Doctrine DBAL 2.3.5 doesn't work...

http://www.doctrine-project.org/downloads/DoctrineDBAL-2.3.5-full.tar.gz

Thanks



 Comments   
Comment by Steve Müller [ 02/Dec/14 ]

ted bohus for now please use this link: https://github.com/doctrine/dbal/archive/v2.3.5.zip
We are having a look into the issue.





[DBAL-1164] Creating SQLite Connections via a URL Does Not Work Created: 07/Mar/15  Updated: 07/Mar/15

Status: Open
Project: Doctrine DBAL
Component/s: Drivers
Affects Version/s: 2.5.1
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: Christopher Davis Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None


 Description   

When creating a SQL light connection put the URL path into the `dbname` param. But the SQLite driver doesn't using the `dbname` param: it uses the `path` parameter.

So doing this:

$conn = DriverManager::getConnection([
'url' => 'sqlite:////some/path/here',
]);

Generates a PDO DSN like this: `sqlite:`. The path is completely ignored.

See the driver code here: https://github.com/doctrine/dbal/blob/6b6143ba16e5f17242835910173c033a8f73f845/lib/Doctrine/DBAL/Driver/PDOSqlite/Driver.php#L81-L88

`DriverManager` either needs some logic to use the path when it sees a sqlite URL or the Driver itself should use `dbname` (eg. check path, check dbname, check memory).



 Comments   
Comment by Christopher Davis [ 07/Mar/15 ]

Same is true for memory databases. `dbname` is set, but the driver never checks it.





[DBAL-1206] Generating Table SQL without indexes is invalid if using AUTO_INCREMENT Created: 24/Apr/15  Updated: 24/Apr/15

Status: Open
Project: Doctrine DBAL
Component/s: Platforms
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: Markus Fasselt Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None
Environment:

MySQL (but maybe other database vendors too)



 Description   

Dumping the following table

CREATE TABLE `users` (
    `id` INT AUTO_INCREMENT NOT NULL,
    PRIMARY KEY (`id`)
);

with the following snippet (0 = NoIndexes)

$platform->getCreateTableSQL($table, 0);

results in

CREATE TABLE `users` (
    `id` INT AUTO_INCREMENT NOT NULL,
);

The problem is, that the table contains an AUTO_INCREMENT column which cannot be used without a primary key. But the primary key is skipped, as I skipped all indexes.

As this SQL is invalid, I suggest to skip the AUTO_INCREMENT argument, too, if the indexes are skipped. Alternatively, the Primary Key always has to be included.

What do you think? I can provide a fix, if you agree with me.






[DBAL-1207] Schema Update Issue with DBAL 2.5 Binary Type Created: 24/Apr/15  Updated: 18/May/15

Status: Open
Project: Doctrine DBAL
Component/s: Schema Managers
Affects Version/s: 2.5
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: Alex Gurrola Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: schematool
Environment:

Ubuntu 14.04.2 LTS, Apache 2.4.7, PHP 2.5.9, MySQL 5.5, Symfony 2.6.6



 Description   

Every time I run a doctrine:schema:update command within Symfony, using DBAL 2.5, it tries to execute this SQL Query, every single time:

SQL Query
ALTER TABLE user_sessions CHANGE sess_id sess_id VARBINARY(128) NOT NULL;

The PHP Annotations I am using for this column is as follows:

Binary Column
/**
 * @var string
 *
 * @ORM\Column(name="sess_id", type="binary", length=128, nullable=false)
 * @ORM\Id
 * @ORM\GeneratedValue(strategy="IDENTITY")
 */
private $sessId;

It seems the binary type is somehow registering a change even though no change has actually been made. I updated to DBAL to 2.5 before getting the binary type supported by the doctrine:schema:update command.

Thanks in advance for any assistance in squashing this odd bug.

--Alex Gurrola



 Comments   
Comment by Nate Baker [ 15/May/15 ]

Do you intentionally have @ORM\GeneratedValue(strategy="IDENTITY"), or is that there from an auto import? I had the same problem but if I changed to strategy="NONE" it doesn't happen anymore. See this issue: http://www.doctrine-project.org/jira/browse/DBAL-353

Comment by Alex Gurrola [ 18/May/15 ]

It was an auto import. Your link resolved the issue, thanks.





[DBAL-1234] Additional slash in dbname when providing settings as URL without scheme Created: 21/May/15  Updated: 21/May/15

Status: Open
Project: Doctrine DBAL
Component/s: Drivers
Affects Version/s: 2.5.1
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: Sebastian Krebs Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None

Issue Links:
Duplicate
duplicates DBAL-1238 [GH-863] Strip leading slash of datab... Open

 Description   

Hi,

I use https://github.com/realestateconz/MssqlBundle to connect to an MSSQL-database and I'd like to provide the connection parameters as URL. Because dblib is not a supported driver I setup the driverClass instead

driver_class:   \Realestate\MssqlBundle\Driver\PDODblib\Driver

So the corresponding URL would look like

//user:pass@127.0.0.1/dabasename

But now it tries to connect to the database /databasename instead of databasename. I can set an arbitrary scheme here as long as it exists and is supported (and is not SQLite)

mysqli://user:pass@127.0.0.1/dabasename

Now it works, but it's a hack.

It seems, that the issue is here
https://github.com/doctrine/dbal/blob/32b1a4f85a078f67752851c27be4065071db1f8b/lib/Doctrine/DBAL/DriverManager.php#L262
As long as there is no scheme the leading slash remains. I'd guess, that it should also take into account, that there might be no driver name, but a concrete driverClass instead

(!isset($url['scheme']) && !isset(isset($url['driverClass']))

?






[DBAL-933] Get Statement Column Metadata Created: 05/Jul/14  Updated: 06/Jul/14

Status: Open
Project: Doctrine DBAL
Component/s: Drivers
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Trivial
Reporter: Benoît Burnichon Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None


 Description   

It would be nice to have a utility method in \Doctrine\Dbal\Driver\ResultStatement to properly retrieve the result column names.

Currently, we have to rely on somthing like
$columnNames = array_keys($statement->fetch(\PDO::FETCH_ASSOC));

Problem: It does not work with empty result-sets, and more checks should be performed to handle these.

With PDO, http://www.php.net/manual/en/pdostatement.getcolumnmeta.php could be used to properly retrieve names.

For Sqlite3 it is easy, http://www.php.net/manual/en/sqlite3result.columnname.php

For Mysql, http://www.php.net/manual/en/mysqli-result.fetch-fields.php would do the trick






[DBAL-1243] Unique Key on two columns overrules three column index causing drop index Created: 09/Jun/15  Updated: 09/Jun/15

Status: Open
Project: Doctrine DBAL
Component/s: Schema Managers
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Trivial
Reporter: Arkadiusz Rzadkowolski Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Running schema compare will result with index being returned for removal.

DROP INDEX OPB_BLG_IDX1 ON OPB_BLOGS;

The reason for that is that OPB_BLG_IDX1 matches two columns (BLG_DOMAIN, BLG_PATH) with unique key OPB_BLG_UK1. It skips check for last column (BLG_STATUS).

Shouldn't spansColumns method be run on same type of index only? Right now OPB_BLG_IDX1 is being removed since doctrine thinks it's overruled by unique key (and I don't think it should be treated that way).

Example annotation (problem is with OPB_BLG_UK1 & OPB_BLG_IDX1 as stated above):

/**
 * OpbBlogs
 *
 * @ORM\Table(name="OPB_BLOGS", uniqueConstraints={@ORM\UniqueConstraint(name="OPB_BLG_UK1", columns={"BLG_DOMAIN", "BLG_PATH"})},
 * indexes={
 *      @ORM\Index(name="OPB_BLG_IDX1", columns={"BLG_DOMAIN", "BLG_PATH", "BLG_STATUS"}),
 *      @ORM\Index(name="OPB_BLG_IDX2", columns={"BLG_USR_ID"}),
 *      @ORM\Index(name="OPB_BLG_IDX3", columns={"BLG_TYP_ID"}),
 *      @ORM\Index(name="OPB_BLG_IDX4", columns={"BLG_CAT_ID"}),
 *      @ORM\Index(name="OPB_BLG_IDX5", columns={"BLG_DOMAIN"}),
 *      @ORM\Index(name="BLG_CREATED_DATE", columns={"BLG_CREATED_DATE"}),
 *      @ORM\Index(name="BLG_ID", columns={"BLG_ID", "BLG_LAST_POST_ID"}),
 *      @ORM\Index(name="BLG_LAST_POST_DATE", columns={"BLG_LAST_POST_DATE"}),
 *      @ORM\Index(name="BLG_SLT_ID", columns={"BLG_SLT_ID", "BLG_DBNAME"})
 * })
 *
*@ORM\Entity
 */





Generated at Thu Aug 27 22:33:45 EDT 2015 using JIRA 6.4.10#64025-sha1:5b8b74079161cd76a20ab66dda52747ee6701bd6.