|
Thats why you shouldn't generate proxies at runtime. The problem happens on windows, because the atomic rename operation doesn't work as perfectly there as on linux.
We cannot fix this in Doctrine.
|
|
Benjamin Eberlei What do you mean "you shouldn't generate proxies at runtime"? I'm not in production, this is in dev. And I'm using the default configuration.
What I don't understand is why will Doctrine regenerate proxies on every request? The warning is reproductible, and even when no PHP entity has been touched.
|
|
Benjamin Eberlei To simplify my previous message (I don't want to bury you under questions) I'll sum it up like that:
What can I do?
Thanks!
|
|
Benjamin Eberlei ping: what can be done?
Can we suppress the error with @rename($tmpFileName, $fileName); ?
https://github.com/doctrine/common/blob/master/lib/Doctrine/Common/Proxy/ProxyGenerator.php#L287
I can make a PR if you think that's a valid solution.
|
|
Matthieu Napoli no, if you have warnings, please disable them via ini setting. With error suppression there, we may have further problems identifying more serious issues.
About proxy generation: that happens EVERY time in dev environments. Generate them once and disable it afterwards.
|
|
Marco Pivetta OK I can disable the auto generation then (I'll have to remember to regenerate them when I edit the model).
Thanks for the feedback
Is that possible to make those proxies generate only if the entity file has been modified since the last generation? (only asking if can and should be done, I can look for implementing it myself if that's the case)
|
|
Matthieu Napoli that would be very obnoxious when changing entities often. I wouldn't do that (generating only if not already available)
|
|
Marco Pivetta Yes but for now they are regenerated at every request when in dev mode (at least with the default configuration http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/configuration.html#obtaining-an-entitymanager)
Which one is worse: generating every proxy class at every request, or generate only those which changed (in dev environment of course, not prod)?
If neither of these options are good (i.e. auto generation should be disabled), I don't understand why the docs say to enable auto generation when in dev environment.
|
|
Matthieu Napoli that's because in dev environments you shouldn't care about that one exception (usually happens when you got concurrent requests).
It is worse to generate only on changes: that's a lot of additional checks, variables to keep in memory and additional logic that is not needed.
Let's keep it as it is (generating at each request) for dev environments: works fine 
Another (eventual) solution for dev environments would be not to write the proxy file, but to eval it.
|
|
eval it would be a good solution IMO, no more "woops the directory is not writable" and it's more neutral for the user filesystem (but not as easy to debug). But OK, I see what you mean, it works let's keep it that way.
Actually the problem on my setup is that PHP errors are turned into exceptions, so on an (poorly designed) AJAX treeview (lots of nodes to load => lots of requests), I end up with some nodes not loaded because of the exception. And it feels weird to either silently log all PHP warnings or silently ignore the specific warning for the rename.
|
|
Matthieu Napoli I'd go with `eval` then. Needs refactoring of the abstract proxy factory and of the proxy generator (proxy generator should no longer write files).
|
|
Re-opening: the proxy factory could directly `eval()` the produced proxy code. The ProxyGenerator should no longer write the generated files to disk automatically.
|
|
I've opened a PR: https://github.com/doctrine/common/pull/269
|
|
Hi
Could you please attach your entities or a failing test case ?
Cheers
|
|
sure
LineRelated.php :
class LineRelated
{
/**
* @ORM\Id
* @ORM\Column(type="integer")
* @ORM\GeneratedValue(strategy="AUTO")
*/
protected $id;
/**
* @ORM\ManyToOne(targetEntity="Line", inversedBy="line_related")
* @ORM\JoinColumn(name="line_id", referencedColumnName="id",nullable=false)
*/
protected $line;
/**
* @ORM\Column(name="line_id_related", type="integer")
*/
protected $line_related;
/**
* @ORM\Column(type="smallint",nullable=false)
*/
protected $count = 0;
###### get/set etc....... #######
Line.php
class Line
{
/**
* @ORM\Id
* @ORM\Column(type="integer")
* @ORM\GeneratedValue(strategy="AUTO")
*/
protected $id;
########## blablabla #############
my action:
public function right_line_relatedAction($line = null, $title='相关线路')
{
$lines = $l->getByUser($user, array());
$related = $this->getDoctrine()->getRepository('MyWebBundle:LineRelated')->findBy(array('line' => $lines), array('count' => 'DESC'), 20);
return $related;
}
|
|
Hi,
How did you get this query string ?
Repository#findBy does not quote the values, It uses PDO:bindParam.
so the expected query string should be someting like :
WHERE t0.line_id IN (?, ? ,?)
I tried to reproduce but in my tests the generated Query binds the parameters as "PDO::PARAM_INT".
I have added a test case.
Could you please can try to change it and make fails.
Cheers
|
|
reproduced :
SELECT t0.id AS id1, t0.line_id_related AS line_id_related2, t0.count AS count3, t0.line_id AS line_id4 FROM line_related t0 WHERE t0.line_id IN ('6059', 4851, '6068', 126352, '6060', '1000000') ORDER BY t0.count DESC LIMIT 20
Parameters: [['6059', 4851, '6068', 126352, '6060', '1000000']]
[Hide runnable query]
Time: 234.53 ms [ Explain query ]
let me have a look on what's going on
|
|
interesting. I've dump(using ladybug_dump) the $lines,and I found out that when the element is a Proxies Object(Object(Proxies_GC_\My\WebBundle\Entity\Line)),then the id of that Object will be with quoted,when the elememt is an Real Entity,then It will be without quote.
for example,in my last comment, the parameters is [['6059', 4851, '6068', 126352, '6060', '1000000']]
the result of dumping $lines is :
array(6)
[0]: object(Proxies_CG_\Zuo\WebBundle\Entity\Line)
[1]: object(Zuo\WebBundle\Entity\Line)
[2]: object(Proxies_CG_\Zuo\WebBundle\Entity\Line)
[3]: object(Zuo\WebBundle\Entity\Line)
[4]: object(Proxies_CG_\Zuo\WebBundle\Entity\Line)
[5]: object(Proxies_CG_\Zuo\WebBundle\Entity\Line)
tell me if you need more information. thanks
|
|
This may be because
in proxies ( https://github.com/doctrine/doctrine2/blob/42e83a2716d19eada4f1cd49ece77d5f5229a239/lib/Doctrine/ORM/Proxy/ProxyFactory.php#L383 ) is not necessarily composed by integers. This could be fixed with DCOM-96. I'll add the tests to my development branch and will ping you back tomorrow
|
|
thanks
|
|
I see what is going on here... But this should not be a problem anyway, since they're bound anyway as "PDO::PARAM_INT", as Fabio B. Silva told you.
That's only a problem with the logger showing them as string. PDO will handle the conversion before the value hits the DB as far as I know.
|
|
I can understand your point,but what I don't really get is that the execute time of sql is very long,that explained the quote should be in the sql,not like what you said,that's only a problem with the logger.
|
|
scourgen can you profile the difference directly in CLI? What about checking the bound parameter type? Are those values bound as INTs in your case?
|
|
@ocramius I wish I could, but I was using doctrine2 with symfony2,So It looks like It will takes some time to simulating all environment and settings that could allow me to reproduced the problem.
but anyway,I will have a try and tell you what happen when I found something.
|
|
scourgen ok, awaiting your reply then
|
|
I've spent some time on playing with native doctrine2. It took me awhile to setup everything. but I just don't get that how to retrive data with its Proxy ojbect(for example Proxies_CG_\My\WebBundle\Entity\Line).
I mean the result of
$this->_em->getRepository("something")->findxxx()
always return an array of real object. I can't reproduced the situation(#comment-19186) that happens on symfony2+doctrine2.
anyway,I can make sure the problem is real exist,Because the execute time of that slow sql from the tool bar of symfony2 is same as I executed it at mysql cli. If the sql shows up on log with quote but running at mysql without quote,the execute time won't be same(actually It will be much more faster,in my case,20x times,from 2xxms to 10ms).
|
|
scourgen you can use
$em->getReference($className, $identifier)
(identifier being a key=>value array) to force proxies.
Give it a try
|
|
looks like I reproduced it.
public function testIssue()
{
$no_used= $this->_em->getRepository(__NAMESPACE__. '\DDC2214Line')->findOneById(1);
$lines=array(
$this->_em->getReference(__NAMESPACE__. '\DDC2214Line',1),
$this->_em->getReference(__NAMESPACE__. '\DDC2214Line','2'),
$this->_em->getReference(__NAMESPACE__. '\DDC2214Line',3),
);
$logger = $this->_em->getConnection()->getConfiguration()->getSQLLogger();
$ids = array_map(function($r){
return $r->id;
}, $this->relatedList);
$related = $this->_em->createQuery('select lr from '.__NAMESPACE__ . '\DDC2214LineRelated lr where lr.id in (:ids)')->setParameter('ids',$lines)->getResult();
$query = end($logger->queries);
$this->assertCount(3, $related);
$this->assertEquals($ids, $query['params'][0]);
$this->assertEquals(\Doctrine\DBAL\Connection::PARAM_INT_ARRAY, $query['types'][0]);
}
}
I use MySql Query log to see what's really happen in database(http://dev.mysql.com/doc/refman/5.5/en/query-log.html)
this is the log from table mysql.general_log
2013-01-08 12:23:44 [root] @ localhost [127.0.0.1] 59 0 Connect root@localhost on doctrine_tests
2013-01-08 12:23:44 root[root] @ localhost [127.0.0.1] 59 0 Query CREATE TABLE DDC2214Line (id INT AUTO_INCREMENT NOT NULL, PRIMARY KEY(id)) DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci ENGINE = InnoDB
2013-01-08 12:23:44 root[root] @ localhost [127.0.0.1] 59 0 Query CREATE TABLE DDC2214LineRelated (id INT AUTO_INCREMENT NOT NULL, line_id INT NOT NULL, count SMALLINT NOT NULL, line_id_related INT NOT NULL, INDEX IDX_D31307994D7B7542 (line_id), PRIMARY KEY(id)) DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci ENGINE = InnoDB
2013-01-08 12:23:44 root[root] @ localhost [127.0.0.1] 59 0 Query ALTER TABLE DDC2214LineRelated ADD CONSTRAINT FK_D31307994D7B7542 FOREIGN KEY (line_id) REFERENCES DDC2214Line (id)
2013-01-08 12:23:44 root[root] @ localhost [127.0.0.1] 59 0 Query START TRANSACTION
2013-01-08 12:23:44 root[root] @ localhost [127.0.0.1] 59 0 Query INSERT INTO DDC2214Line (id) VALUES (null)
2013-01-08 12:23:44 root[root] @ localhost [127.0.0.1] 59 0 Query INSERT INTO DDC2214Line (id) VALUES (null)
2013-01-08 12:23:44 root[root] @ localhost [127.0.0.1] 59 0 Query INSERT INTO DDC2214Line (id) VALUES (null)
2013-01-08 12:23:44 root[root] @ localhost [127.0.0.1] 59 0 Query INSERT INTO DDC2214LineRelated (count, line_id_related, line_id) VALUES (1, 1, 1)
2013-01-08 12:23:44 root[root] @ localhost [127.0.0.1] 59 0 Query INSERT INTO DDC2214LineRelated (count, line_id_related, line_id) VALUES (2, 2, 2)
2013-01-08 12:23:44 root[root] @ localhost [127.0.0.1] 59 0 Query INSERT INTO DDC2214LineRelated (count, line_id_related, line_id) VALUES (3, 3, 3)
2013-01-08 12:23:44 root[root] @ localhost [127.0.0.1] 59 0 Query COMMIT
2013-01-08 12:23:44 root[root] @ localhost [127.0.0.1] 59 0 Query SELECT t0.id AS id1 FROM DDC2214Line t0 WHERE t0.id = 1 LIMIT 1
2013-01-08 12:23:44 root[root] @ localhost [127.0.0.1] 59 0 Query SELECT d0_.id AS id0, d0_.count AS count1, d0_.line_id_related AS line_id_related2, d0_.line_id AS line_id3 FROM DDC2214LineRelated d0_ WHERE d0_.id IN (1, '2', 3)
2013-01-08 12:23:44 root[root] @ localhost [127.0.0.1] 59 0 Quit
you can see,in database level,the second parameter of last query but two has quote ( (1, '2', 3) )
|
|
A related Github Pull-Request [GH-247] was opened
https://github.com/doctrine/common/pull/247
|
|
A related Github Pull-Request [GH-247] was closed
https://github.com/doctrine/common/pull/247
|
|
error log from orm:validate-schema
|
|
Copying from pastebin:
use \Doctrine\ORM\Mapping as ORM;
use \xxx\Doctrine\Annotation\Entity as re;
use \xxx\Doctrine\Annotation\Forms as rf;
use \Doctrine\Common\Collections;
/**
* @ORM\Entity
*/
class EventPicture extends \Picture
{
/**
* @ORM\ManyToOne(targetEntity="Event", inversedBy="eventPicture")
* @ORM\JoinColumn(name="FK_Event", referencedColumnName="id")
*/
protected $event;
}
use \Doctrine\ORM\Mapping as ORM;
use \xxx\Doctrine\Annotation\Entity as re;
use \xxx\Doctrine\Annotation\Forms as rf;
use \Doctrine\Common\Collections;
/** @ORM\MappedSuperclass */
class Picture extends \xxx\Doctrine\Entity\BaseEntity
{
/**
* @ORM\Id
* @ORM\Column(type="integer")
* @ORM\GeneratedValue(strategy="IDENTITY")
* @var type
*/
protected $id;
/**
* @ORM\Column(type="string",unique=true, nullable=false)
* @rf\FileUpload(fileSize="php",uploadType="local",fieldName="link",formControl="FileUploadField",image=true)
*
*/
protected $link;
}
kluk does this happen also with any other simple custom annotation? For example following:
/**
* @Annotation
* @Target({"PROPERTY","ANNOTATION"})
*/
final class Entity implements Annotation
{
/**
* @var string
*/
public $value;
}
|
|
the same error when using simple annotation.
<?php
use \Doctrine\ORM\Mapping as ORM;
use \xxx\Doctrine\Annotation\Entity as re;
use \xxx\Doctrine\Annotation\Forms as rf;
use \Doctrine\Common\Collections;
/** @ORM\MappedSuperclass */
class Picture extends \xxx\Doctrine\Entity\BaseEntity {
/**
* @ORM\Id
* @ORM\Column(type="integer")
* @ORM\GeneratedValue(strategy="IDENTITY")
* @var type
*/
protected $id;
/**
* @ORM\Column(type="integer")
* @rf\SetClass({"class","hide"})
*/
public $value;
/**
* @ORM\Column(type="string",unique=true, nullable=true)
* @rf\FileUpload(fileSize="php",uploadType="local",fieldName="link",formControl="FileUploadField",image=true)
*
*/
protected $link;
}
When i remove $value , $picture from class everything goes ok.
Easy fix for me is change ClassMetadataInfo.
/**
* INTERNAL:
* Adds a field mapping without completing/validating it.
* This is mainly used to add inherited field mappings to derived classes.
*
* @param array $fieldMapping
*
* @return void
*/
public function addInheritedFieldMapping(array $fieldMapping)
{
if(isset($fieldMapping['fieldName'])){
$this->fieldMappings[$fieldMapping['fieldName']] = $fieldMapping;
$this->columnNames[$fieldMapping['fieldName']] = $fieldMapping['columnName'];
$this->fieldNames[$fieldMapping['columnName']] = $fieldMapping['fieldName'];
}
}
But i dont know if this fix can break another part of doctrine.
|
|
Can you put the code of your annotations online? I can't seem to understand why this happens.
|
Unable to find source-code formatter for language: php. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml
namespace libs\Doctrine\Annotation\Entity;
use Doctrine\Common\Annotations\Annotation;
/** @Annotation */
class CustomMapping extends Annotation
{
/**
*
* @var string
*/
public $className;
/**
*
*
* @var IQueryable| string
*/
public $dataSource;
}
|
|
Not a blocker
|
|
What's the result of `EXPLAIN` on a query without the subquery?
|
|
explain without the subquery
|
|
Sergey Gerdel that's not the same query.
|
|
Sergey Gerdel this is still using
Using index; Using temporary; Using filesort
Check your indexes
|
|
Not in the index problem
SELECT DISTINCT id0 FROM (SELECT m0_.id AS id0, m0_.title AS title1, m0_.text AS text2, m0_.price AS price3, m0_.originalPrice AS originalPrice4, m0_.condition_type AS condition_type5, m0_.image_1 AS image_16, m0_.image_2 AS image_27, m0_.image_3 AS image_38, m0_.image_4 AS image_49, m0_.image_5 AS image_510, m0_.video AS video11, m0_.contact_email AS contact_email12, m0_.contact_name AS contact_name13, m0_.contact_phone AS contact_phone14, m0_.contact_type AS contact_type15, m0_.published AS published16, m0_.type AS type17, m0_.status AS status18, m0_.highlight AS highlight19, m0_.urgent AS urgent20, m0_.topads AS topads21, m0_.period AS period22, m0_.hits AS hits23, m0_.ip AS ip24, m0_.created_at AS created_at25, m0_.updated_at AS updated_at26 FROM milla_message m0_ WHERE m0_.status = 1 ORDER BY m0_.published DESC) dctrn_result LIMIT 20 OFFSET 0
Time: 104.614s explain 3
SELECT DISTINCT m0_.id AS id0 FROM milla_message m0_ WHERE m0_.status = 1 ORDER BY m0_.published DESC LIMIT 20 OFFSET 0;
Time: 0.001s explain 4
|
|
Sergey Gerdel the ORM cannot simplify a complex query that way. There may be a conditional on one of the joined results, or generally usage of one of the joined results.
Things that could be optimized here are:
- Removal of the `ORDER BY` clause when grouping (check ORM master, I think somebody already did that)
- Trying to simplify the query by doing some serious hacking on the AST.
The problem I see here is that the chance to spawn random bugs because of the optimization is very high, and you'd have to rewrite `walkSelectStatement`
|
|
Marking as improvement
|
|
Minor? 
i have 100 sec for this query.
200k items are selected for temporary table. wtf?
OK. Programmers may be mistaken in parser
expect ORDER BY m0_.published DESC LIMIT 20 OFFSET 0) dctrn_result
Time: 0.001s
reality ORDER BY m0_.published DESC) dctrn_result LIMIT 20 OFFSET 0
|
|
Sergey Gerdel this problem does not introduce security issues and can be worked around by you while using your own pagination logic. It does not stop you from doing anything, that's why it's minor.
|
|
ok)
i have already created my own paginator.
at last
please see how to fix this problem
https://github.com/Sergic/doctrine2/commit/2733c815387273d3bd199a68acb717e0cbc8ccfe
|
DDC2214Test.php
error.log
EXPLAIN.htmL