[DDC-1682] EntityManager::clear() not working as expected. Created: 05/Mar/12  Updated: 17/Apr/13  Resolved: 27/May/12

Status: Resolved
Project: Doctrine 2 - ORM
Component/s: ORM
Affects Version/s: 2.1.6
Fix Version/s: None
Security Level: All

Type: Bug Priority: Critical
Reporter: German Caseres Assignee: Benjamin Eberlei
Resolution: Incomplete Votes: 0
Labels: None
Environment:

Ubuntu 11.10
Symfony 2



 Description   

I've been reading Doctrine2 Batch Processing documentation.
I've a simplified the code and made a sample where I'm using a Repository inside the loop:

 
for ($i=1; $i<=10000; ++$i) {
     $user = $this->_em->getRepository('some user class')->find($i);
     $this->_em->clear();

     //Clear variables to ensure garbage collections     
     unset($user);
     $user = null;

}

I expect that this script will consume some constant memory in all iterations, but what happens is that every iteration raises memory consumption (more iterations, more memory).... i think that the clear method has some sort of memory leak.

In my production environment (with complex script), i reach a memory limit exception even with 600MB limit... but if I clear the EntityManager on every iteration, shouldn't memory be freed?

Sorry for my bad english.



 Comments   
Comment by Benjamin Eberlei [ 06/Mar/12 ]

You are probably running symfony2 in debug mode? Is the SQL logger enabled? This is probably not a Doctrine problem but something in your code / Symfony that keeps increasing the memory.

Comment by German Caseres [ 06/Mar/12 ]

I've executed the script in debug and prod mode, but I had the same problem in both modes.
I don't think it's a Symfony problem because I had measured memory consumption only before, after and inside the for loop (no symfony methods involved).
About my code, I'm using simple clases with no business code, only simple Doctrine mappings (and standard repository).
Have you tested a similar code? I don't understand why memory consumption continues raising if I'm "destroying" the objects.
I tried with gc_enable and gc_collect_cycles but no success... every iteration increases memory consumption like if the previous loaded objects weren't destroyed... maybe the repository is instancing other objects in every find call that are not destroyed?

Comment by Benjamin Eberlei [ 06/Mar/12 ]

are you using lifecycle listeners? access global state or something?

Comment by Benjamin Eberlei [ 11/Mar/12 ]

Can you generate an xdebug trace for some of the $i's ? say 100 and 1000 with xdebug_start_trace("/tmp/loop".$i); and xdebug_stop_trace(); and upload them? Maybe you can compare yourself, where in the loop the memory increases and if clear even empties it or not.

Comment by Marco Pivetta [ 01/Apr/12 ]

Any news about this one? There's been more than one case where the Symfony data collector (for debug) caused problems like this one... Imo this is not a ORM issue.

Comment by Benjamin Eberlei [ 27/May/12 ]

No feedback given.

Comment by Miha Vrhovnik [ 28/Aug/12 ]

I've been debugging a similar issue today. And Yes, the culprit is the Symfony's data collector. Running the command with --no-debug worked like a charm.

Comment by mathias dusautoy [ 15/Apr/13 ]

same issue here with 2.2.3, php 5.4 & symfony 2.1
have a symfony command running as deamon with --no-debug and no listeners

while(true)

{ $q = $this->em->createQueryBuilder()->select()...->getQuery(); $results = $q->getResult(AbstractQuery::HYDRATE_ARRAY); // commenting this line resolve the memory leak $this->em->clear(); gc_collect_cycles(); // with or without does not change the issue }

the consecutive traces shows that memory does not reduce after clear()

Comment by Marco Pivetta [ 15/Apr/13 ]

mathias dusautoy please check this in insulation (without Symfony2 if possible)

Comment by Benjamin Eberlei [ 15/Apr/13 ]

this may be array hydrator related, not sure that may not cause problems.

Comment by mathias dusautoy [ 16/Apr/13 ]

without symfony:

<?php

use Doctrine\ORM\Tools\Setup;
use Doctrine\ORM\EntityManager;

$loader = require_once __DIR__.'/../app/autoload.php';
$loader->add('Acme\\CoreBundle', __DIR__.'/../src/Acme/CoreBundle/');

$isDevMode = true;
$config = Setup::createAnnotationMetadataConfiguration(array(__DIR__."/../src/Acme/CoreBundle/Entity"), $isDevMode, null, null, false);

$conn = array(
	'driver' => 'pdo_mysql',
	'host' => 'localhost',
	'dbname' => 'dbname',
	'user' => 'root',
	'password' => ''
);

$em = EntityManager::create($conn, $config);

$d = new \DateTime();

while(true) {
	echo memory_get_usage() . PHP_EOL;
	$qb = $em->createQueryBuilder()
		->select('c')
		->from('Acme\\CoreBundle\\Entity\\Consultation', 'c')
		->where('c.date > :date')->setParameter(':date', $d)
		->orderBy('c.date', 'ASC');

	$q = $qb->getQuery();

	$results = $q->getResult();

	foreach($results as $c) {
		echo $c->getDate()->format('H:i:s') . PHP_EOL;
	}
	
	$q->free();

	$em->clear();

	gc_collect_cycles();
	
	echo memory_get_usage() . PHP_EOL . PHP_EOL;
}

output:

7978568
7978568

7978568
7978568

7978568
11:51:27
11474520

11474520
11473368

11473368
11473368

11473368
11473368

11473368
11473368

Am I missing something?

Comment by Marco Pivetta [ 16/Apr/13 ]

Memory usage here seems quite constant (the change from 7978568 to 11474520 may well be because of metadata and hydrators). The output doesn't seem to be conforming your snippet though.

Comment by mathias dusautoy [ 16/Apr/13 ]

yes sorry the above output is for:

foreach($results as $c) {
    echo $c->getDate()->format('H:i:s') . PHP_EOL;
    $d = $c->getDate();
}

with

foreach($results as $c) {
    echo $c->getDate()->format('H:i:s') . PHP_EOL;
}

the output is:

3489864
12:22:27
13502680

13502680
12:22:27
13515496

13515496
12:22:27
13528328

13528328
12:22:27
13541144

13541144
12:22:27
13553976

....


74513560
12:22:27
74526520

74526520
12:22:27
74539520

74539520
12:22:27
74552560

and goes on

Comment by Christophe Coevoet [ 17/Apr/13 ]

Do you have bidirectional relations in your user entity ? If yes, you will still have some references to the object after clearing the EntityManager (in the related object, itself reference by the user)

Generated at Thu Apr 17 07:03:09 UTC 2014 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.