Doctrine 2 - ORM
  1. Doctrine 2 - ORM
  2. DDC-1682

EntityManager::clear() not working as expected.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Incomplete
    • Affects Version/s: 2.1.6
    • Fix Version/s: None
    • Component/s: ORM
    • Security Level: All
    • Labels:
      None
    • Environment:
      Ubuntu 11.10
      Symfony 2

      Description

      I've been reading Doctrine2 Batch Processing documentation.
      I've a simplified the code and made a sample where I'm using a Repository inside the loop:

       
      for ($i=1; $i<=10000; ++$i) {
           $user = $this->_em->getRepository('some user class')->find($i);
           $this->_em->clear();
      
           //Clear variables to ensure garbage collections     
           unset($user);
           $user = null;
      
      }
      

      I expect that this script will consume some constant memory in all iterations, but what happens is that every iteration raises memory consumption (more iterations, more memory).... i think that the clear method has some sort of memory leak.

      In my production environment (with complex script), i reach a memory limit exception even with 600MB limit... but if I clear the EntityManager on every iteration, shouldn't memory be freed?

      Sorry for my bad english.

        Activity

        Hide
        Benjamin Eberlei added a comment -

        You are probably running symfony2 in debug mode? Is the SQL logger enabled? This is probably not a Doctrine problem but something in your code / Symfony that keeps increasing the memory.

        Show
        Benjamin Eberlei added a comment - You are probably running symfony2 in debug mode? Is the SQL logger enabled? This is probably not a Doctrine problem but something in your code / Symfony that keeps increasing the memory.
        Hide
        German Caseres added a comment -

        I've executed the script in debug and prod mode, but I had the same problem in both modes.
        I don't think it's a Symfony problem because I had measured memory consumption only before, after and inside the for loop (no symfony methods involved).
        About my code, I'm using simple clases with no business code, only simple Doctrine mappings (and standard repository).
        Have you tested a similar code? I don't understand why memory consumption continues raising if I'm "destroying" the objects.
        I tried with gc_enable and gc_collect_cycles but no success... every iteration increases memory consumption like if the previous loaded objects weren't destroyed... maybe the repository is instancing other objects in every find call that are not destroyed?

        Show
        German Caseres added a comment - I've executed the script in debug and prod mode, but I had the same problem in both modes. I don't think it's a Symfony problem because I had measured memory consumption only before, after and inside the for loop (no symfony methods involved). About my code, I'm using simple clases with no business code, only simple Doctrine mappings (and standard repository). Have you tested a similar code? I don't understand why memory consumption continues raising if I'm "destroying" the objects. I tried with gc_enable and gc_collect_cycles but no success... every iteration increases memory consumption like if the previous loaded objects weren't destroyed... maybe the repository is instancing other objects in every find call that are not destroyed?
        Hide
        Benjamin Eberlei added a comment -

        are you using lifecycle listeners? access global state or something?

        Show
        Benjamin Eberlei added a comment - are you using lifecycle listeners? access global state or something?
        Hide
        Benjamin Eberlei added a comment -

        Can you generate an xdebug trace for some of the $i's ? say 100 and 1000 with xdebug_start_trace("/tmp/loop".$i); and xdebug_stop_trace(); and upload them? Maybe you can compare yourself, where in the loop the memory increases and if clear even empties it or not.

        Show
        Benjamin Eberlei added a comment - Can you generate an xdebug trace for some of the $i's ? say 100 and 1000 with xdebug_start_trace("/tmp/loop".$i); and xdebug_stop_trace(); and upload them? Maybe you can compare yourself, where in the loop the memory increases and if clear even empties it or not.
        Hide
        Marco Pivetta added a comment -

        Any news about this one? There's been more than one case where the Symfony data collector (for debug) caused problems like this one... Imo this is not a ORM issue.

        Show
        Marco Pivetta added a comment - Any news about this one? There's been more than one case where the Symfony data collector (for debug) caused problems like this one... Imo this is not a ORM issue.
        Hide
        Benjamin Eberlei added a comment -

        No feedback given.

        Show
        Benjamin Eberlei added a comment - No feedback given.
        Hide
        Miha Vrhovnik added a comment -

        I've been debugging a similar issue today. And Yes, the culprit is the Symfony's data collector. Running the command with --no-debug worked like a charm.

        Show
        Miha Vrhovnik added a comment - I've been debugging a similar issue today. And Yes, the culprit is the Symfony's data collector. Running the command with --no-debug worked like a charm.
        Hide
        mathias dusautoy added a comment -

        same issue here with 2.2.3, php 5.4 & symfony 2.1
        have a symfony command running as deamon with --no-debug and no listeners

        while(true)

        { $q = $this->em->createQueryBuilder()->select()...->getQuery(); $results = $q->getResult(AbstractQuery::HYDRATE_ARRAY); // commenting this line resolve the memory leak $this->em->clear(); gc_collect_cycles(); // with or without does not change the issue }

        the consecutive traces shows that memory does not reduce after clear()

        Show
        mathias dusautoy added a comment - same issue here with 2.2.3, php 5.4 & symfony 2.1 have a symfony command running as deamon with --no-debug and no listeners while(true) { $q = $this->em->createQueryBuilder()->select()...->getQuery(); $results = $q->getResult(AbstractQuery::HYDRATE_ARRAY); // commenting this line resolve the memory leak $this->em->clear(); gc_collect_cycles(); // with or without does not change the issue } the consecutive traces shows that memory does not reduce after clear()
        Hide
        Marco Pivetta added a comment -

        mathias dusautoy please check this in insulation (without Symfony2 if possible)

        Show
        Marco Pivetta added a comment - mathias dusautoy please check this in insulation (without Symfony2 if possible)
        Hide
        Benjamin Eberlei added a comment -

        this may be array hydrator related, not sure that may not cause problems.

        Show
        Benjamin Eberlei added a comment - this may be array hydrator related, not sure that may not cause problems.
        Hide
        mathias dusautoy added a comment - - edited

        without symfony:

        <?php
        
        use Doctrine\ORM\Tools\Setup;
        use Doctrine\ORM\EntityManager;
        
        $loader = require_once __DIR__.'/../app/autoload.php';
        $loader->add('Acme\\CoreBundle', __DIR__.'/../src/Acme/CoreBundle/');
        
        $isDevMode = true;
        $config = Setup::createAnnotationMetadataConfiguration(array(__DIR__."/../src/Acme/CoreBundle/Entity"), $isDevMode, null, null, false);
        
        $conn = array(
        	'driver' => 'pdo_mysql',
        	'host' => 'localhost',
        	'dbname' => 'dbname',
        	'user' => 'root',
        	'password' => ''
        );
        
        $em = EntityManager::create($conn, $config);
        
        $d = new \DateTime();
        
        while(true) {
        	echo memory_get_usage() . PHP_EOL;
        	$qb = $em->createQueryBuilder()
        		->select('c')
        		->from('Acme\\CoreBundle\\Entity\\Consultation', 'c')
        		->where('c.date > :date')->setParameter(':date', $d)
        		->orderBy('c.date', 'ASC');
        
        	$q = $qb->getQuery();
        
        	$results = $q->getResult();
        
        	foreach($results as $c) {
        		echo $c->getDate()->format('H:i:s') . PHP_EOL;
        	}
        	
        	$q->free();
        
        	$em->clear();
        
        	gc_collect_cycles();
        	
        	echo memory_get_usage() . PHP_EOL . PHP_EOL;
        }
        

        output:

        7978568
        7978568
        
        7978568
        7978568
        
        7978568
        11:51:27
        11474520
        
        11474520
        11473368
        
        11473368
        11473368
        
        11473368
        11473368
        
        11473368
        11473368
        

        Am I missing something?

        Show
        mathias dusautoy added a comment - - edited without symfony: <?php use Doctrine\ORM\Tools\Setup; use Doctrine\ORM\EntityManager; $loader = require_once __DIR__.'/../app/autoload.php'; $loader->add('Acme\\CoreBundle', __DIR__.'/../src/Acme/CoreBundle/'); $isDevMode = true ; $config = Setup::createAnnotationMetadataConfiguration(array(__DIR__. "/../src/Acme/CoreBundle/Entity" ), $isDevMode, null , null , false ); $conn = array( 'driver' => 'pdo_mysql', 'host' => 'localhost', 'dbname' => 'dbname', 'user' => 'root', 'password' => '' ); $em = EntityManager::create($conn, $config); $d = new \DateTime(); while ( true ) { echo memory_get_usage() . PHP_EOL; $qb = $em->createQueryBuilder() ->select('c') ->from('Acme\\CoreBundle\\Entity\\Consultation', 'c') ->where('c.date > :date')->setParameter(':date', $d) ->orderBy('c.date', 'ASC'); $q = $qb->getQuery(); $results = $q->getResult(); foreach($results as $c) { echo $c->getDate()->format('H:i:s') . PHP_EOL; } $q->free(); $em->clear(); gc_collect_cycles(); echo memory_get_usage() . PHP_EOL . PHP_EOL; } output: 7978568 7978568 7978568 7978568 7978568 11:51:27 11474520 11474520 11473368 11473368 11473368 11473368 11473368 11473368 11473368 Am I missing something?
        Hide
        Marco Pivetta added a comment -

        Memory usage here seems quite constant (the change from 7978568 to 11474520 may well be because of metadata and hydrators). The output doesn't seem to be conforming your snippet though.

        Show
        Marco Pivetta added a comment - Memory usage here seems quite constant (the change from 7978568 to 11474520 may well be because of metadata and hydrators). The output doesn't seem to be conforming your snippet though.
        Hide
        mathias dusautoy added a comment - - edited

        yes sorry the above output is for:

        foreach($results as $c) {
            echo $c->getDate()->format('H:i:s') . PHP_EOL;
            $d = $c->getDate();
        }
        

        with

        foreach($results as $c) {
            echo $c->getDate()->format('H:i:s') . PHP_EOL;
        }
        

        the output is:

        3489864
        12:22:27
        13502680
        
        13502680
        12:22:27
        13515496
        
        13515496
        12:22:27
        13528328
        
        13528328
        12:22:27
        13541144
        
        13541144
        12:22:27
        13553976
        
        ....
        
        
        74513560
        12:22:27
        74526520
        
        74526520
        12:22:27
        74539520
        
        74539520
        12:22:27
        74552560
        

        and goes on

        Show
        mathias dusautoy added a comment - - edited yes sorry the above output is for: foreach($results as $c) { echo $c->getDate()->format('H:i:s') . PHP_EOL; $d = $c->getDate(); } with foreach($results as $c) { echo $c->getDate()->format('H:i:s') . PHP_EOL; } the output is: 3489864 12:22:27 13502680 13502680 12:22:27 13515496 13515496 12:22:27 13528328 13528328 12:22:27 13541144 13541144 12:22:27 13553976 .... 74513560 12:22:27 74526520 74526520 12:22:27 74539520 74539520 12:22:27 74552560 and goes on
        Hide
        Christophe Coevoet added a comment -

        Do you have bidirectional relations in your user entity ? If yes, you will still have some references to the object after clearing the EntityManager (in the related object, itself reference by the user)

        Show
        Christophe Coevoet added a comment - Do you have bidirectional relations in your user entity ? If yes, you will still have some references to the object after clearing the EntityManager (in the related object, itself reference by the user)
        Hide
        Bruno Lemos added a comment - - edited

        Fixed it by doing the following:

        $this
        ->_em
        ->getConnection()
        ->getConfiguration()
        ->setSQLLogger(null);

        Show
        Bruno Lemos added a comment - - edited Fixed it by doing the following: $this ->_em ->getConnection() ->getConfiguration() ->setSQLLogger(null);

          People

          • Assignee:
            Benjamin Eberlei
            Reporter:
            German Caseres
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: