[DCOM-234] Cache file fails on Windows - name too long Created: 19/Feb/14  Updated: 19/Feb/14

Status: Open
Project: Doctrine Common
Component/s: Caching
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Critical
Reporter: Matt Durak Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Windows 8.1 x64
Symfony CMF with Doctrine caching



 Description   

The filename component of the FileCache is unbounded. It is possible to attempt to create files that are too long for the Windows filesystem. This causes an exception.

Example from a Symfony CMF/PHPCR project:

file_put_contents(D:\web_workspace\study\app\cache\dev\doctrine\cache\71d059fe39968db8\3e485dbb27419897\dfb24337887ff42c\a7b85c70bfefa781\nodes[nodes weak references cmsprogrampagesThe Summer ProgramThe Calendar 5 weeks for all Europe, routeContent, default][49].doctrinecache.data): failed to open stream: Invalid argument in D:\web_workspace\study\vendor\doctrine\cache\lib\Doctrine\Common\Cache\FilesystemCache.php line 111

In this case, the ID was
"nodes[nodes weak references cmsprogrampagesThe Summer ProgramThe Calendar 5 weeks for all Europe, routeContent, default][49].doctrinecache.data"

That refers to a phpcr node:
cms/program/pages/The Summer Program/The Calendar 5 weeks for all Europe

Attempting to manually create that file in windows explorer shows that it is trunacted to
"nodes[nodes weak references cmsprogrampagesThe Summer ProgramThe Calendar 5 weeks for all Europe, routeContent, default][49].do" (128 characters)

Shouldn't the ID be truncated so this doesn't happen? Should this be fixed here or in the phpcr that is creating the long ID to begin with?






[DCOM-245] doContains method in MemcacheCache cache provider class returning wrong values Created: 27/Jun/14  Updated: 27/Jun/14

Status: Open
Project: Doctrine Common
Component/s: Caching
Affects Version/s: 2.2
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Javier Mellado Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: orm
Environment:

dev, test and prod



 Description   

Method doContains in MemcacheCache cache provider class looks like this


    /**
     * {@inheritdoc}
     */
    protected function doContains($id)
    {
        return (bool) $this->memcache->get($id);
    }

In the case of an empty array stored, the result of casting an empty array is false but actually there's a value in cache with they key $id. As a matter of fact, when you take a look at MemcachedCache cache provider class, the doContains method looks like this:

    /**
     * {@inheritdoc}
     */
    protected function doContains($id)
    {
        return (false !== $this->memcached->get($id));
    }

Which is more accurate in terms of the existance of a value in cache for $id. I had to have a workaround with the above code to make it work with Memcache.

Is there any reason it is like that in MemcacheCache cache provider class and not in MemcachedCache cache provider class?



 Comments   
Comment by Marco Pivetta [ 27/Jun/14 ]

Javier Mellado can you please open a PR for this? The fix seems trivial, but it just needs a test in order to merge.

Comment by Javier Mellado [ 27/Jun/14 ]

Excuse me Marco, what is a PR? I'll be more than happy to do so.

Comment by Marco Pivetta [ 27/Jun/14 ]

Javier Mellado pull request on github. If you don't then it may just be fixed in future as soon as someone picks it.





[DCOM-238] FileCache does not remove directories on flush Created: 21/Mar/14  Updated: 21/Mar/14

Status: Open
Project: Doctrine Common
Component/s: Caching
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: Ryan Korczykowski Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Windows 7



 Description   

FileCache::doFlush() does not remove the directories that store the cache file. The cache file is correctly deleted. This results in hundreds of hanging directories when the cache is flushed / rebuilt.

The version of doctrine/cache is 1.3.0.



 Comments   
Comment by Marco Pivetta [ 21/Mar/14 ]

This is likely because of the iterator skipping directories:

https://github.com/doctrine/cache/blob/36c4eee5051629524389da376ba270f15765e49f/lib/Doctrine/Common/Cache/FileCache.php#L120-L122





[DCOM-130] Paths in Doctrine\Common\Cache\FileCache could create large directory indexes Created: 23/Oct/12  Updated: 05/Jul/13

Status: Open
Project: Doctrine Common
Component/s: Caching
Affects Version/s: 2.3
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: R Churchill Assignee: Benjamin Eberlei
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Any



 Description   

The way paths are created within FileCache currently, there is a theoretical maximum of 16^12 directories in the cache directory, which is quite a large number. Usually schemes like this are used to restrict the number of files in one directory.

Comparing with git, for example, the dirs are arranged

00/
1c/
...
ff/

and then the object store within those directories, which is a lot more manageable, say if you happen to type ls in the cache directory, you will get a maximum listing of 256 dirs. PhpThumb does something similar when caching images.

How about something like this for getFilename():

$idHash = md5($id);
$path = substr($idHash, 0, 2) . DIRECTORY_SEPARATOR . substr($idHash, 2, 2) . DIRECTORY_SEPARATOR . substr($idHash, 4);
$path = $this->directory . DIRECTORY_SEPARATOR . $path;

return $path . $id . $this->extension;

Not nearly so elegant, but I think this has better properties for the file system. Also I would be tempted to use one of the sha family hashes and not to include the $id within the filename, but perhaps this is helpful for debugging?



 Comments   
Comment by Julian Higman [ 10/May/13 ]

We hit this problem in a live system - with a lot of cached items, the number of subdirectories that FileCache creates can exceed the number that an ext3 filesystem allows in a single directory (about 32000).

After that, an attempt to cache a new item can get an error like this:

mkdir() [function.mkdir]: Too many links

Our solution was similar to that suggested:


    protected function getFilename($id) {
        $path = implode(str_split(md5($id), 2), DIRECTORY_SEPARATOR);
        $path = $this->directory . DIRECTORY_SEPARATOR . $path;
        return $path . DIRECTORY_SEPARATOR . $id . $this->extension;
    }

It splits the md5 of the item id into parts of length 2, rather than the original 12. This creates a deeply nested structure, but which won't ever exceed the limit on number of subdirectories in any one directory. It's the same subdirectory pattern used by default by Apache mod_disk_cache, as well.

Comment by Julian Higman [ 05/Jul/13 ]

After a couple of months in production, we ran into another problem with this - we reached the maximum number of inodes in the fielsystem.

The resulting errors look like this:

mkdir() [function.mkdir]: No space left on device

There is actually disk space left, but looking at the inodes shows that the limit has been hit:

-bash-3.2# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 6553600 6553600 0 100% /

The creation of directories and subdirectories can be constrained slightly by using 3 instead of 2 characters (with hex chars, that will give max of 16^3 = 4096 subdirectories per directory, still less than the ext3 limit of 32000)

$path = implode(str_split(md5($id), 2), DIRECTORY_SEPARATOR);

but ultimately the inodes will still all be used up.

The only other options are pruning the cache at intervals, or switching to a different caching strategy altogether.

Comment by Marco Pivetta [ 05/Jul/13 ]

Julian Higman I'd suggest file-based caching mechanisms are not suited for that environment. The file cache is really meant for all those environments where there's strict constraints (like shared hosting).





Generated at Sat Oct 25 20:46:07 UTC 2014 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.