Re: [squid-users] Number of Objects Inside aufs Cache Directory

Hi Amos, Thanks for your reply.  I've read the "few hundreds" thing in "Squid: The Definitive Guide" by Duane Wessels and I think his recommendation is related to performance of squid more than to file system constraints. I quoted the following from the book:  "Some people think that Squid performs better, or worse, depending on the particular values for L1 and L2. It seems to make sense, intuitively, that small directories can be searched faster than large ones. Thus, L1 and L2 should probably be large enough so that each L2 directory has no more than a few hundred files." Best regards, Firas ________________________________ From: Amos Jeffries <squid3_at_treenet.co.nz> To: squid-users_at_squid-cache.org Sent: Monday, September 2, 2013 3:33 AM Subject: Re: [squid-users] Number of Objects Inside  aufs Cache Directory On 2/09/2013 8:40 a.m., Golden Shadow wrote: > Hello there! > > I've read that the number of first level and second level aufs subdirectories should be selected so that the number of objects inside each second level subdirectory should be no more than few hundreds. Not sure where that came from. "few hundreds" sounds like advice for working with FAT-16 formatted disks - or worse. Most of the modern filesystems can handle several thousands easily. The real reason for these parameters is that some filesystems start producing errors. For example; Squid tores 2^24 objects in its cache_dir but - FAT16 fails with more than 2^14 files in one directory - and IIRC ext2 and/or ext3 starts giving me trouble around 2^16 files in one directory. I'm not sure about other OS, I've not hit their limits myself. > In one of the subdirectories of the following cache_dir on my 3.3.8 squid, there is more than 15000 objects! In other subdirectories, there is ZERO objects, is this normal?! Yes. It depends entirely on how many objects are in the cache. The earlier fileno entries fill up first, so the directories where those fileo map to will show lots of files while later ones do not. NOTE: when changing these L1/L2 values the entire cache_dir fileno->file mapping gets screwed up. So you need to erase the contents and use an empty location/directory to build the new structure shape inside. > cache_dir aufs /mnt/cachedrive2/small 250000 64 256 min-size=32000 max-size=200000 If you want to increase that spread you can change the 64 to 128. It should halve the number of files in the fullest directory. With 250GB storing 32KB-200KB sized objects you are looking at a total of between 1,310,720 and 8,192,000 objects in that particular cache. Amos
Received on Mon Sep 02 2013 - 21:45:32 MDT