My rought estimate is and has always been about 100 bytes per cache
object (may be a little high, but your memory usage is very close to
this).
I would recomment to start by decreasing the cache_dir sizes. Good news
is that there is ongoing development works that will limit the overhead
of Squid, cutting down the difference between accounted ant total
allocated.
The "memory utilization" page should give you a quite good hint on where
memory are allocated.
Also, I'd recommend to upgrade to a recent Squid-2.4 snapshot. There was
a couple of ugly memory related bugfixes that never made it into
Squid-2.3 (and one of these even got broken again before 2.4.STABLE1 was
released, but is fixed again in the current snapshots).
-- Henrik Nordstrom Squid Hacker Damien McIntosh wrote: > > To the list, > > I have some concerns with the amount of memory my squid process is using on one of the caching servers that I manage. > > Version is 2.3.STABLE4 > running on a Mandrake Linux 7.2 on COMPAQ DL380 hardware, 1Gb RAM 1.5Gb swap. Reiserfs is used for the two cache dirs. > > Here is a cut down version of the current config file: > > cache_mem 10485760 bytes > cache_swap_low 80 > cache_swap_high 82 > maximum_object_size 8388608 bytes > minimum_object_size 0 bytes > ipcache_size 1024 > ipcache_low 90 > ipcache_high 95 > fqdncache_size 1024 > cache_dir ufs /var/spool/squid/cache1 33000 77 256 > cache_dir ufs /var/spool/squid/cache2 33000 77 256 > memory_pools on > memory_pools_limit 52428800 bytes > replacement_policy GDSF > reference_age 604800 seconds > > Store Directory Statistics: > Store Entries : 8501395 > Maximum Swap Size : 67584000 KB > Current Store Swap Size: 54066291 KB > Current Capacity : 80% used, 20% free > > Store Directory #0: /var/spool/squid/cache1 > First level subdirectories: 77 > Second level subdirectories: 256 > Maximum Size: 33792000 KB > Current Size: 27039639 KB > Percent Used: 80.02% > Filemap bits in use: 2905712 of 4194304 (69%) > Filesystem Space in use: 35243136/35551996 KB (99%) > Filesystem Inodes in use: 0/-1 (0%) > Flags: > > Store Directory #1: /var/spool/squid/cache2 > First level subdirectories: 77 > Second level subdirectories: 256 > Maximum Size: 33792000 KB > Current Size: 27026652 KB > Percent Used: 79.98% > Filemap bits in use: 2900647 of 4194304 (69%) > Filesystem Space in use: 34979420/35551996 KB (98%) > Filesystem Inodes in use: 0/-1 (0%) > Flags: SELECTED > > >From the File Descriptors page in cachemgr.cgi: > Usually about 127 filedescriptors are open. > > >From the mem utilisation page from cachemgr.cgi the following is provided: > Cumulative allocated volume: 88.95 GB > Current overhead: 174007 bytes (0.031%) > Idle pool limit: 50.00 MB > > Two directories with about 35Gb in each. Currently the cache is 85% full. > Roughly about 55Gb stored in the cache. > > The result returned from top is > > squid SIZE = 908M RSS= 908M SWAP = 259M %CPU= 5.9 %MEM = 88.7 > > Here are some results returned from cachemgr.cgi: > > Connection information for squid: > Number of clients accessing cache: 3 > Number of HTTP requests received: 1162624 > Number of ICP messages received: 0 > Number of ICP messages sent: 0 > Number of queued ICP replies: 0 > Request failure ratio: 0.00% > HTTP requests per minute: 732.8 > ICP messages per minute: 0.0 > Select loop called: 16456617 times, 5.784 ms avg > Cache information for squid: > Request Hit Ratios: 5min: 29.2%, 60min: 31.3% > Byte Hit Ratios: 5min: 10.0%, 60min: 9.1% > Storage Swap size: 54065426 KB > Storage Mem size: 10232 KB > Storage Replacement Threshold: -1.000000 > Mean Object Size: 9.31 KB > Requests given to unlinkd: 0 > Median Service Times (seconds) 5 min 60 min: > HTTP Requests (All): 0.52331 0.52331 > Cache Misses: 0.61549 0.64968 > Cache Hits: 0.02190 0.00919 > Near Hits: 0.44492 0.49576 > Not-Modified Replies: 0.01469 0.01387 > DNS Lookups: 0.04237 0.05078 > ICP Queries: 0.00000 0.00000 > Resource usage for squid: > UP Time: 95191.283 seconds > CPU Time: 6052.090 seconds > CPU Usage: 6.36% > CPU Usage, 5 minute avg: 12.94% > CPU Usage, 60 minute avg: 12.44% > Maximum Resident Size: 0 KB > Page faults with physical i/o: 1199290 > Memory usage for squid via mallinfo(): > Total space in arena: 897457 KB > Ordinary blocks: 888300 KB 45964 blks > Small blocks: 0 KB 0 blks > Holding blocks: 35936 KB 5 blks > Free Small blocks: 0 KB > Free Ordinary blocks: 9156 KB > Total in use: 924236 KB 103% > Total free: 9156 KB 1% > Memory accounted for: > Total accounted: 545391 KB > File descriptor usage for squid: > Maximum number of file descriptors: 1024 > Largest file desc currently in use: 157 > Number of file desc currently in use: 120 > Files queued for open: 0 > Available number of file descriptors: 904 > Reserved number of file descriptors: 100 > Store Disk files open: 12 > Internal Data Structures: > 8499537 StoreEntries > 2328 StoreEntries with MemObjects > 2286 Hot Object Cache Items > 5804570 Filemap bits set > 5804554 on-disk objects > > Obviously there are some major differences between the "total in use" memory and the "accounted for" memory! > As you can see, 734 http requests per min is quite busy. > > Twice I've had the machine run out of memory to the point where I couldn't even login. No memory to fork the login shell. Cold reset was required. > > So can someone PLEASE point me in the right direction for a solution to this problem. Do I reduce the cache_dir sizes? Install more RAM? Upgrade to 2.4.STABLE1? > > Any solution that you can think of would be great! > > I know that the cache_mem parameter is not going to limit mem usage as squid uses memory for other things however I need some solution to limit the memory usage of the squid process. > > Is my cache_dir just to large for the machine? > > This machine is only a squid caching machine. No other services are on the machine, and it is setup as the main caching proxy. Other internal proxies are chained to the squid and then the squid is chained to our ISP proxy. > > Performance has been great, until now. Once the squid process starts using more swap the machine starts to bog down in swapping and the internal proxies can't connect to the squid or the performance becomes pathetic. > > Any help would be greatly appreciated. > If you require further info then please let me know. > > Thanks > Damien > > ------------------------------------------------------------- > Damien McIntosh > Internet Systems Security Administrator > Brisbane City Council > Phone: +61 7 3403 4950 > Email: cioc3@brisbane.qld.gov.au > -------------------------------------------------------------- > > This mail has passed through an insecure network. > All enquires should be directed to the message author.Received on Wed May 16 2001 - 16:09:35 MDT
This archive was generated by hypermail pre-2.1.9 : Tue Dec 09 2003 - 17:00:09 MST