Re: [squid-users] Squid uses way too much RAM and starts swapping ...

On 11/05/11 19:19, guest01 wrote:
> Hi,
>
> I am currently using squid 3.1.12 as forward-proxy without
> harddisk-caching (only RAM is used for caching). Each server is
> running on RHEL5.5 and is pretty strong (16 CPUs, 28GB RAM), but each
> servers starts swapping a few days after start. The workaround at the
> moment is to reboot the server once a week, which I don't really like.
> But swapping leads to serious side effects, e.g. performance troubles,
> ...
>
> way too much swapping:
> http://imageshack.us/m/52/6149/memoryday.png
>
> I already read a lot of posts and mails for similar problems, but
> unfortunately, I was not able to solve this problem. I added following
> infos to my squid.conf-file:
> # cache specific settings
> cache_replacement_policy heap LFUDA
> cache_mem 1600 MB
> memory_replacement_policy heap LFUDA
> maximum_object_size_in_memory 2048 KB
> memory_pools off
> cache_swap_low 85
> cache_swap_high 90
>
> (There are four squid instances per server, which means that 1600*4 =
> 6400MB RAM used for caching, which is not even 1/4 of the total
> available amount of RAM. Plenty enough, don't you think?)

Not that is "for HTTP object caching", emphasis on *caching* and "HTTP
object". In-transit objects and non-HTTP caches (Ip cache, domain name
cache, persistent connections cache, client database, via/fwd database,
network performance cache, auth caches, external ACL caches) and the
indexes for all those caches use other memory.

Then again they should all be using no more than a few GB combined. So
you may have hit a new leak (all the known ones are resolved before 3.1.12).

>
> Very strange are the negative values (Memory usage for squid via
> mallinfo():) from the output below. Maybe that is a reason for running
> out of RAM?

mallinfo() sucks badly when going above 2GB of RAM. It can be ignored.

The section underneath it "Memory accounted for:" is Squids own
accounting and more of a worry. It should not have had negatives since
before 3.1.10.

>
> HTTP/1.0 200 OK
> Server: squid/3.1.12
> Mime-Version: 1.0
> Date: Wed, 11 May 2011 07:06:10 GMT
> Content-Type: text/plain
> Expires: Wed, 11 May 2011 07:06:10 GMT
> Last-Modified: Wed, 11 May 2011 07:06:10 GMT
> X-Cache: MISS from xlsqip03_1
> Via: 1.0 xlsqip03_1 (squid/3.1.12)
> Connection: close
>
> Squid Object Cache: Version 3.1.12
> Start Time: Wed, 27 Apr 2011 11:01:13 GMT
> Current Time: Wed, 11 May 2011 07:06:10 GMT
> Connection information for squid:
> Number of clients accessing cache: 1671
> Number of HTTP requests received: 16144359
> Number of ICP messages received: 0
> Number of ICP messages sent: 0
> Number of queued ICP replies: 0
> Number of HTCP messages received: 0
> Number of HTCP messages sent: 0
> Request failure ratio: 0.00
> Average HTTP requests per minute since start: 810.3
> Average ICP messages per minute since start: 0.0
> Select loop called: 656944758 times, 1.820 ms avg
> Cache information for squid:
> Hits as % of all requests: 5min: 17.4%, 60min: 18.2%
> Hits as % of bytes sent: 5min: 45.6%, 60min: 39.9%
> Memory hits as % of hit requests: 5min: 86.1%, 60min: 88.9%
> Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0%
> Storage Swap size: 0 KB
> Storage Swap capacity: 0.0% used, 0.0% free
> Storage Mem size: 1622584 KB
> Storage Mem capacity: 100.0% used, 0.0% free

Okay 1.6 GB of RAM used for caching HTTP objects. Fully used.

> Mean Object Size: 0.00 KB

Problem #1. It *may* be Squid not accounting for the memory objects in
the mean.

> Requests given to unlinkd: 0
> Median Service Times (seconds) 5 min 60 min:
> HTTP Requests (All): 0.01648 0.01235
> Cache Misses: 0.05046 0.04277
> Cache Hits: 0.00091 0.00091
> Near Hits: 0.01469 0.01745
> Not-Modified Replies: 0.00000 0.00091
> DNS Lookups: 0.00190 0.00190
> ICP Queries: 0.00000 0.00000
> Resource usage for squid:
> UP Time: 1195497.286 seconds
> CPU Time: 22472.507 seconds
> CPU Usage: 1.88%
> CPU Usage, 5 minute avg: 5.38%
> CPU Usage, 60 minute avg: 5.44%
> Process Data Segment Size via sbrk(): 3145032 KB
> Maximum Resident Size: 0 KB
> Page faults with physical i/o: 8634
> Memory usage for squid via mallinfo():
> Total space in arena: -1049140 KB
> Ordinary blocks: -1277813 KB 87831 blks
> Small blocks: 0 KB 0 blks
> Holding blocks: 2240 KB 5 blks
> Free Small blocks: 0 KB
> Free Ordinary blocks: 228673 KB
> Total in use: -1275574 KB 122%
> Total free: 228674 KB -22%
> Total size: -1046900 KB
> Memory accounted for:
> Total accounted: -1375357 KB 131%
> memPool accounted: 2818947 KB -269%
> memPool unaccounted: -3865847 KB 0%
> memPoolAlloc calls: 111
> memPoolFree calls: 8322084644
> File descriptor usage for squid:
> Maximum number of file descriptors: 1024
> Largest file desc currently in use: 563
> Number of file desc currently in use: 472
> Files queued for open: 0
> Available number of file descriptors: 552
> Reserved number of file descriptors: 100
> Store Disk files open: 0
> Internal Data Structures:
> 96996 StoreEntries
> 96996 StoreEntries with MemObjects
> 96980 Hot Object Cache Items
> 0 on-disk objects
>
> Has anyone experienced similar things or does even know a solution?

Not since we fixed Squid's capacity to calculate storage used by >2GB
objects.

Can you find the actual maximum and average object size for the cached
objects in that Squid? (the mgr:vm_objects report should have all their
details, "inmem_hi:" being the object size)

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.12
   Beta testers wanted for 3.2.0.7 and 3.1.12.1