[squid-users] Squid memory leak?

I've been chasing what looks like a memory leak for quite some time. I've seen it all through
the Squid 2.X releases, and it still is present in 2.5STABLE4. I've seen it on machines running
HP-UX 10.20 and on Red Hat Linux. Currently I'm running on three identical Dell 2650 servers,
each with dual 1.8 GHz processors, 2 GB of RAM, and two 20 GB cache partitions. They are all
running Red Hat ES 2.1, kernel 2.4.9-e.25smp.

Squid is built with these configuration options:
    Squid Cache: Version 2.5.STABLE4
    configure options: --prefix=/local/proxy/squid --enable-cache-digests
                        --enable-underscores --with-pthreads --enable-storeio=aufs,ufs
                        --enable-removal-policies=heap --enable-gnuregex

With a full cache, Squid memory utilization starts at about 250 MB. It then grows at about
300 MB per week. The graph at
        ftp://ftp.sas.com/pub/nam/squid-mem.png
shows how the memory grows.
We see about 80 requests per second during business hours. The graph
        ftp://ftp.sas.com/pub/nam/squid-reqs.png
shows our requests per second.
We use DNS round-robin to load balance requests between the three systems.

Each squid process forwards most requests on to a Trend Interscan VirusWall process listening
on port 8080 on the same server. It also sends the URLs through the "adzap" redirector. ACLs
are used to decide which URLs are sent through the redirector and also which URLs are
sent to the virus scanner.

Here's some squid information output from the cachemgr.cgi:
===========================================================
Squid Object Cache: Version 2.5.STABLE4
Start Time: Sat, 20 Sep 2003 19:29:16 GMT
Current Time: Mon, 29 Sep 2003 21:14:57 GMT
Connection information for squid:
        Number of clients accessing cache: 0
        Number of HTTP requests received: 17264989
        Number of ICP messages received: 1961488
        Number of ICP messages sent: 1961585
        Number of queued ICP replies: 0
        Request failure ratio: 0.00
        Average HTTP requests per minute since start: 1321.4
        Average ICP messages per minute since start: 300.3
        Select loop called: 90171815 times, 8.694 ms avg
Cache information for squid:
        Request Hit Ratios: 5min: 62.3%, 60min: 65.9%
        Byte Hit Ratios: 5min: 15.8%, 60min: 21.1%
        Request Memory Hit Ratios: 5min: 4.5%, 60min: 4.2%
        Request Disk Hit Ratios: 5min: 18.4%, 60min: 12.5%
        Storage Swap size: 31876696 KB
        Storage Mem size: 98348 KB
        Mean Object Size: 14.58 KB
        Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
        HTTP Requests (All): 0.02317 0.02317
        Cache Misses: 0.16775 0.16775
        Cache Hits: 0.01309 0.01469
        Near Hits: 0.09736 0.10857
        Not-Modified Replies: 0.01387 0.01469
        DNS Lookups: 0.00464 0.00464
        ICP Queries: 0.00316 0.00541
Resource usage for squid:
        UP Time: 783941.680 seconds
        CPU Time: 118154.720 seconds
        CPU Usage: 15.07%
        CPU Usage, 5 minute avg: 65.24%
        CPU Usage, 60 minute avg: 60.68%
        Process Data Segment Size via sbrk(): 556455 KB
        Maximum Resident Size: 0 KB
        Page faults with physical i/o: 33606
Memory usage for squid via mallinfo():
        Total space in arena: 556455 KB
        Ordinary blocks: 491460 KB 532257 blks
        Small blocks: 0 KB 0 blks
        Holding blocks: 19384 KB 12 blks
        Free Small blocks: 0 KB
        Free Ordinary blocks: 64995 KB
        Total in use: 510844 KB 89%
        Total free: 64995 KB 11%
        Total size: 575839 KB
Memory accounted for:
        Total accounted: 263830 KB
        memPoolAlloc calls: 2122989169
        memPoolFree calls: 2118393830
File descriptor usage for squid:
        Maximum number of file descriptors: 4096
        Largest file desc currently in use: 877
        Number of file desc currently in use: 465
        Files queued for open: 0
        Available number of file descriptors: 3631
        Reserved number of file descriptors: 100
        Store Disk files open: 4
Internal Data Structures:
        2199786 StoreEntries
         13566 StoreEntries with MemObjects
         13530 Hot Object Cache Items
        2186456 on-disk objects
===================================================
What is most interesting from this is that mallinfo says 556455 KB is in the arena, but
squid can only account for 263830 KB of it.

Here's the interesting bits of our configuration:
===================================================
http_port 80 8000
cache_peer 127.0.0.1 parent 8080 7 no-query no-digest default
cache_peer inetgw02.unx.sas.com sibling 80 3130 proxy-only
cache_peer inetgw03.unx.sas.com sibling 80 3130 proxy-only
cache_peer inetgw02.unx.sas.com parent 8080 7 no-query no-digest round-robin
cache_peer inetgw03.unx.sas.com parent 8080 7 no-query no-digest round-robin
cache_mem 96 MB
cache_swap_low 95
cache_swap_high 97
maximum_object_size 700 MB
maximum_object_size_in_memory 32 KB
cache_replacement_policy heap LFUDA
memory_replacement_policy heap GDSF
cache_dir aufs /cache/cache1/squid 16384 53 254
cache_dir aufs /cache/cache2/squid 16384 53 254
cache_store_log none
emulate_httpd_log on
acl all src 0.0.0.0/0.0.0.0
acl noAds myport 80
acl Reads method GET HEAD
acl CIDR_A src 10.0.0.0/255.0.0.0
acl virus_proto proto HTTP
acl novirus-url urlpath_regex -i \.gif(\?.*)?$ \.jpg(\?.*)?$ \.png(\?.*)?$
ident_lookup_access allow CIDR_A
ident_lookup_access deny all
memory_pools on
log_icp_queries off
client_db off
always_direct allow novirus-url
always_direct allow !virus_proto
never_direct allow all
strip_query_terms off
redirect_program /opt/adzap/wrapzap
redirect_children 10
redirector_access deny !Reads
redirector_access allow noAds
redirector_bypass on
===================================================

Here's some messages that are showing up in the cache.log file.

===================================================
2003/09/29 16:38:26| sslReadServer: FD 578: read failure: (104) Connection reset by peer
2003/09/29 16:39:00| sslReadServer: FD 574: read failure: (104) Connection reset by peer
2003/09/29 16:41:44| sslReadServer: FD 560: read failure: (104) Connection reset by peer
2003/09/29 16:42:14| sslReadServer: FD 485: read failure: (104) Connection reset by peer
2003/09/29 16:42:54| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:42:55| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:42:59| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:03| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:07| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:08| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:17| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:21| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:31| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:32| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:33| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:34| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:46| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:47| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:44:00| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:44:00| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
===================================================
Does anyone have an idea of why I'm consuming so much memory? Am I just running into memory
fragmentation issues? Should I be linking with a different version of malloc? Would the
configuration option '--enable-dlmalloc' help?
I've worked around this problem by having a cron job check how large the squid process is.
If it grows too large I restart it. Currently it gets restarted twice a month.
------------------------------------------------------------------------------
Mike Mitchell
Mike.Mitchell@sas.com
(919) 677-8000 X16793
Received on Mon Sep 29 2003 - 16:23:30 MDT