Hi,
Any news on this topic? Unfortunately, RAM is running full within days
and at the moment, our workaround is to do a reboot ... We would
appreciate any other solution!
thanks,
peter
On Wed, May 11, 2011 at 3:47 PM, guest01 <guest01_at_gmail.com> wrote:
> On Wed, May 11, 2011 at 10:47 AM, Amos Jeffries <squid3_at_treenet.co.nz> wrote:
>> On 11/05/11 19:19, guest01 wrote:
>>>
>>> Hi,
>>>
>>> I am currently using squid 3.1.12 as forward-proxy without
>>> harddisk-caching (only RAM is used for caching). Each server is
>>> running on RHEL5.5 and is pretty strong (16 CPUs, 28GB RAM), but each
>>> servers starts swapping a few days after start. The workaround at the
>>> moment is to reboot the server once a week, which I don't really like.
>>> But swapping leads to serious side effects, e.g. performance troubles,
>>> ...
>>>
>>> way too much swapping:
>>> http://imageshack.us/m/52/6149/memoryday.png
>>>
>>> I already read a lot of posts and mails for similar problems, but
>>> unfortunately, I was not able to solve this problem. I added following
>>> infos to my squid.conf-file:
>>> # cache specific settings
>>> cache_replacement_policy heap LFUDA
>>> cache_mem 1600 MB
>>> memory_replacement_policy heap LFUDA
>>> maximum_object_size_in_memory 2048 KB
>>> memory_pools off
>>> cache_swap_low 85
>>> cache_swap_high 90
>>>
>>> (There are four squid instances per server, which means that 1600*4 =
>>> 6400MB RAM used for caching, which is not even 1/4 of the total
>>> available amount of RAM. Plenty enough, don't you think?)
>>
>> Not that is "for HTTP object caching", emphasis on *caching* and "HTTP
>> object". In-transit objects and non-HTTP caches (Ip cache, domain name
>> cache, persistent connections cache, client database, via/fwd database,
>> network performance cache, auth caches, external ACL caches) and the indexes
>> for all those caches use other memory.
>>
>> Then again they should all be using no more than a few GB combined. So you
>> may have hit a new leak (all the known ones are resolved before 3.1.12).
>
> Ok, very strange. But at least it is reproducible, it takes about a
> week until squid is starting to swap ...
> http://img191.imageshack.us/img191/9615/memorymonth.png
>
>>>
>>> Very strange are the negative values (Memory usage for squid via
>>> mallinfo():) from the output below. Maybe that is a reason for running
>>> out of RAM?
>>
>> mallinfo() sucks badly when going above 2GB of RAM. It can be ignored.
>>
>> The section underneath it "Memory accounted for:" is Squids own accounting
>> and more of a worry. It should not have had negatives since before 3.1.10.
>>
>>>
>>> HTTP/1.0 200 OK
>>> Server: squid/3.1.12
>>> Mime-Version: 1.0
>>> Date: Wed, 11 May 2011 07:06:10 GMT
>>> Content-Type: text/plain
>>> Expires: Wed, 11 May 2011 07:06:10 GMT
>>> Last-Modified: Wed, 11 May 2011 07:06:10 GMT
>>> X-Cache: MISS from xlsqip03_1
>>> Via: 1.0 xlsqip03_1 (squid/3.1.12)
>>> Connection: close
>>>
>>> Squid Object Cache: Version 3.1.12
>>> Start Time: Wed, 27 Apr 2011 11:01:13 GMT
>>> Current Time: Wed, 11 May 2011 07:06:10 GMT
>>> Connection information for squid:
>>> Number of clients accessing cache: 1671
>>> Number of HTTP requests received: 16144359
>>> Number of ICP messages received: 0
>>> Number of ICP messages sent: 0
>>> Number of queued ICP replies: 0
>>> Number of HTCP messages received: 0
>>> Number of HTCP messages sent: 0
>>> Request failure ratio: 0.00
>>> Average HTTP requests per minute since start: 810.3
>>> Average ICP messages per minute since start: 0.0
>>> Select loop called: 656944758 times, 1.820 ms avg
>>> Cache information for squid:
>>> Hits as % of all requests: 5min: 17.4%, 60min: 18.2%
>>> Hits as % of bytes sent: 5min: 45.6%, 60min: 39.9%
>>> Memory hits as % of hit requests: 5min: 86.1%, 60min: 88.9%
>>> Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0%
>>> Storage Swap size: 0 KB
>>> Storage Swap capacity: 0.0% used, 0.0% free
>>> Storage Mem size: 1622584 KB
>>> Storage Mem capacity: 100.0% used, 0.0% free
>>
>> Okay 1.6 GB of RAM used for caching HTTP objects. Fully used.
>>
>>> Mean Object Size: 0.00 KB
>>
>> Problem #1. It *may* be Squid not accounting for the memory objects in the
>> mean.
>>
>>> Requests given to unlinkd: 0
>>> Median Service Times (seconds) 5 min 60 min:
>>> HTTP Requests (All): 0.01648 0.01235
>>> Cache Misses: 0.05046 0.04277
>>> Cache Hits: 0.00091 0.00091
>>> Near Hits: 0.01469 0.01745
>>> Not-Modified Replies: 0.00000 0.00091
>>> DNS Lookups: 0.00190 0.00190
>>> ICP Queries: 0.00000 0.00000
>>> Resource usage for squid:
>>> UP Time: 1195497.286 seconds
>>> CPU Time: 22472.507 seconds
>>> CPU Usage: 1.88%
>>> CPU Usage, 5 minute avg: 5.38%
>>> CPU Usage, 60 minute avg: 5.44%
>>> Process Data Segment Size via sbrk(): 3145032 KB
>>> Maximum Resident Size: 0 KB
>>> Page faults with physical i/o: 8634
>>> Memory usage for squid via mallinfo():
>>> Total space in arena: -1049140 KB
>>> Ordinary blocks: -1277813 KB 87831 blks
>>> Small blocks: 0 KB 0 blks
>>> Holding blocks: 2240 KB 5 blks
>>> Free Small blocks: 0 KB
>>> Free Ordinary blocks: 228673 KB
>>> Total in use: -1275574 KB 122%
>>> Total free: 228674 KB -22%
>>> Total size: -1046900 KB
>>> Memory accounted for:
>>> Total accounted: -1375357 KB 131%
>>> memPool accounted: 2818947 KB -269%
>>> memPool unaccounted: -3865847 KB 0%
>>> memPoolAlloc calls: 111
>>> memPoolFree calls: 8322084644
>>> File descriptor usage for squid:
>>> Maximum number of file descriptors: 1024
>>> Largest file desc currently in use: 563
>>> Number of file desc currently in use: 472
>>> Files queued for open: 0
>>> Available number of file descriptors: 552
>>> Reserved number of file descriptors: 100
>>> Store Disk files open: 0
>>> Internal Data Structures:
>>> 96996 StoreEntries
>>> 96996 StoreEntries with MemObjects
>>> 96980 Hot Object Cache Items
>>> 0 on-disk objects
>>>
>>> Has anyone experienced similar things or does even know a solution?
>>
>> Not since we fixed Squid's capacity to calculate storage used by >2GB
>> objects.
>>
>> Can you find the actual maximum and average object size for the cached
>> objects in that Squid? (the mgr:vm_objects report should have all their
>> details, "inmem_hi:" being the object size)
>
> Unfortunately, we had to reboot this server. At the moment there is no
> swapping. But the mean object size is still 0.0kb:
> xlsqip03:/root>/opt/squid/bin/squidclient -h 10.122.125.4
> cache_object://10.122.125.4/info
> HTTP/1.0 200 OK
> Server: squid/3.1.12
> Mime-Version: 1.0
> Date: Wed, 11 May 2011 13:33:58 GMT
> Content-Type: text/plain
> Expires: Wed, 11 May 2011 13:33:58 GMT
> Last-Modified: Wed, 11 May 2011 13:33:58 GMT
> X-Cache: MISS from xlsqip03_1
> Via: 1.0 xlsqip03_1 (squid/3.1.12)
> Connection: close
>
> Squid Object Cache: Version 3.1.12
> Start Time: Wed, 11 May 2011 09:35:49 GMT
> Current Time: Wed, 11 May 2011 13:33:58 GMT
> Connection information for squid:
> Number of clients accessing cache: 814
> Number of HTTP requests received: 504290
> Number of ICP messages received: 0
> Number of ICP messages sent: 0
> Number of queued ICP replies: 0
> Number of HTCP messages received: 0
> Number of HTCP messages sent: 0
> Request failure ratio: 0.00
> Average HTTP requests per minute since start: 2117.6
> Average ICP messages per minute since start: 0.0
> Select loop called: 15248273 times, 0.937 ms avg
> Cache information for squid:
> Hits as % of all requests: 5min: 17.6%, 60min: 15.8%
> Hits as % of bytes sent: 5min: 38.0%, 60min: 34.3%
> Memory hits as % of hit requests: 5min: 88.2%, 60min: 87.0%
> Disk hits as % of hit requests: 5min: 0.1%, 60min: 0.1%
> Storage Swap size: 0 KB
> Storage Swap capacity: 0.0% used, 0.0% free
> Storage Mem size: 1622020 KB
> Storage Mem capacity: 100.0% used, 0.0% free
> Mean Object Size: 0.00 KB
> Requests given to unlinkd: 0
> Median Service Times (seconds) 5 min 60 min:
> HTTP Requests (All): 0.02190 0.01847
> Cache Misses: 0.04277 0.04776
> Cache Hits: 0.00091 0.00091
> Near Hits: 0.02069 0.02069
> Not-Modified Replies: 0.00091 0.00091
> DNS Lookups: 0.00190 0.00094
> ICP Queries: 0.00000 0.00000
> Resource usage for squid:
> UP Time: 14288.578 seconds
> CPU Time: 642.922 seconds
> CPU Usage: 4.50%
> CPU Usage, 5 minute avg: 5.70%
> CPU Usage, 60 minute avg: 5.27%
> Process Data Segment Size via sbrk(): 3035580 KB
> Maximum Resident Size: 0 KB
> Page faults with physical i/o: 0
> Memory usage for squid via mallinfo():
> Total space in arena: -1158592 KB
> Ordinary blocks: -1187512 KB 8338 blks
> Small blocks: 0 KB 0 blks
> Holding blocks: 2240 KB 5 blks
> Free Small blocks: 0 KB
> Free Ordinary blocks: 28920 KB
> Total in use: -1185273 KB 103%
> Total free: 28921 KB -3%
> Total size: -1156352 KB
> Memory accounted for:
> Total accounted: -1290919 KB 112%
> memPool accounted: 2903385 KB -251%
> memPool unaccounted: -4059737 KB 0%
> memPoolAlloc calls: 111
> memPoolFree calls: 263492993
> File descriptor usage for squid:
> Maximum number of file descriptors: 1024
> Largest file desc currently in use: 372
> Number of file desc currently in use: 298
> Files queued for open: 0
> Available number of file descriptors: 726
> Reserved number of file descriptors: 100
> Store Disk files open: 0
> Internal Data Structures:
> 105790 StoreEntries
> 105790 StoreEntries with MemObjects
> 105780 Hot Object Cache Items
> 0 on-disk objects
>
> xlsqip03:/root>/opt/squid/bin/squidclient -h 10.122.125.4
> cache_object://10.122.125.4/vm_objects | head -n 50
> HTTP/1.0 200 OK
> Server: squid/3.1.12
> Mime-Version: 1.0
> Date: Wed, 11 May 2011 13:34:39 GMT
> Content-Type: text/plain
> Expires: Wed, 11 May 2011 13:34:39 GMT
> Last-Modified: Wed, 11 May 2011 13:34:39 GMT
> X-Cache: MISS from xlsqip03_1
> Via: 1.0 xlsqip03_1 (squid/3.1.12)
> Connection: close
>
> KEY 00C06887AD5B919595711834A5195511
> STORE_OK IN_MEMORY SWAPOUT_NONE PING_DONE
> CACHABLE,DISPATCHED,VALIDATED
> LV:1305110370 LU:1305110371 LM:1157000708 EX:-1
> 0 locks, 0 clients, 1 refs
> Swap Dir -1, File 0XFFFFFFFF
> GET http://www.hardwaremuseum.at/mitarbeiter/keske.jpg
> inmem_lo: 0
> inmem_hi: 37138
> swapout: 0 bytes queued
>
> KEY 00A02D3964813C2CC4A7D6553D2CF0A7
> STORE_OK IN_MEMORY SWAPOUT_NONE PING_DONE
> CACHABLE,DISPATCHED,VALIDATED
> LV:1305110512 LU:1305111016 LM:1275291526 EX:-1
> 0 locks, 0 clients, 2 refs
> Swap Dir -1, File 0XFFFFFFFF
> GET http://nyx.at/bikeboard/images/design/icon_hotels.png
> inmem_lo: 0
> inmem_hi: 1507
> swapout: 0 bytes queued
>
> KEY 0020B363AC66E4EAF4B812DCCB561B99
> STORE_OK IN_MEMORY SWAPOUT_NONE PING_DONE
> CACHABLE,DISPATCHED,VALIDATED
> LV:1305110827 LU:1305110833 LM:966956701 EX:-1
> 0 locks, 0 clients, 1 refs
> Swap Dir -1, File 0XFFFFFFFF
> GET http://windfind.ivwbox.de/blank.gif
> inmem_lo: 0
> inmem_hi: 296
> swapout: 0 bytes queued
>
> KEY 00C09B8188CC913834CF292CC2A4D72E
> STORE_OK IN_MEMORY SWAPOUT_NONE PING_DONE
> CACHABLE,DISPATCHED,VALIDATED
> LV:1305110909 LU:1305110991 LM:1153932790 EX:-1
> 0 locks, 0 clients, 3 refs
> Swap Dir -1, File 0XFFFFFFFF
>
> xlsqip03:/root>/opt/squid/bin/squidclient -h 10.122.125.4
> cache_object://10.122.125.4/vm_objects | grep inmem_hi
> inmem_hi: 37138
> inmem_hi: 1507
> inmem_hi: 296
> inmem_hi: 747
> inmem_hi: 5149
> inmem_hi: 1449
> inmem_hi: 5231
> inmem_hi: 3594
> inmem_hi: 7911
> inmem_hi: 699
> inmem_hi: 141554
> inmem_hi: 8997
> inmem_hi: 977
> inmem_hi: 37114
> inmem_hi: 6229
> inmem_hi: 79923
> inmem_hi: 1016
> inmem_hi: 11075
> inmem_hi: 1475
> inmem_hi: 6113
> inmem_hi: 3118
> inmem_hi: 26894
> inmem_hi: 1433
> inmem_hi: 2763
> inmem_hi: 2388
> inmem_hi: 287
> inmem_hi: 2481
> inmem_hi: 297
> inmem_hi: 3399
> inmem_hi: 217
> inmem_hi: 1549
> inmem_hi: 41865
> inmem_hi: 4920
> inmem_hi: 7857
> inmem_hi: 1608
> inmem_hi: 5666
> inmem_hi: 9507
> inmem_hi: 8653
> inmem_hi: 11728
> inmem_hi: 6106
> inmem_hi: 452
> inmem_hi: 2740
> inmem_hi: 1122
> inmem_hi: 2183
> inmem_hi: 3861
> inmem_hi: 14803
> inmem_hi: 861
> inmem_hi: 26080
> inmem_hi: 28724
> inmem_hi: 2094
> inmem_hi: 428
> inmem_hi: 1445
> inmem_hi: 468
> inmem_hi: 21754
>
> I also checked our Test-Squid with Squid 3.2.0.7:
> squid_at_xlsqit01 /etc/squid $ /opt/squid/bin/squidclient -h 10.122.125.2
> cache_object://10.122.125.2/info
> HTTP/1.1 200 OK
> Server: squid/3.2.0.7
> Mime-Version: 1.0
> Date: Wed, 11 May 2011 13:40:19 GMT
> Content-Type: text/plain
> Expires: Wed, 11 May 2011 13:40:19 GMT
> Last-Modified: Wed, 11 May 2011 13:40:19 GMT
> Connection: close
>
> Squid Object Cache: Version 3.2.0.7
> Start Time: Fri, 29 Apr 2011 07:56:13 GMT
> Current Time: Wed, 11 May 2011 13:40:19 GMT
> Connection information for squid:
> Number of clients accessing cache: 28
> Number of HTTP requests received: 44628
> Number of ICP messages received: 0
> Number of ICP messages sent: 0
> Number of queued ICP replies: 0
> Number of HTCP messages received: 0
> Number of HTCP messages sent: 0
> Request failure ratio: 0.00
> Average HTTP requests per minute since start: 2.5
> Average ICP messages per minute since start: 0.0
> Select loop called: 1549800905 times, 1.365 ms avg
> Cache information for squid:
> Hits as % of all requests: 5min: 0.0%, 60min: 0.0%
> Hits as % of bytes sent: 5min: 100.0%, 60min: 57.0%
> Memory hits as % of hit requests: 5min: 0.0%, 60min: 0.0%
> Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0%
> Storage Swap size: 0 KB
> Storage Swap capacity: 0.0% used, 0.0% free
> Storage Mem size: 30452 KB
> Storage Mem capacity: 5.9% used, 94.1% free
> Mean Object Size: 0.00 KB
> Requests given to unlinkd: 0
> Median Service Times (seconds) 5 min 60 min:
> HTTP Requests (All): 0.00189 0.00000
> Cache Misses: 0.00000 0.12262
> Cache Hits: 0.00000 0.00000
> Near Hits: 0.00000 0.00000
> Not-Modified Replies: 0.00000 0.00000
> DNS Lookups: 0.00047 0.00047
> ICP Queries: 0.00000 0.00000
> Resource usage for squid:
> UP Time: 1057446.006 seconds
> CPU Time: 366.647 seconds
> CPU Usage: 0.03%
> CPU Usage, 5 minute avg: 0.08%
> CPU Usage, 60 minute avg: 0.04%
> Process Data Segment Size via sbrk(): 63720 KB
> Maximum Resident Size: 0 KB
> Page faults with physical i/o: 0
> Memory usage for squid via mallinfo():
> Total space in arena: 63984 KB
> Ordinary blocks: 63645 KB 22 blks
> Small blocks: 0 KB 0 blks
> Holding blocks: 18480 KB 12 blks
> Free Small blocks: 0 KB
> Free Ordinary blocks: 339 KB
> Total in use: 339 KB 0%
> Total free: 339 KB 0%
> Total size: 82464 KB
> Memory accounted for:
> Total accounted: 54669 KB 66%
> memPool accounted: 54669 KB 66%
> memPool unaccounted: 27795 KB 34%
> memPoolAlloc calls: 9115016
> memPoolFree calls: 13310573
> File descriptor usage for squid:
> Maximum number of file descriptors: 32768
> Largest file desc currently in use: 60
> Number of file desc currently in use: 41
> Files queued for open: 0
> Available number of file descriptors: 32727
> Reserved number of file descriptors: 200
> Store Disk files open: 0
> Internal Data Structures:
> 1461 StoreEntries
> 1461 StoreEntries with MemObjects
> 1459 Hot Object Cache Items
> 0 on-disk objects
> squid_at_xlsqit01 /etc/squid $ /opt/squid/bin/squidclient -h 10.122.125.2
> cache_object://10.122.125.2/info | grep -i mean
> Mean Object Size: 0.00 KB
>
> Seems to be the same problem:
> squid_at_xlsqit01 /etc/squid $ /opt/squid/bin/squidclient -h 10.122.125.2
> cache_object://10.122.125.2/vm_objects | grep inmem_hi
> inmem_hi: 410
> inmem_hi: 4859
> inmem_hi: 5927
> inmem_hi: 1494
> inmem_hi: 766
> inmem_hi: 681
> inmem_hi: 2215
> inmem_hi: 423
> inmem_hi: 236
> inmem_hi: 386
> inmem_hi: 3899
> inmem_hi: 1060
> inmem_hi: 1093
> inmem_hi: 39037
> inmem_hi: 2321
> inmem_hi: 53163
> inmem_hi: 822
> inmem_hi: 272
> inmem_hi: 420
> inmem_hi: 421
> inmem_hi: 381
> inmem_hi: 2186
> inmem_hi: 307
> inmem_hi: 394
>
> Very strange indeed ... Maybe it is related to our RHEL5.5 environment
> or compile options? Otherwise my guess is that many people would have
> the same problem?
>
> xlsqip03:/root>/opt/squid/sbin/squid -v
> Squid Cache: Version 3.1.12
> configure options: '--enable-ssl' '--enable-icap-client'
> '--sysconfdir=/etc/squid' '--enable-async-io' '--enable-snmp'
> '--enable-poll' '--with-maxfd=32768' '--enable-storeio=aufs'
> '--enable-removal-policies=heap,lru' '--enable-epoll'
> '--disable-ident-lookups' '--enable-truncate'
> '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid'
> '--with-default-user=squid' '--prefix=/opt/squid' '--enable-auth=basic
> digest ntlm negotiate'
> '-enable-negotiate-auth-helpers=squid_kerb_auth'
> --with-squid=/home/squid/squid-3.1.12 --enable-ltdl-convenience
>
> squid_at_xlsqit01 /etc/squid $ /opt/squid/sbin/squid -v
> Squid Cache: Version 3.2.0.7
> configure options: '--enable-ssl' '--enable-icap-client'
> '--sysconfdir=/etc/squid' '--enable-async-io' '--enable-snmp'
> '--enable-poll' '--with-maxfd=32768' '--enable-storeio=aufs'
> '--enable-removal-policies=heap,lru' '--enable-epoll'
> '--disable-ident-lookups' '--enable-truncate'
> '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid'
> '--with-default-user=squid' '--prefix=/opt/squid'
> '-enable-negotiate-auth-helpers=squid_kerb_auth'
> --enable-ltdl-convenience
>
> We are basically only doing authentication (Kerberos, LDAP, NTLM) and
> ICAP authorization.
> Is there anything else I can do?
>
> thanks and regards!
> Peter
>
Received on Mon May 30 2011 - 08:07:23 MDT
This archive was generated by hypermail 2.2.0 : Mon May 30 2011 - 12:00:05 MDT
