[squid-users] latency issues squid2.7 WCCP

Squid 2.7.STABLE1-20080528 on Debian Linux 2.6.19.7
running on quad dual-core 2.6mhz Opterons with 32 gig RAM; 8x140GB disk
partitions
using WCCP L2 redirects transparently from a Cisco 4948 GigE switch

Server has one GigE NIC for the incoming redirects and two GigE NICs for
outbound http requests.
Using IPTables to port forward HTTP to Squid; no ICP, auth, etc.;
strictly a web cache using heap/LFUDA replacement
and 16GB memory allocated with mem pools on, no limit.

Used in an ISP environment, accommodating approx. 8k predominately cable
modem customers during peak.

Issue we're experiencing is some web pages taking in excess of 20
seconds to load, marked latency for customers
running web-based speed tests, etc.
Cache.log and Access.log aren't indicating any errors or timeouts;
system operates 96 DNS instances and 32k file descriptors
(neither has gotten maxed yet).
General Runtime Info from Cachemgr taken during pre-peak usage:
Start Time: Tue, 23 Sep 2008 18:07:37 GMT
Current Time: Tue, 23 Sep 2008 21:00:49 GMT

Connection information for squid:
    Number of clients accessing cache: 3382
    Number of HTTP requests received: 2331742
    Number of ICP messages received: 0
    Number of ICP messages sent: 0
    Number of queued ICP replies: 0
    Request failure ratio: 0.00
    Average HTTP requests per minute since start: 13463.4
    Average ICP messages per minute since start: 0.0
    Select loop called: 11255153 times, 0.923 ms avg
Cache information for squid:
    Request Hit Ratios: 5min: 42.6%, 60min: 40.0%
    Byte Hit Ratios: 5min: 21.2%, 60min: 18.6%
    Request Memory Hit Ratios: 5min: 18.3%, 60min: 17.2%
    Request Disk Hit Ratios: 5min: 33.6%, 60min: 33.3%
    Storage Swap size: 952545580 KB
    Storage Mem size: 8237648 KB
    Mean Object Size: 40.43 KB
    Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
    HTTP Requests (All): 0.19742 0.12106
    Cache Misses: 0.27332 0.17711
    Cache Hits: 0.08265 0.03622
    Near Hits: 0.27332 0.16775
    Not-Modified Replies: 0.02317 0.00865
    DNS Lookups: 0.09535 0.04854
    ICP Queries: 0.00000 0.00000
Resource usage for squid:
    UP Time: 10391.501 seconds
    CPU Time: 4708.150 seconds
    CPU Usage: 45.31%
    CPU Usage, 5 minute avg: 33.29%
    CPU Usage, 60 minute avg: 33.36%
    Process Data Segment Size via sbrk(): 1041332 KB
    Maximum Resident Size: 0 KB
    Page faults with physical i/o: 4
Memory usage for squid via mallinfo():
    Total space in arena: 373684 KB
    Ordinary blocks: 372642 KB 809 blks
    Small blocks: 0 KB 0 blks
    Holding blocks: 216088 KB 21 blks
    Free Small blocks: 0 KB
    Free Ordinary blocks: 1041 KB
    Total in use: 588730 KB 100%
    Total free: 1041 KB 0%
    Total size: 589772 KB
Memory accounted for:
    Total accounted: 11355185 KB
    memPoolAlloc calls: 439418241
    memPoolFree calls: 378603777
File descriptor usage for squid:
    Maximum number of file descriptors: 32000
    Largest file desc currently in use: 9171
    Number of file desc currently in use: 8112
    Files queued for open: 2
    Available number of file descriptors: 23886
    Reserved number of file descriptors: 100
    Store Disk files open: 175
    IO loop method: epoll
Internal Data Structures:
    23570637 StoreEntries
    532260 StoreEntries with MemObjects
    531496 Hot Object Cache Items
    23561001 on-disk objects

Generated Tue, 23 Sep 2008 21:00:47 GMT, by
cachemgr.cgi/2.7.STABLE1-20080528_at_proxy.machlink.com

TCPDUMP shows packets traversing all interfaces as expected; bandwidth
to both upstream providers isn't being maxed
and when Squid is shut down, http traffic loads much faster and without
any noticeable delay.

Where/what else can I look at for the cause of the latency? It becomes
significantly worse during peak use - but as
we're not being choked on bandwidth and things greatly improve when I
shut down squid that narrows it to something
on the server. Is the amount of activity overloading a single squid
process? I'm not seeing any I/O errors in logs and haven't
found any evidence the kernel is under distress.
Any pointers are greatly appreciated.
thanks
-Ryan
Received on Wed Sep 24 2008 - 15:49:14 MDT