I have a squid cache under moderate load (150 req/sec) that is starting
to exhibit some very strange behavior.
It is used primarily to download large files. The symptom is that the
files begin to download and stall around 256k. Maybe they pick up later
and go for a few K, or maybe they just stay stalled out. It's like
squid is spending all its time serving little requests and starving the
big ones.
This is Squid 2.4STABLE4 on OpenBSD 3.0, using the following configure:
CC=cc ./configure --prefix=/nfsn/apps/squid --sysconfdir=/nfsn/conf
--disable-wccp --enable-delay-pools --enable-removal-policies="heap,lru"
--quiet --enable-storeio=diskd,ufs --enable-referer-log
--enable-time-hack
Using 4 diskd cache trees.
I have checked everything I can think of to check.
- There are plenty of file descriptors.
- There is plenty of free RAM.
- There is almost no disk I/O.
- The processor is 95%+ idle.
Apache running on the same machine serving the same files can spit them
out at wire speed.
I have checked my squid.conf for the run_extra_slow directive, but I
don't see it.
The only thing I can think of that has changed is that there is a new
popular file and I have 30-50 clients hitting it simultaneously (web
accelerator). The "starvation" effect is bar far most pronounced on
this file. I notice that on "In Memory & In Transit" this file is
listed more than once. I've attached an example of that at the bottom.
Any advice appreciated, this is causing major problems both for us and
the people that want this file.
Thanks very much!
Jeff
-- NearlyFreeSpeech.NET $1/1GB Web Hosting, no minimums, no monthly fees, no kidding. http://www.nearlyfreespeech.net/ KEY F028292CD172F3F13428E68E14111BA2 GET http://files.reactorcritical.com/nfsn_dl/EAC51BE4468EA853FAEBF1C18CA060E 4/icq2002a-build3706.exe STORE_PENDING NOT_IN_MEMORY SWAPOUT_NONE PING_DONE RELEASE_REQUEST,DISPATCHED,PRIVATE,VALIDATED LV:1016278704 LU:1016278704 LM:1016185237 EX:1016278704 3 locks, 1 clients, 1 refs Swap Dir -1, File 0XFFFFFFFF inmem_lo: 0 inmem_hi: 755808 swapout: 0 bytes queued Client #0, 0x170fc00 copy_offset: 723936 seen_offset: 723936 copy_size: 4096 flags: KEY 0993104B1D82A398D3384FA4B93241F2 GET http://files.reactorcritical.com/nfsn_dl/EAC51BE4468EA853FAEBF1C18CA060E 4/icq2002a-build3706.exe STORE_PENDING NOT_IN_MEMORY SWAPOUT_NONE PING_DONE RELEASE_REQUEST,DISPATCHED,PRIVATE,VALIDATED LV:1016278704 LU:1016278704 LM:1016185237 EX:1016278704 3 locks, 1 clients, 1 refs Swap Dir -1, File 0XFFFFFFFF inmem_lo: 0 inmem_hi: 608064 swapout: 0 bytes queued Client #0, 0x1ba1c00 copy_offset: 588768 seen_offset: 588768 copy_size: 4096 flags: KEY F64427434E83B2030A97668C5653AEC7 GET http://files.reactorcritical.com/nfsn_dl/EAC51BE4468EA853FAEBF1C18CA060E 4/icq2002a-build3706.exe STORE_PENDING NOT_IN_MEMORY SWAPOUT_NONE PING_DONE RELEASE_REQUEST,DISPATCHED,PRIVATE,VALIDATED LV:1016279177 LU:1016279177 LM:1016185237 EX:1016279177 3 locks, 1 clients, 1 refs Swap Dir -1, File 0XFFFFFFFF inmem_lo: 0 inmem_hi: 331864 swapout: 0 bytes queued Client #0, 0x8c4c00 copy_offset: 310240 seen_offset: 310240 copy_size: 4096 flags: KEY B544297F561BD1BA0D4DCCB0AE44D6B0 GET http://files.reactorcritical.com/nfsn_dl/EAC51BE4468EA853FAEBF1C18CA060E 4/icq2002a-build3706.exe STORE_PENDING NOT_IN_MEMORY SWAPOUT_NONE PING_DONE CACHABLE,DISPATCHED,VALIDATED LV:1016278210 LU:1016279353 LM:1016185237 EX:-1 35 locks, 33 clients, 69 refs Swap Dir -1, File 0XFFFFFFFF inmem_lo: 0 inmem_hi: 263280 swapout: 0 bytes queued Client #0, 0x545000 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #1, 0x20d8000 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #2, 0x171c000 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #3, 0x16f0000 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #4, 0x1755800 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #5, 0x54ac00 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #6, 0x1773c00 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #7, 0x172e800 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #8, 0x16fe800 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #9, 0x1ba1800 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #10, 0x1740400 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #11, 0x4d2400 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #12, 0x177c000 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #13, 0x524000 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #14, 0x8c4400 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #15, 0x16fe000 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #16, 0x1755000 copy_offset: 246896 seen_offset: 246896 copy_size: 4096 flags: Client #17, 0x1b7d800 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #18, 0xd04800 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #19, 0x170f000 copy_offset: 246896 seen_offset: 246896 copy_size: 4096 flags: Client #20, 0x1755c00 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #21, 0x1765000 copy_offset: 246896 seen_offset: 246896 copy_size: 4096 flags: Client #22, 0x171c800 copy_offset: 246896 seen_offset: 246896 copy_size: 4096 flags: Client #23, 0x1d98000 copy_offset: 246896 seen_offset: 246896 copy_size: 4096 flags: Client #24, 0xd04c00 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #25, 0x14ee000 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #26, 0xb3b800 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #27, 0x52cc00 copy_offset: 263280 seen_offset: 263280 copy_size: 4096 flags: Client #28, 0x524800 copy_offset: 246896 seen_offset: 246896 copy_size: 4096 flags: Client #29, 0xa34c00 copy_offset: 4062759 seen_offset: 4062759 copy_size: 4096 flags: Client #30, 0xa34800 copy_offset: 246896 seen_offset: 246896 copy_size: 4096 flags: Client #31, 0x172e400 copy_offset: 246896 seen_offset: 246896 copy_size: 4096 flags: Client #32, 0x16f0400 copy_offset: 246896 seen_offset: 246896 copy_size: 4096 flags:Received on Sat Mar 16 2002 - 04:57:29 MST
This archive was generated by hypermail pre-2.1.9 : Tue Dec 09 2003 - 17:06:57 MST