[squid-users] Fastest accelerator setup I could come up with

Hi all,

after I tortured this list for hints on tuning Squid in an accelerator
set up I did some testing and now like to share what came out of it. The
following proved to be the most reliable and fasted set up. Also it goes
easy on the resources.
I haven't done any benchmarking so I can't give you maximum numbers.
Since we use two siblings in our production environment I was able to
compare directly any changes with our old set up by tuning only one of
the two siblings and leaving the other as it was.
I was able to lower most of the performance indicators like CPU, I/O by
30 - 40% with the below set up.
If you have any questions please feel free to ask. I'm sure I forgot
something here.

First our content. We serve only pictures, lots of them. Here is the
read histogram as the cachemgr shows it:
    1- 1: 947 0%
    2- 2: 880 0%
    3- 4: 1705 0%
    5- 8: 2964 0%
    9- 16: 5238 0%
   17- 32: 13366 0%
   33- 64: 25554 0%
   65- 128: 49637 0%
  129- 256: 958687 1%
  257- 512: 26930299 38%
  513- 1024: 1094073 2%
 1025- 2048: 14418961 20%
 2049- 4096: 19701586 28%
 4097- 8192: 3839852 5%
 8193-16384: 2841771 4%
16385-32768: 791308 1%

As you can see we have to handle mostly very, very small files which is
a real pain in the a**. So COSS was my white knight to handle this.

Although we don't use CARP we made sure that the proxies always get the
same requests. We have at our prime time about 40 MBit/s outgoing
traffic which makes about 1000 requests per second.
Also I should mention that we use a standard Debian kernel with no
tuning in any kernel parameters except the following two:
net.ipv4.tcp_max_syn_backlog = 4096
vm.swappiness = 0

Now some numbers...

Hardware
======
CPU: Intel Xeon 2GHz (dual quad-core)
RAM: 16 GB
Disk: 4x 136 GB

Configuration
=========
Squid can use the maximum number of file descriptors: 65536
I left out the ACLs we use and stuff like purging and cachemgr access.
Any configuration items not mentioned here have their default values.
All new options I mark with '->' at the beginning of the line:

# disable all the proxy networking stuff, we are standalone
htcp_port 0
announce_port 0
icp_port 0

# we are a accelerator so listen on standard http port
# IP is that of img5
http_port 80 accel vport defaultsite=img5

# define the backend servers and balance between them
cache_peer ws1 parent 80 0 no-query no-digest originserver round-robin
name=s1
cache_peer ws2 parent 80 0 no-query no-digest originserver round-robin
name=s2
# our sibling for Cache Digest exchange
cache_peer wi10 sibling 80 0 proxy-only name=i10
# this comes from ICP times and I'm not sure if it's still needed with
Cache Digest
acl sibling10 src i10
miss_access deny sibling10

# lowers the response time dramatically
#
http://wiki.squid-cache.org/Features/CollapsedForwarding?highlight=(collapsed)
collapsed_forwarding on

# we have 16GB and use 4 as cache. This is a good trade-off between FS cache
# and Squids memory usage.
-> cache_mem 4096 MB

### Replacement policies
# mem - Heap GSDF: fast and small objects preferred. Better to have more
# small objects in memory than to read them from disk.
-> memory_replacement_policy heap GDSF
# disk - Heap LFUDA: fast and more popular pictures, more "hot" content,
# is preferred.
-> cache_replacement_policy heap LFUDA

# This seems to help speed up things too.
-> quick_abort_min 0 KB
-> quick_abort_max 0 KB
-> quick_abort_pct 100

# we don't need detailed client information
-> client_db off

# close connections fast
-> half_closed_clients off

### Cache directories
# COSS stuff, ca. 16 GB in size, maximum file size is 24kB
-> cache_dir coss /web/cache/1/coss 16373 max-size=24576 block-size=1024
-> cache_dir coss /web/cache/2/coss 16373 max-size=24576 block-size=1024
-> cache_dir coss /web/cache/3/coss 16373 max-size=24576 block-size=1024
-> cache_dir coss /web/cache/4/coss 16373 max-size=24576 block-size=1024
# using async-io on ca. 60 GB
cache_dir aufs /web/cache/1/aufs 61440 290 256 min-size=24576
cache_dir aufs /web/cache/2/aufs 61440 290 256 min-size=24576
cache_dir aufs /web/cache/3/aufs 61440 290 256 min-size=24576
cache_dir aufs /web/cache/4/aufs 61440 290 256 min-size=24576

# since our content changes very fast we rebuild the digest every ten
minutes
digest_rebuild_period 10 minutes
digest_rewrite_period 10 minutes

# don't send the version string
httpd_suppress_version_string on

## LOGGGING STUFF
# don't need the store log
cache_store_log none
# define access.log
#access_log /var/log/squid/access.log

### INCLUDES
include /etc/squid/conf.d/*.conf

Thanks again for all the help,

Markus
Received on Thu Feb 25 2010 - 10:43:14 MST