Re: [squid-users] Frequent cache rebuilding

Matus UHLAR - fantomas wrote:
> On 21.01.09 10:05, Andreev Nikita wrote:
>
>> Squid cache resides on NFS partition on storage system. It is 1Gb link
>> and both sides are connected to the same switch. Squid has dual core
>> Xeon processor and 4GB of RAM.
>>
>> The main concern here is that squid always eats 100% of 1 core. And
>> our clients can't reach full channel throughput (4Gbs). As I can see
>> outside link is half full. Secondly it looks like FS performance is
>> very poor. I tried to clear cache by setting
>> cache_swap_low 0
>> cache_swap_high 0
>> and it took about 15 hours for squid to actually clear the cache!
>>

Yeah. Deleting a million objects over NFS is going to take a while.

>> Why does squid eat 100% of processor if the problem is in FS?

How is your cache_dir defined? aufs (in general) is a better choice
than ufs, diskd might still have some stability issues under load, and
coss is a good supplement as a small object cache. Conceivably if Squid
is set up with a ufs cache_dir mounted as NFS, it's spending a lot of
time in a wait state, blocked while the I/O completes.

>> Maybe it's not an FS problem at all?

I have to agree that using a cache_dir mounted as NFS is (at the very
least) part of the problem...

>> What can I do to find the performance bottleneck?
>>

Use profiling tools like iostat, vmstat and top.

>
> NFS is the bottleneck. Can you connect the disk on storage system directly?
>

Or at least use AoE* or iSCSI... NFS is convenient, but far from ideal
for a Squid cache. AoE and iSCSI are better**, but not neither will
perform as well as direct attached storage. The less latency between
Squid and the cache data, the better.

> btw, which link is 1 gbit and which is 4 gbit? NFS is connected with 1gbit
> and you want squid to be able to saturate 4gbit connection?
>

Can one Squid process even saturate a 4gbit link? In this case, given
the number of clients (~216) and the average number of requests per
minute (432.5) I'd have to guess we are talking about a 4mbit internet
connection.

 From the cache_info dump provided, DNS lookups are taking nearly 200
ms. That's pretty slow too. Then again, I suppose if the Squid server
only has one Ethernet interface and that is saturated with NFS, DNS
queries are going to suffer.

Chris

*ATA over Ethernet (http://en.wikipedia.org/wiki/ATA_over_Ethernet)
**Just a gut feeling with no benchmarks to back it up, but I would think
that AoE would be better than iSCSI, as there is less protocol overhead.
Received on Wed Jan 21 2009 - 20:26:52 MST