Re: efficient acl filtering

From: Kevin Fink <kevin@dont-contact.us>
Date: Fri, 12 Sep 1997 09:53:44 -0700 (PDT)

Squid is not well-designed for filtering large lists of URLs. We had to
completely rewrite the ACL code in order to get acceptable performance.
Some numbers, to establish the baseline:

We use several different lists, but the total number of lines is 254,710.
So we have several more than 5,000...

Our standard servers are Pentium 133s with 128 MB of RAM and 1 GB of disk
space. We have also run P100s, P200s, dual P133s, and Alphas (433 MHz, I
think - the 500 chip was backordered...).

There are two reasons why your system could be slow. 1) The filtering
could be slowing it down. 2) You may not have enough memory.

Have you tried running it without the lists in place? Is performance much
better in that case?

If it is the lists, how much memory do they take up? Notice that if you
are using url_regex that it precompiles the URLs, eating an awful lot of
memory. The filtering list may be pushing you into using swap, which will
definitely destroy your performance.

I suspect that the problem isn't so much the speed of the filtering, as
5,000 lines of ACLs really isn't very many. I suspect that the additional
memory requirements of the filter lists are pushing you into using swap.

If I am wrong, and the server is not swapping, then you will have to do
what we did - completely rewrite the ACL code. Or you could just contract
with us to run your filtering for you... :-)

Kevin

On Fri, 12 Sep 1997, Gustaf Bjorksten wrote:

> We have a proxy server (squid 1.1.16) that we use to filter URLs. The
> acl list has approx. 5000 acls in it.
>
> The machine is an intel P100 with 128mb RAM and 5gb of disk space and
> although it is dedicated to the filtering task it seems to run very
> slowly.
>
> We have tried recompiling squid with the b-tree option but this seems to
> make little difference.
>
> Does anybody out there have experience with running squid efficiently
> with a large list of acls? Is there a better way for us to do this
> (perhaps using an external redirector or recompiling with other
> options?)?
>
> Thank you,
>
> Gus
>

------------------------------------------------------------------------------
 Kevin Fink <kevin@fink.com> N2H2, Creators of Bess
 http://www.fink.com/ 1301 Fifth Avenue, Suite 1501
 http://www.n2h2.com/ Seattle, WA 98101
------------------------------------------------------------------------------
 (206) 971-1400 VOICE (206) 971-1460 FAX (206) 680-7666 PAGER
------------------------------------------------------------------------------
Received on Fri Sep 12 1997 - 10:03:57 MDT

This archive was generated by hypermail pre-2.1.9 : Tue Dec 09 2003 - 16:37:01 MST