Re: [squid-users] Squid and Squidguard using high disk IO

Thanks so much for all the advise and responses :-)

I decided to try Dansguardian.........

Currently I have a working model setup though it needs a bit of tuning
and tweaking but good news is that I am using the SquidGuard blacklists
so all is pretty much good!!

Have been testing; performance is phenomenal though sometimes when Squid
can't connect to a site properly in order to populate the cache etc...
the pages might need a bit of refreshing however, I consider those as
just teething problems.

So yeah.... NET <- NAT <- <Squid + c-icap + Clamd> <- Dansguardian <- PF
is how things look like now :-)

Regards,

Kaya

On 11/09/2013 10:37 PM, Marcus Kool wrote:
> On Sat, Nov 09, 2013 at 11:16:12PM +0100, Loïc BLOT wrote:
>> Hello Kaya,
>> first, don't forget to look at sysctl kern.maxfiles values.
>> Also improve daemon FD values in login.conf for squid. Don't forget each
>> connection is a FD (1 connection for the client, 1 for the transaction
>> to remote site, somes for the caching).
>>
>> Also to improve performances of squidguard, i stored all blacklists DB
>> to a memory fs (mfs) this improve massively squidguard performance
> If the disk I/O is really the bottleneck, consider ufdbGuard.
> ufdbGuard loads the URL database in memory and easily does
> 25,000 URL lookups/sec, much more than you will ever need.
>
> Marcus
>
>> I have wrote an article to improve squid perfs on OpenBSD:
>> http://www.unix-experience.fr/2013/monter-un-proxy-cache-performant-avec-squid-et-openbsd/
>>
>>
>>
>> --
>> Best regards,
>> Loïc BLOT,
>> UNIX systems, security and network engineer
>> http://www.unix-experience.fr
>>
>>
>>
>> Le samedi 09 novembre 2013 à 19:39 +0000, Kaya Saman a écrit :
>>> Just found this is Squid cache log:
>>>
>>> 2013/11/09 19:28:25 kid1| /var/squid/cache/04/7A: (24) Too many open files
>>> 2013/11/09 19:31:31 kid1| WARNING: All 20/20 redirector processes are busy.
>>> 2013/11/09 19:31:31 kid1| WARNING: 20 pending requests queued
>>> 2013/11/09 19:31:31 kid1| WARNING: Consider increasing the number of
>>> redirector processes in your config file.
>>>
>>>
>>> The cache size is 2GB.... though that shouldn't affect performance as
>>> far as I understand.
>>>
>>> On 11/09/2013 05:23 PM, Eliezer Croitoru wrote:
>>>> Hey,
>>>>
>>>> Notes inside.
>>>>
>>>> On 11/09/2013 05:58 PM, Kaya Saman wrote:
>>>>> What can I do to improve performance with this?
>>>>>
>>>>>
>>>>> Is this line too high: url_rewrite_children 500
>>>> YES!!
>>>>
>>>>> or simply have a misconfigured something?
>>>>
>>>>> I additionally have 'c-icap' running with squidclamav coupled to clamd
>>>>> in case that is of importance - not using the squidGuard line in the
>>>>> squidclamav.conf file!!!
>>>>>
>>>>> Basically how can I get the IO usage down and get the system to work
>>>>> again?
>>>> For how many users exactly?
>>>> Just a note that I am not in a favor of any OS by default but I would
>>>> feel better Using Linux.
>>>>
>>>>> - the logs don't indicate anything outside of 'starting squidGuard
>>>>> process' many times.
>>>> The basic assumption of using 500 child process is that you have
>>>> atleast 100 CPUs.
>>>> SquidGuard was design for performance which is lots of urls per sec.
>>>> It can be tested just to clear the point out.
>>>> for example in a rate of 1500k requests per second you should not have
>>>> a need in more then 40-50 children.
>>>> In practice it works a bit different speed since there is a speed
>>>> limit on STDIN and STDOUT which slows down the speed of squid and
>>>> squidguard communication blocking the whole squid instance(in a way).
>>>>
>>>> If you need basic url filtering you can use ICAP which has an option
>>>> to run as a standalone service outside of squid settings and machine.
>>>>
>>>> I have written in the past a small ICAP service for the favor of
>>>> requests manipulation and filtering.
>>>> I have never finished it in a level I was happy with but the basic
>>>> code can be seen here:
>>>> https://github.com/elico/echelon
>>>>
>>>> I know for a fact that ICAP interface adds concurrency by the "nature"
>>>> of it using TCP.
>>>>
>>>> This is not the place to ask about concurrency in squidguard which can
>>>> allow the usage of square less processes(children) for more requests.
>>>>
>>>> In order to find the right number of children start with 40 and see if
>>>> it fits you and then see what is the bottle neck in the whole setup.
>>>>
>>>> Eliezer
>
Received on Sat Nov 09 2013 - 22:46:37 MST