[squid-users] Resolved: Re: [squid-users] Time-based oddity that I can't quite nail down...

On Thu, Nov 12, 2009 at 17:49, Amos Jeffries <squid3_at_treenet.co.nz> wrote:
> Kurt Buff wrote:
> <snip>
>>
>>> By testing by running a browser on the proxy server itself you can
>>> identify if the problem is Squid or something outside Squid.
>>
>> Unless you think lynx (or some other text-based browser, or using
>> wget/curl/other) is sufficient, I am leery of doing this. I'd have to
>> install all the bits to do this, including X and a browser, and all
>> that. I'm all up for learning, though, so your suggestion on how to do
>> this is very warmly welcomed.
>
> The squidclient tool fits into the niche gap between telnet and lynx.
> Allowing a simple input of the request URL and optional other details and
> producing a dump of the results. It has no large dependencies.
>
> To test through the proxy:
>  squidclient http://example.com/
>
> To test without proxy:
>  squidclient -h example.com -p 80 /
>
>
>>
>> I was unable to do any testing in the past few days - a new child is
>> taking me away from working longer hours...
>>
>> I hope to do some work on it this weekend.
>>
>> Kurt
>
> Congratulations!
>
> Amos

Sorry for the very late followup, but I thought I'd outline what happened:

For diagnostics, I finally ended up using cURL on three different
FreeBSD boxes: One is the router for our DS3 and outside our firewall,
and two are internal (used for various network monitoring/management
functions) with one of those using our Active Directory DNS servers,
and the other using our ISPs DNS servers. This allowed me to test
several things at once - inside firewall, outside firewall, internal
DNS servers used and external DNS servers used.

I selected from the squid logs a set of around 2100 URLs from a single
day (basically, any url with 'www' in it), and pared them down to the
bare http://FQDN url, and used those as a comparison set across all
three machines for multiple runs.

Several nights (and days!) I ran the following command at the same
time (using at(1)) on each machine:

     /usr/local/bin/curl -K /root/urls.txt >> /root/DATE-curl-MachineName.out

Each entry in urls.txt file looks like the following:

     url = "http://h30333.www3.hp.com"
     -s
     -w = "%{url_effective}\t%{time_total}\t%{time_namelookup}\n"
     -o = /dev/null

I narrowed it down to that set of options, because I quickly figured
out that it wasn't squid at issue, after using it as a proxy in
urls.txt with the '-x' option.

I had suspected it was the firewall and its name resolution times,
which is why I included the '\t%{time_namelookup}' parameter, but that
was not the issue.

So, to end the suspense - it was the firewall's http(s) proxy, with
off-hours results for cURL behind the firewall at over 20 seconds
average page load time across the URL set, with some taking as long as
360+ seconds. I did no analysis of which pages were worst, nor why.

Applying patches to bring it current for the particular rev of the
software fixed the problem completely.

It was a weird and frustrating problem, but I feel pretty good about solving it.

Kurt
Received on Thu Mar 18 2010 - 22:55:18 MDT