Err, my mail client ate my formatting -- that refresh_pattern is all
on one line in my squid.conf file.
Also, FYI, these logs are all for squid straight from the ubuntu
repos, but I saw the same problem when compiling from source with the
latest stable version (3.1.10).
On Thu, Dec 30, 2010 at 1:37 PM, Chas Williams <chas_at_seomoz.org> wrote:
> Hi Curl,
>
> I'm trying configure squid to cache everything for a minimum
> amount of time, no matter what (I release this is a bit of a
> bastardization of it's purpose and of the http spec -- I'm trying to
> use it as a cache for another program I'm working on, in a space where
> this is reasonable).
>
> I can't seem to make negative caching work -- below are (1) the
> send and receive headers from a curl request for a URL that 404s,
> twice (to show that the second is a cache miss), (2) version
> information (of squid and my box), (3) my squid.conf file, and (4) my
> access, cache, and store logs. These are all from a fresh install on a
> new box.
>
> Any ideas?
>
> Thanks,
>
> Chas Williams
>
> P.S. for bonus points, can anyone tell me if my squid.conf will really
> cache everything for at least 240 minutes?
>
> ----------------------------------------------------------------------------------------------------------------------------
> ubuntu_at_domU-12-31-39-0C-81-A1:~$ curl -x localhost:3128 -v
> "http://hotelburst.es/robots.txt" >/dev/null
> * About to connect() to proxy localhost port 3128 (#0)
> * Trying 127.0.0.1... % Total % Received % Xferd Average Speed
> Time Time Time Current
> Dload Upload Total Spent Left Speed
> 0 0 0 0 0 0 0 0 --:--:-- --:--:--
> --:--:-- 0connected
> * Connected to localhost (127.0.0.1) port 3128 (#0)
>> GET http://hotelburst.es/robots.txt%a0HTTP/1.1
>> User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
>> Host: hotelburst.es
>> Accept: */*
>> Proxy-Connection: Keep-Alive
>>
> * HTTP 1.0, assume close after body
> < HTTP/1.0 404 Not Found
> < Date: Thu, 30 Dec 2010 21:18:14 GMT
> < Server: Apache/2.2.3 (Debian) mod_jk/1.2.18 mod_python/3.2.10
> Python/2.4.4 PHP/5.2.0-8+etch15 mod_ssl/2.2.3 OpenSSL/0.9.8c
> mod_perl/2.0.2 Perl/v5.8.8
> < Content-Length: 406
> < Content-Type: text/html; charset=iso-8859-1
> < X-Cache: MISS from localhost
> < X-Cache-Lookup: MISS from localhost:3128
> < Via: 1.0 localhost (squid/3.1.6)
> * HTTP/1.0 proxy connection set to keep alive!
> < Proxy-Connection: keep-alive
> <
> { [data not shown]
> 100 406 100 406 0 0 1303 0 --:--:-- --:--:--
> --:--:-- 1309* Connection #0 to host localhost left intact
> * Closing connection #0
> ubuntu_at_domU-12-31-39-0C-81-A1:~$ curl -x localhost:3128 -v
> "http://hotelburst.es/robots.txt" >/dev/null
> * About to connect() to proxy localhost port 3128 (#0)
> * Trying 127.0.0.1... % Total % Received % Xferd Average Speed
> Time Time Time Current
> Dload Upload Total Spent Left Speed
> 0 0 0 0 0 0 0 0 --:--:-- --:--:--
> --:--:-- 0connected
> * Connected to localhost (127.0.0.1) port 3128 (#0)
>> GET http://hotelburst.es/robots.txt%a0HTTP/1.1
>> User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
>> Host: hotelburst.es
>> Accept: */*
>> Proxy-Connection: Keep-Alive
>>
> * HTTP 1.0, assume close after body
> < HTTP/1.0 404 Not Found
> < Date: Thu, 30 Dec 2010 21:18:16 GMT
> < Server: Apache/2.2.3 (Debian) mod_jk/1.2.18 mod_python/3.2.10
> Python/2.4.4 PHP/5.2.0-8+etch15 mod_ssl/2.2.3 OpenSSL/0.9.8c
> mod_perl/2.0.2 Perl/v5.8.8
> < Content-Length: 406
> < Content-Type: text/html; charset=iso-8859-1
> < X-Cache: MISS from localhost
> < X-Cache-Lookup: HIT from localhost:3128
> < Via: 1.0 localhost (squid/3.1.6)
> * HTTP/1.0 proxy connection set to keep alive!
> < Proxy-Connection: keep-alive
> <
> { [data not shown]
> 100 406 100 406 0 0 3992 0 --:--:-- --:--:--
> --:--:-- 4060* Connection #0 to host localhost left intact
> * Closing connection #0
> ---------------------------------------------------------------------------------------------------------------
> ubuntu_at_domU-12-31-39-0C-81-A1:~$ squid3 -v
> Squid Cache: Version 3.1.6
> configure options: '--build=x86_64-linux-gnu' '--prefix=/usr'
> '--includedir=${prefix}/include' '--mandir=${prefix}/share/man'
> '--infodir=${prefix}/share/info' '--sysconfdir=/etc'
> '--localstatedir=/var' '--libexecdir=${prefix}/lib/squid3'
> '--disable-maintainer-mode' '--disable-dependency-tracking'
> '--disable-silent-rules' '--srcdir=.' '--datadir=/usr/share/squid3'
> '--sysconfdir=/etc/squid3' '--mandir=/usr/share/man'
> '--with-cppunit-basedir=/usr' '--enable-inline' '--enable-async-io=8'
> '--enable-storeio=ufs,aufs,diskd' '--enable-removal-policies=lru,heap'
> '--enable-delay-pools' '--enable-cache-digests' '--enable-underscores'
> '--enable-icap-client' '--enable-follow-x-forwarded-for'
> '--enable-auth=basic,digest,ntlm,negotiate'
> '--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SASL,SMB,YP,DB,POP3,getpwnam,squid_radius_auth,multi-domain-NTLM'
> '--enable-ntlm-auth-helpers=smb_lm,'
> '--enable-digest-auth-helpers=ldap,password'
> '--enable-negotiate-auth-helpers=squid_kerb_auth'
> '--enable-external-acl-helpers=ip_user,ldap_group,session,unix_group,wbinfo_group'
> '--enable-arp-acl' '--enable-esi' '--disable-translation'
> '--with-logdir=/var/log/squid3' '--with-pidfile=/var/run/squid3.pid'
> '--with-filedescriptors=65536' '--with-large-files'
> '--with-default-user=proxy' '--enable-linux-netfilter'
> 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -g -Wall -O2'
> 'LDFLAGS=-Wl,-Bsymbolic-functions' 'CPPFLAGS=' 'CXXFLAGS=-g -O2 -g
> -Wall -O2' --with-squid=/build/buildd/squid3-3.1.6
>
> ubuntu_at_domU-12-31-39-0C-81-A1:~$ cat /etc/issue
> Ubuntu 10.10 \n \l
> ---------------------------------------------------------------------------------------------------------------
> ubuntu_at_domU-12-31-39-0C-81-A1:~$ sudo cat /etc/squid3/squid.conf
> ######################################
> # Squid3 config for crawl v2 machines
> ######################################
> #Access control
> acl manager proto cache_object
> acl localhost src 127.0.0.1/32
> acl to_localhost dst 127.0.0.0/8
> acl SSL_ports port 443
> acl Safe_ports port 80 # http
> acl CONNECT method CONNECT
> http_access allow manager localhost
> http_access deny manager
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
> http_access allow localhost
> http_access deny all
> icp_access deny all
> htcp_access deny all
> http_port 3128
> #Force minimum cache time for everything
> refresh_pattern . 240 999999% 1440 override-expire override-lastmod
> ignore-no-cache ignore-no-store ignore-private ignore-must-revalidate
> #Cache 404s as well
> negative_ttl 240 minutes
> #No neighbor caches
> icp_port 0
> cache_mem 256 MB
> cache_dir ufs /mnt/cache 1000 16 256
> access_log /var/log/squid3/access.log squid
> cache_log /var/log/squid3/cache.log
> cache_store_log /var/log/squid3/store.log
> coredump_dir /var/spool/squid3
>
> ---------------------------------------------------------------------------------------------------------------
>
> ubuntu_at_domU-12-31-39-0C-81-A1:~$ sudo cat /var/log/squid3/access.log
> 1293743891.380 309 127.0.0.1 TCP_MISS/404 822 GET
> http://hotelburst.es/robots.txt%a0- DIRECT/85.25.120.108 text/html
> 1293743893.629 99 127.0.0.1 TCP_MISS/404 821 GET
> http://hotelburst.es/robots.txt%a0- DIRECT/85.25.120.108 text/html
>
> ubuntu_at_domU-12-31-39-0C-81-A1:~$ sudo cat /var/log/squid3/cache.log
> 2010/12/30 21:17:45| 0 Objects expired.
> 2010/12/30 21:17:45| 0 Objects cancelled.
> 2010/12/30 21:17:45| 0 Duplicate URLs purged.
> 2010/12/30 21:17:45| 0 Swapfile clashes avoided.
> 2010/12/30 21:17:45| Took 0.01 seconds ( 0.00 objects/sec).
> 2010/12/30 21:17:45| Beginning Validation Procedure
> 2010/12/30 21:17:45| Completed Validation Procedure
> 2010/12/30 21:17:45| Validated 25 Entries
> 2010/12/30 21:17:45| store_swap_size = 0
> 2010/12/30 21:17:46| storeLateRelease: released 0 objects
>
> ubuntu_at_domU-12-31-39-0C-81-A1:~$ sudo cat /var/log/squid3/store.log
> 1293743893.629 RELEASE -1 FFFFFFFF 9D744644E77923213EBD01054E48C903
> 404 1293743894 -1 -1 text/html 406/406 GET
> http://hotelburst.es/robots.txt
> ---------------------------------------------------------------------------------------------------------------
>
-- Chas Williams SEOmozReceived on Thu Dec 30 2010 - 21:40:46 MST
This archive was generated by hypermail 2.2.0 : Fri Dec 31 2010 - 12:00:04 MST