Re: squid-2.5.STABLE5 WCCPv2 Layer2 redirect

On Thursday, April 01, 2004 12:44 +0200
Henrik Nordstrom <hno@squid-cache.org> wrote:

|> There is a meeting today during which it will be clarified that the
|> University does not have any issue with surrender of its intellectual
|> property rights to the open source community; then we will roll our
|> changes in to the latest available wccpv2 patch and submit a diff for
|> your consideration.
|
| Sounds great!

Well, it's been a while but prompted into further action by requests for
more information from Michael Dunnell <mdunnell@cox.net> and Simon
Thoustrup <simon@thoustrup.dk> we have now put our changes together for
your consideration.

This is a patch to enable WCCPv2 Layer2 redirection on squid. The patch was
made for 2.5.STABLE5 but 'patch' offsets the hunks and it does apply
cleanly to 2.5.STABLE7 which we have then successfully built and run.

This patch is based on a combination of the other wccpv2 patches that
already exist, particularly (IIRC) the visolve-WCCPv2 patch that is
referenced at squid.org. The tricky bits like actually making the code
changes, extensions and getting it to work for us are the work of James
Goodlet.

The rest of this message body consists of notes relating to the required
router configuration and squid.conf settings. The attachments are
'squid.INSTALL.USCS' which are the local build instructions that we use
and, of course, 'wccpv2-L2.patch' which is the patch itself.

Notes:

Our patch has been tested against IOS Version 12.1(20)E3 running natively
on a Cisco Catalyst 6509 with Supervisor 2 and MSFC2 and PFC2 daughtercards
(and a Switch Fabric Module 2 (SFM2)).

All interfaces on the Cisco which are to participate in WCCP *must* be
configured to use "cef", e.g. individual interfaces or vlans must not use
the "no ip cef" directive. If CEF is disabled, some redirected packets
will be corrupted under load.

Redirection interception is enabled using the directive:

    ip wccp web-cache redirect out

in the interface or vlan for the upstream connection from this router.
Contrary to what we've read elsewhere, there does not appear to be a
problem using wccp V2 redirection in either "in" or "out" interception
modes: we've used both, and have run with "out" capturing about 50% of
traffic from our (6,000 machine) campus network for 6 months now.

WCCP V2 is enabled on the router using the command:

   ip wccp web-cache redirect-list 150

NB this version of IOS defaults to V2. Redirect list 150 (an IP extended
access list) should define which traffic is to be intercepted, and
crucially which should be allowed to pass. The latter set must include the
IP addresses used by your squid servers to talk to the outside world. For
example, if your site network is 192.0.34.0/23, and your two squid servers
are 192.0.34.100 and 192.0.34.101, then ACL 150 should minimally read:

   access-list 150 deny ip host 192.0.34.100 any
   access-list 150 deny ip host 192.0.34.101 any
   access-list 150 permit ip 192.0.34.0 0.0.1.255 any

The squid servers must be layer 2 reachable from the Cisco (this is L2
redirection, of course). For us, this means that the vlan our squid
servers sit on must have its active router end point on this Cisco, and the
traffic on the vlan actually gets to this Cisco over some 802.1q trunks
from elsewhere on our network. Yes, this isn't very efficient, but we want
WCCP for simplicity, not efficiency.

In squid, the directive to enable WCCPv2 and point the cache advertisements
to the Cisco is done using the single directive:

        wccp2_router 192.0.34.1

where 192.0.34.1 is the primary IP address of the Cisco on the interface or
vlan shared by the squid servers and the Cisco router (i.e. the interface
through which squid would normally talk to the router). Note *primary* IP
address. We run Cisco's Hot Standby Routing Protocol (HSRP) on our
interfaces, and this is not compatible with this implementation of WCCP for
a number reasons. The main reason is that it isn't possible to configure
squid to talk to the "standby ip" address, the token address which moves
between the participating routers in the standby arrangement. The router
sees the HERE_I_AM arrive from the cache, but the cache does not receive
the I_SEE_YOU response: we haven't looked very deeply into this, but I
suspect its down to a bug in the Cisco. This means it is necessary to use
the actual address assigned to the interface rather than the standby ip
address. So if the interface were configured as so:

interface vlan 1
        ip address 192.0.34.2 255.255.255.0
        standby ip address 192.0.34.1

whilst normal traffic on this interface should be routed toward the standby
ip address 192.0.34.1, squid would have to be configured to talk directly
to the primary ip address on the interface, 192.0.34.2. If this interface
is disabled, 192.0.34.1 will move to the standby partner, but wccp will
break since squid can no longer talk to 192.0.34.2.

Of course, wccp is incompatible with HSRP (or VRRP, etc) for other reasons
as well: this WCCPv2 only supports one router, and having that router
change without squid's knowledge would be unpredictable at best.
Supporting more than one router would be the starting point to redundant
routing working with wccp.

-----

Other things of note:

1. Much more of the WCCPv2 packets are parsed than is necessary for this
patch. We went about the task of laying a structure over the top of the
inbound and outbound packets pretty much in a straight line without
thinking too hard about what we needed to read and write. We've left this
baggage intact in the patch in case its useful for someone else to extend
the support for WCCPv2.

2. Some of the options need to be parameterised and put into the
configuration file. For instance, this patch will break support for GRE
redirection using WCCPv2. That's not cool. We looked at how to extend the
configuration file to support the options that should be changeable, but
reckoned we'd take a *lot* longer to do that badly than someone else more
familiar with the code. Sorry.

4. Some of the debugging messages are a bit misleading, particularly the
"cache count change" message, which should really be suppressed when the
old cache count is the same as the new cache count (on the designated cache
for WCCPv2) or suppressed entirely on a cache which is not currently
designated (those messages which say something like "existing number of
caches = 0, new number of caches = N" on caches which are not designated
simply mean "this cache is not designated"). Much too chatty, far too
misleading.

6. We've tried to patch the automake and autoconf files as well, but we
don't (knowingly) use them, so we could have goofed there.

7. Cache shutdown still isn't as smooth as we'd like it. We moved the WCCP
connection close to be the first action, before client connection closure,
during shutdown and reconfigure in squid. This was with the intention of
getting squid to remove itself from the cache engine set on the router
*before* it stopped processing requests, since otherwise a set of http
requests would discarded for a period of some 10s of seconds. This
modification isn't good enough, and I think either wccp2ConnectionShutdown
or wccp2ConnectionClose should be modified to send a HERE_I_AM with a
command extension giving the WCCP2_COMMAND_TYPE_SHUTDOWN command to the
router (informing the router that this cache engine is shutting down and
should no longer receive any redirected traffic. If no-one else takes this
up, we'll probably have a go ourselves, since much of our interest in WCCP
stems from wanting to relieve our users of the trouble of even knowing that
we use cache engines.

So... there it is. It works for us. It could stand some polishing but it
does the job that it was intended to. I hope that it makes it into the main
squid branch so that we can continue to use it in the future without all
the manual mods.

-- 
Mark Foster, IT Services, University of Sussex, Falmer, UK. BN1 9QJ
E-mail: m.e.foster@sussex.ac.uk - VOICE/FAX: +44 1273 678902/873135
PGP Fingerprint: 3342 C02C 7BE8 3FE4 AAC5  8BB2 03B7 9263 DDF2 04C1
        --------------------------------------------------
"There are no such useless words as...'I didn't have a chance.'"
                                                    [Driving, HMSO]