Re: Squid-3 and large response headers

On Tue, 2008-01-08 at 03:47 +0100, Henrik Nordström wrote:
> Squid-3 has lots of problems with response headers >4KB. (Bug #2001)
>
> 3.0.STABLE "hangs" and stops reading the response. Cleared by abort or
> timeout only.
>
> 3.HEAD at least grows it's header receive buffer and continues to read
> the data and feed the response down the chain, but this then triggers
> problems in other areas including an assertion failure..
>
> Best would be if we could stop sending the response headers completely
> within the data stream and only parse it once, but that's possibly a
> little too big change for 3.0 and more a 3.1 target.
>
> What I am mostly wondering is how do the buffering work with ICAP? Lets
> say there is a 64KB response header. What buffers would need to be
> adjusted to make this fit?
>
> I already know the "client_side" is a big mess with 4KB hardcoded little
> everywhere, but it's till familiar enough to unwind. But the data flows
> in the ICAP Part is still a bit of unchartered land for me..

In ICAP, header parsing (first ICAP, then HTTP) is done in
ICAPModXact::parseHead(). It uses HttpMsg::parse() method.

The ICAPXaction::readBuf data member stores incoming headers. It is
initialized as follows:

  readBuf.init(SQUID_TCP_SO_RCVBUF, SQUID_TCP_SO_RCVBUF);

Also, keep in mind that ICAP clones HTTP messages in
ICAPModXact::prepEchoing(). This is currently done by writing the
message header into a temporary MemBuf buffer and reading it back(*).
That buffer is initialized as follows:

    httpBuf.init();

Similar default MemBuf::init() calls are used to initialize buffers for
writing ICAP requests to the network.

Overall, it seems that if Squid core can cope with large headers, ICAP
should be able to do the same without major changes because it is
reusing core facilities for parsing. Buffer sizes may need to be
adjusted.

Alex.
(*) The current cloning method is far from ideal from the performance
point of view, of course.
Received on Tue Jan 08 2008 - 23:40:08 MST