Memory Usage with max-object-size (was Re: Download hanging)

On Tue, 8 Aug 2006, Henrik Nordstrom wrote:

> tis 2006-08-08 klockan 07:51 +0800 skrev Steven:
>
> Not sure. I think COSS wants the object to be spooled to the cache_dir
> in one chunk, not small pieces at a time over an extended period of
> time.
>
> But if COSS can handle an object being swapped out slowly then sure.

At a glance, COSS would need to wipe a half-downloaded object from memory
on an abort to avoid problems when rebuilding the cache from disk. Other
than that, the code does appear to accept objects slowly.

The main test would be how much less efficient squid would be using
the COSS stripe buffers rather than the existing per-object buffer. This
would need to be weighed against any problems caused. I think that COSS
will always benefit from the current code, but if someone set
max-object-size to 10MB for a non-COSS cache_dir, the current buffering
behaviour could have bad side-effects. This is worth knowing in case it
comes up in a future bug report.

I've developed another patch to COSS that would add another
advantage to the slow store approach. This new patch allows the
administrator to set a maximum number of stripes that a given COSS
cache_dir will hold in memory. This would allow slightly better control
over how much memory a squid process could grow to. Once again, this is
good to know if a problem does arise.

If you think the above possibilities are worth considering, I'm happy to
modify and test COSS using a slow store method.

> > If e->mem_obj->reply->content_length is > 0, I don;t believe that we need
> > to wait for the object to grow too large before we can make a decision on
> > where to store it.
>
> Agreed. But that part of the code needs to be audited for the change.

If the check needs to stay, what about making it the following code?

if ((e->store_status != STORE_OK) && (swapout_size < store_maxobjsize) &&
(e->mem_obj->reply->content_length < store_maxobjsize)) {

This way objects that we know are too big will bypass this mechanism
sooner.

Steven
Received on Tue Aug 08 2006 - 05:56:13 MDT