RE: Storing partial responses

> -----Original Message-----
> From: Joe Cooper [mailto:joe@swelltech.com]
> Sent: Wednesday, 13 December 2000 4:59 PM
> To: Alex Rousskov
> Cc: Robert Collins; squid-dev@squid-cache.org
> Subject: Re: Storing partial responses
>
>
<SNIP>
>
> Ok, it strikes me that 1 here could be a good part of the
> implementation
> of the cyclic object store that was discussed by Henrik,
> Adrian, Alex,
> and several others a while back and documented on Henrik's webpage at
> Sourceforge. I recently discussed this briefly with Henrik
> and Adrian
> offlist just on performance issues (since it had been hashed to death
> here with no one stepping in to actually work on it), and part of the
> design that was touched on as being the Probable Right
> Way(tm) (at least
> IMHO) was that of a cyclic object store with a fixed object size.
>
> Now, we take Alex's suggestion 1 here, and further limit it (in
> complexity at the FS level) to fixed objects of 8 or 16 or
> 32kb (or some
> other fixed size--whatever proves fastest on benchmarks).
> Every object,
> large or small, gets broken into, or padded (possibly
> combined), to fit
> that object size and then striped onto disk in the same manner as the
> Typhoon news server does (and purportedly Cisco's Cache
> Engine, though I
> can't confirm that one).
>
> This forces the 'meta-level' code to assemble our objects and
> know what
> objects are in what blocks. We recall the idea of rewriting partial
> objects when they become complete and expiring the old ones...or even
> just rewriting them and 'forgetting' the old ones as we sweep
> across the
> disk. (If you haven't reviewed the cyclical object store discussion
> lately, the idea is that writes always go from the head of
> the disk to
> the tail of the disk, and then start over at the head again.
> Reads, of
> course, must still be random, but writes can be done in
> relatively large
> and very cheap operations...while removal is essential zero disk
> operation--just drop it from the in-core hash table or object
> db as it
> will eventually be written over in the normal course of disk events.)
>
>
> > In either case, a meta level that knows how to handle
> partial objects
> > will be required. For example, HTTP "ranges" logic will
> reside on that
> > level.

:blue sky idea:
I think perhaps a happy compromise would be to split the discussion
about meta level information into two levels
- a request satisfaction level and
- a cache store level

Give the request satisfaction level the "Ranges" logic and so forth,
give the cache store level the object assembly smarts, breaking objects
into chunks and so on as per your recap.

> This seems to me to indicate that we're about to add some complexity
> (maybe a lot) to the meta- and FS levels of the code in order to deal
> with an object that can be of varying sizes or divided over the disk.
> One way or another we've got to increase the knowledge and
> logic of the
> file system and the upper levels by a quite large amount. Maybe we
> should take it the extra two steps to solve a few more issues,
> particularly the issue that this change will present--speed.
>
> Now...I should also point out that I'm not here proposing (though it
> certainly could seem so) an actual 'filesystem' be
> implemented at the OS
> kernel level as may have been on the table in the earlier referenced
> discussion--though I don't think it was ever established that
<SNIP>
>
> I hope I'm not rehashing territory that has been considered
> laid to rest
> here. I just wanted to make sure we don't end up doing
> things twice or
> thrice, if we can merge a couple of goals and do things with
> less work
> overall.

For sure :]

Rethinking my blue sky comment above, where would the changes you are
discussing regarding a news style spooling architecture sit? Would it be
a new FS in the current API or would it be a clean break with the
existing FS's?

I was writing with the assumption that you were proposing a clean break,
necessitating the construction of the object splitting logic. If you are
talking about a new FS type, then the logic for that should sit in that
IMO, and therefore we come back to Alex's (0, 1 or 2) choice. And again
I still favour (2a).

Rob
Received on Tue Dec 12 2000 - 23:12:45 MST