Re: [squid-users] Ramdisks

A very good summary. Thanks Adrian.

A little out of thread from the "trying to find the ultimate Squid-FS design"
/ "why WAFL is good/bad" discussion, but a very good summary of what needs to
be done before any of the issues discussed can see the light full proper.

Same last words...

Regards
Henrik

On Tuesday 27 November 2001 12.29, Adrian Chadd wrote:

> Guys, abstracting the store io interface enough to support multiple
> filesystem types _CAN_ be done and _CAN_ be efficient.
> Its on my todo list, and it involves the following:
>
> * replacing storeGet* and the various lookup routines with hooks that
> hook into the store io layer
> * moving the StoreEntry _PUBLIC_ object index into the store io layer,
> which means that the indexes are held per storeio rather than
> globally
> * removing the whole logging/rebuilding code - again, thats a storeio
> thing which the filesystems can do themselves
> * redoing the IO interface (open/close/create/read/write/unlink) to be
> more .. well, useful. Eg.
>
> + make open+read and create+write ops (where a create can be a create
> op with a NULL write block)
> + modify the read/write IO interface to actually read/write as much
> as possible rather than 4k at a time (one could even do this by just
> throwing a stmem chain at the store IO api)
>
> I'm going to wait for squid-2.5 to go out the door before I do this,
> since my plan for this will basically be:
>
> * destroy with extreme prejudice the existing src/fs/* modules, with
> the notable exception of the null layer
> * write a simple, simple FS module (think "apache/cern style disk dir
> layout) to test the whole thing with
> * make the squid internals work with the above FS module
> * get you guys to test it throughly. :-)
> * then play with alternate FS layouts, moving ufs/aufs/diskd/coss/whatever
> into the new order
>
> There's all sorts of benefits inherit in an FS layer like this, which
> include:
>
> * private (home) caches can be built to have gig's of object storage,
> but only use like 20mb of RAM for squid (since one can use an on-disk
> index rather than an in-memory index)
> * I can re-write COSS to use even less RAM for a complete in-memory index,
> by doing things such as grouping the StoreEntry's per COSS stripe into
> an allocated "slab" (eg from andres' slab allocator) which I can then
> walk as an array to find all the objects allocated in a COSS stripe.
> This saves me needing to keep replacement info per StoreEntry, which
> can save at least 4 bytes (and even more, depending .. :-)
> * then, when the commloops code pans out, one can even consider writing
> filesystems that deal with seldom-changing streaming objects better,
> or a FS that can deal with partial objects, or Vary object chains,
> or .....
>
> We know what this boils down to. Someone find me a couple of months of
> part-time funding and I'll have it magically appear for people.
>
> :-)
>
> Adrian
Received on Tue Nov 27 2001 - 05:37:07 MST