Re: [squid-users] Ramdisks

On Tue, Nov 27, 2001, Henrik Nordstrom wrote:

> Perhaps not as bad if you are using a FIFO policy on the files stored on
> a WAFL filesystem, but fragmentation over time still exists.
>
> > hmm, why not? I'd agree that it is not a best FS for small objects <4K
> > (simple COSS would be much better), but as a general "fit-all" squid-FS
> > it could be pretty good, imo.
>
> Do we need a general purpose filesystem for Squid?

No.

Guys, abstracting the store io interface enough to support multiple
filesystem types _CAN_ be done and _CAN_ be efficient.
Its on my todo list, and it involves the following:

* replacing storeGet* and the various lookup routines with hooks that
  hook into the store io layer
* moving the StoreEntry _PUBLIC_ object index into the store io layer,
  which means that the indexes are held per storeio rather than
  globally
* removing the whole logging/rebuilding code - again, thats a storeio
  thing which the filesystems can do themselves
* redoing the IO interface (open/close/create/read/write/unlink) to be
  more .. well, useful. Eg.

  + make open+read and create+write ops (where a create can be a create
    op with a NULL write block)
  + modify the read/write IO interface to actually read/write as much
    as possible rather than 4k at a time (one could even do this by just
    throwing a stmem chain at the store IO api)

I'm going to wait for squid-2.5 to go out the door before I do this,
since my plan for this will basically be:

* destroy with extreme prejudice the existing src/fs/* modules, with
  the notable exception of the null layer
* write a simple, simple FS module (think "apache/cern style disk dir
  layout) to test the whole thing with
* make the squid internals work with the above FS module
* get you guys to test it throughly. :-)
* then play with alternate FS layouts, moving ufs/aufs/diskd/coss/whatever
  into the new order

There's all sorts of benefits inherit in an FS layer like this, which
include:

* private (home) caches can be built to have gig's of object storage,
  but only use like 20mb of RAM for squid (since one can use an on-disk
  index rather than an in-memory index)
* I can re-write COSS to use even less RAM for a complete in-memory index,
  by doing things such as grouping the StoreEntry's per COSS stripe into
  an allocated "slab" (eg from andres' slab allocator) which I can then
  walk as an array to find all the objects allocated in a COSS stripe.
  This saves me needing to keep replacement info per StoreEntry, which
  can save at least 4 bytes (and even more, depending .. :-)
* then, when the commloops code pans out, one can even consider writing
  filesystems that deal with seldom-changing streaming objects better,
  or a FS that can deal with partial objects, or Vary object chains,
  or .....

We know what this boils down to. Someone find me a couple of months of
part-time funding and I'll have it magically appear for people.

:-)

Adrian
Received on Tue Nov 27 2001 - 04:29:37 MST