Re: storeGet() -> storeGetPublic() ?

----- Original Message -----
From: "Adrian Chadd" <adrian@creative.net.au>
To: "Robert Collins" <robert.collins@itdomain.com.au>
Cc: <squid-dev@squid-cache.org>
Sent: Sunday, January 07, 2001 10:13 PM
Subject: Re: storeGet() -> storeGetPublic() ?

> On Sun, Jan 07, 2001, Robert Collins wrote:
> > perhaps provide a set of routines that allow a fs to build a in memory index, then something like reiserFS could optionally pass
all
> > additions/purges through those routines, thus generating a in memory index - and then digest capability. Yes for reiser it's
> > additional code with no purpose but creating digests, but on the other hand it'd be very modular and thus easy to #define on or
off
> > for the cache administrator to choose...
> >
>
> I thought about that, but then the digest implementation would have to
> track the refcount on all the bits. Which could potentially blow up
> the digest creation space by quite a bit.
>
> Not that I mind, if people decide this is how they want to go. It also
> means that additions/deletions could happen without periodic rebuilding
> of the digest..
>
> I really would like to make the cache digests an integral part of the
> storage manager. They could make disk-based schemes such as reiserfs_raw
> nearly as fast as the "traditional" squid method of performing lookups,
> without burdening the kernel with strange cache memory management
> requirements.
>
>
> Adrian

Not quite what I was meaning. - but close enough.

The real question as I see it is: should disk based index's support digest creation?

I think yes - here's a different angle - how big is a digest & how many collisions do production caches see? (I have read some doco
on bloom filters). I suspect that adding a small ref counter to the digest entries (say 3/4 bits) will still result in _less_ memory
usage than we see today with in memory index's. Heck, even a byte per digest filter entry is less than the storage of a URL in
memory per cache entry.

Rob
Received on Sun Jan 07 2001 - 04:23:05 MST