Re: Large Rock Store

On 10/16/2012 04:45 PM, Amos Jeffries wrote:
> On 17.10.2012 11:02, Alex Rousskov wrote:
>> Hello,
>>
>> We have started working on caching "large" (i.e., multi-slot)
>> responses in Rock Store. Initial design overview is available at the
>> following wiki page, along with several design alternatives we have
>> considered. Feedback is welcomed.
>>
>> http://wiki.squid-cache.org/Features/LargeRockStore
>>
>>
>> As a part of Large Rock work, I will also try to fix at least one of
>> Store API quality problems that complicate any Store-related
>> improvements:
>>
>> 1) Store class is a parent of too many, barely related classes:
>> StoreController, StoreHashIndex, SwapDir, MemStore (and TestStore). This
>> "top heavy" design forces us to add pure virtual methods to Store and
>> then supply dummy implementations in many Store kids. And, more
>> importantly, it makes it difficult to understand which part of the
>> storage API each Store kid is responsible for, leading to boundaries
>> violation and other problems.
>>
>> 2) There is no class dedicated to a non-shared memory cache.
>> StoreController currently implements most of non-shared memory cache
>> logic while MemStore implements shared memory cache. StoreController
>> non-shared memory sharing code should be moved to a dedicated class
>> instead.
>>
>> 3) StoreHashIndex should probably become responsible for all disk caches
>> (SwapDirs) as a whole, while StoreController will coordinate disk and
>> memory caching (using StoreHashIndex and MemStore objects). Currently,
>> some disk-related manipulations reside in StoreHashIndex and some in
>> StoreController.
>>
>> 4) Get rid of the global store_table. Make it local to non-shared
>> caching code. As we have discussed previously, the existence of this
>> global makes shared caching life very difficult because we cannot share
>> such a complex table and yet some older code expects to find all entries
>> there. It also leads to problems with entry locking where the code
>> assumes that an idle entry will remain in the global table at least for
>> a while (I believe there are a few bugzilla reports about associated
>> core dumps). I doubt I can solve all the StoreEntry locking problems (it
>> may require significant client side rewrite), but removing store_table
>> global is a step in the right direction.
>>
>>
>> If you have suggestions on priorities or want to add other large-scale
>> Store annoyances to the above list, please let me know.

> Thank you, architecture clarification for store would be very welcome.
>
> IMHO;
> - "Store" should be the namespace - or the name of a base class for
> shared API accessors to a *single* store regardless of type.

Good idea! A common namespace (and source directory?) for core "object
store" classes would be a nice addition. If we need a single store
interface class, it should probably be named "Store". The namespace
would then be "Cache" (too general?), "Swap" (too disk-specific), or
"Storage" (too long?).

> - Low-level virtual methods should be in another class specific to the
> type of store (disk, memory, shared, ...). Which the particular store
> controller inherit from alongside "Store" interface class.

Yes, of course, except I am not sure a common parent for all classes is
actually needed. This will become clear during polishing.

> - Index hash object/s should be independent of the store backend and
> the class re-used by all (separate instances though).

Actually, indexing objects is store's prerogative and its details are
invisible to callers. Today, all indexing classes are required by some
stores and cannot be reused by other stores, and I do not expect that to
change. There is one serious challenge here. I will discuss it in a
separate email.

> - store controller - managing the set of available stores for
> startup/reload/clean/dirty/up/down/search/object-move handling?

Yes, it already more-or-less does that. We just need to polish it by
removing code that is meant to be elsewhere and bringing
Controller-specific code into that class. It will manage/coordinate two
objects: "memory store" and "disk stores".

> Since you mention a difficulty identifying what each of the store API
> classes does, can you suggest anything better than by arch brainstorm
> above?

I think we are in agreement on the high-level stuff. I do not think it
we should try to document most of the polishing/reshuffling details now
because there are too many of them, they are determined by high-level
class roles, the changes should be pretty clear in the patches, and I
may not fix everything anyway. There is one exception that I will
discuss separately.

> For example; its not clear to me in the slightest why StoreHashIndex is
> relevant to disks but not in-memory caches. Surely the in-memory have a
> hash index of their content as well? (or should after global store_table
> is gone) - or is the class name irrelevant to what it actually does
> nowdays?

Please ignore the current StoreHashIndex class name. It will not survive
this polishing.

This class is needed to represent all disk caches taken together and
coordinate activities among individual disk caches (e.g., cache_dir
selection for a given miss object).

Why separate the memory store from disk stores? While there is a lot in
common between disk and memory caches, there are a few significant
differences. For example:

* There may be many disk caches (that need coordination) but there is at
most one memory cache. That is why disk caches need a "disk cache
coordination" class and memory cache does not.

* Memory cache access methods are synchronous. Most disk cache access
methods are not (or should not be).

* An object may be placed in at most one disk cache, in at most one
memory cache, but it can be cached both on disk and in memory. The first
algorithm is (or will be) implemented by the StoreHashIndex replacement,
the second by MemStore itself, and the third by StoreController.

One of the biggest challenges would be to handle a combination of
original non-shared caches and modern shared caches. I will discuss this
in a separate email.

Thank you,

Alex.
Received on Wed Oct 17 2012 - 14:34:24 MDT