Object store based cache_dir(such as swift|kyoto and others)

A summary about Objects store such as swift or kyoto cabinet as DB for
squid.

Now squid uses internal store DB optimized for performance which is
amazing but still the lack of Robustness from the aspect of "very large
store".
a singe squid server can work against only local stores which limits a
cluster and also the size of the cache store.
To allow squid more then just that we can add a cache_dir which will be
a "driver" to object based store for cache storage.
The idea is kind of complex since the current way of handling cache
objects is limiting the driver.

Today squid lookup for a cache object by hash in internal DB based on
full cache objects.
Any way we are going to approach the idea will *put* some new limits to
the cache store.
Since there is a direct access to the HDD of the server we can organize
the cache objects in a very abstract way but with object based store
such as Kyoto cabinet it's not.
In Kyoto cabinet we have the limit of only putting a full and not a
partial objects and also the time limits are fixed.

I havn't used other DB such as swift that we were talking about before.

This is the place to talk about the subject outside of the IRC channel.

There are couple directions of storing objects:
- by url\storeid hash
- by full hash
- by partial hash

metalinks adds the standard which allows partial content to be framed
into a hash object and not into "raw bytes" which what makes the
internet so big in a way.

Squid servers in a way can move forward and remove the DB handling to a
more advanced Databases that gives robustness to the server.
This can lead the way for squid to move from single service process to a
much more then just that by throwing all the HDD IO load on other
machine that fits to this specific task.

I am writing from a very small knowledge and understanding of how squid
store and the other software internals are.
But if we do start thinking about it I think it lead into something big.

Please take the stage from me,
Eliezer
Received on Sun May 05 2013 - 22:00:52 MDT