[squid-users] Re: Automatic StoreID ?

>Actually, two commercial vendors - PeerApp and ThunderCache - claim
their products doesn't use urls to identify the objects, thus they
don't have to maintain StoreID-like de-duplication database manually.

Any ideas how do they do it? <

Instead of first mapping the URL to a memory-resident table, keeping
pointers (file-id, bucket no.) to the real location of the object on disk, a
hash-value, derived from the URL could directly be used to designate the
storage location on disk, avoiding the translation table, squid uses.
This is the principle of every hashed table in a fast database system.
Drawback is, you have to deal with "collisions" on the disk and "overflows":
hashes for different URLs point to same storage location on disk. Different
solutions for this problem available, though (chaining, sequential storage,
secondary storage area etc.). And you have to manage variable sized
"buckets", the storage locations, hashing points to.

Positive consequence: No rebuild of the in-memory-table necessary, as there
is none. Avoids the time-comsuning rebuild of rock-storage-table from disk.

I can imagine, that because of historical reasons (much simpler to
implement), squid uses the translation-table instead of direct hashing,
whereas Thundercache etc. can rely on some low-level DB-system, having
direct hashing "ready to be used".

--
View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Automatic-StoreID-tp4665140p4665198.html
Sent from the Squid - Users mailing list archive at Nabble.com.