Re: memory-mapped files in Squid

>>> Oskar Pearson wrote
> Hi
>
> > > I'd be very interested in finding out more about this. Who is working on
> > > squid-FS?
>
> > That would be me... I've got code here, two people have seen it, it's roug
h
> > as guts, I'm hoping to do something more with it in the next couple of week
s
> > (and yes, I've been saying that since before xmas :(. I unfortunately got
> > hammered by other work, but that's slowing now...
>
> I think that you need to take up the "quick release" motto... release your
> code here. There is obviously interest, but it's difficult to start right
> from the beginning.... I believe that if you release your code someone will
> do some of the hacking that you want to do...

Yeah, I'd like to do that, but the code is really atrocious atm. Give me
time to make sure it at least has clear 'this is where it goes into squid'
points in it, and the more embarassing out-by-x calculation bugs are sorted,
and I'll gladly let people hack on it - I'm anxious to get the thing moving
again too ;) (He says eyeing off the four biggest caches here...)

>
> > storing filenumbers instead of names, where the number corresponds
> > directly to the location on disk of the first block of the file. Keep
>
> In the next few months a group of us here are going to be doing various
> modifications to Linux here to get a really high-end cache going.
>
> Currently our todo list (first to last) is a bit like this:

Your todo list sounds fairly linux-specific - we're running on Solaris,
and any patches I produce I want to be as generic OS-wise as possible -
I don't generally see much point (for me, anyway ;) in writing OS-specific
stuff... However:

> Creating a squid-fs. This is unlikly to be performed at the squid level:
> we are likely to do it in kernel mode: it is simply going to support
> things like 'open' and 'close' in their standard form... Given that
> ext2 is something like 6000 lines of code, a much simpler filesystem
> at the kernel level is going to be a reasonable amount of code.
> Romfs (very, very, very simple) in 2.2 is 709 lines.

The system here basically opens a block device (or a file, if you prefer -
that's how I'm testing), and seeks/reads/writes to that. It provides
replacement read() and write() and so forth. That's the best level I could
see to hook it in.

>
> I am currently trying to find a paper I had that spoke about the read/write
> blocks sizes of modern disks...
>
> I believe that your 4k figure is based on fastfs.ps by McKusic? That paper
> is not exactly new: the principles apply....

Nope, the 4K block size is based primarily on the size distribution of files
in the cache. The system is _supposed_ to have an 8K chunk size as well,
to address some frag issues, but that's fallen by the way at the moment
- can be pushed back into it later.

>
> Ok... it's now about 50 minutes later and I have found the paper... it was
> on a friend's desk ;)
>
> The Paper is entitled 'Embedded Inodes and Explicit Grouping: Exploiting
> Disk Bandwidth for Small files'. It's by "Gregory R. Ganger and M. Frans
> Kaashoek, Laboratory for Computer Science, Massachusetts Institute of
> Technology"

Thanks, I'll check that out. Bear in mind that when you're designing for
something as specific as squid, you can throw a lot of standard concerns
out the window (and a lot of others can be harmful rather than beneficial).
Again, check the archives for some of this discussion previously ;)

KevinL
(Going to hack on it tonight, now... ;)
Received on Tue Jul 29 2003 - 13:15:55 MDT