Re: memory mapped store entries

Stewart Forster wrote:
>> >So you're saying that you want to mlock() all these pages into RAM so they
>> >will never get paged out and cause delays?
>>
>> It would be an option, but I don't think it's necessary.

>What about when paging occurs due to memory shortage. Sure paging is bad
>in any event. I was thinking mlock() would ensure the mmap()ed pages would
>remain resident so they didn't require any page-faults to bring them in
>again after being paged out.

The question is, would we rather have the kernel page other things out,
or allow it to page out the mmapped block (partly) as well.

>> Correct me if I'm wrong, but didn't I just demonstrate that the page-in/out
>> behaviour of the mmapped case is going to be better (marginally) than
>> the behaviour without mmap?

>Maybe. You have assumed ample memory to prevent buffer cache thrash for
>your mmap()ed pages.

I would normally assume that. If we wouldn't have ample memory, the
machine starts swapping. We all know that squid doesn't like that.
The only thing that (apparently) can happen is that the buffer cache
is being overflowed with writes. But this problem exists with and
without mmapped file, so I'm not sure if this is a problem.

>> If you're concerned, you could build in checkpoints, where squid tells
>> the kernel to sync mmap and disk content *now* (this will happen
>> asynchronously again).

>I'd do that with a threaded fsync() happening every N secs.

Actually, the mmap equivalent is called: msync(MS_ASYNC);

>> > Are we
>> >happy to mlock() pages into RAM to ensure mmap()'s performance?

>> Like I said, I don't think this will make much of a difference.

>You'd have to do comparisons first to say that.

Well, no I can say "think" when I think I can predict the result of those
tests (which I presume you are referring to).

>That's pretty low load. I'm talking about 5000 HTTP requests per minute at
>peak loads across 80Gb cache swaps. The mmap() system you propose will

Ok, I know when I've been outnumbered :-).

>What I'm REALLY ranting about is that before mmap()ed indexes are committed
>to squid 1.2 I want to be REALLY certain there isn't some major page-faulting
>about to happen as opposed to the solution that's currently in squid under
>high loads.

Your concerns are completely valid. This is why I ask for comments, before
enforcing the patches upon everyone (as if that would work :-).

>I propose that a side-patch for your stuff is generated and tested against
>some real killer caches before committing it. If it's just a #define at
>compile time, then I'm happy with that too.

Fair enough... If anyone has some feedback on how to make the switch
more cleanly in the source, I'm open to suggestions. As you might gather
from the excerpt I posted, I'm using defines like Skey to hide the
configurable indirection. I'm not particularly proud on this solution,
but it seems to be the least intrusive.

-- 
Sincerely,                                                          srb@cuci.nl
           Stephen R. van den Berg (AKA BuGless).
"To err is human, to debug ... divine."