Re: logfile writing

On Mon, Jul 17, 2006, Henrik Nordstrom wrote:

> Instead of the linked list of separately maintained buffers, have you
> considered using MemBuf:s?

Nope! I just did this quick hack to mirror stuff I've done in the past
under Linux/BSD to try and stuff IPC IO into page-aligned buffers to
exploit COW semantics. (Although I didn't follow through with the last
bit of the COW semantics - ie, keeping the buffer around until it
/had/ been used :)

I'll look at re-using the membuf code later today.

> > * rework the socket IO scheduling code to try and hold onto a buffer until
> > its full or 1 second has passed. It might not really matter but it'll be
> > interesting to profile it at very high (>1500/sec) request rates.
>
> Not sure I agree with this. It should even out anyway under load, and
> under low load it's just annoying to have logs delayed.

They're only delayed to be sent to the logfile process up to a second.
A 32k buffer holds 256 x 128 byte logfile entries; so an actual high
throughput will basically ensure things never really get buffered.

> > * figure out what to do about rotate (as it stands, this code only re-opens
> > the logfiles and therefore pushes logfile maintainence into an external
> > script.)
>
> You flush any pending buffers first, right?

This patch doesn't do that yet. It wasn't that important as buffered data
doesn't get lost during a rotate.

The only thing this patch doesn't address is load shedding where the OS
just can't write to the log file(s) fast enough. Its trivial to add, but
I wanted to make sure it functioned correctly first.

Robert: I do remember your work having that annoying problem of dropping
logfile entries. I've got an idea on how to test it (I'll write a script
to throw lots of consecutively-numbered requests at my code and make sure
they all appear in the logs.)

Adrian
Received on Sun Jul 16 2006 - 19:06:48 MDT