Re: SMP: logging

On 02/23/2010 02:57 PM, Amos Jeffries wrote:
> On Tue, 23 Feb 2010 09:03:51 -0700, Alex Rousskov
> <rousskov_at_measurement-factory.com> wrote:
>> On 02/23/2010 07:44 AM, Kinkie wrote:
>>> 2010/2/23 Tsantilas Christos <chtsanti_at_users.sourceforge.net>:
>>>>> Log rotation will probably need special care to avoid rotating the
> same
>>>>> log many times. It is relatively easy to designate one process to do
>>>>> the
>>>>> rotation (e.g., squid1), but that alone is not enough to synchronize
>>>>> everything. Ideally, I would prefer to avoid locking. Any clever
> tricks
>>>>> applicable here?
>>>> Maybe something like the following:
>>>> - there is a master process
>>>> - master process communicates with one direction pipes with kids,
>>>> sending
>>>> various commands
>>>>
>>>> For logs rotation:
>>>> - master process closes the old file and rename it
>>>> - master process opens a new log file
>>>> - kids still writing to the old file (because they did not close the
>>>> file descriptor yet)
>>>> - master process send a command to all kids saying that the log file
>>>> rotated
>>>> - The kids close the log file and reopen it.
>>>>
>>>> The above will work at least for unix systems (I do not know about
>>>> MS-Windows)
>>>> Master/kids communication using pipes is used by the apache worker
>>>> server.
>>>> I am also using it in my icap server. I am not seeing any problem.
>>> Why not just having a log writing helper to whom clients write log
>>> lines via some kind of sockets?
>>> That would make the log rotation problem just go away - especially the
>>> part where there may be race conditions among workers writing together
>>> to the same file.
>> The logging daemon does not make log rotation problem go away. You still
>> need to implement log rotation. It just becomes slightly easier to do
> so.
>> The O_APPEND solution, if it works, does not have race conditions among
>> workers writing together to the same file. Or, more precisely, we trust
>> the file system layer to handle those race conditions.
>>
>> Please see my earlier email with the same subject for the
>> daemon-vs-append comparison summary. I would not be surprised if there
>> are more pros and cons to consider here, but so far I do not see enough
>> advantages to implement a logging daemon as the default SMP logging
>> mechanism.
>
> We already have a logging daemon producing stdin/stdout/stderr sockets to
> a master process. We have a TCP multi-input daemon coming. The file daemon
> is certainly working swimmingly well here and elsewhere.
>
> All thats needed is to make it the default and...
> a) pass its sockets a little further afield (same as for all other
> helpers if we share them).

If the stdin socket does not have an equivalent of O_APPEND, can
multiple processes write to it and still avoid corrupting each other
entries?

> or
> b) use O_APPEND internally to the daemon so each process sub-child daemon
> can write the same log.

Sure. And this would especially make sense if the daemon is doing more
than just logging (and Squid cannot do that internally). Otherwise, each
process can simply log with O_APPEND to a single file. The choice would
be up to the administrator, with the simplest option being the default.

> The hard bit is cache.log. Though if we go the O_APPEND route thats easy,
> if we want everything daemon logged, it should not be too hard to start a
> daemon for it and convert debugs() to log there.

Ah, I did not realize cache.log daemon logging is not supported yet. One
more reason to start with simple O_APPEND. As a side effect, we would be
able to debug daemon log starting problems as well :-).

Cheers,

Alex.
Received on Tue Feb 23 2010 - 22:48:04 MST