Re: assertion failed: store_dir_diskd.c:839: "rb->flags.need_to_validate"

On 9 Jul 2001, at 20:03, Henrik Nordstrom <hno@hem.passagen.se> wrote:

> Andres Kroonmaa wrote:
> >
> > More bits.
> > I've noticed that sometimes swap.state is too large, say 3 times larger
> > than expected. I suspect that Squid is in some conditions appending to
> > swap.state instead of writing out clean and then rename.
>
> In my testing of HEAD I have noticed that the log is not properly
> cleaned on startup.. cache.log says it is, but still seems to be using
> the same log (or a duplicate thereof). Noticed this when I had a couple
> of thousand "replaced" entries for a single object in the cache (was
> about 3 object total), and swap.state was quite large all the time.

 Actually, I'm abit unhappy with how swap.state is currently managed.
 I don't like that swap.state is opened for write append, for one. This
 makes it required to do slow startup on the whole swap.state, although
 most part of it has been validated during previous clean write.

 I'd propose changing the way swap.state is managed.
  1) swap.state file is opened read-only, always. never written to.
  2) swap.state.new is a transaction log written to during runtime
  3) swap.state.clean is a clean log, a combination of swap.state and new
  4) swap.state.clean is rewritten regularily, or "committed", from full
     inmem storedb.
  5) When commit to clean successful, swap.state.new is truncated and
     we restart appending to it from start.
  6) If commit was successful, rename swap.state.clean to swap.state

 The idea is that during any dirty startup, we always have known good
 swap.state and possibly uncommitted swap.state.new which can be kept
 decently small, that together can define latest state of swap.
 All writes go to temporary files only. Only relatively small amount is
 "in air", and in case of dirty shutdown, we can quickly load last-good
 swap.state, swap.state.new and validate only objects referenced in new.

 Another thing that bothers me is very slow and very cpu-eating CLEAN
 startup. It seems to me that mostly due to writing each storeentry
 individually to state.new as squid reads in swap.state. I can't see the
 need to write each sentry into .new as it goes. Instead I'd expect it to
 readin full swap.state, construct inmem storedb, and then writeout clean
 log in one shot into swap.state.clean. So I'd also propose doing so.

 Comments?

------------------------------------
 Andres Kroonmaa <andre@online.ee>
 CTO, Microlink Online
 Tel: 6501 731, Fax: 6501 725
 Pärnu mnt. 158, Tallinn,
 11317 Estonia
Received on Tue Jul 10 2001 - 02:32:35 MDT