Re: Squid squid-1.NOVM.19 ate my cache/log

You might try changing the value of store_avg_object_size to something
lower than the default of 20k. On my cache 11k is just about ideal. I was
having the same problems as you with the change (although I'm running the
normal VM version of squid), except that my cache_swap loss was the result
of hundreds of "You've run out of swap file numbers. Freeing 1MB"
messages, resulting in a loss of about 300 MB or so of cache objects
before I managed to fix the problem.

Since I redefined store_avg_object_size to 11k my cache has been running
fine (so far -- I fixed it Monday).

        -Bill

On 26 Dec 1997, Daniel S. Riley wrote:

> Perhaps I haven't been paying attention, but it seems to me 1.19 has
> some serious lossage--in fact, it lost most of our cache disk. I just
> upgraded a small cache from 1.NOVM.18 to 1.NOVM.19, and when I started
> up the new version for the first time it logged
>
> 97/12/26 13:38:00| Starting Squid Cache version 1.NOVM.19 for mips-dec-ultrix4.4...
> 97/12/26 13:38:00| With 256 file descriptors available
> [...]
> 97/12/26 13:38:02| Configuring Parent pb.cache.nlanr.net/3128/3130
> 97/12/26 13:38:04| 4096 Lines read so far.
> 97/12/26 13:38:05| 8192 Lines read so far.
> 97/12/26 13:38:06| 12288 Lines read so far.
> 97/12/26 13:38:06| Finished rebuilding storage from disk image.
> 97/12/26 13:38:06| 13239 Lines read from previous logfile.
> 97/12/26 13:38:06| 683 Objects loaded.
> 97/12/26 13:38:06| 0 Objects expired.
> 97/12/26 13:38:06| 0 Duplicate URLs purged.
> 97/12/26 13:38:06| 0 Swapfile clashes avoided.
> 97/12/26 13:38:06| Took 4 seconds ( 170.7 objects/sec).
> 97/12/26 13:38:06| store_swap_size = 6964k
>
> so I had 13,000 objects in the old cache, and 1.NOVM.19 loaded 700 of
> them, losing track of 95% of the cache.
>
> The cause of this seems to be the change
>
> - Changed storeInit() to allocate 1.5 times the estimated number
> of cache objects for the 'filemap' bitmap.
>
> which changed MAX_SWAP_FILE from a constant 1<<21 to
> 3/2*maxSize/avjObjectSize. Other effects of this change have been
> discussed recently, and it does significantly reduce memory usage for
> small caches (a noticeable win for us), but I do not remember seeing
> any mention of how this interacts with
>
> if (sfileno < 0 || sfileno >= MAX_SWAP_FILE)
> continue;
>
> in storeDoRebuildFromDisk(). The effect is that any files that were
> given numbers beyond the newly caclulated MAX_SWAP_FILE are quietly
> discarded from cache/log and the files forgotten about but *not*
> deleted. Worse, since those file numbers are now outside the active
> range, they will never be reused, so the forgotten files will just
> sit there until someone intervenes.
>
> If I've got this right, it seems to me there ought to be a big warning
> in CAPITAL LETTERS attached to 1.19 that
>
> - upgrading may cause squid to lose track of big chunks of
> cache disk space
> - decreasing maxSize or increasing avjObjectSize may lose
> track of smaller chunks of disk space
>
> Since I was down to 700 objects that squid still had track of, and the
> old log had been overwritten, I ended doing an 'rm -rf cache' followed
> by a 'squid -z' to get back to a well defined state.
> --
> Dan Riley dsr@mail.lns.cornell.edu
> Wilson Lab, Cornell University HEPNET/SPAN: lns598::dsr (44630::dsr)
> http://www.lns.cornell.edu/~dsr/ "Distance means nothing/To me" -Kate Bush
>
Received on Fri Dec 26 1997 - 22:51:31 MST