We are running squid version 2.4.STABLE7 on FreeBSD version 4.6-RELEASE #2.
I am beginning to see an particularly confusing error in squid. I have
included an excerpt from the cache.log file for visual assistance.
First thing to look at is on 2002/12/31 14:38:40. It seems to be stuck at
100% of "rebuilding the store". It is in this loop for 12 minutes.
Everything then seems to clear up. However on 2002/12/31 14:53:46 it begins
"rebuild the store" again, which gets "stuck" in a loop until 2002/12/31
16:09:46 and the hard drive is full - taking squid down.
We have attempted to analyze squid's source, and it appears to be getting
caught in a loop when verifying cache objects while attempting to process
NEW connections at the same time.
ANY assistance you can give is GREATLY appreciated.
Brad Holman
Technical Support Manager
brad@s4f.com <mailto:brad@s4f.com>
S4F Internet Management Systems
2448 E. 81st Street . 33rd Floor
Tulsa, OK 74137
Phone 918-524-1010 . Fax 918-524-1011
<BEGIN LOG FILE
CACHE.LOG>------------------------------------------------------------------
---------------
2002/12/31 14:17:43| Rebuilding storage in /usr/local/squid/cache (DIRTY)e
2002/12/31 14:17:43| Using Least Load store dir selection
2002/12/31 14:17:43| Set Current Directory to /usr/local/squid/cache
2002/12/31 14:17:43| Loaded Icons.
2002/12/31 14:17:43| Accepting HTTP connections at 0.0.0.0, port 8080, FD
28.
2002/12/31 14:17:43| Accepting HTTP connections at 0.0.0.0, port 8081, FD
29.
2002/12/31 14:17:43| Accepting ICP messages at 0.0.0.0, port 3130, FD 30.
2002/12/31 14:17:43| WCCP Disabled.
2002/12/31 14:17:43| Ready to serve requests.
2002/12/31 14:17:43| Store rebuilding is 3.2% complete
2002/12/31 14:17:45| Done reading /usr/local/squid/cache swaplog (126340
entries)
2002/12/31 14:17:45| Finished rebuilding storage from disk.
2002/12/31 14:17:45| 126254 Entries scanned
2002/12/31 14:17:45| 0 Invalid entries.
2002/12/31 14:17:45| 0 With invalid flags.
2002/12/31 14:17:45| 126169 Objects loaded.
2002/12/31 14:17:45| 0 Objects expired.
2002/12/31 14:17:45| 85 Objects cancelled.
2002/12/31 14:17:45| 0 Duplicate URLs purged.
2002/12/31 14:17:45| 0 Swapfile clashes avoided.
2002/12/31 14:17:45| Took 2.5 seconds (50767.6 objects/sec).
2002/12/31 14:17:45| Beginning Validation Procedure
2002/12/31 14:17:45| Completed Validation Procedure
2002/12/31 14:17:45| Validated 126169 Entries
2002/12/31 14:17:45| store_swap_size = 1135716k
2002/12/31 14:17:46| storeLateRelease: released 0 objects
2002/12/31 14:20:00| Restarting Squid Cache (version 2.4.STABLE7)...
2002/12/31 14:20:00| FD 28 Closing HTTP connection
2002/12/31 14:20:00| FD 29 Closing HTTP connection
2002/12/31 14:20:00| FD 30 Closing ICP connection
2002/12/31 14:20:00| Cache dir '/usr/local/squid/cache' size remains
unchanged at 4194304 KB
2002/12/31 14:20:00| DNS Socket created on FD 5
2002/12/31 14:20:00| Adding nameserver 209.81.96.49 from /etc/resolv.conf
2002/12/31 14:20:00| Adding nameserver 209.81.96.130 from /etc/resolv.conf
2002/12/31 14:20:00| helperOpenServers: Starting 12 'squidGuard' processes
2002/12/31 14:20:01| helperOpenServers: Starting 5 'ncsa_auth' processes
2002/12/31 14:20:01| Accepting HTTP connections at 0.0.0.0, port 8080, FD 7.
2002/12/31 14:20:01| Accepting HTTP connections at 0.0.0.0, port 8081, FD
28.
2002/12/31 14:20:01| Accepting ICP messages at 0.0.0.0, port 3130, FD 29.
2002/12/31 14:20:01| WCCP Disabled.
2002/12/31 14:20:01| Loaded Icons.
2002/12/31 14:20:01| Ready to serve requests.
2002/12/31 14:25:23| Restarting Squid Cache (version 2.4.STABLE7)...
2002/12/31 14:25:23| FD 7 Closing HTTP connection
2002/12/31 14:25:23| FD 28 Closing HTTP connection
2002/12/31 14:25:23| FD 29 Closing ICP connection
2002/12/31 14:25:23| Cache dir '/usr/local/squid/cache' size remains
unchanged at 4194304 KB
2002/12/31 14:25:23| DNS Socket created on FD 5
2002/12/31 14:25:23| Adding nameserver 209.81.96.49 from /etc/resolv.conf
2002/12/31 14:25:23| Adding nameserver 209.81.96.130 from /etc/resolv.conf
2002/12/31 14:25:23| helperOpenServers: Starting 12 'squidGuard' processes
2002/12/31 14:25:24| helperOpenServers: Starting 5 'ncsa_auth' processes
2002/12/31 14:25:24| Accepting HTTP connections at 0.0.0.0, port 8080, FD 7.
2002/12/31 14:25:24| Accepting HTTP connections at 0.0.0.0, port 8081, FD
28.
2002/12/31 14:25:24| Accepting ICP messages at 0.0.0.0, port 3130, FD 29.
2002/12/31 14:25:24| WCCP Disabled.
2002/12/31 14:25:24| Loaded Icons.
2002/12/31 14:25:24| Ready to serve requests.
2002/12/31 14:36:56| Preparing for shutdown after 1462 requests
2002/12/31 14:36:56| Waiting 30 seconds for active connections to finish
2002/12/31 14:36:56| FD 7 Closing HTTP connection
2002/12/31 14:36:56| FD 28 Closing HTTP connection
2002/12/31 14:38:23| Starting Squid Cache version 2.4.STABLE7 for
i386-portbld-freebsd4.6...
2002/12/31 14:38:23| Process ID 148
2002/12/31 14:38:23| With 7236 file descriptors available
2002/12/31 14:38:23| DNS Socket created on FD 5
2002/12/31 14:38:23| Adding nameserver 209.81.96.49 from /etc/resolv.conf
2002/12/31 14:38:23| Adding nameserver 209.81.96.130 from /etc/resolv.conf
2002/12/31 14:38:23| helperOpenServers: Starting 12 'squidGuard' processes
2002/12/31 14:38:24| helperOpenServers: Starting 5 'ncsa_auth' processes
2002/12/31 14:38:24| Unlinkd pipe opened on FD 27
2002/12/31 14:38:24| Swap maxSize 4194304 KB, estimated 322638 objects
2002/12/31 14:38:24| Target number of buckets: 16131
2002/12/31 14:38:24| Using 16384 Store buckets
2002/12/31 14:38:24| Max Mem size: 81920 KB
2002/12/31 14:38:24| Max Swap size: 4194304 KB
2002/12/31 14:38:24| Store logging disabled
2002/12/31 14:38:24| Rebuilding storage in /usr/local/squid/cache (DIRTY)
2002/12/31 14:38:24| Using Least Load store dir selection
2002/12/31 14:38:24| Set Current Directory to /usr/local/squid/cache
2002/12/31 14:38:24| Loaded Icons.
2002/12/31 14:38:24| Accepting HTTP connections at 0.0.0.0, port 8080, FD
28.
2002/12/31 14:38:24| Accepting HTTP connections at 0.0.0.0, port 8081, FD
29.
2002/12/31 14:38:24| Accepting ICP messages at 0.0.0.0, port 3130, FD 30.
2002/12/31 14:38:24| WCCP Disabled.
2002/12/31 14:38:24| Ready to serve requests.
2002/12/31 14:38:25| Store rebuilding is 3.2% complete
2002/12/31 14:38:27| Restarting Squid Cache (version 2.4.STABLE7)...
2002/12/31 14:38:27| FD 28 Closing HTTP connection
2002/12/31 14:38:27| FD 29 Closing HTTP connection
2002/12/31 14:38:27| FD 30 Closing ICP connection
2002/12/31 14:38:27| Cache dir '/usr/local/squid/cache' size remains
unchanged at 4194304 KB
2002/12/31 14:38:27| DNS Socket created on FD 5
2002/12/31 14:38:27| Adding nameserver 209.81.96.49 from /etc/resolv.conf
2002/12/31 14:38:27| Adding nameserver 209.81.96.130 from /etc/resolv.conf
2002/12/31 14:38:27| helperOpenServers: Starting 12 'squidGuard' processes
2002/12/31 14:38:27| helperOpenServers: Starting 5 'ncsa_auth' processes
2002/12/31 14:38:27| Accepting HTTP connections at 0.0.0.0, port 8080, FD 7.
2002/12/31 14:38:27| Accepting HTTP connections at 0.0.0.0, port 8081, FD
28.
2002/12/31 14:38:27| Accepting ICP messages at 0.0.0.0, port 3130, FD 29.
2002/12/31 14:38:27| WCCP Disabled.
2002/12/31 14:38:27| Loaded Icons.
2002/12/31 14:38:27| Ready to serve requests.
2002/12/31 14:38:40| Store rebuilding is 100.0% complete
2002/12/31 14:38:55| Store rebuilding is 100.0% complete
2002/12/31 14:39:10| Store rebuilding is 100.0% complete
2002/12/31 14:39:25| Store rebuilding is 100.0% complete
2002/12/31 14:39:40| Store rebuilding is 100.0% complete
2002/12/31 14:39:55| Store rebuilding is 100.0% complete
2002/12/31 14:40:10| Store rebuilding is 100.0% complete
2002/12/31 14:40:25| Store rebuilding is 100.0% complete
2002/12/31 14:40:40| Store rebuilding is 100.0% complete
2002/12/31 14:40:55| Store rebuilding is 100.0% complete
2002/12/31 14:41:10| Store rebuilding is 100.0% complete
2002/12/31 14:41:25| Store rebuilding is 100.0% complete
2002/12/31 14:41:40| Store rebuilding is 100.0% complete
2002/12/31 14:41:55| Store rebuilding is 100.0% complete
2002/12/31 14:42:10| Store rebuilding is 100.0% complete
2002/12/31 14:42:25| Store rebuilding is 100.0% complete
2002/12/31 14:42:40| Store rebuilding is 100.0% complete
2002/12/31 14:42:55| Store rebuilding is 100.0% complete
2002/12/31 14:43:10| Store rebuilding is 100.0% complete
2002/12/31 14:43:25| Store rebuilding is 100.0% complete
2002/12/31 14:43:40| Store rebuilding is 100.0% complete
2002/12/31 14:43:55| Store rebuilding is 100.0% complete
2002/12/31 14:44:10| Store rebuilding is 100.0% complete
2002/12/31 14:44:25| Store rebuilding is 100.0% complete
2002/12/31 14:44:40| Store rebuilding is 100.0% complete
2002/12/31 14:44:55| Store rebuilding is 100.0% complete
2002/12/31 14:45:10| Store rebuilding is 100.0% complete
2002/12/31 14:45:25| Store rebuilding is 100.0% complete
2002/12/31 14:45:40| Store rebuilding is 100.0% complete
2002/12/31 14:45:55| Store rebuilding is 100.0% complete
2002/12/31 14:46:10| Store rebuilding is 100.0% complete
2002/12/31 14:46:25| Store rebuilding is 100.0% complete
2002/12/31 14:46:40| Store rebuilding is 100.0% complete
2002/12/31 14:46:55| Store rebuilding is 100.0% complete
2002/12/31 14:47:10| Store rebuilding is 100.0% complete
2002/12/31 14:47:25| Store rebuilding is 100.0% complete
2002/12/31 14:47:40| Store rebuilding is 100.0% complete
2002/12/31 14:47:55| Store rebuilding is 100.0% complete
2002/12/31 14:48:10| Store rebuilding is 100.0% complete
2002/12/31 14:48:25| Store rebuilding is 100.0% complete
2002/12/31 14:48:40| Store rebuilding is 100.0% complete
2002/12/31 14:48:55| Store rebuilding is 100.0% complete
2002/12/31 14:49:10| Store rebuilding is 100.0% complete
2002/12/31 14:49:25| Store rebuilding is 100.0% complete
2002/12/31 14:49:40| Store rebuilding is 100.0% complete
2002/12/31 14:49:55| Store rebuilding is 100.0% complete
2002/12/31 14:50:10| Store rebuilding is 100.0% complete
2002/12/31 14:50:25| Store rebuilding is 100.0% complete
2002/12/31 14:50:30| Preparing for shutdown after 139 requests
2002/12/31 14:50:30| Waiting 30 seconds for active connections to finish
2002/12/31 14:50:30| FD 7 Closing HTTP connection
2002/12/31 14:50:30| FD 28 Closing HTTP connection
2002/12/31 14:50:40| Store rebuilding is 100.0% complete
2002/12/31 14:51:58| Starting Squid Cache version 2.4.STABLE7 for
i386-portbld-freebsd4.6...
2002/12/31 14:51:58| Process ID 169
2002/12/31 14:51:58| With 7236 file descriptors available
2002/12/31 14:51:58| DNS Socket created on FD 5
2002/12/31 14:51:58| Adding nameserver 209.81.96.49 from /etc/resolv.conf
2002/12/31 14:51:58| Adding nameserver 209.81.96.130 from /etc/resolv.conf
2002/12/31 14:51:58| helperOpenServers: Starting 12 'squidGuard' processes
2002/12/31 14:51:58| helperOpenServers: Starting 5 'ncsa_auth' processes
2002/12/31 14:51:58| Unlinkd pipe opened on FD 27
2002/12/31 14:51:58| Swap maxSize 4194304 KB, estimated 322638 objects
2002/12/31 14:51:58| Target number of buckets: 16131
2002/12/31 14:51:58| Using 16384 Store buckets
2002/12/31 14:51:58| Max Mem size: 81920 KB
2002/12/31 14:51:58| Max Swap size: 4194304 KB
2002/12/31 14:51:58| Store logging disabled
2002/12/31 14:51:58| Rebuilding storage in /usr/local/squid/cache (DIRTY)
2002/12/31 14:51:58| Using Least Load store dir selection
2002/12/31 14:51:58| Set Current Directory to /usr/local/squid/cache
2002/12/31 14:51:58| Loaded Icons.
2002/12/31 14:51:58| Accepting HTTP connections at 0.0.0.0, port 8080, FD
28.
2002/12/31 14:51:58| Accepting HTTP connections at 0.0.0.0, port 8081, FD
29.
2002/12/31 14:51:58| Accepting ICP messages at 0.0.0.0, port 3130, FD 30.
2002/12/31 14:51:58| WCCP Disabled.
2002/12/31 14:51:58| Ready to serve requests.
2002/12/31 14:51:59| Store rebuilding is 0.0% complete
2002/12/31 14:52:09| Preparing for shutdown after 16 requests
2002/12/31 14:52:09| Waiting 30 seconds for active connections to finish
2002/12/31 14:52:09| FD 28 Closing HTTP connection
2002/12/31 14:52:09| FD 29 Closing HTTP connection
2002/12/31 14:52:09| Shutting down...
2002/12/31 14:52:09| FD 30 Closing ICP connection
2002/12/31 14:52:09| Closing unlinkd pipe on FD 27
2002/12/31 14:52:09| Not currently OK to rewrite swap log.
2002/12/31 14:52:09| storeDirWriteCleanLogs: Operation aborted.
CPU Usage: 8.519 seconds = 5.348 user + 3.171 sys
Maximum Resident Size: 24364 KB
Page faults with physical i/o: 2
2002/12/31 14:52:09| Open FD WRITING 25
/usr/local/squid/cache/swap.state.new
2002/12/31 14:52:09| Squid Cache (Version 2.4.STABLE7): Exiting normally.
2002/12/31 14:53:29| Starting Squid Cache version 2.4.STABLE7 for
i386-portbld-freebsd4.6...
2002/12/31 14:53:29| Process ID 148
2002/12/31 14:53:29| With 7236 file descriptors available
2002/12/31 14:53:29| DNS Socket created on FD 5
2002/12/31 14:53:29| Adding nameserver 209.81.96.49 from /etc/resolv.conf
2002/12/31 14:53:29| Adding nameserver 209.81.96.130 from /etc/resolv.conf
2002/12/31 14:53:29| helperOpenServers: Starting 12 'squidGuard' processes
2002/12/31 14:53:30| helperOpenServers: Starting 5 'ncsa_auth' processes
2002/12/31 14:53:30| Unlinkd pipe opened on FD 27
2002/12/31 14:53:30| Swap maxSize 4194304 KB, estimated 322638 objects
2002/12/31 14:53:30| Target number of buckets: 16131
2002/12/31 14:53:30| Using 16384 Store buckets
2002/12/31 14:53:30| Max Mem size: 81920 KB
2002/12/31 14:53:30| Max Swap size: 4194304 KB
2002/12/31 14:53:30| Store logging disabled
2002/12/31 14:53:30| Rebuilding storage in /usr/local/squid/cache (DIRTY)
2002/12/31 14:53:30| Using Least Load store dir selection
2002/12/31 14:53:30| Set Current Directory to /usr/local/squid/cache
2002/12/31 14:53:30| Loaded Icons.
2002/12/31 14:53:30| Accepting HTTP connections at 0.0.0.0, port 8080, FD
28.
2002/12/31 14:53:30| Accepting HTTP connections at 0.0.0.0, port 8081, FD
29.
2002/12/31 14:53:30| Accepting ICP messages at 0.0.0.0, port 3130, FD 30.
2002/12/31 14:53:30| WCCP Disabled.
2002/12/31 14:53:30| Ready to serve requests.
2002/12/31 14:53:31| Store rebuilding is 0.0% complete
2002/12/31 14:53:33| Restarting Squid Cache (version 2.4.STABLE7)...
2002/12/31 14:53:33| FD 28 Closing HTTP connection
2002/12/31 14:53:33| FD 29 Closing HTTP connection
2002/12/31 14:53:33| FD 30 Closing ICP connection
2002/12/31 14:53:33| Cache dir '/usr/local/squid/cache' size remains
unchanged at 4194304 KB
2002/12/31 14:53:33| DNS Socket created on FD 5
2002/12/31 14:53:33| Adding nameserver 209.81.96.49 from /etc/resolv.conf
2002/12/31 14:53:33| Adding nameserver 209.81.96.130 from /etc/resolv.conf
2002/12/31 14:53:33| helperOpenServers: Starting 12 'squidGuard' processes
2002/12/31 14:53:33| helperOpenServers: Starting 5 'ncsa_auth' processes
2002/12/31 14:53:33| Accepting HTTP connections at 0.0.0.0, port 8080, FD 7.
2002/12/31 14:53:33| Accepting HTTP connections at 0.0.0.0, port 8081, FD
28.
2002/12/31 14:53:33| Accepting ICP messages at 0.0.0.0, port 3130, FD 29.
2002/12/31 14:53:33| WCCP Disabled.
2002/12/31 14:53:33| Loaded Icons.
2002/12/31 14:53:33| Ready to serve requests.
2002/12/31 14:53:43| Restarting Squid Cache (version 2.4.STABLE7)...
2002/12/31 14:53:43| FD 7 Closing HTTP connection
2002/12/31 14:53:43| FD 28 Closing HTTP connection
2002/12/31 14:53:43| FD 29 Closing ICP connection
2002/12/31 14:53:43| Cache dir '/usr/local/squid/cache' size remains
unchanged at 4194304 KB
2002/12/31 14:53:43| DNS Socket created on FD 5
2002/12/31 14:53:43| Adding nameserver 209.81.96.49 from /etc/resolv.conf
2002/12/31 14:53:43| Adding nameserver 209.81.96.130 from /etc/resolv.conf
2002/12/31 14:53:43| helperOpenServers: Starting 12 'squidGuard' processes
2002/12/31 14:53:44| helperOpenServers: Starting 5 'ncsa_auth' processes
2002/12/31 14:53:44| Accepting HTTP connections at 0.0.0.0, port 8080, FD 7.
2002/12/31 14:53:44| Accepting HTTP connections at 0.0.0.0, port 8081, FD
28.
2002/12/31 14:53:44| Accepting ICP messages at 0.0.0.0, port 3130, FD 29.
2002/12/31 14:53:44| WCCP Disabled.
2002/12/31 14:53:44| Loaded Icons.
2002/12/31 14:53:44| Ready to serve requests.
2002/12/31 14:53:46| Store rebuilding is 1.7% complete
2002/12/31 14:54:01| Store rebuilding is 3.6% complete
2002/12/31 14:54:16| Store rebuilding is 5.4% complete
2002/12/31 14:54:31| Store rebuilding is 7.1% complete
2002/12/31 14:54:46| Store rebuilding is 8.8% complete
2002/12/31 14:55:01| Store rebuilding is 10.5% complete
2002/12/31 14:55:16| Store rebuilding is 12.0% complete
2002/12/31 14:55:31| Store rebuilding is 13.6% complete
2002/12/31 14:55:46| Store rebuilding is 15.0% complete
2002/12/31 14:56:01| Store rebuilding is 16.4% complete
2002/12/31 14:56:16| Store rebuilding is 17.9% complete
2002/12/31 14:56:20| Restarting Squid Cache (version 2.4.STABLE7)...
2002/12/31 14:56:20| FD 7 Closing HTTP connection
2002/12/31 14:56:20| FD 28 Closing HTTP connection
2002/12/31 14:56:20| FD 29 Closing ICP connection
2002/12/31 14:56:20| Cache dir '/usr/local/squid/cache' size remains
unchanged at 4194304 KB
2002/12/31 14:56:20| DNS Socket created on FD 5
2002/12/31 14:56:20| Adding nameserver 209.81.96.49 from /etc/resolv.conf
2002/12/31 14:56:20| Adding nameserver 209.81.96.130 from /etc/resolv.conf
2002/12/31 14:56:20| helperOpenServers: Starting 12 'squidGuard' processes
2002/12/31 14:56:21| helperOpenServers: Starting 5 'ncsa_auth' processes
2002/12/31 14:56:21| Accepting HTTP connections at 0.0.0.0, port 8080, FD 7.
2002/12/31 14:56:21| Accepting HTTP connections at 0.0.0.0, port 8081, FD
28.
2002/12/31 14:56:21| Accepting ICP messages at 0.0.0.0, port 3130, FD 29.
2002/12/31 14:56:21| WCCP Disabled.
2002/12/31 14:56:21| Loaded Icons.
2002/12/31 14:56:21| Ready to serve requests.
2002/12/31 14:56:21| Restarting Squid Cache (version 2.4.STABLE7)...
2002/12/31 14:56:21| FD 7 Closing HTTP connection
2002/12/31 14:56:21| FD 28 Closing HTTP connection
2002/12/31 14:56:21| FD 29 Closing ICP connection
2002/12/31 14:56:21| Cache dir '/usr/local/squid/cache' size remains
unchanged at 4194304 KB
2002/12/31 14:56:21| DNS Socket created on FD 5
2002/12/31 14:56:21| Adding nameserver 209.81.96.49 from /etc/resolv.conf
2002/12/31 14:56:21| Adding nameserver 209.81.96.130 from /etc/resolv.conf
2002/12/31 14:56:21| helperOpenServers: Starting 12 'squidGuard' processes
2002/12/31 14:56:21| helperOpenServers: Starting 5 'ncsa_auth' processes
2002/12/31 14:56:21| Accepting HTTP connections at 0.0.0.0, port 8080, FD 7.
2002/12/31 14:56:21| Accepting HTTP connections at 0.0.0.0, port 8081, FD
28.
2002/12/31 14:56:21| Accepting ICP messages at 0.0.0.0, port 3130, FD 29.
2002/12/31 14:56:21| WCCP Disabled.
2002/12/31 14:56:21| Loaded Icons.
2002/12/31 14:56:21| Ready to serve requests.
2002/12/31 14:56:31| Store rebuilding is 19.1% complete
2002/12/31 14:56:46| Store rebuilding is 20.6% complete
2002/12/31 14:57:01| Store rebuilding is 21.9% complete
2002/12/31 14:57:16| Store rebuilding is 23.2% complete
2002/12/31 14:57:31| Store rebuilding is 24.5% complete
2002/12/31 14:57:46| Store rebuilding is 25.8% complete
2002/12/31 14:58:01| Store rebuilding is 26.9% complete
2002/12/31 14:58:16| Store rebuilding is 28.1% complete
2002/12/31 14:58:31| Store rebuilding is 29.2% complete
2002/12/31 14:58:46| Store rebuilding is 30.4% complete
2002/12/31 14:59:01| Store rebuilding is 31.4% complete
2002/12/31 14:59:16| Store rebuilding is 32.5% complete
2002/12/31 14:59:31| Store rebuilding is 33.5% complete
2002/12/31 14:59:46| Store rebuilding is 34.5% complete
2002/12/31 15:00:01| Store rebuilding is 35.5% complete
2002/12/31 15:00:16| Store rebuilding is 36.5% complete
2002/12/31 15:00:31| Store rebuilding is 37.4% complete
2002/12/31 15:00:46| Store rebuilding is 38.3% complete
2002/12/31 15:01:01| Store rebuilding is 39.2% complete
2002/12/31 15:01:16| Store rebuilding is 40.0% complete
2002/12/31 15:01:31| Store rebuilding is 40.8% complete
2002/12/31 15:01:46| Store rebuilding is 41.7% complete
2002/12/31 15:02:01| Store rebuilding is 42.5% complete
2002/12/31 15:02:16| Store rebuilding is 43.3% complete
2002/12/31 15:02:31| Store rebuilding is 44.0% complete
2002/12/31 15:02:46| Store rebuilding is 44.8% complete
2002/12/31 15:03:01| Store rebuilding is 45.5% complete
2002/12/31 15:03:16| Store rebuilding is 46.2% complete
2002/12/31 15:03:31| Store rebuilding is 46.9% complete
2002/12/31 15:03:46| Store rebuilding is 47.6% complete
2002/12/31 15:04:01| Store rebuilding is 48.3% complete
2002/12/31 15:04:16| Store rebuilding is 49.0% complete
2002/12/31 15:04:31| Store rebuilding is 49.6% complete
2002/12/31 15:04:46| Store rebuilding is 50.2% complete
2002/12/31 15:05:01| Store rebuilding is 50.9% complete
2002/12/31 15:05:16| Store rebuilding is 51.5% complete
2002/12/31 15:05:31| Store rebuilding is 52.1% complete
2002/12/31 15:05:46| Store rebuilding is 52.7% complete
2002/12/31 15:06:01| Store rebuilding is 53.3% complete
2002/12/31 15:06:16| Store rebuilding is 293131.9% complete
2002/12/31 15:06:31| Store rebuilding is 5432.5% complete
2002/12/31 15:06:46| Store rebuilding is 2783.8% complete
2002/12/31 15:07:01| Store rebuilding is 1895.1% complete
2002/12/31 15:07:16| Store rebuilding is 1450.9% complete
2002/12/31 15:07:31| Store rebuilding is 1182.5% complete
2002/12/31 15:07:46| Store rebuilding is 1001.2% complete
2002/12/31 15:08:01| Store rebuilding is 871.0% complete
2002/12/31 15:08:16| Store rebuilding is 774.4% complete
2002/12/31 15:08:31| Store rebuilding is 699.9% complete
2002/12/31 15:08:46| Store rebuilding is 639.8% complete
2002/12/31 15:09:01| Store rebuilding is 590.6% complete
2002/12/31 15:09:16| Store rebuilding is 550.0% complete
2002/12/31 15:09:31| Store rebuilding is 515.2% complete
2002/12/31 15:09:46| Store rebuilding is 485.5% complete
2002/12/31 15:10:01| Store rebuilding is 460.0% complete
2002/12/31 15:10:16| Store rebuilding is 437.2% complete
2002/12/31 15:10:31| Store rebuilding is 417.7% complete
2002/12/31 15:10:46| Store rebuilding is 400.0% complete
2002/12/31 15:11:01| Store rebuilding is 384.1% complete
2002/12/31 15:11:16| Store rebuilding is 369.9% complete
2002/12/31 15:11:31| Store rebuilding is 357.1% complete
2002/12/31 15:11:46| Store rebuilding is 345.4% complete
2002/12/31 15:12:01| Store rebuilding is 334.7% complete
2002/12/31 15:12:16| Store rebuilding is 324.9% complete
2002/12/31 15:12:31| Store rebuilding is 315.9% complete
2002/12/31 15:12:46| Store rebuilding is 307.6% complete
2002/12/31 15:13:01| Store rebuilding is 299.9% complete
2002/12/31 15:13:16| Store rebuilding is 292.8% complete
2002/12/31 15:13:31| Store rebuilding is 286.1% complete
2002/12/31 15:13:46| Store rebuilding is 279.9% complete
2002/12/31 15:14:01| Store rebuilding is 274.1% complete
2002/12/31 15:14:16| Store rebuilding is 268.7% complete
2002/12/31 15:14:31| Store rebuilding is 263.6% complete
2002/12/31 15:14:46| Store rebuilding is 258.8% complete
2002/12/31 15:15:01| Store rebuilding is 254.2% complete
2002/12/31 15:15:16| Store rebuilding is 249.9% complete
2002/12/31 15:15:31| Store rebuilding is 245.8% complete
2002/12/31 15:15:46| Store rebuilding is 242.0% complete
2002/12/31 15:16:01| Store rebuilding is 238.3% complete
2002/12/31 15:16:16| Store rebuilding is 234.9% complete
2002/12/31 15:16:31| Store rebuilding is 231.6% complete
2002/12/31 15:16:46| Store rebuilding is 228.4% complete
2002/12/31 15:17:01| Store rebuilding is 225.4% complete
2002/12/31 15:17:16| Store rebuilding is 222.5% complete
2002/12/31 15:17:31| Store rebuilding is 219.8% complete
2002/12/31 15:17:46| Store rebuilding is 217.2% complete
2002/12/31 15:18:01| Store rebuilding is 214.7% complete
2002/12/31 15:18:16| Store rebuilding is 212.3% complete
2002/12/31 15:18:31| Store rebuilding is 210.0% complete
2002/12/31 15:18:46| Store rebuilding is 207.8% complete
2002/12/31 15:19:01| Store rebuilding is 205.7% complete
2002/12/31 15:19:16| Store rebuilding is 203.6% complete
2002/12/31 15:19:31| Store rebuilding is 201.7% complete
2002/12/31 15:19:46| Store rebuilding is 199.8% complete
2002/12/31 15:20:01| Store rebuilding is 198.0% complete
2002/12/31 15:20:16| Store rebuilding is 196.2% complete
2002/12/31 15:20:31| Store rebuilding is 194.5% complete
2002/12/31 15:20:46| Store rebuilding is 192.9% complete
2002/12/31 15:21:01| Store rebuilding is 191.4% complete
2002/12/31 15:21:16| Store rebuilding is 189.8% complete
2002/12/31 15:21:31| Store rebuilding is 188.4% complete
2002/12/31 15:21:46| Store rebuilding is 186.9% complete
2002/12/31 15:22:01| Store rebuilding is 185.6% complete
2002/12/31 15:22:16| Store rebuilding is 184.2% complete
2002/12/31 15:22:31| Store rebuilding is 182.9% complete
2002/12/31 15:22:46| Store rebuilding is 181.7% complete
2002/12/31 15:23:01| Store rebuilding is 180.4% complete
2002/12/31 15:23:16| Store rebuilding is 179.2% complete
2002/12/31 15:23:31| Store rebuilding is 178.1% complete
2002/12/31 15:23:46| Store rebuilding is 177.0% complete
2002/12/31 15:24:01| Store rebuilding is 175.9% complete
2002/12/31 15:24:16| Store rebuilding is 174.8% complete
2002/12/31 15:24:31| Store rebuilding is 173.8% complete
2002/12/31 15:24:46| Store rebuilding is 172.8% complete
2002/12/31 15:25:01| Store rebuilding is 171.8% complete
2002/12/31 15:25:16| Store rebuilding is 170.9% complete
2002/12/31 15:25:31| Store rebuilding is 170.0% complete
2002/12/31 15:25:46| Store rebuilding is 169.1% complete
2002/12/31 15:26:01| Store rebuilding is 168.2% complete
2002/12/31 15:26:16| Store rebuilding is 167.3% complete
2002/12/31 15:26:31| Store rebuilding is 166.5% complete
2002/12/31 15:26:46| Store rebuilding is 165.7% complete
2002/12/31 15:27:01| Store rebuilding is 164.9% complete
2002/12/31 15:27:16| Store rebuilding is 164.1% complete
2002/12/31 15:27:31| Store rebuilding is 163.4% complete
2002/12/31 15:27:46| Store rebuilding is 162.6% complete
2002/12/31 15:28:01| Store rebuilding is 161.9% complete
2002/12/31 15:28:16| Store rebuilding is 161.2% complete
2002/12/31 15:28:31| Store rebuilding is 160.5% complete
2002/12/31 15:28:46| Store rebuilding is 159.9% complete
2002/12/31 15:29:01| Store rebuilding is 159.2% complete
2002/12/31 15:29:16| Store rebuilding is 158.6% complete
2002/12/31 15:29:31| Store rebuilding is 157.9% complete
2002/12/31 15:29:46| Store rebuilding is 157.3% complete
2002/12/31 15:30:01| Store rebuilding is 156.7% complete
2002/12/31 15:30:16| Store rebuilding is 156.1% complete
2002/12/31 15:30:31| Store rebuilding is 155.5% complete
2002/12/31 15:30:46| Store rebuilding is 155.0% complete
2002/12/31 15:31:01| Store rebuilding is 154.4% complete
2002/12/31 15:31:16| Store rebuilding is 24377.1% complete
2002/12/31 15:31:31| Store rebuilding is 9490.1% complete
2002/12/31 15:31:46| Store rebuilding is 5920.7% complete
2002/12/31 15:32:01| Store rebuilding is 4317.6% complete
2002/12/31 15:32:16| Store rebuilding is 3406.8% complete
2002/12/31 15:32:31| Store rebuilding is 2819.5% complete
2002/12/31 15:32:46| Store rebuilding is 2409.4% complete
2002/12/31 15:33:01| Store rebuilding is 2109.1% complete
2002/12/31 15:33:16| Store rebuilding is 1874.2% complete
2002/12/31 15:33:31| Store rebuilding is 1690.0% complete
2002/12/31 15:33:46| Store rebuilding is 1540.4% complete
2002/12/31 15:34:01| Store rebuilding is 1416.6% complete
2002/12/31 15:34:16| Store rebuilding is 1312.4% complete
2002/12/31 15:34:31| Store rebuilding is 1223.4% complete
2002/12/31 15:34:46| Store rebuilding is 1146.6% complete
2002/12/31 15:35:01| Store rebuilding is 1080.8% complete
2002/12/31 15:35:16| Store rebuilding is 1022.3% complete
2002/12/31 15:35:31| Store rebuilding is 970.3% complete
2002/12/31 15:35:46| Store rebuilding is 924.7% complete
2002/12/31 15:36:01| Store rebuilding is 882.9% complete
2002/12/31 15:36:16| Store rebuilding is 845.8% complete
2002/12/31 15:36:31| Store rebuilding is 811.8% complete
2002/12/31 15:36:46| Store rebuilding is 780.7% complete
2002/12/31 15:37:01| Store rebuilding is 752.5% complete
2002/12/31 15:37:16| Store rebuilding is 726.3% complete
2002/12/31 15:37:31| Store rebuilding is 702.4% complete
2002/12/31 15:37:46| Store rebuilding is 680.2% complete
2002/12/31 15:38:01| Store rebuilding is 659.5% complete
2002/12/31 15:38:16| Store rebuilding is 640.3% complete
2002/12/31 15:38:31| Store rebuilding is 622.6% complete
2002/12/31 15:38:46| Store rebuilding is 605.8% complete
2002/12/31 15:39:01| Store rebuilding is 590.0% complete
2002/12/31 15:39:16| Store rebuilding is 575.2% complete
2002/12/31 15:39:31| Store rebuilding is 561.2% complete
2002/12/31 15:39:46| Store rebuilding is 547.7% complete
2002/12/31 15:40:01| Store rebuilding is 535.1% complete
2002/12/31 15:40:16| Store rebuilding is 523.2% complete
2002/12/31 15:40:31| Store rebuilding is 511.9% complete
2002/12/31 15:40:46| Store rebuilding is 501.3% complete
2002/12/31 15:41:01| Store rebuilding is 491.1% complete
2002/12/31 15:41:16| Store rebuilding is 481.5% complete
2002/12/31 15:41:31| Store rebuilding is 472.2% complete
2002/12/31 15:41:46| Store rebuilding is 463.5% complete
2002/12/31 15:42:01| Store rebuilding is 455.0% complete
2002/12/31 15:42:16| Store rebuilding is 447.1% complete
2002/12/31 15:42:31| Store rebuilding is 439.4% complete
2002/12/31 15:42:46| Store rebuilding is 432.1% complete
2002/12/31 15:43:01| Store rebuilding is 425.3% complete
2002/12/31 15:43:16| Store rebuilding is 418.7% complete
2002/12/31 15:43:31| Store rebuilding is 412.4% complete
2002/12/31 15:43:46| Store rebuilding is 406.1% complete
2002/12/31 15:44:01| Store rebuilding is 400.2% complete
2002/12/31 15:44:16| Store rebuilding is 394.4% complete
2002/12/31 15:44:31| Store rebuilding is 389.0% complete
2002/12/31 15:44:46| Store rebuilding is 383.7% complete
2002/12/31 15:45:01| Store rebuilding is 378.8% complete
2002/12/31 15:45:16| Store rebuilding is 374.0% complete
2002/12/31 15:45:31| Store rebuilding is 369.3% complete
2002/12/31 15:45:46| Store rebuilding is 364.8% complete
2002/12/31 15:46:01| Store rebuilding is 360.4% complete
2002/12/31 15:46:16| Store rebuilding is 356.1% complete
2002/12/31 15:46:31| Store rebuilding is 352.0% complete
2002/12/31 15:46:46| Store rebuilding is 347.9% complete
2002/12/31 15:47:01| Store rebuilding is 343.9% complete
2002/12/31 15:47:16| Store rebuilding is 340.1% complete
2002/12/31 15:47:31| Store rebuilding is 336.4% complete
2002/12/31 15:47:46| Store rebuilding is 332.9% complete
2002/12/31 15:48:01| Store rebuilding is 329.4% complete
2002/12/31 15:48:16| Store rebuilding is 326.1% complete
2002/12/31 15:48:31| Store rebuilding is 322.9% complete
2002/12/31 15:48:46| Store rebuilding is 319.7% complete
2002/12/31 15:49:01| Store rebuilding is 316.6% complete
2002/12/31 15:49:16| Store rebuilding is 313.6% complete
2002/12/31 15:49:31| Store rebuilding is 310.7% complete
2002/12/31 15:49:46| Store rebuilding is 307.9% complete
2002/12/31 15:50:01| Store rebuilding is 305.2% complete
2002/12/31 15:50:16| Store rebuilding is 302.5% complete
2002/12/31 15:50:31| Store rebuilding is 299.9% complete
2002/12/31 15:50:46| Store rebuilding is 297.4% complete
2002/12/31 15:51:01| Store rebuilding is 295.0% complete
2002/12/31 15:51:16| Store rebuilding is 292.6% complete
2002/12/31 15:51:31| Store rebuilding is 290.4% complete
2002/12/31 15:51:46| Store rebuilding is 288.2% complete
2002/12/31 15:52:01| Store rebuilding is 286.1% complete
2002/12/31 15:52:16| Store rebuilding is 284.0% complete
2002/12/31 15:52:31| Store rebuilding is 281.9% complete
2002/12/31 15:52:46| Store rebuilding is 279.9% complete
2002/12/31 15:53:01| Store rebuilding is 278.0% complete
2002/12/31 15:53:16| Store rebuilding is 276.0% complete
2002/12/31 15:53:31| Store rebuilding is 274.1% complete
2002/12/31 15:53:46| Store rebuilding is 272.2% complete
2002/12/31 15:54:01| Store rebuilding is 270.4% complete
2002/12/31 15:54:16| Store rebuilding is 268.6% complete
2002/12/31 15:54:31| Store rebuilding is 266.8% complete
2002/12/31 15:54:46| Store rebuilding is 265.1% complete
2002/12/31 15:55:01| Store rebuilding is 263.4% complete
2002/12/31 15:55:16| Store rebuilding is 261.7% complete
2002/12/31 15:55:31| Store rebuilding is 260.1% complete
2002/12/31 15:55:46| Store rebuilding is 258.5% complete
2002/12/31 15:56:01| Store rebuilding is 256.9% complete
2002/12/31 15:56:16| Store rebuilding is 255.4% complete
2002/12/31 15:56:31| Store rebuilding is 33407.5% complete
2002/12/31 15:56:46| Store rebuilding is 14523.9% complete
2002/12/31 15:57:01| Store rebuilding is 9277.1% complete
2002/12/31 15:57:16| Store rebuilding is 6845.5% complete
2002/12/31 15:57:31| Store rebuilding is 5432.9% complete
2002/12/31 15:57:46| Store rebuilding is 4516.4% complete
2002/12/31 15:58:01| Store rebuilding is 3859.2% complete
2002/12/31 15:58:16| Store rebuilding is 3376.1% complete
2002/12/31 15:58:31| Store rebuilding is 3000.2% complete
2002/12/31 15:58:46| Store rebuilding is 2704.1% complete
2002/12/31 15:59:01| Store rebuilding is 2462.9% complete
2002/12/31 15:59:16| Store rebuilding is 2262.7% complete
2002/12/31 15:59:31| Store rebuilding is 2092.5% complete
2002/12/31 15:59:46| Store rebuilding is 1948.3% complete
2002/12/31 16:00:01| Store rebuilding is 1823.6% complete
2002/12/31 16:00:16| Store rebuilding is 1714.7% complete
2002/12/31 16:00:31| Store rebuilding is 1618.0% complete
2002/12/31 16:00:46| Store rebuilding is 1532.9% complete
2002/12/31 16:01:01| Store rebuilding is 1456.9% complete
2002/12/31 16:01:16| Store rebuilding is 1388.0% complete
2002/12/31 16:01:31| Store rebuilding is 1326.3% complete
2002/12/31 16:01:46| Store rebuilding is 1270.3% complete
2002/12/31 16:02:01| Store rebuilding is 1218.9% complete
2002/12/31 16:02:16| Store rebuilding is 1171.8% complete
2002/12/31 16:02:31| Store rebuilding is 1128.8% complete
2002/12/31 16:02:46| Store rebuilding is 1088.8% complete
2002/12/31 16:03:01| Store rebuilding is 1051.5% complete
2002/12/31 16:03:16| Store rebuilding is 1017.1% complete
2002/12/31 16:03:31| Store rebuilding is 985.2% complete
2002/12/31 16:03:46| Store rebuilding is 955.2% complete
2002/12/31 16:04:01| Store rebuilding is 927.3% complete
2002/12/31 16:04:16| Store rebuilding is 901.3% complete
2002/12/31 16:04:31| Store rebuilding is 876.8% complete
2002/12/31 16:04:46| Store rebuilding is 853.9% complete
2002/12/31 16:05:01| Store rebuilding is 832.4% complete
2002/12/31 16:05:16| Store rebuilding is 811.9% complete
2002/12/31 16:05:31| Store rebuilding is 792.5% complete
2002/12/31 16:05:46| Store rebuilding is 774.3% complete
2002/12/31 16:06:01| Store rebuilding is 757.0% complete
2002/12/31 16:06:16| Store rebuilding is 740.5% complete
2002/12/31 16:06:31| Store rebuilding is 724.7% complete
2002/12/31 16:06:46| Store rebuilding is 709.9% complete
2002/12/31 16:07:01| Store rebuilding is 695.7% complete
2002/12/31 16:07:16| Store rebuilding is 682.1% complete
2002/12/31 16:07:31| Store rebuilding is 669.1% complete
2002/12/31 16:07:46| Store rebuilding is 656.6% complete
2002/12/31 16:08:01| Store rebuilding is 644.8% complete
2002/12/31 16:08:16| Store rebuilding is 633.5% complete
2002/12/31 16:08:31| Store rebuilding is 622.7% complete
2002/12/31 16:08:46| Store rebuilding is 612.3% complete
2002/12/31 16:09:01| Store rebuilding is 602.2% complete
2002/12/31 16:09:16| Store rebuilding is 592.6% complete
2002/12/31 16:09:31| Store rebuilding is 583.3% complete
2002/12/31 16:09:46| Store rebuilding is 574.3% complete
2002/12/31 16:09:50| diskHandleWrite: FD 30: disk write error: (28) No space
left on device
Received on Thu Jan 02 2003 - 09:46:02 MST
This archive was generated by hypermail pre-2.1.9 : Tue Dec 09 2003 - 17:12:24 MST