store.cc isEmpty() assert

Yesterday one of my busy 3.2.6 servers died after a rapid series of 'isEmpty()' asserts in store.cc. The servers use one AUFS cache directory and do not use workers. Here is part of the cache.log leading up to the assert: 2013/01/15 16:53:47 kid1| WARNING: 1 swapin MD5 mismatches 2013/01/15 16:53:47 kid1| Could not parse headers from on disk object 2013/01/15 16:53:47 kid1| BUG 3279: HTTP reply without Date: 2013/01/15 16:53:47 kid1| StoreEntry->key: 9F681734F6726E3C03DFF338A40FC82A 2013/01/15 16:53:47 kid1| StoreEntry->next: 0x27ad79f8 2013/01/15 16:53:47 kid1| StoreEntry->mem_obj: 0x3bc40890 2013/01/15 16:53:47 kid1| StoreEntry->timestamp: -1 2013/01/15 16:53:47 kid1| StoreEntry->lastref: 1358286827 2013/01/15 16:53:47 kid1| StoreEntry->expires: -1 2013/01/15 16:53:47 kid1| StoreEntry->lastmod: -1 2013/01/15 16:53:47 kid1| StoreEntry->swap_file_sz: 0 2013/01/15 16:53:47 kid1| StoreEntry->refcount: 1 2013/01/15 16:53:47 kid1| StoreEntry->flags: CACHABLE,PRIVATE,FWD_HDR_WAIT,VALIDATED 2013/01/15 16:53:47 kid1| StoreEntry->swap_dirn: -1 2013/01/15 16:53:47 kid1| StoreEntry->swap_filen: -1 2013/01/15 16:53:47 kid1| StoreEntry->lock_count: 3 2013/01/15 16:53:47 kid1| StoreEntry->mem_status: 0 2013/01/15 16:53:47 kid1| StoreEntry->ping_status: 1 2013/01/15 16:53:47 kid1| StoreEntry->store_status: 1 2013/01/15 16:53:47 kid1| StoreEntry->swap_status: 0 2013/01/15 16:53:47 kid1| assertion failed: store.cc:1850: "isEmpty()" I cleaned up the system by deleting the swap.state file and letting it rebuild the storage. A bit more than 2 hours later it finished loading 4.3 million objects in the 100 GB directory. It generated 150 "WARNING: Ignoring malformed cache entry." messages and purged 4920 duplicate URLs. I see the 'isEmpty()' assert about once a day across four of my busiest servers. This is the first time I've seen it happen frequently enough to cause the service to terminate. This looks related to http://bugs.squid-cache.org/show_bug.cgi?id=3279 http://bugs.squid-cache.org/show_bug.cgi?id=3483 http://bugs.squid-cache.org/show_bug.cgi?id=3745 Mike Mitchell
Received on Wed Jan 16 2013 - 16:00:45 MST