V 1.2 or V 2.0 panics and dumps cache

From: Mark Reynolds <mark@dont-contact.us>
Date: Sat, 10 Oct 1998 12:24:17

Greetings!

I've been trying to locate a problem I've been
having with Squid 1.2beta22, 1.2beta25 and lately 2.0release.

Squid cache works fine until somewhere above 80% full, this starts

1998/10/10 10:18:28| diskHandleWrite: FD 10: disk write error: (28) No
space left on device
1998/10/10 10:18:28| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:18:28| WARNING: Shrinking cache_dir #1 to 1524642 KB
1998/10/10 10:30:08| diskHandleWrite: FD 16: disk write error: (28) No
space left on device
1998/10/10 10:30:08| storeSwapOutHandle: SwapOut failure (err code = -6).

That keeps happening, (shrinking cache) until I either notice, and restart
squid, or all but 40% ish has been purged.

I want to upgrade hard drive capacity and memory, but am not
keen on doing so until I can get this situation stable.

Any suggestions or further questions to better define the problem
would be most welcome.

I'm tending towards thinking faulty hard drive, or maybe not enough memory.

I've included details of all the things I've been watching.

Summary info
------------

Current squid version is 2.0release
OS is Linux 2.0.34, Redhat 5.1
All running with the help of a P233 CPU.

Boot, squid, squid logs drive is a small recycled IDE drive.
Logs are analysed, and rotated each night. Only 1 copy is kept.

Cache drives are on a single PCI scsi card, using ncr53c8xx module.
Main cache drives are 2 recycled 2gig scsi's, configured:

cache_dir /usr/local/squid/cache1 1800 16 256
cache_dir /usr/local/squid/cache2 1800 16 256

Midnight, last night :
df :
Filesystem 1024-blocks Used Available Capacity Mounted on
/dev/hda2 169819 122450 38599 76% /
/dev/sda1 2028862 1543746 380230 80%
/usr/local/squid/cache1
/dev/sdb1 2028862 1542416 381560 80%
/usr/local/squid/cache2

At the moment:

df
Filesystem 1024-blocks Used Available Capacity Mounted on
/dev/hda2 169819 108722 52327 68% /
/dev/sda1 2028862 1154317 769659 60%
/usr/local/squid/cache1
/dev/sdb1 2028862 1157121 766855 60%
/usr/local/squid/cache2

At midnight, last night,

Store Directory Statistics:
Store Entries : 264477
Maximum Swap Size : 3686400 KB
Current Store Swap Size: 3007967 KB
Current Capacity : 82% used, 18% free

Store Directory #0: /usr/local/squid/cache1
First level subdirectories: 16
Second level subdirectories: 256
Maximum Size: 1843200 KB
Current Size: 1503960 KB
Percent Used: 81.60%
Filemap bits in use: 132595 of 335127 (40%)
Flags: SELECTED

Store Directory #1: /usr/local/squid/cache2
First level subdirectories: 16
Second level subdirectories: 256
Maximum Size: 1843200 KB
Current Size: 1504007 KB
Percent Used: 81.60%
Filemap bits in use: 130227 of 335127 (39%)
Flags:

After last 'shrinking event' :

Store Directory Statistics:
Store Entries : 193008
Maximum Swap Size : 3686400 KB
Current Store Swap Size: 2167526 KB
Current Capacity : 59% used, 41% free

Store Directory #0: /usr/local/squid/cache1
First level subdirectories: 16
Second level subdirectories: 256
Maximum Size: 1843200 KB
Current Size: 1083767 KB
Percent Used: 58.80%
Filemap bits in use: 96912 of 335127 (29%)
Flags: SELECTED

Store Directory #1: /usr/local/squid/cache2
First level subdirectories: 16
Second level subdirectories: 256
Maximum Size: 1843200 KB
Current Size: 1083759 KB
Percent Used: 58.80%
Filemap bits in use: 96091 of 335127 (29%)
Flags:

---
Complied with 256 file descriptors.
(I don't see this as currently being a problem. )
Midnight, last night:
File descriptor usage for squid:
	Maximum number of file descriptors:    256
	Largest file desc currently in use:     47
	Number of file desc currently in use:   30
	Available number of file descriptors:  226
	Reserved number of file descriptors:    64
	Disk files open:                         4
Currently:
File descriptor usage for squid:
        Maximum number of file descriptors:    256
        Largest file desc currently in use:     45
        Number of file desc currently in use:   26
        Available number of file descriptors:  230
        Reserved number of file descriptors:    64
        Disk files open:                         4
---
Memory is 96 meg, which also seems to be fine.
Midnight stats were :
free : 
             total       used       free     shared    buffers     cached
Mem:         95480      94104       1376      13212       5584      14596
-/+ buffers/cache:      73924      21556
Swap:        32808          0      32808
vmstat : 
 procs                  memory    swap        io    system         cpu
 r b w  swpd  free  buff cache  si  so   bi   bo   in   cs  us  sy  id
 0 0 0     0  1356  5584 14604   0   0    4    4  178   55   2   1  98
Stats at the moment are :
free                  
             total       used       free     shared    buffers     cached
Mem:         95480      82468      13012      12564      35416      13036
-/+ buffers/cache:      34016      61464
Swap:        32808          0      32808
vmstat                  
 procs                  memory    swap        io    system         cpu
 r b w  swpd  free  buff cache  si  so   bi   bo   in   cs  us  sy  id
 0 0 0     0 12904 35480 13060   0   0    4    5  173   56   2   1  98
   
---
Midnight last night :
mgr:info : 
Squid Object Cache: Version 2.0.RELEASE
Start Time:	Wed, 07 Oct 1998 08:47:48 GMT
Current Time:	Fri, 09 Oct 1998 18:00:01 GMT
Connection information for squid:
	Number of clients accessing cache:	70
	Number of HTTP requests received:	189450
	Number of ICP messages received:	2683280
	Number of ICP messages sent:	2692803
	Number of queued ICP replies:	1
	Request failure ratio:	 0.00%
	HTTP requests per minute:	55.2
	ICP messages per minute:	1566.4
	Select loop called: 5189231 times, 39.685 ms avg
Cache information for squid:
	Storage Swap size:	3007967 KB
	Storage Mem size:	31120 KB
	Storage LRU Expiration Age:	 90.00 days
	Mean Object Size:	11.44 KB
	Requests given to unlinkd:	10215
Median Service Times (seconds)  5 min    60 min:
	HTTP Requests (All):   0.39928  0.32154
	Cache Misses:          1.54242  1.31166
	Cache Hits:            0.04776  0.07014
	Near Hits:             0.33943  0.68577
	Not-Modified Replies:  0.00405  0.05331
	DNS Lookups:           0.06083  0.11926
	ICP Queries:           0.09506  0.10091
Resource usage for squid:
	UP Time:	205933.908 seconds
	CPU Time:	3707.220 seconds
	CPU Usage:	1.80%
	CPU Usage, 5 minute avg:	4.63%
	CPU Usage, 60 minute avg:	1.33%
	Maximum Resident Size: 0 KB
	Page faults with physical i/o: 149526
Memory usage for squid via mallinfo():
	Total space in arena:   63543 KB
	Ordinary blocks:        63031 KB   2919 blks
	Small blocks:               0 KB      0 blks
	Holding blocks:          2264 KB      6 blks
	Free Small blocks:          0 KB
	Free Ordinary blocks:     512 KB
	Total in use:           65295 KB 103%
	Total free:               512 KB 1%
Memory accounted for:
	Total accounted:        54894 KB
File descriptor usage for squid:
	Maximum number of file descriptors:    256
	Largest file desc currently in use:     47
	Number of file desc currently in use:   30
	Available number of file descriptors:  226
	Reserved number of file descriptors:    64
	Disk files open:                         4
Internal Data Structures:
	264477 StoreEntries
	  6294 StoreEntries with MemObjects
	  6288 Hot Object Cache Items
	262822 Filemap bits set
	262822 on-disk objects
And just now :
mgr:info                                        
 
Squid Object Cache: Version 2.0.RELEASE
Start Time:     Sat, 10 Oct 1998 02:49:52 GMT
Current Time:   Sat, 10 Oct 1998 04:12:14 GMT
Connection information for squid:
        Number of clients accessing cache:      20
        Number of HTTP requests received:       3615
        Number of ICP messages received:        62503
        Number of ICP messages sent:    62612
        Number of queued ICP replies:   0
        Request failure ratio:   0.00%
        HTTP requests per minute:       43.9
        ICP messages per minute:        1518.8
        Select loop called: 124687 times, 39.641 ms avg
Cache information for squid:
        Storage Swap size:      2167907 KB
        Storage Mem size:       2532 KB
        Storage LRU Expiration Age:      90.00 days
        Mean Object Size:       11.23 KB
        Requests given to unlinkd:      117
Median Service Times (seconds)  5 min    60 min:
        HTTP Requests (All):   0.05046  0.16775
        Cache Misses:          1.46131  1.38447
        Cache Hits:            0.05046  0.05331
        Near Hits:             0.00000  1.31166
        Not-Modified Replies:  0.03427  0.04277
        DNS Lookups:           0.06364  0.08717
        ICP Queries:           0.12070  0.15326
Resource usage for squid:
        UP Time:        4942.757 seconds
        CPU Time:       82.550 seconds
        CPU Usage:      1.67%
        CPU Usage, 5 minute avg:        1.02%
        CPU Usage, 60 minute avg:       1.46%
        Maximum Resident Size: 0 KB
        Page faults with physical i/o: 5384
Memory usage for squid via mallinfo():
        Total space in arena:   22171 KB
        Ordinary blocks:        22093 KB   3087 blks
        Small blocks:               0 KB      0 blks
        Holding blocks:           420 KB      1 blks
        Free Small blocks:          0 KB
        Free Ordinary blocks:      78 KB
        Total in use:           22513 KB 102%
        Total free:                78 KB 0%
Memory accounted for:
        Total accounted:        16996 KB
File descriptor usage for squid:
        Maximum number of file descriptors:    256
        Largest file desc currently in use:     45
        Number of file desc currently in use:   30
        Available number of file descriptors:  226
        Reserved number of file descriptors:    64
        Disk files open:                         6
Internal Data Structures:
        193033 StoreEntries
           482 StoreEntries with MemObjects
           479 Hot Object Cache Items
        193028 Filemap bits set
        193026 on-disk objects
----
cache.log from when purging started :
1998/10/10 10:18:28| diskHandleWrite: FD 10: disk write error: (28) No
space left on device
1998/10/10 10:18:28| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:18:28| WARNING: Shrinking cache_dir #1 to 1524642 KB
1998/10/10 10:30:08| diskHandleWrite: FD 16: disk write error: (28) No
space left on device
1998/10/10 10:30:08| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:30:08| WARNING: Shrinking cache_dir #1 to 1436514 KB
1998/10/10 10:34:45| diskHandleWrite: FD 21: disk write error: (28) No
space left on device
1998/10/10 10:34:45| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:34:45| WARNING: Shrinking cache_dir #0 to 1382024 KB
1998/10/10 10:40:20| diskHandleWrite: FD 29: disk write error: (28) No
space left on device
1998/10/10 10:40:20| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:40:20| WARNING: Shrinking cache_dir #1 to 1341462 KB
1998/10/10 10:42:32| diskHandleWrite: FD 35: disk write error: (28) No
space left on device
1998/10/10 10:42:32| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:42:32| WARNING: Shrinking cache_dir #0 to 1279035 KB
1998/10/10 10:43:56| diskHandleWrite: FD 10: disk write error: (28) No
space left on device
1998/10/10 10:43:56| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:43:56| WARNING: Shrinking cache_dir #1 to 1248036 KB
1998/10/10 10:44:22| diskHandleWrite: FD 5: disk write error: (28) No space
left on device
1998/10/10 10:44:22| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:44:22| WARNING: Shrinking cache_dir #0 to 1204931 KB
1998/10/10 10:44:36| diskHandleWrite: FD 5: disk write error: (28) No space
left on device
1998/10/10 10:44:36| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:44:36| WARNING: Shrinking cache_dir #1 to 1173167 KB
1998/10/10 10:47:57| diskHandleWrite: FD 6: disk write error: (28) No space
left on device
1998/10/10 10:47:57| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:47:57| WARNING: Shrinking cache_dir #0 to 1117214 KB
1998/10/10 10:48:38| Preparing for shutdown after 203171 requests
1998/10/10 10:48:38| Waiting 30 seconds for active connections to finish
1998/10/10 10:48:38| FD 11 Closing HTTP connection
1998/10/10 10:49:10| Shutting down...
1998/10/10 10:49:10| FD 46 Closing ICP connection
1998/10/10 10:49:10| FD 47 Closing SNMP socket
1998/10/10 10:49:10| Closing unlinkd pipe on FD 18
1998/10/10 10:49:10| storeDirWriteCleanLogs: Starting...
1998/10/10 10:49:10|     65536 entries written so far.
1998/10/10 10:49:11|    131072 entries written so far.
1998/10/10 10:49:11|   Finished.  Wrote 192482 entries.
1998/10/10 10:49:11|   Took 1 seconds (192482.0 entries/sec).
[end]
.----------------------------------------------.------------------------.
| Mark Reynolds <mark@rts.com.au>              | Phone  08 9474 1211    |
| Network Manager, Reynolds Technology Pty Ltd | Fax    08 9474 4772    |
| 8 Preston Street Como 6152 Western Australia | Pager  08 9480 5884    |
| PO Box 120 Como 6952 Western Australia       | http://www.rts.com.au/ |
`----------------------------------------------^------------------------'
Received on Fri Oct 09 1998 - 21:30:46 MDT

This archive was generated by hypermail pre-2.1.9 : Tue Dec 09 2003 - 16:42:24 MST