Re: [squid-users] 2.4.20-aa and LARGE Squid process -> SIGSEGV

From: Ralf Hildebrandt <Ralf.Hildebrandt@dont-contact.us>
Date: Sat, 21 Dec 2002 15:03:09 +0100

* Robert Collins <robertc@squid-cache.org>:

> > We're encountering sporadic crashes of the squid children (SIGSEGV,
> > signal 11). We were investigating in several directions:
>
> Which squid children?

Dunno. I'm just telling you what squid writes to /var/log/messages
(see below) - it clearly talks of "Squid Parent: child process"!

> Squid only spawns 1 one squid process. If you are using aufs, you may
> see many pids on linux, but that is due to a 'feature' of the linux
> thread support where each thread gets a different pid.

Dec 19 17:21:52 spidergirl -- MARK --
Dec 19 17:41:51 spidergirl -- MARK --
Dec 19 18:01:51 spidergirl -- MARK --
Dec 19 18:06:38 spidergirl squid[2198]: Squid Parent: child process 2200 exited due to signal 11
Dec 19 18:06:41 spidergirl squid[2198]: Squid Parent: child process 2796 started
Dec 19 18:06:42 spidergirl squid[2198]: Squid Parent: child process 2796 exited due to signal 11
Dec 19 18:06:45 spidergirl squid[2198]: Squid Parent: child process 2831 started

> > * the Kernel has highmem support enabled (we have 2GB physical RAM and
> > 4 GB swap)
>
> I've not used linux w/ highmem support - what changes does it make to
> the kernel memory interface?

Dunno.

> > * we closely observed dmesg, messages and syslog. No oddities were
> > found. Squid simply crashes with signal 11.
>
> What about cache.log?

There's no special stuff prior to the crashes. Only our tools querying the stats.

> > * we tried both ufs and aufs as cache filesystems, since the FAQ tells
> > us the async I/O may have bugs. Yet, the crashes still occur.
>
> diskd doesn't have any known bugs, and for any large squid install, ufs
> will perform *very* poorly.

Interesting. Maybe we should switch.

> Squid should trap the crash and log it to cache.log.

It doesn't. It's simply logging the restart, but not the crash!

> > cache_mem 500 MB
> > maximum_object_size 65536 KB
> > cache_dir aufs /squid-cache0 16000 256 256
> > cache_dir aufs /squid-cache1 16000 256 256
>
> This configuration will attempt to use ~ 900Mb of RAM. That should be
> fine.

And alas, this works fine. With "cache_mem 700 MB" it also works
excellently!

> As to your problem, may I suggest the following:
> 1) Use diskd. A nasty aufs bug was found recently that could impact you.
> 2) Check cache.log as your primary source for squid diagnostics.

See attached log - not entirely illuminating (for the unitiated?)

> 3) Check squid memory use via cache manager for statistics.

We're using http://wessels.squid-cache.org/squid-rrd/ to monitor &
graph all kinds of data. I'll attach some pics

> 4) Use squid 2.5-stable-cvs if you really want aufs support, the bug is
> fixed there.

The BUG? Which BUG? :) Note that our problem occurs with both ufs and
aufs.

-- 
Ralf Hildebrandt (Im Auftrag des Referat V a)   Ralf.Hildebrandt@charite.de
Charite Campus Mitte                            Tel.  +49 (0)30-450 570-155
Referat V a - Kommunikationsnetze -             Fax.  +49 (0)30-450 570-916
Deutschland: Kein Weltraum links auf dem Geraet. Pfeife zerbrochen.




fd.day.png objects.day.png memory.day.png
Received on Sat Dec 21 2002 - 07:03:13 MST

This archive was generated by hypermail pre-2.1.9 : Tue Dec 09 2003 - 17:12:10 MST