Re: [squid-users] RE: Squid CPU 100% infinite loop

What is your squid fd limit and your system ulimit (ulimit -n) for squid
user ?

-- 
Best regards,
Loïc BLOT, 
UNIX systems, security and network expert
http://www.unix-experience.fr
Le mercredi 29 mai 2013 à 14:03 -0400, Ron Wheeler a écrit :
> Have you looked at garbage collection as a possible source of the problem?
> If you really have a 300 GB cache, that might take a long time to 
> process during GC.
> 
> You might want to post your GC settings to see if anyone has a 
> suggestion or can eliminate GC as the source of your problem.
> 
> The fact that everything works before, during and after except that your 
> CPU is too busy to talk to you, does make one suspect that it is a 
> normal occurrence as far as Squid knows and not worth logging or noting 
> for humans to read.
> 
> 
> Ron
> 
> On 29/05/2013 1:48 PM, Mike Mitchell wrote:
> > I've hit something similar.  I have four identically configured systems with 16K squid FD limit, 24 GB RAM, 300 GB cache directory.  I've seen the same failure randomly on all four systems.  During the day the squid process handles > 100 requests/second, with a peak FD usage around 8K FDs.  In the evenings the load drops to about 20 requests/second, with an FD usage around 1K FDs.  CPU usage hovers less than 10% during this time.
> > Randomly one of the four systems will start increasing its CPU usage.  It takes about 4 hours to go from less than 10% to 100%.  During the four hours the FD usage stays at 1K and the request rate stays right around 20 requests/second.  Once the CPU reaches 100% the squid service stops responding.  About 20 minutes later it starts responding again with CPU levels back down below 10%.  There is nothing in the cache log to indicate a problem.  The squid process did not core dump, nor did the parent restart a child.
> >
> > I have not seen the problem during the day, only after the load drops.  The hangs do not coincide with the scheduled log rotates.  The one last night recovered a half-hour before the log rotated at 2:00 AM.
> >
> > Every one of my hangs have been proceeded with a rise in CPU usage, and squid recovers on its own without logging anything.
> >
> > I have a script that does
> >    GET cache_object://localhost/info
> >    GET cache_object://localhost/counters
> > every five minutes and puts the interesting (to me) bits into RRD files.
> > Obviously the script fails during the 20 minutes the squid process is non-responsive.
> >
> > ________________________________________
> > From: Stuart Henderson [stu_at_spacehopper.org]
> > Sent: Tuesday, May 28, 2013 12:01 PM
> > To: squid-users_at_squid-cache.org
> > Subject: Re: Squid CPU 100% infinite loop
> >
> > On 2013-05-17, Alex Rousskov <rousskov_at_measurement-factory.com> wrote:
> >> On 05/17/2013 01:28 PM, Loïc BLOT wrote:
> >>
> >>> I have found the problem. In fact it's the problem mentionned on my
> >>> last mail, is right. Squid FD limit was reached, but squid doesn't
> >>> mentionned every time the freeze appear that it's a FD limit
> >>> problem, then the debug was so difficult.
> >> Squid should warn when it runs out of FDs. If it does not, it is a
> >> bug. If you can reproduce this, please open a bug report in bugzilla
> >> and post relevant logs there.
> >>
> >> FWIW, I cannot confirm or deny whether reaching FD limit causes what
> >> you call an infinite loop -- there was not enough information in your
> >> emails to do that. However, if reaching FD limit causes high CPU
> >> usage, it is a [minor] bug.
> > I've just hit this one, ktrace shows that it's in a tight loop doing
> > sched_yield(), I'll try and reproduce on a non-production system and open
> > a ticket if I get more details..
> >
> >
> >
> >
> >
> 
>