Slow ICP under high loads (squid-1.2beta20 & 21)

--MimeMultipartBoundary
Content-Type: text/plain; charset=us-ascii

For some time now I've been encountering very slow ICP responses from caches
under high load (sustained > 30 TCP & > 100 ICP requests/sec). I've managed
to trace this down to some code that was added a little while ago in the
effort to reduce CPU usage.

in comm.c:comm_poll (and comm_select if you use select)

            if (revents & (POLLRDNORM | POLLIN | POLLHUP | POLLERR)) {
                debug(5, 6) ("comm_poll: FD %d ready for reading\n", fd);
                if ((hdl = fd_table[fd].read_handler)) {
                    fd_table[fd].read_handler = NULL;
                    hdl(fd, fd_table[fd].read_data);
                }
** if ((incoming_counter++ & 15) == 0)
** comm_poll_incoming();
            }
            if (revents & (POLLWRNORM | POLLOUT | POLLHUP | POLLERR)) {
                debug(5, 5) ("comm_poll: FD %d ready for writing\n", fd);
                if ((hdl = fd_table[fd].write_handler)) {
                    fd_table[fd].write_handler = NULL;
                    hdl(fd, fd_table[fd].write_data);
                }
** if ((incoming_counter++ & 15) == 0)
** comm_poll_incoming();

        ie. the above code calls comm_poll_incoming every 16 read or write
requests. I've tried lower values, but the only one that seems to work
okay is if I remove the "if ((incoming_counter++ & 15) == 0)" test and just
call comm_poll_incoming for EVERY read or write event. This then results in
timely ICP transactions and packets not being dropped. I've tested this in
a production environment to a sustained 65 TCP hits/sec and 200 ICP hits/sec
with no ICP slowdown evident yet.

        I'm not sure what the best fix is, but the "every 16" approach is only
fine for low-medium loaded caches.

        Stew.

-- 
Stewart Forster (Snr. Development Engineer)
connect.com.au pty ltd, Level 9, 114 Albert Rd, Sth Melbourne, VIC 3205, Aust.
Email: slf@connect.com.au   Phone: +61 3 9251-3684   Fax: +61 3 9251-3666
--MimeMultipartBoundary--