Re: Performance Tuning squid on Sun UE1

From: Matthew Petach <mattp@dont-contact.us>
Date: Thu, 11 Sep 1997 11:31:03 -0700 (PDT)

Recently, Jamie Wilson talked about "Performance Tuning squid on Sun UE1", and said
>
> Hi,
>
> I have squid 1.15 installed on a fairly standard Sun Ultra Enterprise 1
> 170, but I seem to be getting less than optimal performance out of it.
> At the moment, from my brief tests, I seem to be getting throughput of
> around 1.4kbps over a 33.6kbps modem (even when downloading cached
> objects). If I turn off caching I generally get full 3.something kbps
> throughput (our provider has bandwidth to burn at the moment).

Classic problem.

If you used a sniffer on the line from your Solaris box to your
modem bank, you'd probably see about 50% retransmits.

You can get a rough verification of this by using

netstat -s

and looking in the TCP section at these values:

On a properly tuned box, they should look like this:
        tcpOutDataSegs =25696062 tcpOutDataBytes =1874883022
        tcpRetransSegs =267496 tcpRetransBytes =3087495

On a box without tuning, you'll typically see the Retrans numbers
being almost as high, if not higher than the Out numbers.

Make SURE the Solaris 2.5.1 TCP retransmit tuning patch (103582)
has been applied to the machine, and be sure to adjust your
tcp parameters correctly on startup; in our /etc/init.d/inetinit
file, we set them as follows:

#
# Set configurable parameters.
#
ndd -set /dev/tcp tcp_old_urp_interpretation 1
ndd -set /dev/tcp tcp_conn_req_max 1024
ndd -set /dev/tcp tcp_rexmit_interval_initial 2048
ndd -set /dev/tcp tcp_rexmit_interval_max 16384
ndd -set /dev/tcp tcp_rexmit_interval_min 2048
 
#
# Added SYN attack precautions from Sun Security Bulletin #136.
# --MNP 10-18-96
#
# *** SYN Attack Protection ***
# 1) Decrease timeout to 10 seconds (default 180 seconds)
# 2) Increase maximum allowable incoming queue to 10240
# 3) Make incoming queue maximum 8192 (default 32)
ndd -set /dev/tcp tcp_ip_abort_cinterval 10000
echo "tcp_param_arr+14/W 0t10240" | adb -kw /dev/ksyms /dev/mem
ndd -set /dev/tcp tcp_conn_req_max 8192
                                                                 

> I figure there must be some tweaking that needs to be done - either to
> squid or Solaris, but, unfortunately, i'm not really sure where to start
> digging around.

Start with your OS. "standard" Solaris sucks large boulders through
a very small hose at velocities approaching C. Once tuned, you
should be MUCH happier with your performance.
 
> If anyone could help (even if it's only saying that I need to provide
> more information!) I would appreciate it - privately or to the list.

I think this might be useful information for anyone running Solaris.

> Regards,
> Jamie Wilson
> zusammen limited

Here's the full text of the TCP retransmit bug as submitted
to Sun (bugID 1233827) (this is very long and involved):
++++++++++++++++++++++ begin included text +++++++++++++++++++++++++
Bug Id: 1233827
 Category: kernel
 Subcategory: tcp-ip
 State: integrated
 Release summary: 5.5, s495, s495_fcs_phase, 5.4, 1.0.3_beta4, 2.5, s495_beta, 5
.3
 Synopsis: tcp retransmits too much for short connections as seen at web sites
        Integrated in releases: s297_12
 Patch id:
Description:
 
 
Architext (www.excite.com) is a prominent Web search engine site. They
are reporting excessive retransmissions using the Solaris 2.5 default values.
 
Email record of some emails is included:
 
 
==============
>From mukesh@caribe Wed Jan 10 11:27 PST 1996
Return-Path: <mukesh@caribe>
Received: from Eng.Sun.COM (engmail2) by caribe.eng.sun.com (5.x/SMI-SVR4)
        id AA08218; Wed, 10 Jan 1996 11:27:09 -0800
Received: from haven.eng.sun.com by Eng.Sun.COM (5.x/SMI-5.3)
        id AA15313; Wed, 10 Jan 1996 11:27:35 -0800
Received: from caribe.eng.sun.com by haven.eng.sun.com (SMI-8.6/SMI-SVR4)
        id LAA23510; Wed, 10 Jan 1996 11:26:21 -0800
Received: from lucknow.eng.sun.com by caribe.eng.sun.com (5.x/SMI-SVR4)
        id AA08213; Wed, 10 Jan 1996 11:26:57 -0800
Received: by lucknow.eng.sun.com (5.x/SMI-SVR4)
        id AA17408; Wed, 10 Jan 1996 11:25:28 -0800
Date: Wed, 10 Jan 1996 11:25:28 -0800
From: mukesh@caribe (Mukesh Kacker)
Message-Id: <9601101925.AA17408@lucknow.eng.sun.com>
To: ben@atext.com
Subject: Re: tcp retransmit parameter
Cc: mukesh@caribe, suryakumar.josyula@Corp
X-Sun-Charset: US-ASCII
Content-Type: text
Content-Length: 2514
X-Lines: 59
Status: RO
 
 
> hi mukesh,
>
> i was hoping you could give me a little advice about tcp
> nretransmissions. i and others have noticed that our web site
> (www.excite.com) has a very high retransmission rate. i believe it's
> because of this:
> www00# ndd /dev/tcp tcp_rexmit_interval_min
> 200
> i think that the web machine is spitting out a duplicate packet
> without waiting long enough to see if the target host got it.
>n
> given this, can you recommend values for
> tcp_rexmit_interval_initial
> tcp_rexmit_interval_max
> tcp_rexmit_interval_min
>
....
>
>
> for the time being, i set
> www00# ndd -set /dev/tcp tcp_rexmit_interval_min 600
> www00# ndd -set /dev/tcp tcp_rexmit_interval_initial 1000
>
 
I was hoping people with operational experience such as you will
tell *us* what are the good values for these :-)
 
These parameters do affect TCP retransmission rates
and I suppose you do understand what these do and are picking the right ones.
Since no two TCP connections are alike, there is no one-size-fits-all
I think your operational experience with your site can best help you
determine the value.
 
As we discussed before in earlier e-mail exchanges, high values these
parameters will help if the problem is "unnecessary" retransmissions
because the TCP round-trip time estimation algorithms takes a certain minimum
amount of data transfer before its estimate approaches close the the actual
value. This assumes that most TCP connections to your site transfer data
less than the minimum needed for estimating round-trip time.
 
If on the other hand, these retransmissions are because Internet is
congested and routers are dropping data, then high values of these
parameters will *hurt* performance as seen by your users since they
will have to wait longer for data that got lost.
 
Also, you have not specified why you cosnider the retransmission rates
you observe as "high". They are "high" compared to what ?Again as we
discussed earlier, if you are comparing with SunOS 4.X (or any other BSD
based implmentation) with the default SunOS 5.X (Solaris 2.X) values,
there will be a difference because of use of a finer granularity timers.
[ The BSD based OS has avery coarse grain timer of 500ms unit and that
skews TCP behavior which is not necessarily a good thing ]. Ofcurse you
should pursue with Sun specific performance problems observed at your
site but a statistics like retransmission rates may not be the cause
of whatever problems you might have observed.
 
-Mukesh Kacker
 Internet Engineering.
 
 
===========
>From ben@atext.com Wed Jan 10 14:04 PST 1996
Return-Path: <ben@atext.com>
Received: from Eng.Sun.COM (engmail1) by caribe.eng.sun.com (5.x/SMI-SVR4)
        id AA21193; Wed, 10 Jan 1996 14:04:07 -0800
Received: from mercury.Sun.COM (mercury.EBay.Sun.COM) by Eng.Sun.COM (5.x/SMI-5.
3)
        id AA16054; Wed, 10 Jan 1996 14:04:35 -0800
Received: from mas.atext.com by mercury.Sun.COM (Sun.COM)
        id OAA07689; Wed, 10 Jan 1996 14:04:34 -0800
Received: (from picklock@localhost) by mas.atext.com (8.6.9/8.6.9) id OAA27945;
Wed, 10 Jan 1996 14:04:34 -0800
Date: Wed, 10 Jan 1996 14:04:34 -0800
From: Ben Lutch <ben@atext.com>
Message-Id: <199601102204.OAA27945@mas.atext.com>
To: Mukesh.Kacker@Eng
Cc: suryakumar.josyula@Corp
Subject: Re: tcp retransmit parameter
Content-Type: text
Content-Length: 680
X-Lines: 19
Status: RO
 
 
 
> I was hoping people with operational experience such as you will
> tell *us* what are the good values for these :-)
 
heh. well, i *can* tell you that the default values are way to low
for someone running a web site. since we have people connecting from
all sorts of links - from 14.4 up to T3's, i've found that the default
values cause a *lot* - like 75% - of retransmitted packets, most of
which don't need to be retransmitted.
 
OS - i'm running solaris 2.5 on our web servers. it's pretty vanilla;
the backlog param is 256.
 
i'll keep you posted about what i find ..
thanks for the info,
ben
ps - we're getting ~4 million hits/day now. had to upgrade to a
20mbit link.
 
============
 
The description field as copied from bug report 1233973 follows:
 
Well we finally converted everything to 2.5 FCS, and one of the
first things I noticed is that async PPP has really slowed up.
 
I can boot either 2.4 or 2.5 on my home machine, so I'm in a good
position to test. My standard test is ttcp, which will send very
compressible data (through V42BIS) and is basically serial port
limited. On 2.4 I get about 28 kilobits of throughput on 2.5 I
get about 10 kilobits this on a 38.4 dte rate.
 
In my tests the sending machine is using the bundled async ppp
talking ppp to an annex, and sending to a fixed machine on
the lan side of the annex. For diagnostic purposes I can capture
packets on the lan side, and can do netstat -s on either side.
 
Looking at netstat -s before and after on the sending machine I see:
 
2.4: tcpoutdatasegs 116 , tcpretranssegs 11
2.5: tcpoutdatasegs 147 , tcpretranssegs 232
 
 
I have full netstat -s before and after on 2.4/2.5 as well as
tcpdump's for both sessions if your interested. The stime
graph on the 2.4 session is straight line after the first few
seconds, the 2.5 is jagged for the whole session.
 
Any ideas, or should I just submit this through sunservice?
 
Also, do you know of a version of tcpdump that works on the
the ipdptp* interfaces. The standard version says its working
but never captures anything. Snoop works, but the tcpdump output
is easier to analyze with the awk scripts provided.
 
===
I'm also seeing degraded NFS performance over ISDN (V3 TCP on 2.5
vs V2 UDP on 2.4). I haven't looked at that closely but it looks
like lots of retrans, and I wonder if its the same problem.
 
Ken Mandelberg | km@mathcs.emory.edu PREFERRED
Emory University | {rutgers,gatech}!emory!km UUCP
Dept of Math and CS | km@emory.bitnet NON-DOMAIN BITNET
Atlanta, GA 30322 | Phone: Voice (404) 727-7963, FAX 727-5611
 
The description field as copied from bug report 1235408 follows:
 
There seems to be a performance problem when ftp'ing a file from a Solaris
2.4 system to a Solaris 2.5 system in both directions at the same time.
 
I'm only using 1B channel with no compression's and PPP encapsulation. It
looks like one ftp works at full speed saturating the B chan "out" on one
system and "in" on the other at 64Kbits/sec where as the other ftp (in the
other direction) operates at less than half speed. The system that has it's
B "out" at 64k has t's B "in" at less than 32K most of the time.
 
The two system's I'm using are:
 
System A :- sc1000 64Mbytes of mem, 4 cpu's, Solaris 2.4
System B :- Neutron+ 32Mbytes of mem, 1 cpu, Solaris 2.5
 
 
So it's not that the systems are not fast enough to keep up or are slow
writing to disk (I've done the ftp's over ethernet to prove this). The bottle
neck seems to be somewhere in the isdn driver or ppp module and it's only
a problem when both ftp's are active together. If two separate ftp's are done
you get a much higher throughput, also there is no problem withtwo Solaris 2.4
systems doing two simultaneous ftp's. The throughput results are shown below.
 
The Solaris 2.4 to 2.4 ftp's have throughput's of > 6Kbytes/sec in both dir's
where as Solaris 2.4 to 2.5 only achieves 4.5 Kbyes/sec with the simultaneous
Solaris 2.5 to 2.4 ftp achieving from 2.6Kbyes/sec!!! to 7.5Kbyes/sec.
 
This seems to be random and way below expected performance. 2.6Kbytes/sec
translates into < 20Kbits/sec!!. And ISDN is supposed to be 64Kbits/sec
FULL DUPLEX!!
 
 
ISDN ftp stat's in KBytes/sec
=============================
 
SS20 SC1000 SC1000 Neutron
 to to to to
SC1000 SS20 Neutron SC1000 Comments
===== ====== ======= ======= =======
6 6 4.5 2.6 Simultaneous ftp's After reboot.
 
6.3 6.5 4.2 7.5 Simultaneous ftp's After above ftp's
 
6.2 6.1 4.4 2.6 Simultaneous ftp's After above ftp's
 
- - 4.6 2.6 Simultaneous Neutron <-> SC1000 ftp's
                                  After above ftp's
 
- - 7.1 - SC1000 to Neutron only After above ftp's
 
- - - 7.6 Neutron to SC1000 only After above ftp's
 
 
 
The Ethernet ftp stats below show that the cpu power and disk acces of the
systems involved are not bottle necks.
 
 
Ethernet ftp stat's in KBytes/sec
==========-======================
 
SS20 SC1000 SC1000 Neutron
 to to to to
SC1000 SS20 Neutron SC1000 Comments
===== ====== ======= ====== ========
660 530 470 540 Simultaneous ftp's After reboot.
 
420 350 310 330 Simultaneous ftp's After above ftp's
 
410 360 310 340 Simultaneous ftp's After above ftp's
 
- - 530 520 Simultaneous Neutron <-> SC1000 ftp's
                                        After above ftp's
 
- - 720 - SC1000 to Neutron only After above ftp's
 
- - - 730 Neutron to SC1000 only After above ftp's
 
 
 
path_stat's Stat's When both ftp's active
=========================================
 
sc1000
------
PPP over ISDN Statistics: Wed Jan 24 12:56:44 1996
 
dir name type state in / out directory_number
----------------------------------------------------------------------
 
     ss20 p inactive
     ss20 p inactive
 
     isdn1 p inactive
 
     neutron p inactive
in neutron p ip 64000/35488_B1 16052139
 
 
Neutron
-------
PPP over ISDN Statistics: Wed Jan 24 12:54:21 1996
 
dir name type state in / out directory_number
----------------------------------------------------------------------
 
     isdn1 p inactive
 
out sc1000-1 p ip 35352/64000_B1 6052118
     sc1000-1 p inactive
 
 
path_stats after sc10000 to neutron ftp finished
===============================================
 
sc10000
-------
PPP over ISDN Statistics: Wed Jan 24 13:00:31 1996
 
dir name type state in / out directory_number
----------------------------------------------------------------------
 
     ss20 p inactive
     ss20 p inactive
 
     isdn1 p inactive
 
     neutron p inactive
in neutron p ip 64000/01920_B1 16052139
 
 
 
 
Neutron
-------
PPP over ISDN Statistics: Wed Jan 24 12:57:55 1996
 
dir name type state in / out directory_number
----------------------------------------------------------------------
 
     isdn1 p inactive
 
out sc1000-1 p ip 01824/64000_B1 6052118
     sc1000-1 p inactive
 
jmcl - Wed Jan 31 11:16:32 PST 1996
 
I have verified this on two systems here. One is cat-atm15, a galaxy running 2.5
and cat-atm16, an SS2 running 2.4. Ftp's run in both directions one at a time
work fine, however two simultaneous ftp's cause a problem. It is always in
the 2.4 system to 2.5 system direction that gets slowed down. I have tried
this with the new MLP encapsulation as well as with the "old" 1.0.2 BOD method
and it makes no difference - the 2.5 to 2.4 direction is slower (and oddly it
looks like the 2.5 to 2.4 direction gets faster??!!)
 
Here's the ftp output from the 2.4 system (cat-atm16):
ftp> bin
200 Type set to I.
ftp> put /kernel/unix /tmp/unix
200 PORT command successful.
150 Binary data connection for /tmp/unix (129.146.204.18,36239).
226 Transfer complete.
local: /kernel/unix remote: /tmp/unix
902540 bytes sent in 67 seconds (13 Kbytes/s)
ftp> put /kernel/unix /tmp/unix
200 PORT command successful.
150 Binary data connection for /tmp/unix (129.146.204.18,36243).
226 Transfer complete.
local: /kernel/unix remote: /tmp/unix
902540 bytes sent in 1e+02 seconds (8.6 Kbytes/s)
ftp> quir
?Invalid command
ftp> quit
221 Goodbye.
cat-atm16#
 
The first transfer giving 13Kbytes/s was done when it was running by itself. The
second was done while another ftp was running in the other direction. The perfor
mance
of the 2nd ftp is actually worse than shown because the first ftp run in the oth
er
direction finished much sooner giving this ftp a chance to "catch-up".
 
Here is the ftp output from the 2.5 system (cat-atm15):
ftp> lcd /tmp
Local directory now /tmp
ftp> put /kernel/genunix /tmp/genunix
200 PORT command successful.
150 Binary data connection for /tmp/genunix (129.146.204.17,33030).
226 Transfer complete.
local: /kernel/genunix remote: /tmp/genunix
876724 bytes sent in 59 seconds (15 Kbytes/s)
ftp> quit
221 Goodbye.
cat-atm15#
 
Normally, the transfer get's only 13 or 14 Kbytes/s whereas this time it got 15K
which is faster than usual. Very odd.
 
I also have snapshots of netstat -i and netstat -s commands run on both machines
before and after the ftp transfer in ~jmcl/bugs/1235408
 
Both machines are running the generic OS with no patches however. Doing the
same test over ethernet does not appear to show this problem, however it's
harder to get a good test comparison as the ftp test runs so much faster over et
hernet.
One additional test would be to test ISDN 1.0.2 on 2.5 as well.
 
jmcl - Wed Jan 31 19:37:53 PST 1996
 
I installed SunISDN 1.0.2 on cat-atm15 and ran it against cat-atm16 and it
had the same problems exactly. One transfer at a time worked just fine but
two simultaneous transfers had performance problems. The 2.4 to 2.5 direction
was very slow and the 2.5 to 2.4 direction seemed slightly faster.
 
Since we see this with old ISDN software, and without any of the new features
added in SunISDN 1.0.3, this problem is "probably" not ISDN related or is at lea
st
not 1.0.3 related. We should get folks from the IP group involved in the investi
gation -
they may have some insight as to what changes between 2.4 and 2.5 would cause
this behaviour.
Work around:
 
 
Set tcp_rexmit_interval_min to 3 seconds
 
The work around field as copied from bug report 1235408 follows:
 
 
Try
 
 
ndd -set /dev/tcp tcp_rexmit_interval_initial 4000
ndd -set /dev/tcp tcp_rexmit_interval_min 4000
 
-Alain
 
 
05/09/96 pmonot@france
 
Note: This workaround is not acceptable in general: the modification of
tcp_rexmit_interval_xxx will affect all the interfaces connected to the
system. Our customers cannot use it in production systems.
 Summary:
Version numbers will be added to this table when they are available.
 
Patch id by release:
===================
 
        SPARC:
 
                                                                 module
                2.4 2.5 2.5.1 affected
        |--------|--------|--------|-----------------|
        | 101945 | 103169 | 103582 | /kernel/drv/ip |
        | 101945 | 103447 | 103630 | /kernel/drv/tcp |
        |--------|--------|--------|-----------------|
 
        X86:
                                                                     module
              2.4 2.5 2.5.1 affected
        |--------|--------|--------|-----------------|
        | 101946 | 103170 | 103581 | /kernel/drv/ip |
        | 101946 | 103448 | 103631 | /kernel/drv/tcp |
        |--------|--------|--------|-----------------|
 
        PowerPC:
                                 module
             2.5.1 affected
        |--------|-----------------|
        | 103583 | /kernel/drv/ip |
        | 103632 | /kernel/drv/tcp |
        |--------|-----------------|
 
+++++++++++++++++++++++++++ End of included text ++++++++++++++++++++++++

Matt

-- 
InterNex Information Services   |           Matthew Petach
Network Engineering             |           mpetach@internex.net
2306 Walsh Avenue               |           Tel: (408) 327-2211
Santa Clara, CA  95051          |           Fax: (408) 496-5484
Received on Thu Sep 11 1997 - 12:09:00 MDT

This archive was generated by hypermail pre-2.1.9 : Tue Dec 09 2003 - 16:37:00 MST