RE: [squid-users] Squid isn't using persistent server-side connections upon POST/PUT/DELETE

Hello,

thanks for your help. We followed your advice and built the newest Squid version (squid/3.2.0.13-20111019-r11381) - same issues. The option server_persistent_connections is ON. Also the FIN definitely stems from the Squid.

The problem still is: Squid closes its server-side TCP-Connection upon PUT {body}, POST, POST {body}, DELETE {body}. The connections remains persistent upon GET, PUT, DELETE (without content):

Method Client <-> Squid Squid <-> Server
GET persistent persistent
PUT persistent persistent
PUT {body} persistent non-persistent
POST persistent non-persistent
POST {body} persistent non-persistent
DELETE persistent persistent
DELETE {body} persistent non-persistent

Here is as a visual presentation of the problem: http://betablogger.info/persistent_connection.png

This behaviour appears both if the server answers with 204 No Content and 200 Success. For instance:

Squid says:

PUT /PUT HTTP/1.1
Content-Type: text/plain
Host: 192.168.14.200
Content-Length: 14
Via: 1.1 ubuntu (squid/3.2.0.13-20111019-r11381)
X-Forwarded-For: 192.168.14.200
Cache-Control: max-age=259200
Connection: keep-alive

put, put, put.

Server replies:

HTTP/1.1 200 OK
Date: Thu Oct 20 2011 16:29:14 GMT+0200 (Mitteleurop.ische Sommerzeit)
Server: Restlet-Framework/2.1rc1
Content-Length: 17
Content-Type: text/plain
Connection: Keep-Alive

I am your Server.

Squid sends FIN ACK in reply.

If you want to reproduce this obviously buggy behaviour, here is a NodeJs-client and server you can easily use to reproduce it (you need a Squid) (http://nodejs.org ). Just change the respective IP-addresses.

Client.js (you can also download it here: http://betablogger.info/client.js )

var http = require('http');

var originServer = '192.168.14.200';
var proxyServer = '192.168.14.194';
var proxyPort = 3128;

var Repeater = function(method, content, times) {
        var self = this;
        var options = {
                host: proxyServer,
                port: proxyPort,
                path: 'http://' + originServer + '/' + method,
                method: method,
                headers: {
                        'Proxy-Connection' : 'keep-alive',
                        'Content-Type' : 'text/plain',
                        Host: originServer,
                        Connection: 'keep-alive'
                }
        };
        if(content == null)
                content = '';
        if(times == null)
                times = 10;
        
        this.makeRequests = function() {
                for(var i = 1; i <= times; i++) {
                        console.log(options.method + '-Request no ' + i + ' ist being sent...');
                        var req = http.request(options, onResponse);
                        var body = new Buffer(content);
                        req.setHeader('Content-Length', body.length);
                        req.end(body);
                }
        }
        
        var onEnd = function() {
                console.log('Finished receiving Response for ' + options.method + '-Request.');
        };
        
        var onResponse = function(res) {
                res.on('end', onEnd);
        };
}

function test(method, content, times) {
        r = new Repeater(method, content, times);
        r.makeRequests();
};

test('GET');
test('PUT');
test('PUT', 'put, put, put.');
test('POST');
test('POST','My new post.');
test('DELETE');
test('DELETE', 'Die!');

and Server.js (http://betablogger.info/server.js )

var http = require('http');

var originServer = '192.168.14.200';
var content = 'I am your Server.';

http.createServer(function (req, res) {
        console.log(req.method + '-Request received.');
        var body = new Buffer(content);
        req.on('end', function() {
                res.writeHead(content === '' ? 204 : 200, {
                        'Date': new Date(),
                        'Server': 'Restlet-Framework/2.1rc1',
                        'Content-Length': body.length,
                        'Content-Type': 'text/plain',
                        'Connection': 'Keep-Alive'
                });
                res.end(body);
        });
}).listen(80, originServer);

console.log('Server running at http://' + originServer + ':80/');

Use a network monitor like Wireshark (http://www.wireshark.org/ ) or Microsoft Network Monitor to observe the faulty behaviour of the Squid.

Should we report this as a bug?

We are frankly quite aghast, that a bug of these proportions and effects went unobserved for such a long time.

Thanks in advance,
Florian and Felix

----------------------------------------
>
> NP: inserted a few empty lines to make the text paragraphs
> identifiable.
>
> On Tue, 11 Oct 2011 20:25:06 +0200, Franz Kafka wrote:
> > Good evening Squid experts,
> > We have a serious issue with persistent connections. We are testing a
> > large REST-Application which heavily depends on caching and correct
> > cache behaviour in particular concering the rather scarcely used
>
> Then if you have not already done so please look over the compliance
> checklist at:
> http://wiki-squid-cache.org/Features/HTTP11
>
> It covers milestone releases in the common Squid versions which
> advertise 1.1. There are still numbers of networks using Squid 2.6 and
> 2.5 though which have 1.0 compliance. And networks using un-rated
> releases. So this is not a complete picture, just an idea of what can be
> expected out in the Internet.
>
> HTTP/1.1 correct behaviour is an ongoing effort. Sponsorship is
> appreciated, working patches even more so.
>
> <snip>
> > So only if the PUT doesn't contain an HTTP-Body, the squid<->server
> > connection is persistent. The client<->squid connection on the other
> > hand is persistent (we use both the non-standard Proxy-Connection:
> > keep-alive and the correct hopy-by-hop Connection: keep-alive in our
> > request).
>
> You should be able to drop Proxy-Connection entirely. Squid-3.1 emits
> it by mistake in some releases. All squid accept Connection:.
>
> > Here is a detailed example of a successful GET:
> > 1. Client sends to Squid:
> > GET http://192.168.14.194/ HTTP/1.1Cache-Control: no-cache,
> > max-age=0Date: Tue, 11 Oct 2011 16:53:44 GMTContent-Length: 0Accept:
> > application/x-java-serialized-object,
> > application/x-java-serialized-object+xmlHost:
> > 192.168.14.194User-Agent: Restlet-Framework/2.1rc1Connection:
> > keep-aliveProxy-Connection: keep-alive
> > 2. Squid sends to Server:
> > GET / HTTP/1.1Date: Tue, 11 Oct 2011 16:53:44 GMTContent-Length:
> > 0Accept: application/x-java-serialized-object,
> > application/x-java-serialized-object+xmlHost:
> > 192.168.14.194User-Agent: Restlet-Framework/2.1rc1Via: 1.1 localhost
> > (squid/3.1.11)X-Forwarded-For: 192.168.14.194Cache-Control: no-cache,
> > max-age=0Connection: keep-alive
> > 3. Server sends to Squid:
> > HTTP/1.1 200 OKDate: Tue, 11 Oct 2011 16:53:44 GMTServer:
> > Restlet-Framework/2.1rc1Vary: Accept-Charset, Accept-Encoding,
> > Accept-Language, AcceptConnection: keep-aliveContent-Length:
> > 5Content-Type: text/plain; charset=UTF-8
> > blubb
> >
> > 4. Squid sends to Client:
> > HTTP/1.0 200 OKDate: Tue, 11 Oct 2011 16:53:44 GMTServer:
> > Restlet-Framework/2.1rc1Vary: Accept-Charset, Accept-Encoding,
> > Accept-Language, AcceptContent-Length: 5Content-Type: text/plain;
> > charset=UTF-8X-Cache: MISS from localhostX-Cache-Lookup: MISS from
> > localhost:3128Via: 1.0 localhost (squid/3.1.11)Connection: keep-alive
> > blubb
> > 5. Both TCP-Connections Client<->Squid and Squid<->Server are still
> > open and used for subsequent GET Requests.
> > And now an example of a failed persistent connection between squid
> > and server:
> > 1. Client sends to Squid:
> > POST http://192.168.14.194/ HTTP/1.1Cache-Control: no-cache,
> > max-age=0Date: Tue, 11 Oct 2011 16:53:46 GMTContent-Length:
> > 4Content-Type: text/plain; charset=UTF-8Accept:
> > application/x-java-serialized-object,
> > application/x-java-serialized-object+xmlHost:
> > 192.168.14.194User-Agent: Restlet-Framework/2.1rc1Connection:
> > keep-aliveProxy-Connection: keep-alive
> > test
> > 2. Squid does Threeway-SYN-SYN/ACK-ACK-Handshake with Server and
> > sends:
> > POST / HTTP/1.1Date: Tue, 11 Oct 2011 16:53:45 GMTContent-Length:
> > 4Content-Type: text/plain; charset=UTF-8Accept:
> > application/x-java-serialized-object,
> > application/x-java-serialized-object+xmlHost:
> > 192.168.14.194User-Agent: Restlet-Framework/2.1rc1Via: 1.1 localhost
> > (squid/3.1.11)X-Forwarded-For: 192.168.14.194Cache-Control: no-cache,
> > max-age=0Connection: keep-alive
> > test
> > 3. Server replies over that connection:
> > HTTP/1.1 200 OKDate: Tue, 11 Oct 2011 16:53:45 GMTServer:
> > Restlet-Framework/2.1rc1Vary: Accept-Charset, Accept-Encoding,
> > Accept-Language, AcceptConnection: keep-aliveContent-Length:
> > 4Content-Type: text/plain; charset=UTF-8
> > test
> > 4. Wih FIN-ACK the Squid closes the connection to the server and
> > replies to the client:
> > HTTP/1.0 200 OKDate: Tue, 11 Oct 2011 16:53:46 GMTServer:
> > Restlet-Framework/2.1rc1Vary: Accept-Charset, Accept-Encoding,
> > Accept-Language, AcceptContent-Length: 4Content-Type: text/plain;
> > charset=UTF-8X-Cache: MISS from localhostX-Cache-Lookup: MISS from
> > localhost:3128Via: 1.0 localhost (squid/3.1.11)Connection: keep-alive
> > test
> > 5. For the subsequent requests, the client uses the same connection
> > to the Squid, which is still open and the Squid opens a new
> > connection
> > to server to forward a PUT or POST. If the Squid isn't used at all,
> > the client<->squid connection is persistent.
>
> > To us that behaviour seems inconsistent and faulty. RFC 2616 doesn't
> > make any comments about closing a persistent connection if a
> > POST/PUT/DELETE was send. Thus it's not reasonable at all. Is it a
> > bug?
>
> > Btw, yesterday we already reported an unrelated TCP issue of Squid,
> > which totally abandons a Connection if a Request having an If-Match
> > Header results in a Cache Hit
> > (http://bugs.squid-cache.org/show_bug.cgi?id=3379 ).
> > We are very thankful for any suggestions concerning our persistent
> > connection problem, which keeps us up at night. We are trying to run
> > a
> > benchmark comparison and the TCP-Connection-kills totally ruin the
> > performance.
>
> At least you are getting realistic results out of it.
>
>
> I think sending keep-alive then FIN is a bug. if in fact that FIN is
> actually coming from Squid. The rest seems perfectly compliant
> behaviour. Annoying and inefficient, but compliant.
>
> Check you have server_persistent_connections ON. I think OFF there
> would send close and FIN. But its possible to be what you see.
>
> RFC 2616 also specifies that POST is non-idempotent. So there are
> tricky conditions around whether new connections are required. Whether a
> particular Squid uses persistence or not varies. Though they should at
> least be keeping it alive afterwards since Content-Length is present.
>
> PUT and DELETE, should have gone through the persistent connections
> AFAIK, but may have been caught up in the same logics as POST.
>
> Please also check whether this still occurs with the latest daily
> bundle before reporting. 3.1.11 is a few months old now and we have had
> a focus on performance fixes this year.
>
> Amos
>
Received on Thu Oct 20 2011 - 15:46:06 MDT