Re: stale client blocking other downloads

2012-09-19 00:21:44 +0300, Eliezer Croitoru:
> On 9/17/2012 4:49 PM, Stephane Chazelas wrote:
> >It is a 3.1.20 (debian wheezy amd64) in a transparent proxy
> >configuration (TPROXY).
> >
> >I can come up with some perl code to reproduce the issue if that
> >may help.
> >
> >Is that some known issue? Would there be a workaround (like
> >disabling the merging of requests)?
[...]
> if you can talk IP + PORT it will be more efficient to understand
> the basic problem.
> What is the IP of the origin server?
> What is the IP of the tproxy server?
> What is the IP of the two clients?
>
> Squid 3.1 dosnt implement (if I understood right)
> collapsed_forwarding which means that the situation is different.
> Two clients tries to access some server and one dosnt get reply.
> So no merging of request should be in your situation.
[...]

Those are requests to rules.emergingthreats.net which has 2 IP
addresses: 69.195.137.28 and 216.40.222.19. The two clients are
on the same machine (10.10.10.10). The tproxy is 10.0.0.1 with
an external IP address of 4.5.6.7 (IP addresses
massaged here)

The two connections to the proxy are:

10.10.10.10:33502 -> 69.195.137.28:80
10.10.10.10:33503 -> 69.195.137.28:80

Which are tproxied to

10.10.10.10:46426 -> 216.40.222.19
(and then SNATed but I don't think squid is or should be aware
of that). (I see other occurrences where the tproxied connection
goes to the same 69.195.137.28 address, so I don't think the two
IP address is part of the problem).

Here is the output of ss on the proxy machine during the
deadlock:

$ ss -n | grep -e 69.195.137.28 -e 216.40.222.19
ESTAB 0 47448 69.195.137.28:80 10.10.10.10:33502
ESTAB 178440 0 10.10.10.10:46426 216.40.222.19:80
ESTAB 0 0 69.195.137.28:80 10.10.10.10:33503

See how the first two are blocked with full receive or sent
buffers and that causes the 3rd one to stall as well even though
the send buffer is empty.

I log every connection with conntrackd and I could definitely
only see *one* outgoing connection to any of emergingthreats IP
addresses. Which is why I assumed there was some sort of
merging/collapsing in squid.

I could reproduce it consistently yesterday (on a different
client machine as well) with:

-------------------
#! /usr/bin/perl -w
use IO::Socket::INET;

$s1 = IO::Socket::INET->new("rules.emergingthreats.net:80");
$s2 = IO::Socket::INET->new("rules.emergingthreats.net:80");

print $s1 qq(GET /open/snort-2.8.6/emerging.rules.tar.gz HTTP/1.1\r
Host: rules.emergingthreats.net\r
Connection: close\r
User-agent: bar\r\n\r\n) ;
print $s2 qq(GET /open/snort-2.8.6/emerging.rules.tar.gz HTTP/1.1\r
Host: rules.emergingthreats.net\r
Connection: close\r
User-agent: bar\r\n\r\n) ;

system("ss -n");
while (<$s2>) {}
------------------------------------------

But only when providing with a User-Agent oddly enough (and also
when inserting a 10 second delay between the two requests).
Then, I tried to do the same query from a different client and
since then, I cannot reproduce it anymore (even if I do a
different request), and I see no merging in conntrackd. So I
don't have the full picture here, and reckon it's going to be
difficult to investigate as I don't know how to reproduce it
consistently.

I've got network captures, but only on the down side (between
the client and the proxy).

-- 
Stephane