Re: Improving qos_flows mark feature - obtaining mark later

On Mon, 2011-10-17 at 10:34 -0600, Alex Rousskov wrote:
> On 10/16/2011 04:23 PM, Andrew Beverley wrote:
>
> > I've been using the qos_flows feature for preserving a netfilter mark,
> > but have run into some problems.
> >
> > Currently, the netfilter mark for the connection is obtained in
> > forward.cc, during the stages of opening a connection to the remote
> > server. The problem with this is that the connection mark may change
> > once the reply is received.
> >
> > So, I would like to move Ip::Qos::getNfmarkFromServer() from
> > FwdState::dispatch to a function that is called during the server reply
> > stage. However, I can't work out where it should go.
>
> Could you define "server reply stage" more precisely, please? If you
> want that function to be called once per [HTTP] response, then the
> definition should be that of a single response "state change" and not a
> continuous "reply stage". If you want that function to be called once
> per connection, then the definition should be based on
> connection-specific state changes and not on the response state change. Etc.

What I meant was during the receiving of data from the server, as
opposed to (currently) during the sending of the request to the server.
Having thought about this further, I think what I was trying to achieve
was getting the mark every time a packet was received.

>
> > I've looked at moving it to client_side_reply.cc, but I can't find a way
> > to retrieve the remote server connection information. I've also looked
> > at comm_read() in comm.cc, but I can't find the client connection
> > information and it looks like the wrong place anyway.
>
> client_side_reply.cc is dealing with receiving responses from Store.
> Such responses may have no connection associated with them. If you must
> have access to the server connection, then you should look on the server
> side (forward.cc, Server.cc, http.cc, ftp.cc, etc. with tunnel.cc being
> a special case). See Amos' email for more details.

I tried some test code in processReplyHeader() as per Amos' suggestion,
but as that function is only called once per reply it doesn't fulfil the
requirement of retrieving the mark every time new data arrives.

> BTW, code common to HTTP and FTP handling should go into Server.cc to
> the extent possible.

I have tried putting the relevant code into
ServerStateData::addVirginReplyBody() which seems to work for different
protocols as well as being called for each packet received.

Is there any problems with this approach? If not, I will refine and
submit a patch.

> > Where is the best place to move it to, where it has access to both the
> > client and server side connection, but where it is late enough to read
> > the mark value if it changes during the server reply?
>
> Please note that "during the server reply" may take an hour. Do you want
> to do something once per reply, every time there are new bytes read from
> the origin server, or every time a new piece of response body is given
> to the client side, for example?

Ideally it would be for every packet of data received, as each packet of
data will have its own mark value. Doing it every time
ServerStateData::addVirginReplyBody() is called seems to be pretty
close.

To give some more detail as to what I was actually trying to achieve:

I was trying to mark the *connection* with one value whilst a packet was
leaving the internet-facing interface, but a different value as the
reply was received. This was because I wanted a different mark value for
the packets going to the client from Squid, as to the upstream
connection going to the server.

However, this just doesn't work very well for reasons that are probably
obvious, and I've found with all the code above that the mark value was
almost random (presumably because by the time the mark value has been
read, the next packet is on its way out).

So... I still plan to progress the work above, assuming that you are
happy, but my plan for my particular application is to add a mask option
for the "qos_flows mark miss" option. I'll send a patch separately once
I've finished it.

Thanks,

Andy
Received on Mon Oct 31 2011 - 21:03:33 MDT