Re: further to store & message flow discussion

I've skipped the bits we agree on...

----- Original Message -----
From: "Henrik Nordstrom"

> Robert Collins wrote:
>
>
> > This raises another question: for internally generated objects,
> > say gopher responses and error messages, does the intermediary
> > buffer them until the client reads and sends the message, or does
> > the intermediary _appear_ to buffer them, but actually loop them
> > through the store as currently happens. (I have no preference here,
> > just raising the issue).
>
> Well.. the internal objects should be "asyncronous" just like any
> protocol. Very few of the internal objects actually requires buffering.

Yes they need to be async, thats the issue:
An example of my concern: the cache_object protocol is able to generate som elarge objects (say the list of all cache objects). The
client side is unlikely to be ready to send all of that to the wire immediately. So some area in squid has to buffer that temporary
object.
Options:
* the intermediary has limited buffer capabilities - can handle memobjects itself, and these temporary objects never involve the
store.
* the intermediary has no buffer capabilities, it uses the store to do that.
* ???

> > Design points:
> > * From upstream the intermediary recieves clearly marked data &
> > metadata, and the recieved data need not be contiguous.
>
> Yes, but separated in protocol meta data (headers and internal
> information) and data ranges..

Exactly.

> > * The intermediary is needed to satisfy every request because it
> > is the interface through which client_side reaches the store
>
> and the upstream protocols.

Yes. I was picking on client_side to make the case during cache_hits & internal objects clear.

> > * The intermediary is not http-specific. It can deal with an
> > incoming gopher response as accurately and effectively as an incoming
> > http response. Corollary: server side protocols must be able to split
> > recieved data into metadata and message data.
>
> Here I am not so sure on how things should be done, but probably yes. It
> is a question of where and when HTTP gatewaying of the replies takes
> place. As long as Squid is a pure HTTP proxy the gatewaying should take
> place in the protocol implementation, but if we consider mixing then it
> might be a interesting idea to move the gatewaying closer to the client
> and store native protocol objects..

That is a possibility. Certainly we need to have the ability to generate the native message and enough metadata to create a native
reply. I think there are two basic approaches:
* store native objects, and on every reply translate as needed unless the client protocol matches
* store generic messages and translate on every reply.

Now every message will have some metadata that is squid-specific - Age for example. I think that the generic->specific translation
will be lightweight enough that storing generic messages will be vrey effective. We should store the message type, and possibly use
that in store selection (streaming media anyone?).

Ok I've now got a picture of what I need to do:

create a new function for setting the received metadata (which includes message headers) from server protocols, and a matching one
for getting the metadata on the client side. The store doesn't support headers as metadata yet, so some temporary glue will be
needed to parse store objects and create the metadata for the client_side.
I need wrapper functions for storeAppend and storeClientCopy and storeComplete that take an offset parameter. For now I'll work
around the store not supporting these - it'll clarify what the store needs to do.
move the logics in client_side to http as appropriate, and the generic stuff (cache_hits for example) into the new intermediary.

Oh, change all the internal API's that pass a storeobject for generating internal entities to use the intermediarie's object type.
Hmm. Just had an idea. Should we have a server side protocol for internal entity generation, and have things like the config dump
functions call the internal functions' handle _read or equivalent to write data to the entity, rather than calling the
intermediaries wrapper for storeAppend?

Rob
Received on Sun Feb 18 2001 - 14:39:37 MST