multi-streamed http transfers

From: Rod Walker <rwalker@dont-contact.us>
Date: Thu, 16 Dec 2004 21:30:14 -0800 (PST)

Hi,
I`m hoping to use Squid for the retrieval and caching of large data
files(~1GB) for High Energy Physics applications. One of the considerations
is the file transfer rates relative to gridFTP, which can use multiple
parallel streams to increase the transfer rate.
On googling around a little I found several multi-streamed 'wget-like'
http clients, e.g. aget, prozilla, that get transfer speeds comparable to
gridftp. These do not respect the http_proxy environment variable and do not
use the squid cache, probably for the very good reason that splitting a
file into several chunks for transfer will make it very hard to cache.

I can't see any way around this, but in my proposed architecture the last
hop, from the squid cache to the application, is a short one, so a single
stream is fine.

What I`m asking is whether it would be possible to have
the transfers between squid peers, and those from the source web server to
the cache, be multi-streamed.

Cheers,
Rod.

-- 
Rod Walker +1 6042913051
Received on Thu Dec 16 2004 - 23:17:07 MST

This archive was generated by hypermail pre-2.1.9 : Fri Dec 31 2004 - 12:00:05 MST