Re: prefetching using cache-server statistics

From: Wojtek Sylwestrzak <>
Date: Sat, 3 Aug 1996 22:22:18 +0200 (MET DST)

Are these any generic servers or software or just custom built traps ?
Are they distinguishable in any way ?


> Beware, there are robot traps built into some web servers, that if they
> detected a robot trying to download all of their pages, they will stop
> serving pages to the robot (some are designed to even cause the robot
> harm). This is especially bad if an organization's proxy server triggers
> the trap, and so prevents the entire organization for accessing that
> site's pages (this happened at PARC a couple of months ago, and we had to
> write to that site to be taken off their list of robots).
> Ed
Received on Sat Aug 03 1996 - 13:23:55 MDT

This archive was generated by hypermail pre-2.1.9 : Tue Dec 09 2003 - 16:32:45 MST