Re: [squid-users] Caching/spidering an entire site

From: Henrik Nordstrom <hno@dont-contact.us>
Date: Thu, 23 Dec 2004 23:34:32 +0100 (CET)

On Thu, 23 Dec 2004, Ow Mun Heng wrote:

> Cool. Didn't thought about that possibility. Any crawlers recommended?

wget works reasonably well on structured web sites.

> The other thing about spidering is chances are it will spider more depth
> than I want it to.

Then tell the spider how deep it should crawl.

Regards
Henrik
Received on Thu Dec 23 2004 - 15:34:34 MST

This archive was generated by hypermail pre-2.1.9 : Sat Jan 01 2005 - 12:00:03 MST