Web Crawlers, including: Grub (search Engine), Wget, Nutch, Curl, Httrack, Yacy, Dataparksearch, Libwww, Heritrix, Mnogosearch, Gwget, Methabot, ... Xenon (program), Focused Crawler, Faroo, Cuil

Hephaestus Books represents a new publishing paradigm, allowing disparate content sources to be curated into cohesive, relevant, and informative books. To date, this content has been curated from Wikipedia articles and images under Creative Commons licensing, although as Hephaestus Books continues to increase in scope and dimension, more licensed and public domain content is being added. We believe books such as this represent a new and exciting lexicon in the sharing of human knowledge. This particular book contains chapters focused on Web crawlers, and Free web crawlers.

More info: A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion. Other terms for Web crawlers are ants, automatic indexers, bots, or Web spiders, Web robots, or-especially in the FOAF community-Web scutters.