Stephen E. Arnold: Web Crawling ToolKit
Short Honk: Crawl the Web at Scale Short honk: I read “Aduana: Link Analysis to Crawl the Web at Scale.” The write up explains an open source project which can copy content “dispersed all over the Web.” Keep in mind that the approach focuses primarily on text. Aduana is a special back end for the …