Working with Web Robots (Crawlers)

Some useful information to start with when attempting to work with web spiders. Just to learn the basis, these links could be useful for those to begin dealing with this subject:

• http://www.robotstxt.org/ > Basic information and links to known robots and open source projects
• http://www.robotstxt.org/wc/faq.html > The FAQ
• http://en.wikipedia.org/wiki/Web_crawler > Some additional information and links to open source bots
• http://www.ficstar.com/web_grabber/web_crawler.html > A private crawler software
• http://www.newprosoft.com/web-spider.htm > Another private crawler software with a competitive price