Tag Archives: web robot

Basic web mapper

Sometimes it is useful to have an automated tool to get the full web map of your site. Perhaps not your own web site, since you have already implemented some kind of automatic generation and notification to Google (have not yet?), but a client’s one.

There are a few tools to map an external web site, I tried some in my particular case. They were just adware, or demos, or they obscured the links in the final report… Yeah, of course, sometimes a $30 license is worth it, but you might not want to acquire a new piece of proprietary software every time you need a new feature, might you?

So I decided to write it myself in PHP, not for the money, but for the fun :)

Continue reading Basic web mapper

Working with Web Robots (Crawlers)

Some useful information to start with when attempting to work with web spiders. Just to learn the basis, these links could be useful for those to begin dealing with this subject: