Using LinkController: Robots

[ < ] | [ > ] |  | [ << ] | [ Up ] | [ >> ] |  |  |  |  | [Top] | [Contents] | [Index] | [ ? ]

# 15. Robots and Sensible Behaviour

The most important thing about a program like this is to realise that if you set it up incorrectly and used it in the wrong way, you could upset a large number of people who have set up their web servers in the assumption that they would be used normally by human beings browsing through on Netscape.

Probably it's true that the only way forward is for every WWW site to begin to set up robot defences and detect when someone starts to download from at an unreasonable rate and then cut off the person doing the downloading. I suggest that you don't do this for at least two reasons.

- respect for the person's time
- a wish not to be the person who is cut off

There are probably many other reasons, but that's one for the good side in you and one for the selfish. What more do you need.

For suggestions about what constitutes `correct' behaviour, it's worth seeing the Robots World Wide Web page. http://info.webcrawler.com/mak/projects/robots/robots.html

There are a number of points which make LinkController relatively safe as a link. These are all related to the design and limitations on `test-link`.

- `test-link` does not recurse. It only tests links that are specifically listed in the schedule database.
- There is a limit to the number of links that will be tested in one run. This defaults to 1000, but can be configured.
- The schedule for link testing is designed to spread the testing of links across time
- The testing system will not test links at a given site faster than a certain rate.

The last limitation is inherited from the `LWP::RobotUA` module and the documentation for that covers the details of how it works. `test-link` trys to re-order testing of links as needed so that a limit on the rate of visits to one site does not cause a limit on overall testing speed.

[ << ] | [ >> ] |  |  |  |  |  | [Top] | [Contents] | [Index] | [ ? ]

This document was generated by Michael De La Rue on December, 30 2001 using texi2html