	The ideas, and history

LinkController was originally inspired by MOMspider and having the
MOMspider code available was very useful during the creation of this
kit, but, it shares almost no code with MOMspider, other than what has
comes to it from the LibWWW-Perl library.

Philosophically, the MOMspider heritage is obvious in the wish to
handle big jobs efficiently.  In the working practice there are
far more differences than similarities.  

Firstly, the availability of multi-level arrays and hashes in perl has
made much of the tricky and difficult to maintain code in MOMspider
obsolete.  Whereas momspider had to keep information in lots of
different tables, I just bundle it all into Link objects and take them
in and out of a Berkle DB database file at will (in a somewhat
inefficient manner, but that's not so important given that link
checking takes many times as long). 

Secondly, I decided to completely separate the exploration of the
local infostructure, looking for links to be checked, from the actual
checking process.  This means that checking can be spread over a large
number of days and still run efficiently.  The major disadvantage is
the need to build a separate link-database building program and the
fact that we can't check local links as we build this database.  I
don't really think this is a problem.  In fact it means that we treat
checking of local links exactly the same as outside links, and, for
most setups, is more likely to spot a local configuration error.  

The basic aim of this link checking kit is to be able to efficiently
handle any size of link checking job.  At the bottom end we have
checking new pages as they are written.  Here we want to use
information from previous checks to avoid having to check all of each
page every time.  At the other end we have massive info structures
(sites) which deal in many thousands of links and could not possibly
all be checked in one day.  For this latter case the aim is to be able
to efficiently spread the link checking load into all available low
usage periods.

My primary aim in writing this was not to write very efficient code
(takes minimum time to do everything), but rather code which would
scale well.  If your system can check 1000 links in two days, it will
hopefully be able to check almost 7000 links in two weeks.  I'm trying
to make sure all data structures which grow with the number of links
are kept on disk.  
