I'll try and provide some rudimentary documentation of what exactly
is going on here.

1.) overview:

nescience is a document retrieval/comparison system based on the
vector-space model proposed by Salton back in the 60's. i wacked it
out because i got tired of dealing with smart (similar in basic
function to nescience, save with about 15x the code and buggy as maine
in july). it works okay; for speed it's about twice as fast as smart,
and it needs far fewer peculiar configuration/other files to actually
function. i hope its pretty fault tolerant, too (ie. won't die with
'Illegal Instruction' like some programs we could mention).

nescience turns text into vectors. the plan is as follows:

1.) locate a word by scanning through given text, throwing away non-alpha
    characters, calling a word a contiguous chunk of text.
2.) compare the word to a list of common words. if it is a common word,
    toss it.
3.) stem the word using the porter algorithm (don't ask me, man, i just
    work here), add it to a list of stemmed, non-common words that were
    found in this document.
4.) well you've scanned the entire document, sort the stemmed words and
    weight according to the number of times each appears in the document
    (we can discuss weighting later). nescience keeps both the stemmed
    word and a hash of the stemmed word around as representatives of
    each entry in the vector.

example: the text ``i saw your mommy and your mommy's dead''

words: i       (common)       
       saw     (common)
       your    (common)
       mommy   (stems to 'mommi')
       and     (common)
       your    (common)
       mommy   (stems to 'mommi')
       s       (common)
       dead    (stems to 'dead')

which reduces to the (normalized to length 1) vector:

word    hash       multiplicity  weight

dead    139497472       1       0.447214
mommi   452375840       2       0.894427

2.) what's in the code:

nescience:
  conf.c: the code to deal with configuration files
  gettext.c: deals with recursively searching given directory/ies, finding
    text files, pulling out chunks of text of the format specified in the
    configuration files. the usefull routine is:
      find_dirs(): give it a directory, it'll return a linked list of
        locations of text, following rules given in the configuration
        file.
  index.c: code to take a wad of text and index it. the salient routines
    are: 
      nes_index(): takes the output of find_dirs() and does the right
        thing with it.
      vec_file(): takes a file descriptor, start, and length, and 
        returns the vector associated with that passage.
      vec_query(): turns an in-core chunk of text, returns it's vector 
        representation.
      show_vect(): prints a vector's features (like above there).
  retrieve.c: code dealing with retrieving/comparing.
    cmp_vects(): does a dot product. should be pretty fast.
    get_query(): 'enter your query, end with a dot on a line by itself'.
      you get the idea.

nescience.c: a quick example of how to interface to the libnes stuff. if
  you just run it without any arguments, it'll harrass you briefly, then
  tell you the salient features. there should be some demonstration stuff
  all set up in this directory.

cluster.c: this thing i wrote for my urop. it tries to find clusters
  (chunks of text that talk about the same thing) by way of nescience.

3.) buiding

to make libnes.a and nescience, just type 'make'. to make cluster,
do a 'make cluster'. the makefile was just thrown together, but
it works everywhere i've tried it so far (snakes running hp-ux, indys
running irix, pcs running linux).

4) weights

this is about all i can think of that's really worthy of comment. the
idea is this, using the previos example. suppose i've got a vector...

word    hash       multiplicity  weight

dead    139497472       1       0.447214
mommi   452375840       2       0.894427

and i gratuitously square every multiplicity, and re-normalize to
length one. then i get:

word    hash       multiplicity  weight

dead    139497472       1       0.242536
mommi   452375840       2       0.970142

voila. i've increased the importance of redundant words. if you think
about it a bit, this is like turning up the contrast... vectors will
hug the basis vectors more closely. similarly, taking the square root
of the multiplicity will make things murkier.

these are, by no means, the only ways to weight a vector. i think smart
has some much better thought-out ones, but these are a step in the
right direction.

anyway. libnes has a global int (*weight_func)(). you just set it
to point to your favourite weighting function before you do
any indexing (with, like, vect_file() or vect_query()). there are
two in the index.c, good_weight() and stupid_weight() (the former
doing a square, and the latter doing a rough (but fast) approximate
of a square root).

5.) questions

mail me anything relevant, i guess.

-bayard wenzel
biomorph@athena.mit.edu
