Savant internals, yahoo!


Current versions of the savant source files can be found in ~cabbage/RA
(not ~cabbage/savant!)

common.c - functions which reject common words
config.c - read .savantrc: set variables and build the "templates"
index.c - encode documents in vector format, using info from process.c
inter.c - retrieve documents based on a query
process.c - find document files, decide what format they are, parse documents
savant.c - deal with command line options and call appropriate functions
stem.c - stems suffixes off words (we didn't write this one)
tree.c - binary tree functions to convert DocVecs to WordVecs (see below)
vector.c - functions for converting text to vectors, used by index.c
common.h - defines an array of common words to be ingnored
extern.h - external variable and function declarations
savant.h - standard system #includes and some #defines
stem.h - came with stem.c
template.h - typedefs for savant's template system and file processing
vector.h - typedefs for savant's vector and tree system

To begin with, main() in savant.c reads all the command line options and
decides whether it should do an index or an interactive retrieval.  It also
calls load_config() (in config.c), which sets user variables and parses the
templates for document recognition.  See the section below on templates.
Each of savant_index() and savant_inter() calls open_database(), with an "r"
or "w" argument respectively, and then calls core_comm_local() (in common.c)
which intializes the list of common, or "stop" words, which are to be
ignored.  core_comm_local() uses an array of words defined in common.h, hence
local.  There is also a core_comm() which takes a filename argument to read
common words from.  I'm not using that currently, but it would be easy to
patch back in.

Indexing 
-------- 
savant_index() goes through the array of source-files specified by the -s
option and calls find_files() (in process.c) on each.  find_files()
processes the file, or if it is a directory it begins a recursive search and
processes all files it finds, and returns a linked list of file information,
which savant_index() strings together into the variable all_files.
find_files() also receives the array of exclude-files specified by the -e
option, and will ignore these files if it comes across them.  It will also
ignore binary files and files beginning with # or ending with ~.

When it finds an acceptable file, find_files() calls process_file() on it.
process_file() calls recognize(), which decides which template's 
recognize-pattern best matches the file.  Then this template's
format-pattern is used to pick out the individual documents in the file.
match_pattern() is called repeatedly on the file until no more documents
are found.  match_pattern() is hideously complicated!  If it can find 
matches for the subject, source, date and body of a document, then it will
place the offset and length of each into the List_of_Docs argument passed
to it.  If it can't find one of these, it places -1 in this variable.
The result of this is a linked list of type List_of_Docs, which 
process_file() uses as the "docs" field of a Processed_File struct.
This struct will also contain the filename in question and then number of
documents in the linked list.  find_files() takes this and throws it
into a List_of_Files linked list and when all files are done, returns this.

Now savant_index() calls s_index() (index.c) on this linked list of files.
s_index() goes through this list one by one.  It finds the file owner and
modification time for each; in the case a document had no source or date
found by match_pattern(), these will be used.  The filename will be used if
a document's subject was not found.  After this, s_index() goes through
the list of docs associated with each file and calls do_vector() on it.
This function breaks the document body up into "windows," overlapping 
regions of text.  This is an option which can be shut off, in which case
each document will consist of a single window.  For each window, do_vector()
calls vect_file() (in vector.c) on it and receives back a DocVec (see
vector.h).  To make a long story short, vector.c is more or less the 
same as nescience's vector.c.  do_vector() then takes the DocVec and 
calls wvtree_add_vec() (tree.c) on it.  As all the documents are 
vectorized, the global variable Tree is constructed by this funtion.  It
is a binary tree, keyed on the "hashes" of the words being vectorized, and
containing a linked list of "docweights".  Each of these docweights is
an unsigned int, some bits of which are devoted to identifying the DocVec
in question and the rest of which indicating the weight of the word in that
DocVec.  When a DocVec is put into this tree, each entry's hash is located 
in the tree and a new cell is added to the linked list at that location.  
I know it sounds like a mess, but it's not that bad.

do_vector() also writes information to some files.  At the beginning of 
processing the document it writes the filename to DOCLOC_FILE 
(document-location file).  Then for each window it writes the number of 
the document to WMAP_FILE (window-map file), and to DLOFF_FILE 
(document-location offset file) it writes the position in DOCLOC_FILE of
the filename it wrote, along with the document body offset and length in
the file, and the offset from there of the current window.  Many of these
values are written redundantly, but that's ok because it makes retrieval
faster.

do_vector() returns the number of windows it processed, and this is passed
to do_title().  do_title() uses the values of the source, date and 
subject variables associated with a document to make a "title" of the form
"SOURCE | DATE | SUBJECT".  It uses make_nice_name() and make_nice_date()
to deal witht he many possible formats of email addresses and dates,
respectively.  do_title() writes one copy of this to TITLE_FILE.  Then it
writes to TOFF_FILE (title-offset file) the offset in TITLE_FILE where this 
title can be found.  It writes this value a number of times equal to the
number of windows do_vector() found.

When all documents have been vectorized in this way, the binary tree of
WordVec_Lists is complete and normalize_tree() and write_tree().  The first
of these determines the magnitude of all the original DocVecs (this could 
probably be done more efficiently in do_vector(), I guess) and these are 
written to the DOCVECMAGS_FILE.  These are written now in order to avoid
the slow calculations (involving square roots) during retrieval.  
write_tree() does an in-order traversal of the tree.  For each linked list
of vec_weights it writes to WORDVEC_FILE an equivalent array, and notes
in WVOFF_FILE (wordvec-offset file) the hash in question and the offset
in WORDVEC_FILE where this last wordvec was written.

That's all there is to indexing.

Retrieval
---------
savant_inter(), after opening the database files and doing the 
core_comm_local() thing, just calls s_inter().  s_inter() starts a
big user-input loop which accepts the commands query, repeat, print, and
quit.  

query gets a buffer full of user input text, then calls vect_query()
(in vector.c, similar to vect_file()) to get a DocVec of this text.
Then it calls find_matches() on this DocVec.  Essentially the process
behind this is that for each entry in the query vector, the corresponding
WordVec is pulled off disk, and the weight of this word in each document
in the collection is multiplied with the weight of it in the query, and 
this product is added to a running total for the document.  The top 5 
(or whatever) of these are kept in order.  When each word in the
query is taken care of, you can convince yourself that the resulting
totals are the dot products of the query with each of the original 
DocVecs.  The information is stored on disk as WordVecs because they
are much faster to retrieve: to actually take all those dot products 
if DocVecs were stored on disk would mean reading in the entire database,
using too much memory, or read it in a vector at a time for each query,
much too slow.  Here we can read only the wordvectors we actually need.

With the top matches in hand, savant prints their titles for the user.
If verbose, it also prints their location.  After a query, a repeat 
command just prints these summaries again.  A print n command prints out 
the entire text of the original document, preceded by an offset into
the file where the relevant window can be found. 

And that's all...

Templates
---------
Savant uses a system of templates to locate documents in a file.
Each template has two patterns, a recognize-pattern and a format-pattern.
The recognize-pattern is used to decide what kind of file is being dealt
with, and the format-pattern is used to find the documents in it.
The pattern system is simple but flexible.  The basic elements are strings
which savant tries to find in the file, and these commands:
startline - The next string or subpattern should be found at the head of a line
ignore - The next string or subpattern may not appear immediately-ignore text
	until it does appear
anyorder - The rest of strings and subpatterns in this subpattern may appear
	in any order
anyof - Matching any one of the rest of the strings and subpatterns in this
	subpattern should indicate success.
icase - the next string should be matched case-insensitive
optional - the next string or subpattern is optional 

You can impart structure to these patterns with curly braces, which indicate
a separate self-contained pattern (that is what I mean by sub-pattern).
For format-patterns, there are also variables source, subject, date, and
body that are filled.  See savantrc in the remembrance distribution for
examples of all this stuff.





