=head1 mod_perl and dbm files


=head1 Where and Why to use dbm files

Some of the earliest databases implemented on Unix were dbm files, and
many are still in use today.  As of this writing the Berkeley DB is
the most powerful dbm implementation (http://www.sleepycat.com).

If you need a light database, with an easy API, using simple key-value
pairs to store and manipulate a relatively small number of records,
this is a solution that should be amongst the first you consider.

With dbm, it is rare to read the whole database into memory.  Combine
this feature with the use of smart storage techniques, and dbm files
can be manipulated much faster than flat files.  Flat file databases
can be very slow on insert, update and delete operations, when the
number of records starts to grow into the thousands.  Sort algorithms
on flat files can be very time-consuming.

The maximum practical size of a dbm database depends on many factors -
your data, your hardware and the desired response times of course
included - but as a rough guide consider 5,000 to 10,000 records to be
reasonable.

Several different indexing algorithms can be used with dbm:

=over

=item *

The C<HASH> algorithm gives an C<0(1)> complexity of search and
update, fast insert and delete, but a slow sort (which you have to
implement yourself).

=item *

The C<BTREE> algorithm allows arbitrary key/value pairs to be stored
in a sorted, balanced binary tree.  This allows us to get a sorted
sequence of data pairs in C<0(1)>, but at the expense of much slower
insert, update, delete operations than is the case with C<HASH>.

=item *

The C<RECNO> algorithm is more complicated, and enables both
fixed-length and variable-length flat text files to be manipulated
using the same key/value pair interface as in C<HASH> and C<BTREE>.
In this case the key will consist of a record (line) number.

=back

Most often you will want to use the C<HASH> method, but there are many
considerations and your choice may be dictated by your application.

In recent years dbm databases have been extended to allow you to store
more complex values, including data structures.  The C<MLDBM> module
can store and restore the whole symbol table of your script, including
arrays and hashes.

It is important to note that you cannot simply switch a dbm file from
one storage algorithm to another.  The only way to change the
algorithm is to dump the data to a flat file and then restore it using
the new storage method.  You can use a script like this:

  #!/usr/bin/perl -w
  
  #
  # This script takes as its parameters a list of Berkeley DB
  # file(s) which are stored with the DB_BTREE algorithm.  It
  # will back them up using the .bak extension and create
  # instead dbms with the same records but stored using the
  # DB_HASH algorithm
  #
  # Usage: btree2hash.pl filename(s)
  
  use strict;
  use DB_File;
  
    # Do checks 
  die "Usage: btree2hash.pl filename(s))\n" unless @ARGV;
  
  foreach my $filename (@ARGV) {
  
    die "Can't find $filename: $!\n" 
      unless -e $filename and -r $filename;
  
      # First backup the file
    rename "$filename", "$filename.btree" 
      or die "can't rename $filename $filename.btree:$!\n";
  
    my %btree;
    my %hash;
  
      # tie both dbs (db_hash is a fresh one!)
    tie %btree , 'DB_File',"$filename.btree", O_RDWR|O_CREAT, 
        0660, $DB_BTREE or die "Can't tie %btree";
    tie %hash ,  'DB_File',"$filename" , O_RDWR|O_CREAT, 
        0660, $DB_HASH  or die "Can't tie %hash";
  
      # copy DB
    %hash = %btree;
  
      # untie
    untie %btree ;
    untie %hash ;
  }

Note that some dbm implementations come with other conversion
utilities as well.

=head1 mod_perl and dbm

Where does mod_perl fit into the picture?

If you are using a read only dbm file you can have it work faster if
you keep it open (tied) all the time, so that when your CGI script
wants to access the database it is already tied and ready to be used.
This will work with dynamic (read/write) databases as well, but you
need to use locking and data flushing to avoid data corruption.

Although mod_perl and dbm can give huge performance gains to CGI
scripts which manipulate flat files, you should be very careful.  In
addition to the need for locking, you need to consider the
consequences of C<die()> and unexpected process deaths.

If your locking mechanism cannot handle dropped locks, a stale lock
can deactivate your whole site.  You can enter a deadlock situation if
two processes simultaneously try to acquire locks on two separate
databases.  Each has locked only one of the databases, and cannot
continue without locking the second.  Yet this will never be freed
because it is locked by the other process.  If your processes all ask
for their DB files in the same order, this situation cannot occur.

If you modify the DB you should be make very sure that you flush the
data and synchronize it, especially when the process serving your CGI
unexpectedly dies.  In general your application should be tested very
thoroughly before you put it into production to handle important data.

=head1 Locking dbm handlers

Let's make the lock status a global variable, so it will persist from
request to request.  Before we request a lock - I<READ> (shared) or
I<WRITE> (exclusive) - we should first obtain the current lock status.

If we are making a I<READ> lock request, it is granted as soon as the
file becomes unlocked or if it is already I<READ> locked.  The lock
status becomes I<READ> on success.

If we make a I<WRITE> lock request, it is granted as soon as the file
becomes unlocked.  The lock status becomes I<WRITE> on success.

The treatment of the I<WRITE> lock request is most important.

If the DB is I<READ> locked, a process that makes a I<WRITE> request
will poll until there are no reading or writing processes left.  Lots
of processes can successfully read the file, since they do not block
each other.  This means that a process that wants to write to the file
(so first it needs to obtain an exclusive lock) may never get a chance
to squeeze in. The following diagram represents a possible scenario
where everybody can read but no one can write:

  [-p1-]                 [--p1--]
     [--p2--]
   [---------p3---------]
                 [------p4-----]
     [--p5--]   [----p5----]

The result is a starving process, which will timeout the request, and
it will fail to update the DB.  This is a good reason not to cache the
dbm handle with dynamic dbm files.  It will work perfectly with static
DBM files without any need to lock files at all.

Ken Williams solved the above problem with his
L<C<Tie::DB_Lock>|dbm/Tie__DB_Lock> module, which I will discuss in
one of the following sections.

There are several locking wrappers for C<DB_File> in CPAN right now.
Each one implements locking differently and has different goals in
mind.  It is therefore worth knowing the difference, so that you can
pick the right one for your application.


=head1 Flawed Locking Methods Which Must Not Be Used

I<Caution>: The suggested locking methods in the Camel book and
C<DB_File> man page (at least before version 1.72) are flawed.  If you
use them in an environment where more than one process can modify the
dbm file, it can get corrupted!!!  The following is an explanation of
why this happens.

You may not use a tied file's filehandle for locking, since you get
the filehandle after the file has been already tied.  It's too late to
lock.  The problem is that the database file is locked I<after> it is
opened.  When the database is opened, the first 4k (in Berkley dbm
library) is read and then cached in memory.  Therefore, a process can
open the database file, cache the first 4k, and then block while
another process writes to the file.  If the second process modifies
the first 4k of the file, when the original process gets the lock is
now has an inconsistent view of the database.  If it writes using this
view it may easily corrupt the database on disk.

This problem can be difficult to trace because it does not cause
corruption every time a process has to wait for a lock.  One can do
quite a bit of writing to a database file without actually changing
the first 4k.  But once you suspect this problem you can easily
reproduce it by making your program modify the records in the first 4k
of the DB.


=head1 Lock Wrappers Overview

There are five locking wrappers known to me:

=over

=item *

C<Tie::DB_Lock> -- C<DB_File> wrapper which creates copies of the
database file for read access, so that you have kind of a
multiversioning concurrent read system. However, updates are still
serial.  Use this for databases where reads may be lengthy and
consistency problems may occur.  L<More information|dbm/Tie__DB_Lock>.

=item *

C<Tie::DB_LockFile> -- C<DB_File> wrapper that has the ability to lock
and unlock the database while it is being used.  Avoids the
tie-before-flock problem by simply re-tie-ing the database when you
get or drop a lock.  Because of the flexibility in dropping and
re-acquiring the lock in the middle of a session, this can be massaged
into a system that will work with long updates and/or reads.  There
are the hints in the POD documentation.  Refer to the
C<Tie::DB_LockFile> manpage for more information.

=item *

C<DB_File::Lock> -- extremely lightweight C<DB_File> wrapper that
simply flocks a lockfile before tie-ing the database and drops the
lock after the untie.  Allows one to use the same lockfile for
multiple databases to avoid deadlock problems, if desired.  Use this
for databases where updates and reads are quick and simple flock
locking semantics are enough.  Refer to C<DB_File::Lock> manpage for
more information.

=item *

L<C<DB_File::Lock2>|dbm/DB_File__Lock2> -- does the same thing as
C<DB_File::Lock>, but has a slightly different implementation.  I
wrote it before David Harris released his C<DB_File::Lock> and I
didn't want to kill mine, so I'll keep it here for a while :).

=item *

On some Operating Systems (FreeBSD is one example) it is possible to
lock on tie:

  tie my %t, 'DB_File', $TOK_FILE, O_RDWR | O_EXLOCK, 0664;

and only release the lock by un-tie-ing the file.  Check if the
C<O_EXLOCK> flag is available on your operating system before you try
to use this method!

=back


=head1 Tie::DB_Lock

C<Tie::DB_Lock> ties hashes to databases using shared and exclusive
locks.  This module, by Ken Williams, solves the problems raised in
the previous section.

The main difference from what I have described above is that
C<Tie::DB_Lock> copies a dbm file on read.  Reading processes do not
have to keep the file locked while they read it, and writing processes
can still access the file while others are reading.  This works best
when you have lots of long-duration reading, and a few short bursts of
writing.

The drawback of this module is the heavy IO performed when every
reader makes a fresh copy of the DB.  With big dbm files this can be
quite a disadvantage and can slow the server down considerably.

An alternative would be to have one copy of the dbm image shared by
all the reading processes.  This can cut the number of files that are
copied, and puts the responsibility of copying the read-only file on
the writer, not the reader.  It would need some care to make sure it
does not disturb readers when putting a new read-only copy into place.




=head1 DB_File::Lock2

=code DB_File-Lock2.pm Here is C<DB_File::Lock2> which does the
locking by using an external lockfile.

This allows you to gain the lock before the file is tied.  Note that
it's not yet on CPAN and so is listed here in its entirety.  Note also
that this code still needs some testing, so I<be careful> if you use
it on a production machine.

You use it like this:

  use DB_File::Lock2 ();

A simple tie, READ lock and untie

  use DB_File::Lock2 ();
  my $dbfile = "/tmp/test";
  tie my %mydb, 'DB_File::Lock2', $dbfile, 'read';
  print $mydb{foo} if exists $mydb{foo};
  untie %mydb;

You can even skip the C<untie()> call.  When C<$mydb> goes out of
scope everything will be done automatically.  However it is better to
use the explicit call, to make sure the critical sections between lock
and unlock are as short as possible.  This is especially important
when requesting an exclusive (write) lock.

The following example shows how it might be convenient to skip the
explicit C<untie()>.  In this example, we don't need to save the
intermediate result, we just return and the cleanup is done
automatically.

  use DB_File::Lock2 ();
  my $dbfile = "/tmp/test";
  print user_exists("stas") ? "Yes" : "No";
  sub user_exists{
    my $username = shift || '';
  
    warn("No username passed\n"), return 0 unless $username;
  
    tie my %mydb, 'DB_File::Lock2', $dbfile, 'read';
  
    # if we match the username return 1, else 0
    return $mydb{$username} ? 1 : 0;
  
  } # end of sub user_exists

Now let's write all the upper case characters and their respective
ASCII values to a dbm file. Then read the file and print the contents
of the DB, unsorted.

  use DB_File::Lock2 ();
  my $dbfile = "/tmp/test";
  
    # write 
  tie my %mydb, 'DB_File::Lock2', $dbfile,'write';
  for (0..26) {
    $mydb{chr 65+$_} = $_;
  }
  untie %mydb;
  
    # now, read them and printout (unsorted) 
    # notice that 'read' is a default lock mode
  tie %mydb, 'DB_File::Lock2', $dbfile;
  while (my($k,$v) = each %mydb) {
    print "$k => $v\n";
  }
  untie %mydb;

If your CGI script is interrupted, the C<DESTROY> block will take care
of unlocking the dbm file and flush any changes.  So your DB will be
safe against possible corruption because of unclean program
termination.

=cut
