NAME
    ALBD README

  SYNOPSIS
        This package consists of Perl modules along with supporting Perl
        programs that perform Literature Based Discovery (LBD). The core 
        data from which LBD is performed are co-occurrences matrices 
        generated from UMLS::Association. ALBD is based on the ABC
        co-occurrence model. Many options can be specified, and many
        ranking methods are available. The novel ranking methods that use
        association measure are available as well as frequency based
        ranking methods. See samples/lbd for more info. Can perform open and
        closed LBD as well as time slicing evaluation.

        ALBD requires UMLS::Association both to compute the co-occurrence
        database that the co-occurrence matrix is derived from, but also for 
        ranking the generated C terms.

        UMLS::Association requires the UMLS::Interface module to access 
        the Unified Medical Language System (UMLS) for semantic type filtering
        and to determine if CUIs are valid.

        The following sections describe the organization of this software
        package and how to use it. A few typical examples are given to help
        clearly understand the usage of the modules and the supporting
        utilities.

  INSTALL
        To install the module, run the following magic commands:

          perl Makefile.PL
          make
          make test
          make install

        This will install the module in the standard location. You will, most
        probably, require root privileges to install in standard system
        directories. To install in a non-standard directory, specify a prefix
        during the 'perl Makefile.PL' stage as:

          perl Makefile.PL PREFIX=/home/programs

        It is possible to modify other parameters during installation. The
        details of these can be found in the ExtUtils::MakeMaker documentation.
        However, it is highly recommended not messing around with other
        parameters, unless you know what you're doing.

  CO-OCCURRENCE MATRIX SETUP
    ALBD requires that a co-occurrence matrix of CUIs has been created. This
    matrix is stored as a flat file, in a sparse matrix format such that
    each line contains three tab seperated values, cui_1, cui_2, n_11 = the
    count of their co-occurrences. Any matrix with that format is
    acceptable, however the intended method of matrix generation is to
    convert a UMLS::Association database into a flat matrix file. These
    databases are created using the CUICollector tool of UMLS::Association,
    and are run over the MetaMapped Medline baseline. With that file, run
    utils/datasetCreator/fromMySQL/dbToTab.pl to convert the desired
    database into a matrix file. Notice that code in dbToTab.pl is just a
    sample mysql command. If the input database is created in another
    method, a different command may be needed. As long as the resulting
    co-occurrence matrix is in the correct format LBD may be run on it. This
    allows flexibility in where co-occurrence information comes from.

    Note: utils/datasetCreator/fromMySQL/removeQuotes.pl may need to be run
    on the resulting tab seperated file, if quotes are inlcuded in the
    resulting co-ocurrence matrix file.

  Set Up Dummy UMLS::Association Database
    UMLS::Association requires that a database can be connected to that is
    in the correct format. Although this database is not required for ALBD
    (since co-occurrence data is loaded from a co-occurrence matrix), it is
    required to run UMLS:Association. If you ran UMLS::Association to
    generate a co-occurrence matrix, you should be fine. Otherwise you will
    need to create a dummy database that it can connect to. This can be done
    in a few steps:

    1) open mysql type mysql at the terminal

    2) create the default database in the correct format, type: CREATE
    DATABASE cuicounts; use cuicounts; CREATE TABLE N_11(cui_1 CHAR(10),
    cui_2 CHAR(10), n_11 BIGINT(20));

  INITIALIZING THE MODULE
    To create an instance of the ALBD object, using default values for all
    configuration options: %options = (); $options{'lbdConfig'} =
    'configFile'; my $lbd = LiteratureBasedDiscovery->new(\%options);
    $lbd->performLBD();

    The following configuration options are also provided though:

    'assocConfig' path to a UMLS::Association configuration file. Default
    location is 'config/association'. Replace this file for your computer to
    avoid having to specify each time

    'interfaceConfig' path to a UMLS::Interface configuration file. Default
    location is '../config/interface'. Replace this file for your computer
    to avoid having to specify each time.

    These are passed through a hash. For example:

        my %options = ();
        $options{'assocConfig'}   = '/home/share/ALBD/config/association';
        $options{'interfaceConfig'} = '/home/shar/ALBD/config/interface';
        $options{'lbdConfig'} = 'configFile'
        my $lbd = LiteratureBasedDiscovery->new(\%options);
        $lbd->performLBD();

  CONTENTS
    All the modules that will be installed in the Perl system directory are
    present in the '/lib' directory tree of the package.

    The package contains a utils/ directory that contain Perl utility
    programs. These utilities use the modules or provide some supporting
    functionality.

    runDiscovery.pl -- runs LBD using the parameters specified in the input
    file, and outputs to an output file.

    The package contains a large selection of functions to manipulate CUI
    Co-occurrence matrices in the utils/datasetCreator/ directory. These are
    short scripts and generally require modifying the code at the top with
    user input paramaters specific for each run. These scripts include:

    applyMaxThreshold.pl -- applies a maximum co-occurrence threshold to the
    co-occurrence matrix

    applyMinThreshold.pl -- applies a minimum co-occurrence threshold to the
    co-occurrence matrix

    applySemanticFilter.pl -- applies a semantic type and/or group filter to
    the co-occurrence matrix.

    combineCooccurrenceMatrices.pl -- combines the co-occurrence counts of
    multiple co-occurrence matrices

    makeOrderNotMatter.pl -- makes the order of CUI co-occurrences not
    matter by updating the co-occurrence matrix file. (UMLS::Association
    generates co-occurrence files where order does matter, so the sentence
    'cui1 cui2' will only mark a co-occurrence between cui1 and cui2, but
    not between cui2 and cui1).

    removeCUIPair.pl -- removes all occurrences of the specified CUI pair
    from the co-occurrence matrix

    removeExplicit.pl -- removes any keys that occur in an explicit
    co-occurrence matrix from another co-occurrence matrix (typically the
    squared explicit co-occurrence matrix itself, which generates a
    prediction matrix, or the post cutoff matrix used in time slicing to
    generate a gold standard file)

    testMatrixEquality.pl -- checks to see if two co-occurrence matrix files
    contain the same data

    Also included are several subfolders with more specific purposes. Within
    the dataStats subfolder are scripts to collect various statistics about
    the co-occurrence matrices used in LBD. These scriptsinclude:

    getCUICooccurrences.pl -- a data statistics file that gets the number of
    co-occurrences, and number of unique co-occurrences for every CUI in the
    dataset

    getMatrixStats.pl -- determines the number of rows, columns, and entries
    of a co-occurrence matrix

    metaAnalysis.pl -- determines the number of rows, columns, vocabulary
    size, and total number of co-occurrences of a co-occurrence file, or set
    of co-occurrence files

    There is another folder containing scripts to square co-occurrence
    matrices. Squaring an explicit (A to B) co-occurrence matrix results in
    a co-occurrence matrix containing all implicit (A to C) connections.
    This is useful for time slicing and other analysis. Removal of the
    original explicit matrix is an additional step that is required if you
    wish to create a predictions matrix file for every CUI. This can be done
    with the removeExplicit.pl script. Squaring a co-occurrence matrix can
    be very computationally expensive, both in terms of ram and cpu. For
    this reason MATLAB scripts are preferred over perl scripts. Even using
    MATLAB ram can become an issue, and squaring sections of a matrix and
    combining them into a single output matrix may be necassary, but takes
    much longer. Scripts in the squaring folder include:

    convertForSquaring_MATLAB.pl -- functions to convert to and from ALBD
    and MATLAB sparse matrix formats

    squareMatrix.m -- MATLAB script to square a matrix while holding
    everything in ram. Faster, but requires more ram.

    squareMatrix_partial.m -- MATLAB script to square a matrix in chunks.
    Only loads parts of the matrix into ram at a time which makes squaring
    any size matrix possible, but potentially take impracticle amounts of
    time.

    squareMatrix_perl.pl -- squares a matrix in perl, but requires the most
    ram of any squaring method. The easiest method to use, but only
    practical for small datasets.

    The fromMySQL folder contains scripts that convery UMLS::Association
    databases to ALBD co-occurrence matrices. The files contained are:

    dbToTab.pl -- converts a UMLS::Association co-occurrence database to a
    sparse format co-occurrence matrix used for ALBD

    removeQuotes.pl -- removes quotes from lines in the co-occurrence matrix
    file after converting from a database (sometimes needed)

  REFERENCING
        If you write a paper that has used UMLS-Association in some way, we'd 
        certainly be grateful if you sent us a copy.

  CONTACT US
    If you have any trouble installing and using ALBD, please contact us
    directly if you prefer :

        Sam Henry: henryst at vcu.edu

        Bridget McInnes: btmcinnes at vcu.edu

  SOFTWARE COPYRIGHT AND LICENSE
        Copyright (C) 2017 Sam Henry & Bridget McInnes

        This suite of programs is free software; you can redistribute it and/or
        modify it under the terms of the GNU General Public License as published
        by the Free Software Foundation; either version 2 of the License, or (at
        your option) any later version.

        This program is distributed in the hope that it will be useful, but
        WITHOUT ANY WARRANTY; without even the implied warranty of
        MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
        Public License for more details.

        You should have received a copy of the GNU General Public License along
        with this program; if not, write to the Free Software Foundation, Inc.,
        59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.

        Note: The text of the GNU General Public License is provided in the file
        'GPL.txt' that you should have received with this distribution.

