# ABSTRACT: User Guide for Forty-Two

__END__

=head1 Aim and features

The aim of C<42> is to add (and optionally align) sequences to a preexisting
multiple sequence alignment (MSA) while controlling for orthology relationships
and potentially contaminating sequences.

Sequences to add are either nucleotide transcripts resulting from transcriptome
assembly or already translated protein sequences. In theory, one can also use
genomic nucleotide sequences (because C<42> can splice introns), but this
possibility has not been extensively tested so far.

=for todo
    TODO: amend these paragraphs after publication...

The working hypothesis of C<42> is that its orthology-controlling heuristics can
enrich not only MSAs of single-copy genes but also more complicated MSAs
including terminally duplicated genes (in-paralogues) and/or corresponding to
multigenic families featuring different out-paralogues of different ages.
Preliminary tests on a broadly sampled eukaryotic data set suggest that the
orthology relationships enforced by C<42> are in good agreement with those
inferred with I<OrthoFinder> software [Emms and Kelly (2015) I<Genome Biol>
16:157]. To this end, it relies on complete proteomes of reference organisms

Regarding contamination, C<42> implements a system of taxonomic filters (based
on I<NCBI Taxonomy>) allowing it to flag (or to reject) any new sequence for
which the taxonomic affiliation is doubtful. However, the power of this
mechanism is currently dependent on the taxonomic breadth of each MSA.

=for todo
    TODO: amend this after decoy / para

C<42> is exclusively setup through a structured text file (e.g., C<YAML>
format). Archiving of this file allows a user to document all the configuration
details for a given run.

=for todo
    TODO: Example of a complete YAML config file

C<42>'s verbosity is configured directly on the command line. C<42> can be very
introspective if asked to be so. At the highest verbosity level, the numerous
C<BLAST> reports are not deleted after the run and are thus available for manual
inspection (e.g., for debugging purposes).


=head1 Design principles

In a single run, C<42> can process an arbitrary large number of MSAs (specified
using shell jokers on the command line). Moreover, one can search for
orthologous sequences in as many organisms as wanted.

The configuration (C<config>) file has two main parts: one with the options that
apply globally to the run and one that lists the organisms (C<orgs>) to search
and their specific options, including the path (C<bank_dir>) to the
corresponding sequence databases (C<banks> in C<42>'s parlance). The C<config>
file includes a mechanism of default values (C<defaults>) that apply to all
organisms except when otherwise specified in individual C<org> subsections
(e.g., C<code>).

When C<42> enriches a MSA, it processes each organism in turn following the
order of C<org> subsections in the C<config> file. Several I<out-of-order>
optimisations ensure that similar computations (e.g., C<BLAST> searches) are not
repeated uselessly.


=head1 Functional overview

=for todo
    TODO: Include Mick's version of graphical overview

=head2 Orthology-controlling heuristics

Each run of C<42> must specify a set of candidate organisms C<orgs> that are
going to be mined for orthologs, a set of reference organisms (C<ref_orgs>), for
which the complete proteomes have to be available (C<ref_bank_dir>,
C<ref_org_mapper>), and a set of query organisms (C<query_orgs>), which should
be represented in most MSAs to be enriched. These two latter sets of organisms
do not need to be identical but certainly can. They will apply to all organisms
(C<orgs>) to be added to yield the new MSAs (C<out_suffix>).


=head3 Collection of queries

For each C<org>, C<42> extracts all sequences belonging to the C<query_orgs> in
order to assemble a list of C<query_seqs>. Those are used to mine C<orgs> for
homologs (candidate orthologs) and to generate a list of 'validating' orthologs
out of C<ref_orgs>. If a MSA does not contain any sequence fulfilling the
selection criteria, C<42> warns the user and falls back to selecting the longest
sequence instead, which leads to a singleton C<query_seqs>.


=head3 Preflight check of orthology relationships

To ensure that it can accurately enrich MSAs in orthologous sequences, C<42>
verifies that C<query_seqs> and C<ref_orgs> themselves satisfy its orthology
criteria. This two-step process is carried out separately for each MSA.

First, an average C<BLASTP> bit score is computed for each C<ref_org> based on
the individual best hits of each C<query_seq> against the corresponding complete
proteomes. C<query_seqs> without any hit in a given C<ref_org> are taken into
account by contributing a value of zero to the average bit score for the
C<ref_org>. How exactly first hits are considered best hits is explained in
L<Identification of best hits for
queries|#identification-of-best-hits-for-queries>.

C<ref_orgs> without any hit to C<query_seqs> are automatically discarded,
whereas the remaining ones are ranked in descending order on the average bit
score. Low-scoring C<ref_orgs> can be optionally discarded by specifying a value
< 1.0 for the C<ref_org_mul> parameter of the C<config> file. For example,
assuming the user lists 10 different C<ref_orgs> and set C<ref_mul_org> to 0.7,
at most 7 C<ref_orgs> will be retained for assessing orthology relationships.
This could be the result of the automatic removal of two C<ref_orgs> without any
hit and of an additional low-scoring one to honor the C<ref_mul_org> setting.

Second, the best hits for each C<ref_org> are C<BLAST>ed (C<BLASTP>) against the
complete proteomes of other C<ref_orgs> to check that they indeed recover the
same best hits as the C<query_seqs>. If any C<ref_org> fails with any of the
other C<ref_orgs>, a message is issued to warn the user, but C<42> proceeds
normally. More details about the logic behind this are available in
L<Identification of orthologues among
homologues|#identification-of-orthologues-among-homologues>. Otherwise, the
preflight check is considered successful.


=head3 Search for homologues using queries

Each one of the C<query_seqs> is C<BLAST>ed in turn against each one of the
C<banks> for the current C<org>. The exact C<BLAST> flavour is either C<TBLASTN>
or C<BLASTP>, depending on the sequence type of C<org>'s C<banks>. Moreover,
default options of this first C<BLAST> can be overridden by specifying key/value
pairs in the subsection C<homologues> under the section C<blast_args> of the
C<config> file (e.g., low-complexity filters, E-value threshold, maximum number
of hits).

The whole set of hits corresponding to all C<query_seqs> is consolidated into a
single list of B<homologous> sequences. These sequences can be optionally
trimmed to the segment really covered by the matching C<query_seqs>. This
behaviour is useful to avoid non-core regions to perturb orthology assessment.
It is controlled by the C<seq_trimming> parameter of the C<config> file.


=head3 Identification of best hits for queries

Each C<query_seq> is furthermore C<BLAST>ed (C<BLASTP>) against the complete
proteome of each C<ref_org>. Again, C<BLAST> options can be overridden if needed
(subsection C<references> under section C<blast_args>). For each C<query_seq>,
the best hit in the C<ref_org> is recorded. However, when bit scores of
subsequent hits are nearly equal to the bit score of the best hit, the
corresponding sequences are interpreted as closely related in-paralogues and
also added to the list of B<best hits>. This behaviour can be tweaked using the
C<bitscore_mul> parameter of the C<config> file.

As a consequence, several best hits can be recorded for a single
C<query_seq>/C<ref_org> pair, either because several sequences are available for
the C<query_org> (in-paralogues or out-paralogues in the case of a multigenic
family) or because several sequences match a single C<query_seq> in the C<org>'s
C<banks> (which should be co-orthologues then), or for both reasons. In
contrast, if a C<ref_org> has no homologue for the current MSA, C<42> warns the
user and drops it from the list of C<ref_orgs> considered by the
orthology-controlling engine.


=head3 Identification of orthologues among homologues

To sort out orthologous sequences from paralogous sequences, each homologue in
the current C<org> is C<BLAST>ed (C<BLASTX> or C<BLASTP>) against the complete
proteome of each C<ref_org> (C<BLAST> options in subsection C<orthologues> under
section C<blast_args>). And now, here's the heart of C<42>'s heuristics... To be
considered as an orthologue, a homologue must satisfy the following criterion
for every one of the (active) C<ref_orgs> without exception: its best hit in the
corresponding complete proteome must be found in the original list of best hits
assembled using the C<query_seqs>.

It is important to note that C<42> does not care about which particular
C<query_seq> (or C<query_seqs>) recovered the homologue in the C<org> nor about
those that recovered the best hits in the complete proteomes of the C<ref_orgs>.
The only thing that matters is that I<the loop is closed>. The set of homologues
for which this condition holds then become the B<orthologues>. If the parameter
C<brh_mode> of the C<config> file is set to C<disabled>, all homologues are
automatically considered as orthologues (but see L<PARA files|"Optimized
enrichment of multigenic families"> just below).


=head3 Optimized enrichment of multigenic families

For multigenic families split over multiple MSAs, one can also optionally
assemble C<PARA> files. Such a file should contain sequences representative of
the other sub-families of a multigenic family, so as to help C<42> to even
better discriminate between orthologous and paralogous sequences. Sequences in
C<PARA> files are in C<FASTA> format but do not need to be aligned. To be
considered as an orthologue, a homologue must obtain a best hit C<BLAST> bit
score that is higher when compared to the sequences of the MSA than those of the
C<PARA> file.

For example, let us say we have a family composed of 4 subfamilies (A-D). The
initial orthologous group would include the 4 types of paralogues in a single
MSA. Based on a phylogenetic analysis of this MSA, we could split this
orthologous group into 4 distinct MSAs (A-D). If we consider the enrichment of
subfamily A (C<famA.fasta>), then the sequences of the other subfamilies (B, C
and D) should be used to build the C<PARA> file (C<famA.para>). Hence, any
homologous sequence that would be more similar to a sequence in the C<PARA> file
(say of type B) than to any sequence (of type A) in the MSA would then be
rejected as paralogous.

In some cases, C<PARA> files are an adequate replacement for the main heuristics
of C<42>. Yet, both approaches can be used jointly for maximal accuracy.


=head2 Orthologue post-processing

Once orthologues are identified, each one is C<BLAST>ed (C<BLASTX> or C<BLASTP>)
against the MSA itself to recover its closest relatives (C<BLAST> options in
subsection C<templates> under section C<blast_args>).


=head3 Family affiliation and orthologue naming

If the most closely related sequence in the MSA belongs to a given family (e.g.,
C<mt->), the orthologue is affiliated to the same family, as did the original
C<forty>. This allows enriching MSAs corresponding to multigenic families. Note
that only the most closely related sequence can be used to infer the
orthologue's family.

The orthologue identifier is built using the C<org> name and the accession of
the corresponding sequence in the C<org>'s C<banks>, which helps tracking down
all the sequences added to a MSA by C<42> (e.g., for debugging purposes). This
is thus different from the original C<forty>, in which most sequences were
I<contigs> having lost all connection with the nucleotide sequences in the
C<org>'s C<banks>.


=head3 Contamination detection and handling

C<42> then seeks to determine whether the orthologue is a genuine orthologue or
a xenologue contaminating the C<org>'s C<banks>. To this end, it infers the
orthologue's taxonomy by analysing the identifiers of the five closest sequences
in the MSA. More precisely, it considers each of them in turn and stops as soon
as one of them can be reliably affiliated to a I<NCBI Taxonomy> entry.

If the taxon corresponding to the entry satisfies the taxonomic filter
(C<tax_filter> parameters in the C<config> file), the orthologue is simply added
to the MSA. Otherwise, it is further tagged as a contaminant (C<c#>). When an
orthologue is tagged as a contaminant, the binomial of the organism at the
origin of the taxonomic inference is appended to its identifier (i.e.,
C<...Genus_species>).

Taxonomic filters are optional and require a local copy of the I<NCBI Taxonomy>
database (C<tax_dir> parameter in the C<config> file). It can be installed using
C<setup-taxdir.pl> (see L<Installation>).


=head3 Alignment and MSA integration

To integrate the orthologue into the MSA, C<42> chooses the most appropriate
B<template(s)> for alignment among the closest relatives. As for taxonomic
inference, it considers each of them in turn and stops once the coverage of the
orthologue cannot be significantly improved. This allows C<42> to select a
slightly less related sequence as a template provided it aligns with a longer
part of the orthologue. By how much exactly coverage has to be improved for a
close sequence to be retained as a template can be fine-tuned with the
C<coverage_mul> parameter of the C<config> file.

Then comes the alignment itself. With nucleotide C<banks>, both C<BLAST> and
C<exonerate> aligners are available, whereas only C<BLAST> can be used with
protein C<banks>. The preferred aligner can be specified using the C<aligner>
parameter of the C<config> file.

The C<BLAST> aligner has been much improved with respect to the aligner of the
original C<forty>. It extracts all the HSPs for the selected template(s) from
the XML C<BLAST> report and uses them as guides for integrating the orthologue
fragments into the MSA. Then, once all fragments have been integrated for all
candidate organisms, it merges them into a single contiguous sequence per
orthologue. When fragments overlap, the merger gives precedence to the fragments
corresponding to the highest-scoring templates and HSPs.

When the new C<exonerate> aligner is preferred, only the longest selected
template is used. In most cases, the orthologue can be aligned as a single large
fragment. If not, C<42> emits different types of warnings depending on the exact
issue. In worst cases (e.g., C<exonerate> crashing), the orthologue cannot be
integrated, often due to structural rearrangements between the orthologue and
the template. To avoid discarding the orthologue in such cases, one can enable
C<BLAST> as a fall-back for exonerate failures by setting the C<aligner>
parameter to C<exoblast>.

Aligned orthologues are integrated into the MSA all together at the end of the
file but in the following arrangement: first by family, then by candidate
organism and then by accession. Contaminants are interspersed with genuine
orthologues but can be easily identified thanks to their tag (C<c#>).


=head3 Redundancy detection and handling

Independently of the aligner, C<42> never integrates twice the same sequence for
a given organism, even if obtained from multiple orthologues. Further, it
filters out subsequences included in sequences from the same organism that are
either already present in the MSA or that are listed in the C<NON> counterpart
of the MSA. C<NON> files are a bit like C<PARA> files (non-aligned sequences in
C<FASTA> format) except that matches must be exact. Finally, when a newly added
orthologue includes a sequence already present in the MSA for the same organism,
the latter can be either kept or removed, depending on the value of the
parameter C<ls_action> in the C<config> file.


=head1 Usage


=head2 Installation

C<42> is written I<Modern Perl> but relies on two external dependencies:
I<NCBI-BLAST+> and I<Exonerate>. You should download and install the
corresponding binaries the way you feel the most appropriate for your system.

=over

=item - L<ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/>

=item - L<https://www.ebi.ac.uk/about/vertebrate-genomics/software/exonerate>

=back

Most other dependencies can be handled automatically by using C<cpanm> in a
I<Perlbrew> environment L<https://perlbrew.pl/>. Below are a set of commands to
setup such an environment on I<Ubuntu>.

    # install development tools
    $ sudo apt-get update
    $ sudo apt-get install build-essential

    # download the perlbrew installer...
    $ wget -O - http://install.perlbrew.pl | bash

    # initialize perlbrew
    $ source ~/perl5/perlbrew/etc/bashrc
    $ perlbrew init

    # search for a recent stable version of the perl interpreter
    $ perlbrew available
    # install the last even version (e.g., 5.24.x, 5.26.x, 5.28.x)
    # (this will take a while)
    $ perlbrew install perl-5.26.2
    # install cpanm (for Perl dependencies)
    $ perlbrew install-cpanm

    # enable the just-installed version
    $ perlbrew list
    $ perlbrew switch perl-5.26.2

    # make perlbrew always available
    # if using bash (be sure to use double >> to append)
    $ echo "source ~/perl5/perlbrew/etc/bashrc" >> ~/.bashrc
    # if using zsh  (only the destination file changes)
    $ echo "source ~/perl5/perlbrew/etc/bashrc" >> ~/.zshrc

Major C<42> dependencies are the C<Bio::MUST> series of modules. Install them as
follows.

    $ cpanm Bio::FastParsers
    $ cpanm Bio::MUST::Core
    $ cpanm Bio::MUST::Drivers

Since C<Bio::MUST> modules rely external bioinformatics programs and come with
complex test suites, they sometimes raise errors during installation. If you
encounter any such error, consider enabling C<--force> and/or C<--notest>
options of C<cpanm>.

    $ cpanm --force Bio::MUST::Drivers

Install C<42> itself. All remaining dependencies can also be taken care of by
C<cpanm>.

    $ cpanm Bio::MUST::Apps::FortyTwo

Finally install a local mirror of the I<NCBI Taxonomy>. It will be used by C<42>
to taxonomically affiliate inferred orthologous sequences.

    $ setup-taxdir.pl --taxdir=taxdump/


=head2 Input and configuration files

To help with the configuration of the numerous parameters of the software, we
designed a C<config> file generator: C<yaml-generator-42.pl>. When run with the
C<--wizard> option, it will guide you through the configuration by prompting for
all required parameters (pressing C<ENTER> selects the default value). At the
end of process, it will produce a YAML C<config> file named
C<config-$SUFFIX.yaml> and a file (C<build-$SUFFIX.sh>) providing the command to
reproduce the exact same configuration without using the wizard.


=head3 MSAs (C<*.fasta>)

Your MSAs collection can be in C<FASTA> format (aligned or not) and sequence
identifiers must obey a set of basic rules. Each identifier has to hold the
organism name (C<org>) followed by a separator (C<@>) and by a protein/gene
accession number. The organism name is usually the binomial name. Genus and
species must be separated by a whitespace (C<>) or underscore character (C<_>).
In addition, strain name and/or NCBI taxon id are also allowed after the species
name but each preceded by an underscore character (C<_>). If both are used in
the sequence identifier, the taxon id has to come last. Finally, all sequence
identifiers must be unique within each MSA. See examples below.

=for todo
    TODO: explain other formats and subtleties (e.g., families)

    # Genus species@protacc
    >Arabidopsis thaliana@AAL15244
    # Genus species_taxonid@protacc
    >Arabidopsis thaliana_3702@AAO44026
    # Genus species_subspecies_taxonid@protacc
    >Arabidopsis lyrata_lyrata_81972@EFH60692
    # Genus species_taxonid@protacc
    >Archaeoglobus fulgidus_2234@WP_048095550
    # Genus species_strain_taxonid@protacc
    >Archaeoglobus fulgidus_DSM4304_224325@AAB90113
    # Genus species_strain_taxonid@protacc
    >archaeon 13_1_20CM_2_54_9_1805008@OLE74253


=head3 Reference organisms (C<ref_orgs>, C<ref_banks>)

The reference proteome set must be described in the C<config> file. Firstly,
each of the reference proteomes must be in C<FASTA> format in order to be
formatted as a C<BLAST> database with the C<makeblastdb> command.

    $ for REFORG in *.faa; do makeblastdb -in $REFORG -dbtype prot \
        -out `basename $REFORG .faa` -parse_seqids; done

Then, C<yaml-generator-42.pl> will read a file describing the reference proteome
set (C<ref_org_mapper.idm>). This file is composed of two columns separated by a
tabulation character (C<\t>) with the first column being the database basename
(C<ref_bank>) and the second being the organism name (C<ref_org>).

If your banks are like this:

    $ ls Arabidopsis_thaliana_3702_bank.*

    Arabidopsis_thaliana_3702_bank.faa
    Arabidopsis_thaliana_3702_bank.phr
    Arabidopsis_thaliana_3702_bank.pin
    Arabidopsis_thaliana_3702_bank.pog
    Arabidopsis_thaliana_3702_bank.psd
    Arabidopsis_thaliana_3702_bank.psi
    Arabidopsis_thaliana_3702_bank.psq

Then the C<ref_org_mapper> file should look like this:

    Arabidopsis_thaliana_3702_bank    Arabidopsis thaliana_3702

Note: Organism names must follow the same rules as those described above for the
sequence identifiers in your MSAs!


=head3 Query organisms (C<query_orgs>)

C<query_orgs> should be listed in a file (C<queries.txt>) and spelled exactly as
in your MSAs. This file will be processed by C<yaml-generator-42.pl> to populate
the C<config> file. To easily draft a list of C<query_orgs>, you can for example
use the 10 to 20 most represented organisms across all you MSAs (prior to
enrichment).

    $ grep -h \> *.fasta | cut -f1 -d'@' | sort | uniq -c | sort -rn | head -n10

    22498 >Danio_rerio
    21071 >Homo_sapiens
    20722 >Mus_musculus
    18933 >Monodelphis_domestica
    18616 >Loxodonta_africana
    17762 >Latimeria_chalumnae
    17678 >Canis_familiaris
    17114 >Xenopus_tropicalis
    16665 >Anolis_carolinensis
    16611 >Sarcophilus_harrisii


=head3 Candidate organisms (C<orgs>, C<banks>)

The candidate organisms set must be described in the C<config> file. Firstly,
each of the candidate organism files must be in C<FASTA> format in order to
produce a C<BLAST> database with the C<makeblastdb> command:

    $ for ORG in *.fna; do makeblastdb -in $ORG -dbtype nucl \
        -out `basename $ORG .fna` -parse_seqids; done

Within each C<BLAST> database, sequence identifiers must be unique. C<42> will
use the first run of non-whitespace characters as the accession. If this first
chunk is composed of multiple parts separated by pipe characters (C<|>), only
the last part is taken as the sequence accession (see L<orthologue
naming|"Family affiliation and orthologue naming">).

    sequence identifier                      accession

    >seq37                                   seq37
    >comp12_c0_seq1                          comp12_c0_seq1
    >EH093040.1 Sl_SlB_01N04_T7 SLB ...      EH093040.1
    >MMETSP0151_2-20130828|7_1 len=174       7_1
    >gi|301500844|ref|YP_003795256.1| ...    YP_003795256.1

Then, as for C<ref_orgs> above, you need to produce a C<bank_mapper.idm> file
composed of two columns separated by a tabulation character (C<\t>) with the
first column being the database basename (C<bank>) and the second being the
organism name (C<org>).

    Euglena_bank    Euglena gracilis

Note: Again, organism names must follow the same rules as those described above
for the sequence identifiers in your MSAs!


=head3 Taxonomic filters (C<tax_filter>)

=for todo
    TODO: explain purpose of tax-filter and simplify this explanation

You can define a taxonomic filter for each organism by adding a third column to
the C<bank_mapper.idm> file or by letting the generator choose them for you.
With the latter option, you must define at which taxonomic level(s) you want to
set the filter. When specifying several levels (C<--levels=> option), the script
will try to check for the next level in case one is missing. You can put as many
levels as you want separated by a comma (C<,>, no whitespace character) when
using the C<--wizard> option and by a whitespace character (C<>) as a command
line argument. Another possibility is to choose manually from the NCBI lineage
for those that fail (in this case use the C<--choose_tax_filter=1> argument). If
you want to select manually for each candidate organisme set the argument
C<--choose_tax_filter=2>.

Now that you are done preparing files, let's run the wizard!

    yaml-generator-42.pl --wizard


=head2 Assisted configuration using the C<wizard>

Using the C<--wizard> flag enables an interactive mode where you will be asked
to enter each parameter in the terminal.

Note: Pressing the C<ENTER> key selects the default value encoded in C<42>.

Two C<run_mode> are available C<metagenomic> or C<phylogenomic>. The
C<phylogenomic> mode is designed to enrich MSAs with orthologues for subsequent
phylogenomic analysis. In contrast, the C<metagenomic> mode is designed to
estimate the contamination level of transcriptomic data using reference
ribosomal protein MSAs. The latter mode does not modify the MSAs but instead
produces one taxonomic report per MSA listing the lineage of each identified
orthologous sequence. When not specified, C<run_mode> internally defaults to
C<phylogenomic>.

=for todo
    TODO: explain how to debrief metagenomic mode for evaluating transcriptomes

Note: the C<phylogenomic> mode also produces taxonomic reports but deprived of
taxonomic affiliations for the purpose of C<one-on-one.pl> (not currently
distributed on I<CPAN>).
