# Copyright 2018 Jeffrey Kegler
# This file is part of Marpa::R3.  Marpa::R3 is free software: you can
# redistribute it and/or modify it under the terms of the GNU Lesser
# General Public License as published by the Free Software Foundation,
# either version 3 of the License, or (at your option) any later version.
#
# Marpa::R3 is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser
# General Public License along with Marpa::R3.  If not, see
# http://www.gnu.org/licenses/.

=head1 NAME

Marpa::R3::External::Model - Details on input models

=head1 About this document

The alternative input models described in this document are an
advanced technique.
If you are starting out with Marpa, you
probably want to ignore this document.
If you are an experienced Marpa user,
it is still safe to ignore this document,
but you might find the possibilities it discusses
interesting.

This document contains a taxonomy of alternative input models,
detailing their behavior in terms of G1 location and earlemes.
The primary purpose is to aid a programmer in designing new
alternative input models.
A secondary purpose is to clarify Marpa's behavior for all
input models.

Understanding the
settings of G1 location
and of current, closest and furthest earleme is
crucial to working with advanced input models.
For this reason the next sections will go
through the possibilities carefully.
The presentation will start with the most traditional
and restrictive models.
It will proceed to less restrictive models.

This is not the document to which to turn first.
Before reading this document, you should be familiar with
L<Marpa::R3::External::Basic>
and 
L<Marpa::R3::External::Low>.

For brevity, this document speaks only of the
C<lexeme_alternative()> method.
Note that, unless otherwise stated,
everything said in this document
applies to the 
C<lexeme_alternative_literal()> method as well.

This document describes only the behavior
of B<successful> method calls.
For the behavior on failure,
see L<Marpa::R3::External::Low>.

=head1 The initial earleme settings

All input models have the same initial values.
Initially the current Earley set ID (G1 location)
in always 0.
Initially,
the current, closest, and furthest earleme
are always earleme 0.

=head1 The standard model of input

An application will rarely implement
the standard model of input
using the low level external scanning methods.
The high level methods are much simpler
and handier.

But if an application does implement
the standard model of input
using the low level methods,
it does so by making
calls to C<lexeme_alternative()>
and C<lexeme_complete()>
in pairs.
It will make, first, exactly one call
to C<lexeme_alternative()>,
and for a lexeme with length 1.
Following the call to C<lexeme_alternative()>
must come the second call of the pair --
a call
to C<lexeme_complete()>.
For an input of length I<n>, there will be
exactly I<n> such paired calls.

Suppose, in the standard model that,
for a successful call
to C<lexeme_alternative()>,
the following is true:

=over 4

=item *

The current earleme before the call is I<c>.

=back

Because this is the standard model,
this means that we also know that

=over 4

=item *

The current Earley set before the call is I<c>.

=item *

The closest earleme before the call is I<c>.

=item *

The furthest earleme before the call is I<c>.

=back

In that case,
after the successful call to
C<lexeme_alternative()>,
the following will be true:

=over 4

=item *

The current earleme will still be I<c>.

=item *

The current Earley set ID will still be I<c>.

=item *

The closest earleme will be I<c>+1.

=item *

The furthest earleme will be I<c>+1.

=back

Suppose, in the standard model that,
before a successful call
to C<lexeme_complete()>,
the following is true:

=over 4

=item *

The current earleme before the call is I<c>.

=back

Because this is the standard model,
this means that we also know that

=over 4

=item *

The current Earley set ID before the call is I<c>.

=item *

The pending lexeme queue contains
exactly one pending lexeme set,
a singleton set at earleme I<c+1>.

=item *

The closest earleme before the call is I<c+1>.

=item *

The furthest earleme before the call is I<c+1>.

=back

In that case,
after the successful call
to C<lexeme_complete()>,
the current, closest, and furthest earlemes will
be the same.
Specifically,
the following will be true:

=over 4

=item *

The current earleme will be advanced to I<c+1>.

=item *

The current Earley set ID will be advanced to I<c+1>.

=item *

The pending lexeme queue will be empty.

=item *

The closest earleme will remain at I<c+1>.

=item *

The furthest earleme will remain at I<c+1>.

=back

=head1 Ambiguous input

As a first loosening of the standard model,
we no longer require calls to C<lexeme_alternative()>
to be paired with calls to
C<lexeme_complete()>.
Instead,
we allow a series of one or more calls
to C<lexeme_alternative()>
before each call to
C<lexeme_complete()>.

We still require that there be at least one call
to C<lexeme_alternative()>
before each call to
C<lexeme_complete()>.
We also still require that all lexemes have
a length of exactly 1.

In the ambiguous input model, the behavior
of the current Earley set ID and
of the current,
closest, and furthest earlemes are exactly
as described for the standard model, with these exceptions:

=over 4

=item *

Where the current earleme is I<c>,
the value of the closest earleme before the second and subsequent
successful calls of C<lexeme_alternative()>
will be
I<c>+1.

=item *

Where the current earleme is I<c>,
the value of the furthest earleme before the second and subsequent
successful calls of C<lexeme_alternative()>
will be
I<c>+1.

=item *

The pending lexeme queue will still always contains
exactly one pending lexeme set,
and it will still be
at earleme I<c+1>.
But after the C<n>'th successful call to C<lexeme_alternative()>
it will contain C<n> lexemes.

=back

=head1 Variable length lexemes

Our final loosening of the restrictions is to allow
variable length lexemes.
That is, instead of requiring that all lexemes
be of length 1,
we allow lexemes to be of length 1 or longer.

=head2 General rule for pending lexemes

In this subsection we state
a general rule for the pending lexeme queue,
which applies at all times in all input models;

If the pending lexeme queue is empty
the closest and furthest earlemes are equal
to the current earleme.

If the pending lexeme queue is non-empty,
let it be

=for Marpa::R3::Display
ignore: 1

    lexset[1] ... lexset[n],

=for Marpa::R3::Display::End

where C<< n >= 1 >>.
Then the end earleme of the earlemes in C<lexset[1]>
is the closest earleme;
and 
the end earleme of the earlemes in C<lexset[n]>
is the furthest earleme.

=head2 Call of lexeme_alternative()

Suppose, in the variable length lexeme model that,
for a successful call
to C<lexeme_alternative()>,
the following is true:

=over 4

=item *

The current Earley set ID before the call is I<esid>.

=item *

The current earleme before the call is I<c>.

=item *

The length of the lexeme is I<length>.

=back

In that case,
after a successful call to
C<lexeme_alternative()>,
the following will be true:

=over 4

=item *

The current Earley set ID will be I<esid>.
The current Earley set is never changed by a call
to C<lexeme_alternative()>.

=item *

The current earleme will still be I<c>.
The current earleme is never changed by a call
to C<lexeme_alternative()>.

=item *

The pending lexeme queue will be non-empty,
and the closest and furthest earleme will obey
the L<general rule|/"General rule for pending lexemes">.

=item *

The furthest earleme will be greater than or
equal to I<c>+I<length>.

=item *

The closest earleme will be less than or
equal to I<c>+I<length>.

=back

=head2 Call of lexeme_complete()

Suppose, in the variable length lexeme model that,
before a successful call
to C<lexeme_complete()>,
the following is true:

=over 4

=item *

The current Earley set ID before the call is I<esid>.

=item *

The current earleme before the call is I<c>.

=item *

The closest earleme before the call is I<old_closest>.

=back

In that case,
after the successful call
to C<lexeme_complete()>,
the following will be true:

=over 4

=item *

The current Earley set ID will be I<esid>C< + 1>.

=item *

The current earleme will be I<old_closest>.

=item *

The pending lexeme queue may or may not be empty,
and the closest and furthest earleme will obey
the L<general rule|/"General rule for pending lexemes">.

=back

=head1 General rules for the earleme variables

The following
will always be the case,
no matter which of the input models is in use.

=over 4

=item *

The current earleme is always greater than
or equal to the current Earley set ID.

=item *

The closest earleme is always greater than
or equal to the current earleme.

=item *

The furthest earleme is always greater than
or equal to the closest earleme.

=back

=head1 Copyright and License

=for Marpa::R3::Display
ignore: 1

  Copyright 2018 Jeffrey Kegler
  This file is part of Marpa::R3.  Marpa::R3 is free software: you can
  redistribute it and/or modify it under the terms of the GNU Lesser
  General Public License as published by the Free Software Foundation,
  either version 3 of the License, or (at your option) any later version.

  Marpa::R3 is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
  Lesser General Public License for more details.

  You should have received a copy of the GNU Lesser
  General Public License along with Marpa::R3.  If not, see
  http://www.gnu.org/licenses/.

=for Marpa::R3::Display::End

=cut

