A source filter for PDL: take the pain out of slicing !

Perl source filtering is a way to experiment with new syntax
and the power is only limited by the sophistication of the
source parser.

Build the module in the usual way:

  perl Makefile.PL
  make test
  make install

Everything is still experimental but the
approach seems promising.

The file 'local.perldc' contains some useful definitions to
experiment with the new syntax within the perldl shell.

NAME
    PDL::NiceSlice - toward a nicer slicing syntax for PDL

SYNOPSYS
      use PDL::NiceSlice;

      $a(1:4) .= 2;             # concise syntax for ranges
      print $b((0),1:$end);
      $a->xchg(0,1)->(($pos-1)) .= 0;
  
      $idx = long 1, 7, 3, 0;   # a piddle of indices
      $a(-3:2:2,$idx) += 3;     # mix explicit indexing and ranges

      $a($a!=3;?)++;            # short for $a->where($a!=3)++

DESCRIPTION
    `PDL''s slice syntax sucks. This module tries to rectify the situation
    to some degree. Using Perl's ability to use *source filtering* (which
    you can think of as a very powerful macro facility, see the perlfilter
    manpage) it introduces a more reasonable syntax for slicing PDL objects
    (AKA piddles). `PDL::NiceSlice' implements the source filter that
    provides this new feature.

    *NOTE*: `PDL::NiceSlice' relies on several modules from CPAN that make
    source filtering and parsing easier. In particular it requires
    Filter::Simple (which in turn requires a module from the Filter
    distribution) and Text::Balanced. To make your life easier it is
    recommended that you use the CPAN module to resolve dependencies during
    installation automatically. Using the cpan installation shell
    installation should be as easy as this:

       $ perl -MCPAN -e shell
       cpan> install PDL::NiceSlice

The new slicing syntax
    Using `PDL::NiceSlice' slicing piddles becomes so much easier since,
    first of all, you don't need to make explicit method calls. No

      $pdl->slice(....);

    calls, etc. Instead, `PDL::NiceSlice' introduces two ways in which to
    slice piddles without too much typing:

    * using parentheses directly following a scalar variable name, for
      example

         $c = $b(0:-3:4,(0));

    * using the so called *default method* invocation in which the piddle
      object is treated as if it were a reference to a subroutine (see also
      the perlref manpage). Take this example that slices a piddle that is
      part of a perl list `@b':

        $c = $b[0]->(0:-3:4,(0));

    The format of the argument list is the same for both types of invocation
    and will be explained in more detail below.

  Parentheses following a scalar variable name

    An arglist in parentheses following directly after a scalar variable
    name that is *not* preceded by `&' will be resolved as a slicing
    command, e.g.

      $a(1:4) .= 2;         # only use this syntax on piddles
      $sum += $a(,(1));

    However, if the variable name is immediately preceded by a `&', for
    example

      &$a(4,5);

    it will not be interpreted as a slicing expression. Rather, to avoid
    interfering with the current subref syntax, it will be treated as an
    invocation of the code reference `$a' with argumentlist `(4,5)'.

  The *default method* syntax

    The second syntax that will be recognized is what I called the *default
    method* syntax. It is the method arrow `->' directly followed by an open
    parenthesis, e.g.

      $a->xchg(0,1)->(($pos)) .= 0;

    Note that this conflicts with the use of normal code references, since
    you can write in plain Perl

      $sub = sub { print join ',', @_ };
      $sub->(1,'a');

    NOTE: Once `use PDL::NiceSlice' is in effect (see below how to switch it
    off again in a source file) the source filter will incorrectly replace
    the above call to `$sub' with an invocation of the slicing method. This
    is one of the pitfalls of using a source filter that doesn't know
    anything about the runtime type of a variable (cf. the Implementation
    section).

    This shouldn't be a major problem in practice; a simple workaround is to
    use the `&'-way of calling subrefs, e.g.:

      $sub = sub { print join ',', @_ };
      &$sub(1,'a');

  When to use which syntax?

    Why are there two different ways to invoke slicing? The first syntax
    `$a(args)' doesn't work with chained method calls. E.g.

      $a->xchg(0,1)(0);

    won't work. It can *only* be used directly following a valid perl
    variable name. Instead, use the *default method* syntax in such cases:

      $a->xchg(0,1)->(0);

    Similarly, if you have a list of piddles `@pdls':

      $b = $pdls[5]->(0:-1);

  The argument list

    The argument list is a comma separated list. Each argument specifies how
    the corresponding dimension in the piddle is sliced. In contrast to
    usage of the slice method the arguments should *not* be quoted. Rather
    freely mix literals (1,3,etc), perl variabales and function invocations,
    e.g.

      $a($pos-1:$end,myfunc(1,3)) .= 5;

    There can even be other slicing commands in the arglist:

      $a(0:-1:$pdl($step)) *= 2;

    NOTE: If you use function calls in the arglist make sure that you use
    parentheses around their argument lists. Otherwise the source filter
    will get confused since it splits the argument list on commas that are
    not protected by parentheses. Take the following example:

      sub myfunc { return 5*$_[0]+$_[1] }
      $a = sequence 10;
      $sl = $a(0:myfunc 1, 2);
      print $sl;
     PDL barfed: Error in slice:Too many dims in slice
     Caught at file /usr/local/bin/perldl, line 232, pkg main

    The simple fix is

      $sl = $a(0:myfunc(1, 2));
      print $sl;
     [0 1 2 3 4 5 6 7]

    Note that using prototypes in the definition of myfunc does not help. At
    this stage the source filter is simply not intelligent enough to make
    use of this information. So beware of this subtlety.

    Another pitfall to be aware of: currently, you can't use the conditional
    operator in slice expressions (i.e., `?:', since the parser confuses
    them with ranges). For example, the following will cause an error:

      $a = sequence 10;
      $b = rand > 0.5 ? 0 : 1; # this one is ok
      print $a($b ? 1 : 2);    # error !
     syntax error at (eval 59) line 3, near "1,

    For the moment, just try to stay clear of the conditional operator in
    slice expressions (or provide us with a patch to the parser to resolve
    this issue ;).

  Modifiers

    Following a suggestion originally put forward by Karl Glazebrook the
    latest versions of `PDL::NiceSlice' implement *modifiers* in slice
    expressions. Modifiers are convenient shorthands for common variations
    on PDL slicing. The general syntax is

        $pdl(<slice>;<modifier>)

    Three modifiers are currently implemented:

    *   `_': flatten the piddle before applying the slice expression. Here
        is an example

           $b = sequence 3, 3;
           print $b(0:-2;_); # same as $b->flat->(0:-2)
         [0 1 2 3 4 5 6 7]

        which is quite different from the same slice expression without the
        modifier

           print $b(0:-2);
         [
          [0 1]
          [3 4]
          [6 7]
         ]

    *   `|': sever the link to the piddle, e.g.

           $a = sequence 10;
           $b = $a(0:2;|);  # same as $a(0:2)->sever
           $b++;
           print $a; # check if $a has been modified
         [0 1 2 3 4 5 6 7 8 9]

    *   `?': short hand to indicate that this is really a where expression

        As expressions like

          $a->where($a>5)

        are used very often you can write that shorter as

          $a($a>5;?)

        With the `?'-modifier the expression preceding the modifier is *not*
        really a slice expression (e.g. ranges are not allowed) but rather
        an expression as required by the where method. For example, the
        following code will raise an error:

          $a = sequence 10;
          print $a(0:3;?);
         syntax error at (eval 70) line 3, near "0:"

        That's about all there is to know about this one.

    Modifiers are a pretty new feature of `PDL::NiceSlice'. So don't be
    surprised if things don't work quite as expected. Feedback is welcome as
    usual. The modifier syntax may change in the future.

  Argument formats

    In slice expressions you can use ranges and secondly, piddles as 1D
    index lists (although compare the description of the `?'-modifier for
    exceptions).

    * ranges
      You can access ranges using the usual `:' separated format:

        $a($start:$stop:$step) *= 4;

      Note that you can omit the trailing step which then defaults to 1.
      Double colons (`::') are not allowed to avoid clashes with Perl's
      namespace syntax. So if you want to use steps different from the
      default you have to also at least specify the stop position. Examples:

        $a(::2);   # this won't work (in the way you probably intended)
        $a(:-1:2); # this will select every 2nd element in the 1st dim

      Just as with slice negative indices count from the end of the
      dimension backwards with `-1' being the last element. If the start
      index is larger than the stop index the resulting piddle will have the
      elements in reverse order between these limits:

        print $a(-2:0:2);
       [8 6 4 2 0]

      A single index just selects the given index in the slice

        print $a(5);
       [5]

      Note, however, that the corresponding dimension is not removed from
      the resulting piddle but rather reduced to size 1:

        print $a(5)->info
       PDL: Double D [1]

      If you want to get completely rid of that dimension enclose the index
      in parentheses (again similar to the slice syntax):

        print $a((5));
       5

      In this particular example a 0D piddle results. Note that this syntax
      is only allowed with a single index. All these will be errors:

        print $a((0,4));  # will work but not in the intended way
        print $a((0:4));  # compile time error

      An empty argument selects the whole dimension, in this example all of
      the first dimension:

        print $a(,(0));

      Alternative ways to select a whole dimension are

        $a = sequence 5, 5; 
        print $a(:,(0));
        print $a(0:-1,(0));
        print $a(:-1,(0));
        print $a(0:,(0));

      Arguments for trailing dimensions can be omitted. In that case these
      dimensions will be fully kept in the sliced piddle:

        $a = random 3,4,5;
        print $a->info;
       PDL: Double D [3,4,5]
        print $a((0))->info;
       PDL: Double D [4,5]
        print $a((0),:,:)->info;  # a more explicit way
       PDL: Double D [4,5]
        print $a((0),,)->info;    # similar
       PDL: Double D [4,5]

    * piddle index lists
      The second way to select indices from a dimension is via 1D piddles of
      indices. A simple example:

        $a = random 10;
        $idx = long 3,4,7,0;
        $b = $a($idx);

      This way of selecting indices was previously only possible using dice
      (`PDL::NiceSlice' attempts to unify the `slice' and `dice'
      interfaces). Note that the indexing piddles must be 1D or 0D. Higher
      dimensional piddles as indices will raise an error:

        $a = sequence 5, 5;
        $idx2 = ones 2,2;
        $sum = $a($idx2)->sum;
       piddle must be <= 1D at /home/XXXX/.perldlrc line 93

      Note that using index piddles is not as efficient as using ranges. If
      you can represent the indices you want to select using a range use
      that rather than an equivalent index piddle. In particular, memory
      requirements are increased with index piddles (and execution time
      *may* be longer). That said, if an index piddle is the way to go use
      it!

    As you might have expected ranges and index piddles can be freely mixed
    in slicing expressions:

      $a = random 5, 5;
      $b = $a(-1:2,pdl(3,0,1));

  piddles as indices

    You can use piddles to specify indices in ranges. No need to turn them
    into proper perl scalars with the new slicing syntax. However, make sure
    they contain not more than one element! Otherwise a runtime error will
    be triggered. First a few examples that illustrate proper usage:

      $a = sequence 5, 5;
      $rg = pdl(1,-1,3);
      print $a($rg(0):$rg(1):$rg(2),2);
     [
      [11 14]
     ]
      print $a($rg+1,:$rg(0));
     [
      [2 0 4]
      [7 5 9]
     ]

    The next one raises an error

      print $a($rg+1,:$rg(0:1));
     multielement piddle where only one allowed at XXX/Core.pm line 1170.

    The problem is caused by using the 2-element piddle `$rg(0:1)' as the
    stop index in the second argument `:$rg(0:1)' that is interpreted as a
    range by `PDL::NiceSlice'. You *can* use multielement piddles as index
    piddles as described above but not in ranges. And `PDL::NiceSlice'
    treats any expression with unprotected `:''s as a range. *Unprotected*
    means as usual *"not occurring between matched parentheses"*.

Use in scripts and `perldl' shell
    Source filtering can be switched on and off in scripts and perl modules
    by using or unloading `PDL::NiceSlice'.

    Note: this will *not* work in the perldl shell (see below how to enable
    the new slicing syntax within perldl).

    But now back to scripts and modules. Everything after `use
    PDL::NiceSlice' will be translated and you can use the snew slicing
    syntax. Source filtering will continue until the end of the file is
    encountered. You can stop sourcefiltering before the end of the file by
    issuing a `no PDL::NiceSlice' statement.

    Here is an example:

      use PDL::NiceSlice;

      # this code will be translated
      # and you can use the new slicing syntax

      no PDL::NiceSlice;

      # this code won't
      # and the new slicing syntax will raise errors!

    See also the Filter::Simple manpage and example in this distribution for
    further examples.

    To use the filter in the `perldl' shell you need to add the following
    two lines to your .perldlrc file:

       use PDL::NiceSlice;
       $PERLDL::PREPROCESS = \&PDL::NiceSlice::perldlpp;

    A more complete tool box of commands for experimentation is in the file
    local.perldlrc in the `PDL::NiceSlice' source directory. Just include
    the code in that file in your usual ~/.perldlrc and you can switch
    source filtering with PDL::NiceSlice on and off by typing `trans' and
    `notrans', respectively. To see what and how your commands are
    translated switch reporting on:

      perldl> report 1;

    Similarly, switch reporting off as needed

      perldl> report 0;

    Note that these commands will only work if you included the contents of
    local.perldlrc in your perldl startup file.

  evals and `PDL::NiceSlice'

    Due to `PDL::NiceSlice' being a source filter it won't work in the usual
    way within evals. The following will *not* do what you want:

      $a = sequence 10;
      eval << 'EOE';

      use PDL::NiceSlice;
      $b = $a(0:5);

      EOE
      print $b;

    Instead say:

      use PDL::NiceSlice;
      $a = sequence 10;
      eval << 'EOE';

      $b = $a(0:5);

      EOE
      print $b;

    Source filters *must* be executed at compile time to be effective. And
    `PDL::NiceFilter' is just a source filter (although it is not
    necessarily obvious for the casual user).

Implementation
    `PDL::NiceSlice' exploits the ability of Perl to use source filtering
    (see also the perlfilter manpage). A source filter basically filters (or
    rewrites) your perl code before it is seen by the compiler.
    `PDL::NiceSlice' searches through your Perl source code and when it
    finds the new slicing syntax it rewrites the argument list appropriately
    and splices a call to the `nslice' method using the modified arg list
    into your perl code. You can see how this works in the perldl shell by
    switching on reporting (see above how to do that).

    The `nslice' method is an extended version of mslice that knows how to
    deal with index piddles (and therefore combines slicing and dicing).
    Full documentation of `nslice' will be in the next PDL release.

BUGS
    Error checking is probably not yet foolproof.

    The conditional operator can't be used in slice expressions (see above).

    There is currently an undesired interaction between `PDL::NiceSlice' and
    the new Inline::Pdlpp module (currently only in PDL CVS). Since PP code
    generally contains expressions of the type `$var()' (to access piddles,
    etc) `PDL::NiceSlice' recognizes those incorrectly as slice expressions
    and does its substitutions. For the moment (until hopefully the parser
    can deal with that) it is best to explicitly switch `PDL::NiceSlice' off
    before the section of inlined Pdlpp code. For example:

      use PDL::NiceSlice;
      use Inline::Pdlpp;

      $a = sequence 10;
      $a(0:3)++;
      $a->inc;

      no PDL::NiceSlice;

      __DATA__

      __C__

      ppdef (...); # your full pp definition here

    Feedback and bug reports are welcome. Please include an example that
    demonstrates the problem. Log bug reports in the PDL bug database at

      http://sourceforge.net/bugs/?group_id=612

    or send them to the pdl-porters mailing list
    <pdl-porters@jach.hawaii.edu>.

COPYRIGHT
    Copyright (c) 2001, Christian Soeller. All Rights Reserved. This module
    is free software. It may be used, redistributed and/or modified under
    the same terms as PDL itself (see http://pdl.perl.org).

