\input texinfo   @c -*-texinfo-*-
@setfilename abs-Qshell.info
@settitle A Programming-Language Shell
@setchapternewpage odd

@ifinfo
@node Top, Abstract, (dir), (dir)

@menu
* Abstract::			
* Shell-paper::			A Programming-Language Shell

 --- The Detailed Node Listing ---

A Programming-Language Shell

* Introduction::		
* Related work::		
* Programming language::	A Programming Language
* Internal structure::		
* Evaluation::			Evaluated vs. unevaluated args.
* Writing macros::		Primitives useful for writing macros
* Regular expressions::		
* Filenames::			File and Directory Objects
* Commands::			
* Pipeline implementation::	Pipeline implementation
* Control::			
* Future work::			
* Summary::			
@end menu

@end ifinfo

@node Abstract, Shell-paper, Top, Top
@unnumbered Abstract

Modern high-level programming languages provide multiple
data types (such as numbers, strings, and lists), as well as
first-class function values.  Most utility languages
(awk, perl, tcl) and most shells only have one (or a few) data
types (strings), but they are very convenient for
manipulating text or invoking programs.
This paper discusses the issues involved in
getting the best of both worlds, in the context
of the Q programming language.

For example, executing a program has the same syntax
as a function call, a pipe is function composition,
and a disk file just a persistent string.
The implications of these features on the syntax and
structure of the programming language are discussed.

@node Shell-paper,  , Abstract, Top
@chapter A Programming-Language Shell

@ifinfo
@menu
* Introduction::		
* Related work::		
* Programming language::	A Programming Language
* Internal structure::		
* Evaluation::			Evaluated vs. unevaluated args.
* Writing macros::		Primitives useful for writing macros
* Regular expressions::		
* Filenames::			File and Directory Objects
* Commands::			
* Pipeline implementation::	Pipeline implementation
* Control::			
* Future work::			
* Summary::			
@end menu
@end ifinfo

@node Introduction, Related work, Shell-paper, Shell-paper
@section Introduction

First, we will contrast programming languages
with command and utility languages.  The next section
will discuss some previous attempts to bridge the gap.
The most important difference is in the supported data types.
Programming languages normally support many data-types,
including user-defined ones.
There are ways to group data objects together into
arrays, lists, and structures.  Command languages have
just a few data types, sometimes only strings.
On the other hand, very high-level data types
(associative tables, files) may be supported,
but, their treatment may be ad hoc.

Modern functional languages support first-class functions,
and lexical scoping.
Programming languages do syntax checking
and "compilation" before evaluation. Command languages
are completely interpretive, which limits their efficiency.
It also delays all error checking to run-time, making
debugging harder.  Command languages
are usually interactive, programming languages are
generally batch-oriented.
Command languages are designed for writability,
and have a terse syntax; programming languages
are less terse and more reader-friendly.

This paper discusses these issues in the context of Q in some detail.

@node Related work, Programming language, Introduction, Shell-paper
@section Related work

There are many small Scheme and Lisp implementations designed
for embedding into
applications, to allow the latter to be conveniently
programmable and customizable:
xlisp, xscheme, elk, Scheme->C, and GNU emacs Lisp.
Many people dislike Lisp's syntax.
Also, Lisp is too low-level for a shell.

Tcl [Ousterhout] and Python [Guido van Rossum, Amsterdam] are
also for embedding in applications.
Tcl takes a very minimalist approach: The only data type is string.
This is enough for an embedded command language, but not
for non-trivial programs. 
Python has more sophisticated syntax and data types.
However, Python is very procedural (statement-oriented),
and is not suitable as a shell.

EZ [Fraser and Hanson] assumes a self-contained environment
and is not well integrated into Unix. Also, the syntax
could be terser.

Perl is rather low level (for a shell), and it was not designed
for orthogonality or elegance.

@node Programming language, Internal structure, Related work, Shell-paper
@section A Programming Language

User input will be prefixed by an interactive prompt;
the next line is the output:
@example
Q1> 3+4  # This is a comment
7
@end example
Alternatively:
@example
3+4 ==> 7
@end example
means that @code{3+4} evaluates to @code{7}.

A colon before an identifier indicates a declaration:
@example
Q3> :a=[10 20]
Q4> a+a
20 40
@end example
Arrays are created with @code{[ ]}.
In @code{Q3}, @code{a} is declared as a new variable.
Note that the @code{=} is not assignment, it is unification
(bi-directional pattern matching, as in logic programming):
@example
Q5> [:x :y] = a
Q6> format "x is %s, y is %s.\n" x y
x is 10, y is 20.
@end example
The builtin @code{format} is similar to C's
@code{printf} family, with output to a file or a string (as here).

String literals can contain C-style escapes.
They can also contain an expression following a @code{$},
which cancels one level of quotation.
If @code{a} has the value "5" then @code{"3+$a"}
evaluates to @code{"3+5"}.

@comment Missing?
Q does not need parentheses or commas for a function call.
Such delimiters are inconvenient for interactive use.
@example
Q7> :(Factorial :x) = if x=0 => 1 || x * (Factorial x-1)
Q8> 2 / Factorial 20
1/1216451004088320000
@end example

Q uses spaces to avoid parentheses.
All operations within one word are done first:
@example
3+10 * 10 ==> 130
3 + 10*10 ==> 103
@end example
@Comment different
Using spaces improves terseness, is easier to scan visually
than parentheses, and is consistent with the word-splitting
a shell does.

Newlines separate "statements" (which are
just expressions).  A newline is equivalent with @samp{;}.
Requiring an explicit statement terminator would be clumsy
and error-prone in an interactive language.

@node Internal structure, Evaluation, Programming language, Shell-paper
@section Internal structure

Though dynamic scoping has its uses, static scoping
is preferred in functional languages, because
it permits natural definition of first-class operators.

Lexical scoping means that name binding
and deciding if closures are needed must be done before
evaluation.  Because we want to support mutual recursion
without forward declarations, this step must be a
separate pass, after parsing and before evaluation.
This pass, the name binding pass,
is also when macro expansion is done.


@node Evaluation, Writing macros, Internal structure, Shell-paper
@section Argument evaluation

Shells put implicit quotes around command arguments,
and you have to use an evaluate operator to undo the quoting.
For example the operand @code{xx$(a)yy} is implicitly translated
into @code{(string_concat "xx" a "yy")}.

Implicit quoting is convenient when most arguments
are constants, as in interactive use.
Consider the alternative:
@example
mv "-i" "foo.c" "foo.c.old"
@end example

On the other hand, implicit quoting only really works with strings.
For example in:
@example
abs -10
@end example
we want to pass the integer @code{-10} to the @samp{abs} routine,
not the string "-10".

Q's solution is to provide macros, similar to those in Lisp:
A macro is a function that is run at compile time
to transform an expression into a different expression.
Q macros are applied during the name binding pass, after parsing is complete.
(Because Q allows mutually recursive declarations, we do not know
at parse time if an identifier is a macro or not.)

@node Writing macros, Regular expressions, Evaluation, Shell-paper
@section Primitives useful for writing macros

General macro systems can do arbitrary computation while
expanding a macro.  However, most of the time one only needs
simple textual replacement, perhaps with parameters, as suggested
by experience with the C pre-processor.  The @code{parse}
primitive function provides this functionality.

@subsection parse

The "parse" routine takes any number of strings and/or
expressions.  If there is only a single
string argument, "parse" parses the string,
and yields the resulting parse tree expression.
If there are multiple arguments, @samp{parse} effectively parses their
concatenation:  when it gets to the end of one argument,
it continues with the next.  If one of the arguments is
an expression, it is substituted in place.

The "increment" macro @code{incr} is a simple example:
@example
:(macro incr :Y) = parse Y ":=" Y "+1
@end example
The application:
@example
incr X
@end example
is effectively:
@example
X:=X+1
@end example

@subsection The @code{quote} macro

The macro @code{quote} puts quotes around its operands.
Unless there are unquote operators (using @samp{$}),
the result is constant.
For example:
@example
quote ~/Q/mi/parse* 3+5
==> ["~/Q/mi/parse*" "3+5"]
@end example

The @code{quote} macro is the primitive used by all commands
that do not evaluate their arguments.
For example, the command "cd" is defined as:
@example
:(macro cd :x@@)= parse "__cd (quote " x@@ ")"
@end example

(The first @samp{x@@} means the list of all the remaining arguments.
The second "splices" the list @samp{x} into @samp{quote}'s argument list.)
The function @code{__cd} does the actual work of @code{cd}.

@subsection Features of @code{quote}

Parentheses, quoted strings, and backquotes
are included in the output from @code{quote}:
@example
quote (3+4) "is"\: 10
==> ["(3+4)" "\"is\"\\:" "10"]
@end example

Note that @code{quote}
works on @emph{vectors} of strings, not individual strings.
If there are multiple expressions in parentheses,
the result has multiple strings:
@example
quote (a b)
==> ["(a)" "(b)"]
@end example
Concatenation of string vectors yields a distributive "join":
@example
quote /(a b)/(c d)
==>
["/(a)/(c)" "/(a)/(d)" "/(b)/(c)" "/(b)/(d)"]
@end example
This mechanism subsumes the @code{@{...@}}-feature of csh.

@node Regular expressions, Filenames, Writing macros, Shell-paper
@section Regular expressions

Unix is cursed with a number of incompatible syntaxes
for regular expression patterns, used by different
programs, and with various features.
The shell globbing patterns are used most often.
These are simple and terse, but they are not fully general
regular expressions.  Q's solution
extends the conventional globbing syntax, as in
the Korn Shell.

For example, @samp{*} matches any number of characters
except @samp{\0} or @samp{\n}.
When matching against file names, also does not match a @samp{/},
nor an initial @samp{.}.
It also remembers the matched characters for replacements.
More generally, @code{*(@var{pattern})}
matches any number of instances of the @var{pattern}.

The builtin function @code{match} tries to match a string
against a quoted pattern:
@example
"abcd" match a*d  # Succeeds
@end example
There may be a second replacement pattern:
@example
"abcd" match a*d A*D
==> "AbcD"
@end example

@subsection glob

The @code{glob} function takes a single string, interprets
it as a globbing pattern, and returns a sorted vector of matching
file names.  The result is the empty vector if there are
no matches.

To find every @code{Makefile} in any sub-directory of @code{dld-*} do:
@example
glob "dld-*/*(*/)Makefile"
@end example
Note that the above example is
2-3 times faster than GNU find, because
find has to look into @emph{every}
sub-directory of @code{.}, while Q's @code{glob}
only looks at sub-directories matching @code{dld-*}.
(See full paper for Q's algorithm.)

@subsection globlist

@code{globlist} takes a vector
of strings, does tilde-expansion, and calls @code{glob} on each pattern.
Where there is no match, the answer
is replaced by the original pattern, but with quotes and parentheses removed.
The answers are concatenated to one vector.
@example
globlist (quote x(3+4)y p"ar"se* f(oo).*)
==>
["x3+4y" "parserule.o" "parsemacros.o" "parse.o" "foo.*"]
@end example

A simplified implementation of @code{echo}:
@example
Q1> :(macro echo :X@@)= parse "__echo (quote " X@@ ")"
Q2> :(__echo :L)= format "%@{%s%^ %@}\n" (globlist L)
Q3> echo parse*
parserule.o parsemacros.o parse.o
@end example
(The funny format directives do looping.)

In Unix, the shell traditionally does globbing.  This is
usually convenient, but sometimes the standard
expansion is inappropriate, such as the patterns used by
@code{grep} and @code{find}.
In Q, though, it is straightforward to use @code{glob} and @code{match}
to define a @code{ren} macro for a @code{mv} variant that
handles:
@example
ren parse*.c parse*.c.old
@end example

@node Filenames, Commands, Regular expressions, Shell-paper
@section File and Directory Objects

Since files and strings are both sequences of characters,
it is reasonable to use the same operations for both.
But we need to distinguish files from file names,
just as we distinguish identifiers from the values
they evaluate to.
Q uses the prefix @samp{./} to indicate a filename,
which evaluates to a file "object":
@example
./Makefile # The file named Makefile.
@end example

File names can be specified at runtime:
@example
Q1> :fname = "Makefile"
Q2> ./$fname # Same as ./Makefile
@end example

Two consecutive @samp{/} means to go to root:
@example
.//etc/passwd  # The file name /etc/passwd
@end example

All generic string operations work for filenames:
@example
Q2> "xyz" 1 # Indexing
y
Q3> ./Makefile
Q.tar.Z:
	tar cf *|compress -c>Q.tar.Z
Q4> ./Makefile 1
.
Q6> size "xyz"
3
Q7> size ./Makefile
39
Q8> # Reversing Makefile:
Q9> format "%s\n" (./Makefile (38 downto 0))

Z.rat.Q>c- sserpmoc|* fc rat
:Z.rat.Q     
@end example

Filenames can be used for assignment:
@example
Q10> ./foo := ./Makefile
@end example

A filename that names a directory is viewed as an associative mapping
from names to other filenames.
@example
(./unix "Makefile") ==> ./unix/Makefile
size (.//etc "passwd")
==> # Length of /etc/passwd
@end example

(The Unix process environment is also
an associative mapping, from strings to strings.)

@node Commands, Pipeline implementation, Filenames, Shell-paper
@section Commands

The macro @code{run} is used to run a program:
@example
run ls -l ../lib/*.o
@end example
The macro @code{exec} is like @code{run}, but it does
an @code{execve} without a prior @code{fork}.

An implicit @code{run} is added before an undeclared
identifier that matches an executable in @code{PATH}:
@example
ls -l ../lib/*.o
@end example

@subsection Standard input and output

Q allows infix functions, with a varying number of right
parameters. The left argument (if any)
to a program is by convention standard input.

We can pass a string to a program:
@example
Q1> "Hello World!\n" run /usr/bin/tr a-zA-Z A-Za-z
hELLO wORLD!
@end example

The stdin argument need not be a string, but can be
any printable object:
@example
Q2> 250 for 5
250 251 252 253 254
Q3> (250 for 10) tr 0-9 1-90
361 362 363 364 365
@end example

Input can be a filename:
@example
Q4> ./Makefile wc
     199     848    6895
@end example

The output of a command can be saved as a string:
@example
Q5> :S=(ls -ld ../test)
Q6> format "(%#s)" S
("drwxrwxr-x  2 bothner       512 Jul 25 17:41 ../test\n")
@end example

The composition of two programs is a pipeline:
@example
Q7> ls -ld ../test) tr a-zA-Z A-Za-z
DRWXRWXR-X  2 BOTHNER       512 jUL 25 17:41 ../TEST
@end example
(Unbalanced parentheses are allowed here.)

@node Pipeline implementation, Control, Commands, Shell-paper
@section Pipeline implementation

It is important that the features just described be implemented
without extraneous @code{forks} or copying of temporary values.

(See the full paper for details.)

@node Control, Future work, Pipeline implementation, Shell-paper
@section Control

@subsection Output printing

The format directive @samp{%+s} prints the argument as suitable
for printing a result. A newline is included, if appropriate:
@example
(format "|%+s|%+s|" 3+4 "Hello") ==> "|7\n|Hello|"
@end example
A newline is @emph{not} added when printing a string
because running a program is viewed as calling a function that returns
a string.  Since shells do not print a newline after the
output from a program, we should not add a newline when printing a string.

@subsection True and false

Statements that are evaluated solely for their side-effects
(such as assignments and declarations)
should evaluate to "nothing", that is @code{""}.

A declaration in Q is just a unification
expression.  For consistency, other standard relations (such as
@code{2<3}) also evaluate to @code{""} if they succeed.
The default "true" value in Q is therefore @code{""}.

"Exception" values represent "false."
If a function argument is an exception, the function
returns the exception.  Exceptions are not normal first-class
values, and they use a condition-handling
mechanism using non-local jumps.

An "uncaught" exception is normally just printed:
@example
Q1> 2<3   # True, prints nothing.
Q2> 2>3   # False, raises exception.
Exception: Failed comparison (>).
@end example

Exceptions that represent failures (but not serious errors)
are caught using the @code{if} construct.
If a program returns a non-zero exit code, that raises
a failure exception:
@example
Q4> sh -c "exit 1"
Exception: Program sh (pid: 5292) failed with return code 1.
Q5> if sh -c "exit 1" => "sh ok\n" || "sh failed\n"
sh failed
@end example

@subsection Statement lists

Most languages support some kind of "statement list" form, where each
"statement" is evaluated in turn.
In Q, we define the effect of evaluating @code{@var{expr1}; @var{expr2}}
as the following:
First evaluate @var{expr1}, then evaluate @var{expr2}.
If either result is @code{""} (or @code{[]}),
then return the other result.
If neither is null, the result is the concatenation of the two values,
as if computed by @code{format "%+s%+s" @var{expr1} @var{expr2}}.

This arcane definition is chosen because it has the following
desirable properties:
@itemize @bullet
@item
The functional language [Haskell?] form
@example
let x = 3+4 in x*x # Not Q
@end example
(which evaluates to 49) is written:
@example
(:x=3+4; x*x)
@end example
@item
The functional language form
@example
x*x where x = 3+4 # Not Q
@end example
is written:
@example
(x*x; :x=3+4)
@end example
@item
The output from a statement
list is the concatenation of the output from
each sub-command. For example:
@example
:foo = (cat file1; cat file2)
@end example
makes @code{foo} a string of the concatenation of @code{file1}
and @code{file2}.
This is consistent with how shells work.
@item
Associativity: @code{(@var{a}; @var{b}); @var{c}} = @code{@var{a}; (@var{b}; @var{c})}
@item
@code{"";@var{a}} = @code{@var{a};""} = @code{@var{a}}
@item
@code{("string1"; "string2")} = @code{"string1string2"}
@end itemize

@node Future work, Summary, Control, Shell-paper
@section Future work

Job control is currently being worked on.

Make Q a practical (robust) default shell.

Add Garbage collection.

@node Summary,  , Future work, Shell-paper
@section Summary

(Full paper discusses here how Q fits Ousterhout's design goals for Tcl.)

It is practical to combine the features of a shell into
a full-fledged programming languages.
I hope I have shown that careful orthogonal design
can give a moderate-sized very powerful multi-purpose language.

Using C++ has simplified the implementation.

All the given examples work.

@bye
