\input texinfo   @c -*-texinfo-*-
@c
@c $Id: semantic.texi,v 1.6 2001/08/16 03:37:34 andyp Exp $
@c
@setfilename semantic.info
@settitle Semantic Bovinator: Parser generation for Emacs

@ifinfo
@format
START-INFO-DIR-ENTRY
* semantic: (semantic).       Semantic Parsing for Emacs
END-INFO-DIR-ENTRY
@end format
@end ifinfo

@titlepage
@sp 10
@center @titlefont{Semantic}
@vskip 0pt plus 1 fill
Copyright @copyright{} 1999, 2000, 2001 Eric M. Ludlam
@end titlepage

@node Top, Install, (dir), (dir)
@comment  node-name,  next,  previous,  up

Semantic is a program for Emacs which includes, at its core, a lexer,
and a compiler compiler (bovinator).  Additional tools include a
bnf->semantic table converter, example tables, and a speedbar tool.

The core utility is the ``semantic bovinator'' which has similar
behaviors as yacc or bison.  Since it is not designed to be as feature
rich as these tools, it uses the term ``bovine'' for cow, a lesser
cousin of the yak and bison.

In it's current state, the bovinator is both fast (because it uses Emacs
tricks) and slow (it is recursive, and not table (array) based).  If you
have experience with compiler compilers (bison, yacc, etc) your help to
make these updates would be greatly appreciated.


@menu
* Install::                     Installing semantic.
* Lexing::                      Setting up the lexer for your language.
* Bovinating::                  Setting up the parser for your language.
* BNF conversion::              Using the BNF converter to make tables.
* Compiling::                   Running the bovinator on a source file.
* Debugging::                   Debugging bovine tables
* Utilities::                   How to use the nonterminal stream.
* Current Context::             How to get the current code context.
* Tools::                       User Tools which use semantic.
* Index::                       
@end menu

@node Install, Lexing, Top, Top
@chapter Installation

To install semantic, untar the distribution into a subdirectory, such as
@file{/usr/share/emacs/site-lisp/semantic-#.#}.  Next, add the following
lines into your individual @file{.emacs} file, or into
@file{site-lisp/site-start.el}.

@example
(setq semantic-load-turn-everything-on t)
(load-file "/path/to/semantic/semantic-load.el")
@end example

If you would like to turn individual tools on or off in your init
file, skip the first line.

@node Lexing, Bovinating, Install, Top
@chapter Preparing your language for Lexing

In order to reduce a source file into a token list, it must first be
converted into a token stream.  Tokens are syntactic elements such as
whitespace, symbols, strings, lists, and punctuation.

The lexer uses the major-mode's syntax table for conversion.  As long
as that is set up correctly (along with the important
@code{comment-start} and @code{comment-start-skip} variable) the lexer
should already work for your language.

There are three ways to extend the lexer.

@defvar semantic-flex-extensions
Buffer local extensions to the lexical analyzer.
This should contain an alist with a key of a regex and a data element of
a function.  The function should both move point, and return a lexical
token of the form:

@example
( TYPE START . END)
@end example

@code{nil} is also a valid return.
@var{TYPE} can be any type of symbol, as long as it doesn't occur as a
nonterminal in the language definition.
@end defvar

@defvar semantic-flex-syntax-modifications
Updates to the syntax table for a given buffer.
These changes are active only while this file is being flexed.
This is a list where each element is of the form:
@example
(CHAR CLASS) 
@end example

Where CHAR is the char passed to `modify-syntax-entry',
and CLASS is the string also passed to `modify-syntax-entry' to define
what class of syntax CHAR is.

@example
(setq semantic-flex-syntax-modifications '((?. "_"))
@end example

Will convert the period @asis{.} to be a symbol constituant.  (ie, if
filenames are prevalent as in Makefiles.)
@end defvar

@defvar semantic-flex-enable-newlines
When flexing, report @code{'newlines} as syntactic elements.
Useful for languages where the newline is a special case terminator.
Only set this on a per mode basis, not globally.
@end defvar

@section Keywords

Another important piece of the lexer is the keyword table. @ref{Settings}.
You language will want to set up a keyword table for fast conversion of
symbol strings to language terminals.

The keywords table can also be used to store additional information
about those keywords.  The following programming functions can be useful
when examining text in a language buffer.

@defun semantic-flex-keyword-p text
Return a symbol if @var{TEXT} is a keyword in the keyword table.
@end defun

@defun semantic-flex-keyword-put text property value
For keyword @var{TEXT}, set @var{PROPERTY} to have @var{VALUE}.
@end defun

@defun semantic-flex-keyword-get text property
For the keyword found as @var{text}, get the value of @var{property}.
@end defun

@defun semantic-flex-map-keywords fun &optional property
Call function @var{FUN} on every semantic keyword.
If optional @var{PROPERTY} is non-nil call @var{FUN} only on every
keyword which have a @var{PROPERTY} value.
@var{FUN} receives a semantic keyword as argument.
@end defun

@defun semantic-flex-keywords &optional property
Return a list of semantic keywords.
If optional @var{PROPERTY} is non-nil return only keyword which have a
PROPERTY value.
@end defun

Keyword properties can be set up in a BNF file for ease of maintenance.
While examining the text in a language buffer, this can provide an easy
and quick way of storing details about text in the buffer.

@section Standard Keyword Properties

Add known properties here when they are known.

@node Bovinating, BNF conversion, Lexing, Top
@chapter Preparing a bovine table for your language

When converting a source file into a nonterminal token stream
(parse-tree) it is important to specify rules to accomplish this.  The
rules are stored in the buffer local variable
@code{semantic-toplevel-bovine-table}.

While it is certainly possible to write this table yourself, it is most
likely you will want to use the BNF converter.  @xref{BNF conversion}.
This is an easier method for specifying your rules.  You will still need
to specify a variable in your language for the table, however.  A good
rule of thumb is to call it @code{language-toplevel-bovine-table} if it
part of the language, or @code{semantic-toplevel-language-bovine-table}
if you donate it to the semantic package.

When initializing a major-mode for your language, you will set the
variable @code{semantic-toplevel-bovine-table} to the contents of your
language table.  @code{semantic-toplevel-bovine-table} is always buffer
local.

Since it is important to know the format of the table when debugging ,
you should still attempt to understand the basics of the table.

Please see the documentation for the variable
@code{semantic-toplevel-bovine-table} for details on it's format.

* add more doc here *

@node BNF conversion, Compiling, Bovinating, Top
@chapter Using the BNF converter to make bovine tables

The BNF converter takes a file in "Bovine Normal Form" which is similar
to "Backus-Naur Form".  If you have ever used yacc or bison, you will
find it similar.  The BNF form used by semantic, however, does not
include token precedence rules, and several other features needed to make
real parser generators.

It is important to have an Emacs Lisp file with a variable ready to take
the output of your table @xref{Bovinating}.  Also, make sure that the
file @file{semantic-bnf.el} is loaded.  Give your language file the
extension @file{.bnf} and you are ready.

The comment character is @asis{#}.

When you want to test your file, use the keyboard shortcut @kbd{C-c C-c}
to parse the file, generate the variable, and load the new definition
in.  It will then use the settings specified above to determine what to
do.  Use the shortcut @kbd{C-c c} to do the same thing, but spend
extra time indenting the table nicely.

Make sure that you create the variable specified in the
@code{%parsetable} token before trying to convert the bnf file.  A
simple definition like this is sufficient.

@example
(defvar semantic-toplevel-lang-bovine-table
   nil
   "Table for use with semantic for parsing LANG.")
@end example

If you use tokens (created with the @code{%token} specifier), also
make sure you have a keyword table available, like this:

@example
(defvar semantic-lang-keyword-table
   nil
   "Table for use with semantic for keywords.")
@end example

Specify the name of the keyword table with the @code{%keywordtable}
specifier.

The BNF file has two sections.  The first is the settings section, and
the second is the language definition, or list of semantic rules.

@menu
* Settings::                    
* Rules::                       
* Optional Lambda Expression::  
* Examples::                    
@end menu

@node Settings, Rules, BNF conversion, BNF conversion
@section Settings

A setting is a keyword starting with a @asis{%}.  (This syntax is taken
from yacc and bison. @xref{(bison)}.)

There are several settings that can be made in the settings section.
They are:

@table @code
@item %start <nonterminal>
Specify an alternative to @code{bovine-toplevel}.  (See below)
@item %scopestart <nonterminal>
Specify an alternitive to @code{bovine-inner-scope}.
@item %outputfile <filename>
Required.  Specifies the file into which this files output is stored.
@item %parsetable <lisp-variable-name>
Required.  Specifies a lisp variable into which the output is stored.
@item %keywordtable <lisp-variable-name>
Required if there are @code{%token} keywords.
Specifies a lisp variable into which the output of a keyword table is
stored.  This obarray is used to turn symbols into keywords when applicable.
@item %token <name> "<text>"
Optional.  Specify a new token @var{NAME}.  This is added to a lexical
keyword list using @var{TEXT}.  The symbol is then converted into a new
lexical terminal.  This requires that the @code{%keywordtable} specified
variable is available in the file specified by @code{%outputfile}.
@item %token <name> type "<text>"
Optional.  Specify a new token @var{NAME}.  It is made from an existing
lexical token of type @var{TYPE}.  @var{TEXT} is a string which will be
matched explicitly.  @var{NAME} can be used in match rules as though they were
flex tokens, but are converted back to @var{TYPE} "text" internally.
@item %put <NAME> symbol <VALUE>
@itemx %put <NAME> ( symbol1 <VALUE1> symbol2 <VALUE2> ... )
@itemx %put ( <NAME1> <NAME2>...) symbol <VALUE>
Tokens created without a type are considered keywords, and placed in a
keyword table.  Use @code{%put} to apply properties to that keyword.
@ref{Lexing}.
@item %languagemode <lisp-function-name>
@itemx %languagemode ( <lisp-function-name1> <lisp-function-name2> ... )
Optional.  Specifies the Emacs major mode associated with the language
being specified.  When the language is converted, all buffers of this
mode will get the new table installed.
@item %quotemode backquote
Optional.  Specifies how symbol quoting is handled in the Optional
Lambda Expressions. (See below)
@item %setupfunction <lisp-function-name>
Name of a function into which setup code is to be inserted.
@item %( <lisp-expression> )%
Specify setup code to be inserted into the @code{%setupfunction}.
It will be inserted between two specifier strings, or added to
the end of the function.
@end table

When working inside @code{%( ... )%} tokens, any lisp expression can be
entered which will be placed inside the setup function.  In general, you
probably want to set variables that tell Semantic and related tools how
the language works.

Here are some variables that control how different programs will work
with your language.

@defvar semantic-flex-depth
Default flexing depth.
This specifies how many lists to create tokens in.
@end defvar

@defvar semantic-flex-extensions
Buffer local extensions to the lexical analyzer.
This should contain an alist with a key of a regex and a data element of
a function.  The function should both move point, and return a lexical
token of the form:
@example
( TYPE START . END)
@end example

@code{nil} is also a valid return.
@var{TYPE} can be any type of symbol, as long as it doesn't occur as a
nonterminal in the language definition.
@end defvar

@defvar semantic-flex-syntax-modifications
Updates to the syntax table for this buffer.
These changes are active only while this file is being flexed.
This is a list where each element is of the form:
@example
(CHAR CLASS)
@end example
Where @var{CHAR} is the char passed to @dfn{modify-syntax-entry},
and @var{CLASS} is the string also passed to @dfn{modify-syntax-entry} to define
what class of syntax @var{CHAR} is.
@end defvar

@defvar semantic-flex-enable-newlines
When flexing, report @code{'newlines} as syntactic elements.
Useful for languages where the newline is a special case terminator.
Only set this on a per mode basis, not globally.
@end defvar

@defvar semantic-ignore-comments
Default comment handling.
@code{t} means to strip comments when flexing.  @code{Nil} means to keep comments
as part of the token stream.
@end defvar

@defvar semantic-symbol->name-assoc-list
Association between symbols returned, and a string.
The string is used to represent a group of objects of the given type.
It is sometimes useful for a language to use a different string
in place of the default, even though that language will still
return a symbol.  For example, Java return's includes, but the
string can be replaced with @code{Imports}.
@end defvar

@defvar semantic-case-fold
Value for @code{case-fold-search} when parsing.
@end defvar

@defvar semantic-expand-nonterminal
Function to call for each nonterminal production.
Return a list of non-terminals derived from the first argument, or @code{nil}
if it does not need to be expanded.
Languages with compound definitions should use this function to expand
from one compound symbol into several.  For example, in @var{C} the
definition
@example
int a, b;
@end example
is easily parsed into one token, but represents multiple variables.  A
functions should be written which takes this compound token and turns
it into two tokens, one for @var{A}, and the other for @var{B}.

Within the language definition (the @file{.bnf} sources), it is often
useful to set the NAME slot of a token with a list of items that
distinguish each element in the compound definition.

This list can then be detected by the function set in
@code{semantic-expand-nonterminal} to create multiple tokens.
This function has one additional duty of managing the overlays created
by semantic.  It is possible to use the single overlay in the compound
token for all your tokens, but this can pose problems identifying
all tokens covering a given definition.

Please see @file{semantic-java.el} for an example of managing overlays
when expanding a token into multiple definitions.
@end defvar

@defvar semantic-override-table
Buffer local semantic function overrides alist.
These overrides provide a hook for a `major-mode' to override specific
behaviors with respect to generated semantic toplevel nonterminals and
things that these non-terminals are useful for.
Each element must be of the form: (SYM . FUN)
where SYM is the symbol to override, and FUN is the function to
override it with.

Available override symbols:

@multitable {@code{abbreviate-nonterminal}} {(token & nosnarf)} {Find token in buffer}
@item SYBMOL                 @tab PARAMETERS        @tab DESCRIPTION
@item find-dependency        @tab (token)           @tab Find the dependency file
@item find-nonterminal       @tab (token & parent)  @tab Find token in buffer.
@item find-documentation     @tab (token & nosnarf) @tab Find doc comments.
@item abbreviate-nonterminal @tab (token & parent)  @tab Return summery string.
@item summarize-nonterminal  @tab (token & parent)  @tab Return summery string.
@item prototype-nonterminal  @tab (token)           @tab Return a prototype string.
@item prototype-file         @tab (buffer)          @tab Return a file in which prototypes are placed
@item nonterminal-children   @tab (token)           @tab Return first rate children. These are children which may contain overlays.
@item nonterminal-protection @tab (token & parent)  @tab Return protection as a symbol.
@item beginning-of-context   @tab (& point)         @tab Move to the beginning of the
					                 current context.
@item end-of-context         @tab (& point)         @tab Move to the end of the
					                 current context.
@item up-context             @tab (& point)         @tab Move up one context level.
@item get-local-variables    @tab (& point)         @tab Get local variables.
@item get-all-local-variables@tab (& point)         @tab Get all local variables.
@item get-local-arguments    @tab (& point)         @tab Get arguments to this function.

@item end-of-command         @tab                   @tab Move to the end of the current
                                                         command
@item beginning-of-command   @tab                   @tab Move to the beginning of the
                                                         current command
@item ctxt-current-symbol    @tab (& point)         @tab List of related symbols.
@item ctxt-current-assignment@tab (& point)         @tab Variable being assigned to.
@item ctxt-current-function  @tab (& point)         @tab Function being called at point.
@item ctxt-current-argument  @tab (& point)         @tab The index to the argument of
                                                         the current function the cursor
                                                         is in.
@end multitable
                                                   
Parameters mean:

@table @code
@item &
Following parameters are optional
@item buffer
The buffer in which a token was found.
@item token
The nonterminal token we are doing stuff with
@item parent
If a TOKEN is stripped (of positional infomration) then this will be the
parent token which should have positional information in it.
@end table

@end defvar

@defvar semantic-type-relation-separator-character
Character strings used to separation a parent/child relationship.
This list of strings are used for displaying or finding separators
in variable field dereferencing.  The first character will be used for
display.  In @var{C}, a type field is separated like this: ``type.field''
thus, the character is a ``.''.  In @var{C}, and additional value of ``->''
would be in the list, so that ``type->field'' could be found.
@end defvar


@defvar semantic-dependency-include-path
Defines the include path used when searching for files.
This should be a list of directories to search which is specific to
the file being included.
This variable can also be set to a single function.  If it is a
function, it will be called with one arguments, the file to find as a
string, and  it should return the full path to that file, or nil.
@end defvar

This configures Imenu to use semantic parsing.

@defvar imenu-create-index-function
The function to use for creating a buffer index.

It should be a function that takes no arguments and returns an index
of the current buffer as an alist.

Simple elements in the alist look like @samp{(INDEX-NAME . INDEX-POSITION)}.
Special elements look like @samp{(INDEX-NAME INDEX-POSITION FUNCTION ARGUMENTS...)}.
A nested sub-alist element looks like (INDEX-NAME SUB-ALIST).
The function @code{imenu--subalist-p} tests an element and returns t
if it is a sub-alist.

This function is called within a @code{save-excursion}.

The variable is buffer-local.
@end defvar


These are specific to the document tool.

@table @code
@item document-comment-start
Comment start string.
@item document-comment-line-prefix
Comment prefix string.  Used at the beginning of each line.
@item document-comment-end
Comment end string.
@end table

@node Rules, Optional Lambda Expression, Settings, BNF conversion
@section Rules

Writing the rules should be very similar to bison for basic syntax.
Each rule is of the form

@example
RESULT : MATCH1 (optional-lambda-expression)
       | MATCH2 (optional-lambda-expression)
       ;
@end example

@var{RESULT} is a non-terminal, or a token synthisized in your grammar.
@var{MATCH} is a list of elements that are to be matched if @var{RESULT}
is to be made.  The optional lambda expression is a list containing
simplified rules for concocting the parse tree.

In bison, each time an element of a @var{MATCH} is found, it is
"shifted" onto the parser stack.  (The stack of matched elements.)  When
all of @var{MATCH1}'s elements have been matched, it is "reduced" to
@var{RESULT}.  @xref{(bison)Algorithm}.

The first @var{RESULT} written into your language specification should
be @code{bovine-toplevel}, or the symbol specified with @code{%start}.
When starting a parse for a file, this is the default token iterated
over.  You can use any token you want in place of @code{bovine-toplevel}
if you specify what that nonterminal will be with a @code{%start} token
in the settings section.

@var{MATCH} is made up of symbols and strings.  A symbol such as
@code{foo} means that a syntactic token of type @code{foo} must be
matched.  A string in the mix means that the previous symbol must have
the additional constraint of exactly matching it.  Thus, the
combination:

@example
symbol "moose"
@end example

means that a symbol must first be encountered, and then it must
@code{string-match "moose"}.  Be especially careful to remember that the
string is a regular expression.  The code:

@example
punctuation "."
@end example

will match any punctuation.

For the above example in bison, a LEX rule would be used to create a new
token @var{MOOSE}.  In this case, the @var{MOOSE} token would appear.
For the bovinator, this task was mixed into the language definition to
simplify implementation, though Bison's technique is more efficient.

To make a symbol match explicity for keywords, for example, you can use
the @code{%token} command in the settings section to create new symbols.

@example
%token MOOSE "moose"

find_a_moose: MOOSE
            ;
@end example

will match ``moose'' explicitly, unlike the previous example where moose
need only appear in the symbol.  This is because ``moose'' will be
converted to @var{MOOSE} in the lexical analysis stage.  Thus the symbol
@var{MOOSE} won't be available any other way.

If we specify our token in this way:

@example
%token MOOSE symbol "moose"

find_a_moose: MOOSE
            ;
@end example

then @code{MOOSE} will match the string "moose" explicitly, but it won't
do so at the lexical level, allowing use of the text "moose" in other
forms of regular expressions.

Non symbol tokens are also alowed.  For example:

@example
%token PERIOD punctuation "."

filename : symbol PERIOD symbol
         ;
@end example

will explicitly match one period when used in the above rule.

@node Optional Lambda Expression, Examples, Rules, BNF conversion
@section Optional Lambda Expressions

The OLE (Optional Lambda Expression) is converted into a bovine lambda
@xref{Bovinating}. This lambda has special short-cuts to simplify
reading the emacs bnf definition.  An OLE like this:

@example
( $1 )
@end example

results in a lambda return which consists entirely of the string
or object found by matching the first (zeroeth) element of match.
An OLE like this:

@example
( ,(foo $1) )
@end example

executes `foo' on the first argument, and then splices it's return
into the return list whereas:

@example
( (foo $1) )
@end example

executes foo, and that is placed in the return list.

Here are other things that can appear inline:
@table @code
@item $1
the first object matched.
@item ,$1
the first object spliced into the list (assuming it is a list from a
non-terminal)
@item '$1
the first object matched, placed in a list.  ie ( $1 )
@item foo
the symbol foo (exactly as displayed)
@item (foo)
a function call to foo which is stuck into the return list.
@item ,(foo)
a function call to foo which is spliced into the return list.
@item '(foo)
a function call to foo which is stuck into the return list in a list.
@item (EXPAND $1 nonterminal depth)
a list starting with EXPAND performs a recursive parse on the token
passed to it (represented by $1 above.)  The semantic list is a common
token to expand, as there are often interesting things in the list.
The @var{nonterminal} is a symbol in your table which the bovinator will
start with when parsing.  @var{nonterminal}'s definition is the same as
any other nonterminal.  @var{depth} should be at least 1 when
descending into a semantic list.
@item (EXPANDFULL $1 nonterminal depth)
is like EXPAND, except that the parser will iterate over
@var{nonterminal} until there are no more matches.  (The same way the
parser iterates over @code{bovine-toplevel}. This lets you have
much simpler rules in this specific case, and also lets you have
positional information in the returned tokens, and error skipping.
@item (ASSOC symbol1 value1 symbol2 value2 ... )
This is used for creating an association list.  Each @var{SYMBOL} is
included in the list if the associated @var{VALUE} is non-nil.  While
the items are all listed explicitly, the created structure is an
association list of the form:
@example
( ( symbol1 . value1) (symbol2 . value2) ... )
@end example
@end table

If the symbol @code{%quotemode backquote} is specified, then use
@code{,@@} to splice a list in, and @code{,} to evaluate the expression.
This lets you send @code{$1} as a symbol into a list instead of having
it expanded inline.

@node Examples,  , Optional Lambda Expression, BNF conversion
@section Examples

The rule:

@example
SYMBOL : symbol
@end example

is equivalent to

@example
SYMBOL : symbol
         ( $1 )
@end example

which, if it matched the string "A", would return

@example
( "A" )
@end example

If this rule were used like this:

@example
ASSIGN: SYMBOL punctuation "=" SYMBOL
        ( $1 $# )
@end example

it would match "A=B", and return

@example
( ("A") ("B") )
@end example

The letters A and B come back in lists because SYMBOL is a nonterminal,
not an actual lexical element.

to get a better result with nonterminals, use @asis{,} to splice lists
in like this;

@example
ASSIGN: SYMBOL punctuation "=" SYMBOL
        ( ,$1 ,$3 )
@end example

which would return

@example
( "A" "B" )
@end example

@node Compiling, Debugging, BNF conversion, Top
@chapter Compiling a language file with the bovinator

From a program you can use the function @code{semantic-bovinate-toplevel}.
This function takes one optional parameter specifying if the cache
should be refreshed.  By default, the cached results of the last parse
are always used.  Specifying that the cache should be checked will cause
it to be flushed if it is out of date.

Another function you can use is @code{semantic-bovinate-nonterminal}.
This command takes a token stream returned by the function
@code{semantic-flex} followed by a DEPTH (as above).  This takes an
additional optional argument of NONTERMINAL which is the nonterminal in
your table it is to start parsing with.

@deffn Command bovinate &optional clear
Bovinate the current buffer.  Show output in a temp buffer.
Optional argument @var{CLEAR} will clear the cache before bovinating.
@end deffn

@deffn Command semantic-clear-toplevel-cache
Clear the toplevel bovin cache for the current buffer.
Clearing the cache will force a complete reparse next time a token
stream is requested.
@end deffn

@defun semantic-bovinate-toplevel &optional checkcache
Bovinate the entire current buffer.
If the optional argument @var{CHECKCACHE} is non-@code{nil}, then flush the cache iff
there has been a size change.
@end defun

@node Debugging, Utilities, Compiling, Top
@comment  node-name,  next,  previous,  up
@chapter Debugging

Writing language files using BNF is significantly easier than writing
then using regular expressions in a functional manner.  Debugging
them, however, can still prove challenging.

There are two ways to debug a language definition if it is not
behaving as expected.  One way is to debug against the source @file{.bnf}
file.  The second is to debug against the lisp table created from the
@file{.bnf} source, or perhaps written by hand.

If your language definition was written in BNF notation, debugging is
quite easy.  The command @code{bovinate-debug} will start you off.

@deffn Command bovinate-debug
Bovinate the current buffer and run in debug mode.
@end deffn

If you prefer debugging against the Lisp table, find the table in a
buffer, place the cursor in it, and use the command
@code{semantic-bovinate-debug-set-table} in it.

@deffn Command semantic-bovinate-debug-set-table
Set the table for the next debug to be here.
@end deffn

After the table is set, the @code{bovinate-debug} command can be run
at any time for the given language.

While debugging, two windows are visible.  One window shows the file
being parse, and the syntactic token being tested is highlighted.  The
second window shows the table being used (either in the BNF source, or
the Lisp table.) with the current rule highlighted.  The cursor will
sit on the specific match rule being tested against.

In the minibuffer, a brief summary of the current situation is
listed.  The first element is the syntactic token which is a list of
the form:

@example
(TYPE START . END)
@end example

The rest of the display is a list of all strings collected for the
currently tested rule.  Each time a new rule is entered, the list is
restarted.  Upon returning from a rule into a previous match list, the
previous match list is restored, with the production of the dependent
rule in the list.

Use @kbd{C-g} to stop debugging.  There are no commands for any
fancier types of debugging.

@node Utilities, Current Context, Debugging, Top
@comment  node-name,  next,  previous,  up
@chapter Utilities

Once a source file has been parsed, the follwing APIs can be used to
write programs that use the token stream most effectivly.

@menu
* Token Queries::               Getting info about a parsed token (nonterminal).
* Nonterminal Streams::         Working with streams of nonterminals
* Nonterminal Completion::      Completing read functions.
* Override Methods::            Language dependent functions.
* Parser Hooks::                How to know when tags change.
@end menu

@node Token Queries, Nonterminal Streams, Utilities, Utilities
@comment  node-name,  next,  previous,  up
@section Token Queries

When writing programs that use the bovinator, the following functions
are needed to find get details out of a nonterminal.

@defun semantic-token-token token
Retrieve from @var{TOKEN} the token identifier.
ie, the symbol @code{'variable}, @code{'function}, @code{'type}, or other.
@end defun

@defun semantic-token-name token
Retrieve the name of @var{TOKEN}.
@end defun

@defun semantic-token-docstring token &optional buffer
Retrieve the documentation of @var{TOKEN}.
Optional argument @var{BUFFER} indicates where to get the text from.
If not provided, then only the @var{POSITION} can be provided.
@end defun

@defun semantic-token-overlay token
Retrieve the @var{OVERLAY} part of @var{TOKEN}.
The returned item may be an overlay or an unloaded buffer representation.
@end defun

@defun semantic-token-extent token
Retrieve the extent (@var{START} @var{END}) of @var{TOKEN}.
@end defun

@defun semantic-token-start token
Retrieve the start location of @var{TOKEN}.
@end defun

@defun semantic-token-end token
Retrieve the end location of @var{TOKEN}.
@end defun

@defun semantic-token-type token
Retrieve the type of @var{TOKEN}.
@end defun

@defun semantic-token-put token property value
On @var{token}, set @var{property} to @var{value}.
@end defun

@defun semantic-token-get token property
For @var{token} get the value of @var{property}.
@end defun

@defun semantic-token-extra-spec token spec
Retrieve a specifier for the variable @var{TOKEN}.
@var{SPC} is the symbol whose modifier value to get.
This function can get specifiers from any type of @var{TOKEN}.
Do not use this function if you know what type of token you are dereferencing.
Instead, use the function specific to that token type.  It will be faster.
@end defun

@defun semantic-token-type-parts token
Retrieve the parts of the type @var{TOKEN}.
@end defun

@defun semantic-token-type-parent token
Retrieve the parent of the type @var{TOKEN}.
The return value is a list.  @var{A} value of @code{nil} means no parents.
The @dfn{car} of the list is either the parent class, or a list
of parent classes.  The @dfn{cdr} of the list is the list of
interfaces, or abstract classes which are parents of @var{TOKEN}.
@end defun

@defun semantic-token-type-parent-superclass token
Retrieve the parent superclasses of type type @var{TOKEN}.
@end defun

@defun semantic-token-type-parent-implement token
Retrieve the parent interfaces of type type @var{TOKEN}.
@end defun

@defun semantic-token-type-modifiers token
Retrieve the type modifiers for the type @var{TOKEN}.
@end defun

@defun semantic-token-type-extra-specs token
Retrieve the extra specifiers for the type @var{TOKEN}.
@end defun

@defun semantic-token-type-extra-spec token spec
Retrieve a extra specifier for the type @var{TOKEN}.
@var{SPEC} is the symbol whose modifier value to get.
@end defun

@defun semantic-token-function-args token
Retrieve the arguments of the function @var{TOKEN}.
@end defun

@defun semantic-token-function-modifiers token
Retrieve the type modifiers of the function @var{TOKEN}.
@end defun

@defun semantic-token-function-extra-specs token
Retrieve the extra specifiers of the function @var{TOKEN}.
@end defun

@defun semantic-token-function-extra-spec token spec
Retrieve a specifier for the function @var{TOKEN}.
@var{SPEC} is a symbol whose specifier value to get.
@end defun

@defun semantic-token-function-throws token
Retrieve the throws signal of the function @var{TOKEN}.
This is an optional field, and returns @code{nil} if it doesn't exist.
@end defun

@defun semantic-token-function-parent token
The parent of the function @var{TOKEN}.
A function has a parent if it is a method of a class, and if the
function does not appear in body of it's parent class.
@end defun

@defun semantic-token-variable-const token
Retrieve the status of constantness from the variable @var{TOKEN}.
@end defun

@defun semantic-token-variable-default token
Retrieve the default value of the variable @var{TOKEN}.
@end defun

@defun semantic-token-variable-modifiers token
Retrieve type modifiers for the variable @var{TOKEN}.
@end defun

@defun semantic-token-variable-extra-specs token
Retrieve extra specifiers for the variable @var{TOKEN}.
@end defun

@defun semantic-token-variable-extra-spec token spec
Retrieve a specifier value for the variable @var{TOKEN}.
@var{SPEC} is the symbol whose specifier value to get.
@end defun

@defun semantic-token-include-system token
Retrieve the flag indicating if the include @var{TOKEN} is a sysmtem include.
@end defun

For override methods that query a token, see @xref{Token Details}. 

@node Nonterminal Streams, Nonterminal Completion, Token Queries, Utilities
@section Nonterminal streams

These functions take some key, and returns information found inside the
nonterminal stream returned by the bovinator.  Some will return one
token (the first matching item found.)  Others will return a list of all
items matching a given criterion.

These functions query the current buffer's overlay system for tokens.

@defun semantic-find-nonterminal-by-overlay &optional positionormarker buffer
Find all nonterminals covering @var{POSITIONORMARKER} by using overlays.
If @var{POSITIONORMARKER} is @code{nil}, use the current point.
Optional @var{BUFFER} is used if @var{POSITIONORMARKER} is a number, otherwise the current
buffer is used.  This finds all tokens covering the specified position
by checking for all overlays covering the current spot.  They are then sorted
from largest to smallest via the start location.
@end defun

@defun semantic-find-nonterminal-by-overlay-in-region start end &optional buffer
Find all nonterminals which exist in whole or in part between @var{START} and @var{END}.
Uses overlays to determine positin.
Optional @var{BUFFER} argument specifies the buffer to use.
@end defun

@defun semantic-current-nonterminal
Return the current nonterminal in the current buffer.
If there are more than one in the same location, return the
smallest token.
@end defun

These functions search the entire stream for tokens matches a specific
criteria.

@defun semantic-find-nonterminal-by-name name streamorbuffer &optional search-parts search-include
Find a nonterminal @var{NAME} within @var{STREAMORBUFFER}.  @var{NAME} is a string.
If @var{SEARCH-PARTS} is non-@code{nil}, search children of tokens.
If @var{SEARCH-INCLUDE} is non-@code{nil}, search include files.
@end defun

@defun semantic-find-nonterminal-by-property property value streamorbuffer &optional search-parts search-includes
Find all nonterminals with @var{PROPERTY} equal to @var{VALUE} in @var{STREAMORBUFFER}.
Properties can be added with @dfn{semantic-token-put}.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDES} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-extra-spec spec streamorbuffer &optional search-parts search-includes
Find all nonterminals with a given @var{SPEC} in @var{STREAMORBUFFER}.
@var{SPEC} is a symbol key into the modifiers association list.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDES} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-extra-spec-value spec value streamorbuffer &optional search-parts search-includes
Find all nonterminals with a given @var{SPEC} equal to @var{VALUE} in @var{STREAMORBUFFER}.
@var{SPEC} is a symbol key into the modifiers association list.
@var{VALUE} is the value that @var{SPEC} should match.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDES} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-position position streamorbuffer &optional nomedian
Find a nonterminal covering @var{POSITION} within @var{STREAMORBUFFER}.
@var{POSITION} is a number, or marker.  If @var{NOMEDIAN} is non-@code{nil}, don't do
the median calculation, and return nil.
@end defun

@defun semantic-find-innermost-nonterminal-by-position position streamorbuffer &optional nomedian
Find a list of nonterminals covering @var{POSITION} within @var{STREAMORBUFFER}.
@var{POSITION} is a number, or marker.  If @var{NOMEDIAN} is non-@code{nil}, don't do
the median calculation, and return nil.
This function will find the topmost item, and recurse until no more
details are available of findable.
@end defun

@defun semantic-find-nonterminal-by-token token streamorbuffer &optional search-parts search-includes
Find all nonterminals with a token @var{TOKEN} within @var{STREAMORBUFFER}.
@var{TOKEN} is a symbol representing the type of the tokens to find.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDE} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-standard streamorbuffer &optional search-parts search-includes
Find all nonterminals in @var{STREAMORBUFFER} which define simple token types.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDE} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-type type streamorbuffer &optional search-parts search-includes
Find all nonterminals with type @var{TYPE} within @var{STREAMORBUFFER}.
@var{TYPE} is a string which is the name of the type of the token returned.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDES} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-function function streamorbuffer &optional search-parts search-includes
Find all nonterminals in which @var{FUNCTION} match within @var{STREAMORBUFFER}.
@var{FUNCTION} must return non-@code{nil} if an element of @var{STREAM} will be included
in the new list.

If optional argument @var{SEARCH-PARTS} is non-@code{nil}, all sub-parts of tokens
are searched.  The overloadable function @dfn{semantic-nonterminal-children} is
used for the searching child lists.  If @var{SEARCH-PARTS} is the symbol
@code{'positiononly}, then only children that have positional information are
searched.

If @var{SEARCH-INCLUDES} is non-@code{nil}, then all include files are also
searched for matches.
@end defun

@defun semantic-find-nonterminal-by-function-first-match function streamorbuffer &optional search-parts search-includes
Find the first nonterminal which @var{FUNCTION} match within @var{STREAMORBUFFER}.
@var{FUNCTION} must return non-@code{nil} if an element of @var{STREAM} will be included
in the new list.
If optional argument @var{SEARCH-PARTS}, all sub-parts of tokens are searched.
The overloadable function @dfn{semantic-nonterminal-children} is used for
searching.
If @var{SEARCH-INCLUDES} is non-@code{nil}, then all include files are also
searched for matches.
@end defun

@defun semantic-recursive-find-nonterminal-by-name name buffer
Recursivly find the first occurance of @var{NAME}.
Start search with @var{BUFFER}.  Recurse through all dependencies till found.
The return item is of the form (@var{BUFFER} @var{TOKEN}) where @var{BUFFER} is the buffer
in which @var{TOKEN} (the token found to match @var{NAME}) was found.
@end defun


@node Nonterminal Completion, Override Methods, Nonterminal Streams, Utilities
@section Nonterminal completion

These functions provide ways reading the names of items in a buffer with
completion.


@defun semantic-read-symbol prompt &optional default stream filter
Read a symbol name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@var{FILTER} is provides a filter on the types of things to complete.
@var{FILTER} must be a function to call on each element.  (See
@end defun

@defun semantic-read-variable prompt &optional default stream
Read a variable name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@end defun

@defun semantic-read-function prompt &optional default stream
Read a function name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@end defun

@defun semantic-read-type prompt &optional default stream
Read a type name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@end defun


@node Override Methods, Parser Hooks, Nonterminal Completion, Utilities
@section Override Methods

These functions are called `override methods' because they provide
generic behaviors, which a given language can override.  For example,
finding a dependency file in Emacs lisp can be done with the
`locate-library' command (which overrides the default behavior.)  In C,
a dependency can be found by searching a generic search path which can
be passed in via a variable.

@menu
* Token->Text:: Converting Tokens into text strings
* Token Details:: Arbitrary token detail fetching
* Local Context:: Deriving information about a language specific local
                  context.
* Making New Methods:: How to add your own methods for a tool
@end menu

@node Token->Text, Token Details, Override Methods, Override Methods
@subsection Token->Text

Any given token consists of Meta information which is best viewed in
some textual form.  This could be as simple as the token's name, or as
a prototype to be added to header file in C.  Not only are there
several default converters from a Token into text, but there is also
some convenient variables that can be used with them.  Use these
variables to allow options on output forms when displaying tokens in
your programs.

@defvar semantic-token->text-functions
List of functions which convert a token to text.
Each function must take the parameters @var{TOKEN} &optional @var{PARENT} @var{COLOR}.
@var{TOKEN} is the token to convert.
@var{PARENT} is a parent token or name which refers to the structure
or class which contains @var{TOKEN}.  @var{PARENT} is @var{NOT} a class which a @var{TOKEN}
would claim as a parent.
@var{COLOR} indicates that the generated text should be colored using
@code{font-lock}.
@end defvar

@defvar semantic-token->text-custom-list
@var{A} List used by customizeable variables to choose a token to text function.
Use this variable in the @code{:type} field of a customizable variable.
@end defvar

Every token to text conversion function must take the same parameters,
which are @var{TOKEN}, the token to be converted, @var{PARENT}, the
containing parent (like a structure which contains a variable), and
@var{COLOR}, which is a flag specifying that color should be applied
to the returned string.

When creating, or using these strings, particularly with color, use
@dfn{concat} to build up larger strings instead of @dfn{format}.  This
will preserve text properties.

@defun semantic-name-nonterminal token &optional parent color
Return the name string describing @var{TOKEN}.
The name is the shortest possible representation.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-summarize-nonterminal token &optional parent color
Summarize @var{TOKEN} in a reasonable way.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-prototype-nonterminal token &optional parent color
Return a prototype for @var{TOKEN}.
This function should be overloaded, though it need not be used.
This is because it can be used to create code by language independent
tools.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-prototype-file buffer
Return a file in which prototypes belonging to @var{BUFFER} should be placed.
Default behavior (if not overriden) looks for a token specifying the
prototype file, or the existence of an @var{EDE} variable indicating which
file prototypes belong in.
@end defun

@defun semantic-abbreviate-nonterminal token &optional parent color
Return an abbreviated string describing @var{TOKEN}.
The abbreviation is to be short, with possible symbols indicating
the type of token, or other information.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-concise-prototype-nonterminal token &optional parent color
Return a concise prototype for @var{TOKEN}.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-uml-abbreviate-nonterminal token &optional parent color
Return a @var{UML} style abbreviation for @var{TOKEN}.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@node Token Details, Local Context, Token->Text, Override Methods
@subsection Token Details

These functions help derive information about tokens that may not
be obvious for non-traditional languages with their own token types.

@defun semantic-nonterminal-children token &optional positionalonly
Return the list of top level children belonging to @var{TOKEN}.
Children are any sub-tokens which may contain overlays.
The default behavior (if not overriden with @code{nonterminal-children}
is to return type parts for a type, and arguments for a function.

If optional argument @var{POSITIONALONLY} is non-@code{nil}, then only return valid
children if they contain positions.  Some languages may choose to create
lists of children without position/overlay information.

If this function is overriden, use @dfn{semantic-nonterminal-children-default}
to also include the default behavior, and merely extend your own.

Note for language authors:
  If a mode defines a language that has tokens in it with overlays that
should not be considered children, you should still return them with
this function.  If you do not, then token reparsing, and database
saving will fail.
@end defun

@defun semantic-nonterminal-protection token &optional parent
Return protection information about @var{TOKEN} with optional @var{PARENT}.
This function returns on of the following symbols:
   @code{nil}      - No special protection.  Language dependent.
   @code{'public}  - Anyone can access this @var{TOKEN}.
   @code{'private} - Only methods in the local scope can access @var{TOKEN}.
   @code{'friend}  - Like private, except some outer scopes are allowed
              access to token.
Some languages may choose to provide additional return symbols specific
to themselves.  Use of this function should allow for this.

The default behavior (if not overriden with @code{nonterminal-children}
is to return a symbol based on type modifiers.
@end defun

@defun semantic-find-dependency token
Find the filename represented from @var{TOKEN}.
@var{TOKEN} may be a stripped element, in which case @var{PARENT} specifies a
parent token that has positinal information.
Depends on @code{semantic-dependency-include-path} for searching.  Always searches
`.' first, then searches additional paths.


@defvar semantic-dependency-include-path
Defines the include path used when searching for files.
This should be a list of directories to search which is specific to
the file being included.
This variable can also be set to a single function.  If it is a
function, it will be called with one arguments, the file to find as a
string, and  it should return the full path to that file, or nil.
@end defvar


@end defun


@defun semantic-find-nonterminal token &optional parent
Find the location of @var{TOKEN}.
@var{TOKEN} may be a stripped element, in which case @var{PARENT} specifies a
parent token that has position information.
Different behaviors are provided depending on the type of token.
For example, dependencies (includes) will seek out the file that is
depended on, and functions will move to the specified definition.
@end defun

@defun semantic-find-documentation token
Find documentation from @var{TOKEN} and return it as a clean string.
@var{TOKEN} might have @var{DOCUMENTATION} set in it already.  If not, there may be
some documentation in a comment preceding TOKEN's definition which we
cal look for.  When appropriate, this can be overridden by a language specific
enhancement.
@end defun

@node Local Context, Making New Methods, Token Details, Override Methods
@subsection Local Context

@defun semantic-up-context &optional point
Move point up one context from @var{POINT}.
Return non-@code{nil} if there are no more context levels.
Overloaded functions using @code{up-context} take no parameters.
@end defun

@defun semantic-beginning-of-context &optional point
Move @var{POINT} to the beginning of the current context.
Return non-@code{nil} if there is no upper context.
The default behavior uses @dfn{semantic-up-context}.  It can
be overridden with @code{beginning-of-context}.
@end defun

@defun semantic-end-of-context &optional point
Move @var{POINT} to the end of the current context.
Return non-@code{nil} if there is no upper context.
Be default, this uses @dfn{semantic-up-context}, and assumes parenthetical
block delimiters.  This can be overridden with @code{end-of-context}.
@end defun

@defun semantic-get-local-variables &optional point
Get the local variables based on POINT's context.
Local variables are returned in Semantic token format.
Be default, this calculates the current bounds using context blocks
navigation, then uses the parser with @code{bovine-inner-scope} to
parse tokens at the beginning of the context.
This can be overriden with @code{get-local-variables}.
@end defun

@defun semantic-get-local-arguments &optional point
Get arguments (variables) from the current context at @var{POINT}.
Parameters are available if the point is in a function or method.
This function returns a list of tokens.  If the local token returns
just a list of strings, then this function will convert them to tokens.
Part of this behavior can be overridden with @code{get-local-arguments}.
@end defun

@defun semantic-get-all-local-variables &optional point
Get all local variables for this context, and parent contexts.
Local variables are returned in Semantic token format.
Be default, this gets local variables, and local arguments.
This can be overridden with @code{get-all-local-variables}.
Optional argument @var{POINT} is the location to start getting the variables from.
@end defun

These next set of functions handle local context parsing.   This means
looking at the code (locally) and navigating, and fetching information
such as a the type of the parameter the cursor may be typing in.

@defun semantic-end-of-command
Move to the end of the current command.
Be default, uses @code{semantic-command-separation-character}.
Override with @code{end-of-command}.
@end defun

@defun semantic-beginning-of-command
Move to the beginning of the current command.
Be default, users @code{semantic-command-separation-character}.
Override with @code{beginning-of-command}.
@end defun

@defun semantic-ctxt-current-symbol &optional point
Return the current symbol the cursor is on at @var{POINT} in a list.
This will include a list of type/field names when applicable.
This can be overridden using @code{ctxt-current-symbol}.
@end defun

@defun semantic-ctxt-current-assignment &optional point
Return the current assignment near the cursor at @var{POINT}.
Return a list as per @dfn{semantic-ctxt-current-symbol}.
Return @code{nil} if there is nothing relevant.
Override with @code{ctxt-current-assignment}.
@end defun

@defun semantic-ctxt-current-function &optional point
Return the current function the cursor is in at @var{POINT}.
The function returned is the one accepting the arguments that
the cursor is currently in.
This can be overridden with @code{ctxt-current-function}.
@end defun

@defun semantic-ctxt-current-argument &optional point
Return the current symbol the cursor is on at @var{POINT}.
Override with @code{ctxt-current-argument}.
@end defun

Local Context analysis functions can look at the local context,
and return informatin such as the type of the data that should be
entered where the cursor is.  As of this writing, these functions
need some more work.

@defun semantic-suggest-lookup-item name &optional tokentype returntype
Find a token definition matching @var{NAME} with @var{TOKENTYPE}.
Optional @var{RETURNTYPE} is a return value to match against also.
@end defun

@defun semantic-suggest-variable-token-hierarchy
Analyze the current line, and return a series of tokens.
The tokens represent a hierarchy of dereferences.  For example, a
variable name will return a list with one token representing that
variable's declaration.  If that variable is being dereferenced, then
return a list starting with the variable declaration, followed by all
fields being extracted.

For example, in c, ``foo->bar'' would return a list (@var{VARTOKEN} @var{FIELDTOKEN})
where @var{VARTOKEN} is a semantic token of the variable foo's declaration.
@var{FIELDTOKEN} is either a string, or a semantic token representing
the field in foo's type.
@end defun


@node Making New Methods, , Local Context, Override Methods
@subsection Local Context

@node Parser Hooks, , Override Methods, Utilities
@section Parser Hooks

If you write a program that uses the stream of tokens in a persistent
display or database, it is necessary to know when tokens change so
that your displays can be updated.  This is especially important as
tokens can be replaced, changed, or deleted, and the associated
overlays will then throw errors when you try to use them.  Complete
integration with token changes can be achieved via several very
important hooks.

One interesting way to interact with the parser is to let it know that
changes you are going to make will not require reparsing.

@defvar semantic-edits-are-safe
When non-@code{nil}, modifications do not require a reparse.
This prevents tokens from being marked dirty, and it
prevents top level edits from causing a cache check.
Use this when writing programs that could cause a full
reparse, but will not change the tag structure, such
as adding or updating top-level comments.
@end defvar

Next, it is sometimes useful to know what the current parsing state
is.  These function can let you know what level of reparsing may be
needed.  Careful choices on when to reparse can make your program much
faster.

@defun semantic-bovine-toplevel-full-reparse-needed-p &optional checkcache
Return non-@code{nil} if the current buffer needs a full reparse.
Optional argument @var{CHECKCACHE} indicates if the cache check should be made.
@end defun

@defun semantic-bovine-toplevel-partial-reparse-needed-p &optional checkcache
Return non-@code{nil} if the current buffer needs a partial reparse.
This only returns non-@code{nil} if @dfn{semantic-bovine-toplevel-full-reparse-needed-p}
returns nil.
Optional argument @var{CHECKCACHE} indicates if the cache check should be made
when checking @dfn{semantic-bovine-toplevel-full-reparse-needed-p}.
@end defun

If you need very close interaction with the user's editing, then these
two hooks can be used to find out when a given tag is being changed.
These hooks could even be used to cut down on reparsing if used correctly.

For all hooks, make sure you are careful to add it as a local hook if
you only want to effect a single buffer.  Setting it globally can
cause unwanted effects if your program is concerned with a single buffer.

@defvar semantic-dirty-token-hooks
Hooks run after when a token is marked as dirty (edited by the user).
The functions must take @var{TOKEN}, @var{START}, and @var{END} as a parameters.
This hook will only be called once when a token is first made dirty,
subsequent edits will not cause this to run a second time unless that
token is first cleaned.  Any token marked as dirty will
also be called with @code{semantic-clean-token-hooks}, unless a full
reprase is done instead.
@end defvar

@defvar semantic-clean-token-hooks
Hooks run after a token is marked as clean (reparsed after user edits.)
The functions must take a @var{TOKEN} as a parameter.
Any token sent to this hook will have first been called with
@code{semantic-dirty-token-hooks}.  This hook is not called for tokens
marked dirty if the buffer is completely reparsed.  In that case, use
@code{semantic-after-toplevel-bovinate-hook}.
@end defvar

@defvar semantic-change-hooks
Hooks run when semantic detects a change in a buffer.
Each hook function must take three arguments, identical to the
common hook @code{after-change-function}.
@end defvar

Lastly, if you just want to know when a buffer changes, use this hook.

@defvar semantic-after-toplevel-bovinate-hook
Hooks run after a toplevel token parse.
It is not run if the toplevel parse command is called, and buffer does
not need to be fully reparsed.
This function is also called when the toplevel cache is flushed, and
the cache is emptied.
For language specific hooks, make sure you define this as a local hook.
@end defvar

@defvar semantic-before-toplevel-cache-flush-hook
Hooks run before the toplevel nonterminal cache is flushed.
For language specific hooks, make sure you define this as a local hook.
This hook is called before a corresponding
@code{semantic-after-toplevel-bovinate-hook} which is also called during a
flush when the cache is given a new value of nil.
@end defvar

@node Current Context, Tools, Utilities, Top
@comment  node-name,  next,  previous,  up
@chapter Deriving the Current Context

This chapter deals with how to derive the current context, and also
how a language maintainter can get the current context API to work
with their language.

By default, the behavior will function in C like languages.  This
means languages with parenthetical blocks, and type dereferencing
which uses a similar form.

@menu
* Blocks::                      
* Local Variables::             Getting lists of local variables.
* Derived Context::             What goes at a given location?
* Suggestions::                 Get suggestions on one goes in a derived context.
@end menu

@node Blocks, Local Variables, Current Context, Current Context
@section Blocks and Navigation

Source code is typically built up of control structures, and blocks of
context, or lexical scope.  Semantic terms these lexical scopes as a
``context''.  The following functions can be used to navigate contexts.
Some of them are override functions.  Language authors can override
a subset of them to make them work for their language.

@defun semantic-up-context &optional point
Move point up one context from @var{POINT}.
Return non-@code{nil} if there are no more context levels.
Overloaded functions using @code{up-context} take no parameters.
@end defun

@defun semantic-beginning-of-context &optional point
Move @var{POINT} to the beginning of the current context.
Return non-@code{nil} if there is no upper context.
The default behavior uses @code{semantic-up-context}.  It can
be overridden with @code{beginning-of-context}.
@end defun

@defun semantic-end-of-context &optional point
Move @var{POINT} to the end of the current context.
Return non-@code{nil} if there is no upper context.
Be default, this uses @code{semantic-up-context}, and assumes parenthetical
block delimiters.  This can be overridden with @code{end-of-context}.
@end defun

These next set of functions can be used to navigate across commands.

@defun semantic-end-of-command
Move to the end of the current command.
Be default, uses @code{semantic-command-separation-character}.
Override with @code{end-of-command}.
@end defun

@defun semantic-beginning-of-command
Move to the beginning of the current command.
Be default, users @code{semantic-command-separation-character}.
Override with @code{beginning-of-command}.
@end defun

@node Local Variables, Derived Context, Blocks, Current Context
@section Deriving local variables

Within a given context, or block of code, local variables are often
defined.  These functions can be used to retrieve lists of locally
scoped variables.

@defun semantic-get-local-variables &optional point
Get the local variables based on POINT's context.
Local variables are returned in Semantic token format.
Be default, this calculates the current bounds using context blocks
navigation, then uses the parser with @code{bovine-inner-scope} to
parse tokens at the beginning of the context.
This can be overriden with @code{get-local-variables}.
@end defun

@defun semantic-get-local-arguments &optional point
Get arguments (variables) from the current context at @var{POINT}.
Parameters are available if the point is in a function or method.
This function returns a list of tokens.  If the local token returns
just a list of strings, then this function will convert them to tokens.
Part of this behavior can be overridden with @code{get-local-arguments}.
@end defun

@defun semantic-get-all-local-variables &optional point
Get all local variables for this context, and parent contexts.
Local variables are returned in Semantic token format.
Be default, this gets local variables, and local arguments.
This can be overridden with @code{get-all-local-variables}.
@end defun

@node Derived Context, Suggestions, Local Variables, Current Context
@section Deriving the Current Context

While a context has already been used to describe blocks of code,
other context include more local details, such as the symbol the
cursor is on, or the fact we are assinging into some other variable.

These context deriving functions can be overridden to provide language
specific behavior.  By default, it assumes a C like language.

@defun semantic-ctxt-current-symbol &optional point
Return the current symbol the cursor is on at @var{POINT} in a list.
This will include a list of type/field names when applicable.
This can be overridden using @code{ctxt-current-symbol}.
@end defun

@defun semantic-ctxt-current-assignment &optional point
Return the current assignment near the cursor at @var{POINT}.
Return a list as per @code{semantic-ctxt-current-symbol}.
Return @code{nil} if there is nothing relevant.
Override with @code{ctxt-current-assignment}.
@end defun

@defun semantic-ctxt-current-function &optional point
Return the current symbol the cursor is on at @var{POINT}.
The function returned is the one accepting the arguments that
the cursor is currently in.
This can be overridden with `ctxt.current-function'.
@end defun

@defun semantic-ctxt-current-argument &optional point
Return the current symbol the cursor is on at @var{POINT}.
Override with @code{ctxt-current-argument}.
@end defun

@node Suggestions,  , Derived Context, Current Context
@section Getting Suggestions for the Current Context

The suggestion API uses all the context deriving functions to make
intelligent guesses as to what belongs in a given location.  These
functions can be used to create smart completion mechanisms, or
displays.

Unfortunatly, they are not fully implemented in Semantic 1.4.


@node Tools, Index, Current Context, Top
@comment  node-name,  next,  previous,  up
@chapter Tools

Several tools come with Semantic which would not be possible without it.
In general, these tools will work with any language supported by
Semantic.

@menu
* speedbar::                    How to use Semantic speedbar support
* imenu::                       Special support for Imenu.
* semanticdb::                  Cache your parsed buffers between sessions.
* senator::                     The Semantic Navigator
* document::                    Document generation functions
* miscellaneous::               
@end menu

@node speedbar, imenu, Tools, Tools
@comment  node-name,  next,  previous,  up
@section Speedbar

Speedbar supports the display of tags through the Semantic parser.  To
use this utility, add a line like this to your @file{.emacs} file:

@example
(add-hook 'speedbar-load-hook (lambda () (require 'semantic-sb)))
@end example

or you can simply add:

@example
(require 'semantic-sb)
@end example

Once installed, speedbar will use semantic to find tokens, and will
display them appropriately.  Tags from semantic will have additional
details which can be seen, such as return type, or arguments to
functions.

@node imenu, semanticdb, speedbar, Tools
@comment  node-name,  next,  previous,  up
@section Imenu support

There is special support for creating Imenu entries using semantic.
This is a highly customizable tool which can create specialized menu
systems for navigating your source file.

By default, each language that wants special imenu support will set
itself up for it.  To setup imenu for your buffers, use this command
in your @file{.emacs} file:

@example
(add-hook 'semantic-init-hooks (lambda ()
                               	   (imenu-add-to-menubar "TOKENS")))
@end example

Also supported is @dfn{which-func-mode}.  This usually uses imenu
tags to show the current function.  The semantic support for this
function uses overlays, which is much faster.

You can customize imenu with the following options:

@deffn Option semantic-imenu-summary-function
Function to use when creating items in Imenu.
Some useful functions are:
@dfn{semantic-abbreviate-nonterminal}
@dfn{semantic-summarize-nonterminal}
@dfn{semantic-prototype-nonterminal}
@end deffn

@deffn Option semantic-imenu-bucketize-file
Non-@code{nil} if tokens in a file are to be grouped into buckets.
@end deffn

@deffn Option semantic-imenu-buckets-to-submenu
Non-@code{nil} if buckets of tokens are to be turned into submenus.
This option is ignored if @code{semantic-imenu-bucketize-file} is nil.
@end deffn

@deffn Option semantic-imenu-expand-type-parts
Non-@code{nil} if types should have submenus with parts in it.
@end deffn

@deffn Option semantic-imenu-bucketize-type-parts
Non-@code{nil} if elements of a type should be placed grouped into buckets.
@code{Nil} means to keep them in the same order.
Overriden to @code{nil} if @code{semantic-imenu-bucketize-file} is nil.
@end deffn

@deffn Option semantic-imenu-sort-bucket-function
Function to use when sorting tags in the buckets of functions.
@end deffn

@deffn Option semantic-imenu-index-directory
Non @code{nil} to index the entire directory for tags.
Doesn't actually parse the entire directory, but displays tags for all files
currently listed in the current Semantic database.
This variable has no meaning if semanticdb is not active.
@end deffn

@deffn Option semantic-imenu-auto-rebuild-directory-indexes
If non-@code{nil} automatically rebuild directory index imenus.
That is when a directory index imenu is updated, automatically rebuild
other buffer local ones based on the same semanticdb.
@end deffn

When adding support to a language, this variable may need to be set:

@defvar semantic-imenu-expandable-token
Tokens of this token type will be given submenu with children.
By default, a @code{type} has interesting children.  In Texinfo, however,
a @code{section} has interesting children.
@end defvar


@node semanticdb, senator, imenu, Tools
@comment  node-name,  next,  previous,  up
@section Semantic Database

Semanticdb is a utility which tracks your parsed files, and saves the
parsed information to files.  When you reload your source files,
semanticdb automatically associates the file with the cached copy,
saving time by not reparsing your buffer.

Semanticdb also provides an API for programs to use.  These functions
will return token information without loading the source file into
memory by checking the disk cache.

To use semanticdb, add the following to your @file{.emacs} file:

@example
(require 'semanticdb)
(global-semanticdb-minor-mode 1)
@end example

If you have a tool which optionally uses the semantic database, it may
be important to track if the database mode is turned on or off.

@deffn Option semanticdb-mode-hooks
Hooks run whenever @dfn{global-semanticdb-minor-mode} is run.
Use @dfn{semanticdb-minor-mode-p} to determine if the mode has been turned
on or off.
@end deffn

@deffn Option semanticdb-persistent-path
List of valid paths that semanticdb will cache tokens to.
When @dfn{global-semanticdb-minor-mode} is active, token lists will
be saved to disk when Emacs exits.  Not all directories will have
tokens that should be saved.
The value should be a list of valid paths.  @var{A} path can be a string,
indicating a directory in which to save a variable.  An element in the
list can also be a symbol.  Valid symbols are @code{never}, which will
disable any saving anywhere, @code{always}, which enables saving
everywhere, or @code{project}, which enables saving in any directory that
passes a list of predicates in @code{semantic-project-predicates}.
@end deffn

@deffn Option semanticdb-project-roots
List of directories, where each directory is the root of some project.
All subdirectories of a root project are considered a part of one project.
Values in this string can be overriden by project management programs
via the @code{semanticdb-project-root-functions} variable.
@end deffn

The important difference between these two is that you may put just
``~'' in @code{semanticdb-persistent-path}, but you may put individual
project directories into @code{semanticdb-project-roots} so that
different database lists don't get cross referenced incorrectly.

@subsection Searching

You can search for tokens in the database using the following
functions.

@defun semanticdb-find-nonterminal-by-name name &optional databases search-parts search-includes diff-mode find-file-match
Find all occurances of nonterminals with name @var{NAME} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES}, @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
@end defun

@defun semanticdb-find-nonterminal-by-name-regexp regex &optional databases search-parts search-includes diff-mode find-file-match
Find all occurances of nonterminals with name matching @var{REGEX} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
@end defun

@defun semanticdb-find-nonterminal-by-type type &optional databases search-parts search-includes diff-mode find-file-match
Find all nonterminals with a type of @var{TYPE} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
@end defun

@defun semanticdb-find-nonterminal-by-property property value &optional databases search-parts search-includes diff-mode find-file-match
Find all nonterminals with a @var{PROPERTY} equal to @var{VALUE} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
Return a list ((@var{DB-TABLE} . @var{TOKEN-LIST}) ...).
@end defun

@defun semanticdb-find-nonterminal-by-extra-spec spec &optional databases search-parts search-includes diff-mode find-file-match
Find all nonterminals with a @var{SPEC} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
Return a list ((@var{DB-TABLE} . @var{TOKEN-LIST}) ...).
@end defun

@defun semanticdb-find-nonterminal-by-extra-spec-value spec value &optional databases search-parts search-includes diff-mode find-file-match
Find all nonterminals with a @var{SPEC} equal to @var{VALUE} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
Return a list ((@var{DB-TABLE} . @var{TOKEN-LIST}) ...).
@end defun

@defun semanticdb-find-nonterminal-by-function function &optional databases search-parts search-includes diff-mode find-file-match
Find all occurances of nonterminals which match @var{FUNCTION}.
Search in all @var{DATABASES}.  If @var{DATABASES} is @code{nil}, search a range of
associated databases.
When @var{SEARCH-PARTS} is non-@code{nil} the search will include children of tokens.
When @var{SEARCH-INCLUDES} is non-@code{nil}, the search will include dependency files.
When @var{DIFF-MODE} is non-@code{nil}, search databases which are of a different mode.
@var{A} Mode is the @code{major-mode} that file was in when it was last parsed.
When @var{FIND-@var{FILE}-MATCH} is non-@code{nil}, the make sure any found token's file is
in an Emacs buffer.
@end defun

@defun semanticdb-file-stream file
Return a list of tokens belonging to @var{FILE}.
If file has database tokens available in the database, return them.
If file does not have tokens available, then load the file, and create them.
@end defun

@node senator, document, semanticdb, Tools
@comment  node-name,  next,  previous,  up
@section Semantic Navigator

Senator stands for SEmantic NAvigaTOR and was written by David Ponce.

This library defines commands and a minor mode to navigate between
semantic language tokens in the current buffer.

@subsection Commands

The following user level commands are provided by Senator.

@subsubsection Navigation

@deffn Command senator-next-token
Move to the next token in the current buffer.
@end deffn

@deffn Command senator-previous-token
Move to the previous token in the current buffer.
@end deffn

@deffn Command senator-jump sym
Jump to the semantic symbol @var{SYM}.
If called interactively and a prefix argument is supplied jump in the
local type's context (see function @dfn{senator-current-type-context}).
@end deffn

@subsubsection Searching

Searching using senator mode restricts the search only to the
definition text, such as the name of the functions or variables in a
given buffer.

@deffn Command senator-isearch-toggle-semantic-mode
Toggles semantic search in isearch mode.  When semantic search is
enabled, isearch is restricted to token names.
@end deffn

@deffn Command senator-search-forward string
@deffnx Command senator-search-backward string
Search forward and backward for a token matching @var{string}.
@end deffn

@deffn Command re-search-forward regex
@deffnx Command re-search-backward regex
Search forward and backward for a token matching the regular expression
@var{regex}.
@end deffn

@deffn Command word-search-forward word
@deffnx word-search-backward word
Search forward and backward for a token whose name matches @var{word}.
@end deffn

@subsubsection Completion

Completion in senator scans all known definitions in the local file,
and uses that information to provide the completion.

@deffn Command senator-complete-symbol
Complete the current symbol under point.
@end deffn

@deffn Command senator-completion-menu-keyboard-popup
Popup a completion menu for the symbol at point.
@end deffn

@subsubsection Copy/Paste

Token Copy/Paste is a high level form of the typical copy yank used by
Emacs.  Copying a token saves the meta-information related to the
function or item the cursor is currently in.  When that information is
yanked into a new buffer, the form of the text created is based on the
current status of the programming buffer.

For example, pasting a function into a different file results in a
function call template to be inserted.  In a Texinfo file, a @@deffn
is created with documentation for that function or command.

@deffn Command senator-copy-token
Take the current token, and place it in the token ring.
@end deffn

@deffn Command senator-kill-token
Take the current token, place it in the token ring, and kill it.
Killing the token removes the text for that token, and places it into
the kill ring.  Retrieve that text with @code{yank}.
@end deffn

@deffn Command senator-yank-token
Yank a token from the token ring.
The form the token takes is differnet depending on where it is being
yanked to.
@end deffn

@deffn Command senator-copy-token-to-register register &optional kill-flag
Copy the current token into @var{REGISTER}.
Optional argument @var{KILL-FLAG} will delete the text of the token to the
kill ring.
@end deffn

For programmers, to provide specialized pasting, created an override
function for @code{insert-foreign-token}. @xref{Settings}.

@subsubsection Minor Mode

@deffn Command senator-minor-mode
Toggle the SEmantic NAvigaTOR key bindings in the current buffer.

The following default key bindings are provided when semantic minor
mode is enabled:


@table @key
@item C-c , n
senator-next-token
@item C-c , p
senator-previous-token
@item C-c , i
senator-isearch-toggle-semantic-mode
@item C-c , j
senator-jump
@item C-c , TAB
senator-complete-symbol
@item C-c , SPC
senator-completion-menu-keyboard-popup
@item C-c , C-y
senator-yank-token
@item C-c , C-w
senator-kill-token
@item C-c , M-w
senator-copy-token
@item C-c , 
@end table
@end deffn

@subsection Customization

To enable the Senator keymap in all modes that support semantic parsing,
use this:

@example
(add-hook 'semantic-init-hooks 'senator-minor-mode)
@end example

To customize navigation around different types of tokens, use the
following variables:

@deffn Option senator-step-at-token-ids
List of token identifiers where to step.
Token identifier is symbol @code{'variable}, @code{'function}, @code{'type}, or other.  If
@code{nil} navigation steps at any token found.  This is a buffer local
variable.  It can be set in a mode hook to get a specific language
navigation.
@end deffn

@deffn Option senator-step-at-start-end-token-ids
List of token identifiers where to step at start and end.
Token identifier is symbol @code{'variable}, @code{'function}, @code{'type}, or other.  If
@code{nil} navigation only step at beginning of tokens.  If @code{t} step at start
and end of any token where it is allowed to step.  Also, stepping at
start and end of a token prevent stepping inside its children.  This
is a buffer local variable.  It can be set in a mode hook to get a
specific language navigation.
@end deffn

To have a mode specific customization, do something like this in a hook:

@example
(add-hook 'mode-hook
          (lambda ()
            (setq senator-step-at-token-ids '(function variable))
            (setq senator-step-at-start-end-token-ids '(function))
            ))
@end example

This will cause navigation and search commands to stop only between
functions and variables, and to step at start and end of functions
only.

@subsection Contact information for Senator

Any comments, suggestions, bug reports or upgrade requests are
welcome.  Please send them to David Ponce at david@@dponce.com

@node document, miscellaneous, senator, Tools
@comment  node-name,  next,  previous,  up
@section Document generation

The document program uses semantic token streams to aid in the
creation of texinfo documentation.  The main entry point for the
documentation generator are the following commands:

@deffn Command document &optional resetfile
Document the function or variable the cursor is in.
Optional argument @var{RESETFILE} is provided w/ universal argument.
When non-@code{nil}, query for a new documentation file.
@end deffn

@deffn Command document-inline
Document the current function with an inline comment.
@end deffn

@deffn Command document-insert-defun-comment nonterm buffer
Insert mode-comment documentation about @var{NONTERM} from @var{BUFFER}.
@end deffn

@deffn Command document-insert-new-file-header header
Insert a new header file into this buffer.  Add reference to @var{HEADER}.
Used by @code{prototype} if this file doesn't have an introductory comment.
@end deffn

In addition to these base documentation commands, the texinfo semantic
parser includes a two convenience functions when working directly with
texinfo files.

@deffn Command semantic-texi-update-doc &optional token
Update the documentation for @var{TOKEN}.
If the current buffer is a texinfo file, then find the source doc, and
update it.  If the current buffer is a source file, then get the
documentation for this item, find the existing doc in the associated
manual, and update that.
@end deffn

@deffn Command semantic-texi-goto-source &optional token
Jump to the source for the definition in the texinfo file @var{TOKEN}.
If @var{TOKEN} is @code{nil}, it is derived from the deffn under @var{POINT}.
@end deffn

@node miscellaneous,  , document, Tools
@comment  node-name,  next,  previous,  up
@section Miscellaneous commands

@deffn Command semantic-show-dirty-mode arg
Toggle highlighting of dirty tokens.  When optional argument @var{arg}
is negative, disable this mode.  A positive argument enables it.  If
@var{arg} is nil, toggle.

When active, tokens in the current buffer marked ``dirty'' are
highlighted.  A token is considered dirty if the token was generated
before the most recent change to that token's text.
@end deffn


@node Index,  , Tools, Top
@comment  node-name,  next,  previous,  up
@chapter Index

@unnumbered Function Index
@printindex fn

@bye

