% Document Type: LaTeX
% Master File: UserManual.tex
% 
% Date: %G
% File: %W


\documentstyle[a4,11pt,parskip,epsf,twoside,epic,eepic]{report}

\makeindex

\pagestyle{headings}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% MACRO \screendump
% =================
\newcommand{\screendump}[2]{%
  \begingroup                % keep redefinition of \epsfsize local
    \def\epsfsize##1##2{#1##1}% scale to #1% of natural size
    \epsfbox{#2}%            % read image
  \endgroup
}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% MARGIN - Values
% ===============
\evensidemargin0cm
\oddsidemargin0.6cm
\setlength{\textwidth}{15cm}
%\sloppy
    

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% COUNTER - VALUES
% =================
\setcounter{secnumdepth}{3}
\setcounter{tocdepth}{3}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% INCLUDE - Files:
% ================
% enter files that should be compiled by LaTeX seperated by comma.
% (useful while editing just one chapter, leaving all
% other files untouched)
%\includeonly{kernel_implem}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% D O C U M E N T
% ===============
\begin{document}

\bibliographystyle{alpha}
\nocite{Fahlma88,FahLeb91}

\include{title_ext}

\clearpage
\pagenumbering{roman}

\renewcommand{\baselinestretch}{0.6}\small\normalsize
\tableofcontents
%\thispagestyle{empty}
\renewcommand{\baselinestretch}{1.0}\small\normalsize

\clearpage
\pagenumbering{arabic}

% \part{Introduction}

\chapter{The Purpose of this Document}

There are roughly two new releases of SNNS every year. At this pace it
is clear that the changes from version to version are not fundamental
and that the corresponding changes and additions in the documentation
affect only a few pages of the manual.

In order to save a few trees it may be advisable to not print out all
the documentation, that actually hasn't changed much, again and again.
So we decided to provide a sort of an update document to the manual.
This document is prepared for all those users that already have an
older manual and wish to receive only an update for the new
functionality.

We do like to point out, however, that this document can only contain
those parts that are entirely new. All those little improvements,
cross references, and additions in chapters that are mainly unchanged
can not be included here. So we encourage the users either to use the
complete manual online or at least to print out every other release of
the manual in order to become aware of these changes.

Beginners anyways should start with the full documentation and ignore
these excerpts.


\section{New Features of Release 4.0}

Users already familiar with SNNS and its usage may be interested in
the differences between the versions 3.3 and 4.0. New users of SNNS
may skip this section and proceed with the next chapter.

New Features of Release 4.0:

\begin{enumerate}
  
\item A new distributed version of the kernel. SNNS can now be spread
  out in a workstation cluster for faster learning.

\item Improved version of the C-code generator snns2c

\item Validation sets now can check on the performance of the network
  during training.

\item New error function in tool analyze that can handle
  single-output-unit-networks. 

\item New statistic tool that can predict the generalization
  capabilities of the network.

\item Reorganization of the manager panel with new selection mechanism
  for the major SNNS windows. 

\item The remote panel was alternated significantly and renamed
  `control panel'. 

\item Selection possibility in the graph panel between plot of SSE,
  MSE, and SSE/output.

\item New learning algorithm RBF\_DDA

\item New learning algorithm simulated annealing

\item New learning algorithm Monte Carlo. 

\item New learning algorithm Pruned-Cascade-Correlation

\item Several new initialization algorithms for Kohonen and
  Counterpropagation networks, since the two old ones had serious
  flaws.

\item Key codes to bring up all the main panels of SNNS (e.g Alt-c for the
  control panel).

\item Support for NeXT systems.

\item More information printed to the shell during training and with
  the INFO button.

\item The SOURCE and TARGET part of the info panel have been swapped
  for a more intuitive setup of the information.

\item Extensive debugging (as usual).

\end{enumerate}


\clearpage

\setcounter{chapter}{4}
\setcounter{section}{2}
\setcounter{subsection}{2}
\subsection{Manager Panel}
\label{ManagerPanel}
\index{Manager Panel}
\index{Panel!Manager Panel}


Figure~\ref{FigManagerPanel} shows the manager panel. From the
manager panel all other elements that have a different, independent
window assigned can be called. Because this window is of such central
importance, it is recommended to keep it visible all the time.

\begin{figure}[bthp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.65}{xgui_figs/FigManagerPanel.ps}
  \end{minipage}
\caption{Manager panel}
\label{FigManagerPanel}
\end{center}
\end{figure}

The user can request several displays or help windows, but only one
control panel or text window. The windows called from the manager panel
may also be called via key codes as follows (`Alt-' meaning the
alternate key in conjunction with some other key).


\begin{center}
\begin{tabular}{lrp{1cm}lr}
\fbox{\tt FILE~~~~~~~}&{\tt Alt-f}&&\fbox{\tt CONTROL~~~~}&{\tt Alt-c}\\[0.3ex]
\fbox{\tt INFO~~~~~~~}&{\tt Alt-i}&&\fbox{\tt DISPLAY~~~~}&{\tt Alt-d}\\[0.3ex]
\fbox{\tt 3D DISPLAY~}&{\tt Alt-3}&&\fbox{\tt GRAPH~~~~~~}&{\tt Alt-g}\\[0.3ex]
\fbox{\tt BIGNET~~~~~}&{\tt Alt-b}&&\fbox{\tt PRUNING~~~~}&{\tt }\\[0.3ex]
\fbox{\tt CASCADE~~~~}&{\tt }&     &\fbox{\tt KOHONEN~~~~}&{\tt Alt-k}\\[0.3ex]
\fbox{\tt WEIGHTS~~~~}&{\tt Alt-w}&&\fbox{\tt PROJECTION~}&{\tt Alt-p}\\[0.3ex]
\fbox{\tt ANALYZER~~~}&{\tt Alt-a}&&\fbox{\tt INVERSION~~}&{\tt }\\[0.3ex]
\fbox{\tt PRINT~~~~~~}&{\tt }&     &\fbox{\tt HELP~~~~~~~}&{\tt Alt-h}\\[0.3ex]
\fbox{\tt QUIT~~~~~~~}&{\tt Alt-q}&&&\\
\end{tabular}
\end{center}

SNNS Status Message:
\index{Manager Panel!Status Message}

This line features messages about a current operation or its
termination. It is also the place of the command sequence display of
the editor.  When the command is activated, a message about the
execution of the command is displayed. For a listing of the command
sequences see chapter 6.
 
Status line
\index{Manager Panel!Status Line}

This line shows the current position of the mouse in a display, the
number of selected units, and the position of flags, set by the
editor.

X:0 Y:0 gives the current position of the mouse in the display in SNNS
unit coordinates.

The next icon shows a small selected unit. The corresponding value is
the number of currently selected units. This is important, because
there might be selected units not visible in the displays. The
selection of units affects only editor operations (see
chapter 6 and 6.3).

The last icon shows a minature flag. If {\tt safe} appears next to the
icon, the safety flag was set by the user (see chapter 6). In this
case XGUI forces the user to confirm any delete actions.


\subsection{Info Panel}
\label{UnitInfoPanel}
\index{Info Panel}
\index{Panel!Info Panel}


\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.65}{xgui_figs/FigUnitInfoPanel.ps}
  \end{minipage}
\caption{Info panel}
\label{FigUnitInfoPanel}
\end{center}
\end{figure}

\index{Source Unit}
\index{Target Unit}
The info panel displays all data of two units and the link between
them. The unit at the beginning of the link is called {\tt SOURCE},
the other {\tt TARGET}. One may run sequentially through all
connections or sites of the {\tt TARGET} unit with the arrow buttons
and look at the corresponding source units and vice versa.

This panel is also very important for editing, since some operations
refer to the displayed {\tt TARGET} unit or ({\tt
SOURCE}$\rightarrow${\tt TARGET}) link. A default unit can also be
created here, whose values (activation, bias, IO-type, subnet number,
layer numbers, activation function, and output function) are
copied into all selected units of the net. 

The source unit of a link can also be specified in a 2D display by
pressing the middle mouse button, the target unit by releasing it. To
select a link between two units the user presses the middle mouse
button on the source unit in a 2D display, moves the mouse to the
target unit while holding down the mouse button and releases it at the
target unit. Now the selected units and their link are displayed in
the info panel. If no link exists between two units selected in a 2D
display, the {\tt TARGET} is displayed with its first link, thereby
changing {\tt SOURCE}.

In table~\ref{InfoItems} the various fields are listed. The fields in
the second line of the SOURCE or TARGET unit display the name of the
activation function, name of the output function, name of the f-type
(if available). The fields in the line {\tt LINK} have the following
meaning: weight, site value, site function, name of the site. Most
often only a link weight is available. In this case no information
about sites is displayed. 


\begin{table}[hbtp]
\begin{center}
\begin{tabular}{|l|l|l|c|l|}
\hline
Name    & Meaning    & Type  & set by & value range\\ 
\hline
no.     & unit no.   & Label &        & $1$..$2^{31}$\\
subn.   & subnet no. & Label &        & $-32736$..$32735$\\
io      & IO-type    &Label &\fbox{\tt OPTIONS} & I(nput), O(utput), H(idden),\\
        &            &       &                    & D(ual), S(pecial)\\
act.    &activation  & Text  & input  & float;\\
iact.   &initial act.& Text  & input  & float;\\
out.    &output      & Text  & input  & float;\\
bias    &bias value  & Text  & input  & float\\
name    &unit name   & Text  & input  & string, starting with letter\\
\multicolumn{2}{|l|}{activation function}&Label&\fbox{\tt OPTIONS}& as available\\
\multicolumn{2}{|l|}{output function}& Label & \fbox{\tt OPTIONS} & as available\\
link    &weight      & Text  & input  & float\\
\multicolumn{2}{|l|}{site value}  & Label &        & float\\
\multicolumn{2}{|l|}{site function}& Label &        & as available\\
\multicolumn{2}{|l|}{site name} & Label &        & as available at {\tt TARGET}\\
\hline
\end{tabular}
\end{center}
\caption{Table of the unit, link and site fields in the Info panel}
\label{InfoItems}
\end{table}

Unit number, unit subnet number, site value, and site function cannot
be modified. To change attributes of type text, the cursor has to be
exactly in the corresponding field.

%\clearpage

There are the following buttons for the units (from left to right):

\begin{enumerate}

\item Arrow button \epsfbox{xgui_figs/xgui_button_first.ps}: The
  button below TARGET selects the first target unit (of the given
  source unit); the button below SOURCE selects the first source unit
  (of the given target unit);

\item Arrow button \epsfbox{xgui_figs/xgui_button_next.ps}: The
  button below TARGET selects the next target unit (of the given
  source unit); the button below SOURCE selects the next source unit
  (of the given target unit);

\item \fbox{\tt FREEZE}: Unit is frozen, if this button is inverted.
  Changes become active only after \fbox{\tt SET} is clicked.

\item \fbox{\tt DEF}: The default unit is assigned the displayed
  values of {\tt TARGET} and {\tt SOURCE} (only activation, bias,
  IO-type, subnet number, layer numbers, activation function and
  output function).

\item \fbox{\tt OPTIONS}: Calls the following menu:
%\clearpage

\begin{center}
\begin{tabular}{|l|l|}
\hline
{\tt change io-type}    & change the IO-type\\
{\tt change f-type}     & change f-type\\
{\tt display activation function} 
                        &  graph of the activation function\\
{\tt change activation function} 
                        &  change activation function\\
                        & note: f-type gets lost!\\
{\tt display output function} 
                        &  graph of the output function\\
{\tt change output function} 
                        & change output function\\
                        & note: f-type gets lost!\\
{\tt assign layers}     & assign unit to layers\\
{\tt list all sources}  & list all predecessors\\
{\tt list all targets}  & list all successors\\
\hline
\end{tabular}
\end{center}

\item \fbox{\tt SET}: Only after clicking this button the attributes
of the corresponding unit are set to the specified value. The unit is
also redrawn. Therefore the values can be changed without immediate
effect on the unit.
\end{enumerate}

\bigskip
There exist the following buttons for links (from left to right):

\begin{enumerate}
\item \hspace{0.2cm} \epsfbox{xgui_figs/xgui_button_first.ps}: Select
first link of the {\tt TARGET} unit.

\item \hspace{0.2cm} \epsfbox{xgui_figs/xgui_button_next.ps}: Select
next link of the {\tt TARGET} unit.

\item \fbox{\tt OPTIONS}: Calls the following menu:

\begin{center}
\begin{tabular}{|l|l|}
\hline
{\tt list current site of TARGET} 
                        & list of all links of current site.\\
{\tt list all sites of TARGET} 
                        & list all sites of the {\tt TARGET}\\
{\tt list all links from SOURCE} 
                        & list all links starting at {\tt SOURCE}\\
{\tt delete site}       & delete displayed site\\
                        & note: f-type gets lost!\\
{\tt add site}          & add new site to {\tt TARGET}\\
                        & note: f-type gets lost!\\
\hline
\end{tabular}
\end{center}

\item \fbox{\tt SET}: Only after clicking this button the link weight
is set.
\end{enumerate}    



\setcounter{section}{3}


\section {Update Functions}
\index{Update Functions}

Why is an update mode important? It is neccessary to visit the neurons 
of a net in a specific sequential order to perform operations on them. 
This order depends on the topology of the net and greatly influences 
the outcome of a propagation cycle. For each net 
with its own characteristics, it is very important to choose the update 
function associated with the net in order to get the desired behavior 
of the neural network. If a wrong update function is given, SNNS will display 
 a error message on your screen. Click \fbox {\tt OPTIONS} in the control panel 
to select an update function.  
 
The following update functions are available for the various network types:

\begin{center}
\begin{tabular}{|l|l|}
\hline
ART1\_stable \hspace{0.5cm} & \hspace{0.5cm} for ART1 networks\\ 
ART1\_synchronous \hspace{0.5cm} & \hspace{0.5cm} for ART1 networks\\
ART2\_Stable \hspace{0.5cm} & \hspace{0.5cm} for ART2 networks\\
ART2\_Synchronous  \hspace{0.5cm} & \hspace{0.5cm} for ART2 networks\\
ARTMAP\_Stable  \hspace{0.5cm} & \hspace{0.5cm} for ARTMAP networks\\
ARTMAP\_Synchronous  \hspace{0.5cm} & \hspace{0.5cm} for ARTMAP networks\\
Auto\_Synchronous  \hspace{0.5cm} & \hspace{0.5cm} for Autoassociative Memory networks\\
BAM\_Order \hspace{0.5cm} & \hspace{0.5cm} for Bidirectional Associative Memory networks\\
BBTT\_Order \hspace{0.5cm} & \hspace{0.5cm} for Backpropagation-Through-Time networks\\
CC\_Order \hspace{0.5cm} & \hspace{0.5cm} for Cascade- Correlation networks\\
\hline
\end {tabular}
\end{center}

\begin{center}
\begin{tabular}{|l|l|}
\hline
CounterPropagation \hspace{0.5cm} & \hspace{0.5cm} for Counterpropagation networks\\
Dynamic\_LVQ \hspace{0.5cm} & \hspace{0.5cm} for Dynamic-Learning-Vector-Quantization networks\\
Hopfield\_Fixed\_Act \hspace{0.5cm} & \hspace{0.5cm} for Hopfield networks\\
Hopfield\_Synchronous \hspace{0.5cm} & \hspace{0.5cm} for Hopfield networks\\
JE\_Order \hspace{0.5cm} & \hspace{0.5cm} for Jordan or Elman network\\
JE\_Special \hspace{0.5cm} & \hspace{0.5cm} for Jordan or Elman network\\
Kohonen\_Oder \hspace{0.5cm} & \hspace{0.5cm} for Self-Organizing Maps (SOMS)\\
RCC\_Order \hspace{0.5cm} & \hspace{0.5cm} for
reccurrent-Cascade-Correlation networks\\ 
Random\_Order \hspace{0.5cm} & \hspace{0.5cm} for any network\\
Random\_Permutation \hspace{0.5cm} & \hspace{0.5cm} for any network\\
Serial\_Order \hspace{0.5cm} & \hspace{0.5cm} for any network\\
Synchronous\_Order \hspace{0.5cm} & \hspace{0.5cm} for any network\\
TimeDelay\_Order \hspace{0.5cm} & \hspace{0.5cm} for Time-Delay networks\\
Topological\_Order \hspace{0.5cm} & \hspace{0.5cm} for any network\\
\hline
\end {tabular}
\end{center}

All these functions receive their input from the five update parameter fields
in the control panel. See figure 4.16 

The following parameters are required for ART, Hopfield, and
Autoassociative networks. The description of the update functions will
indicate which parameter is needed.

\begin{tabbing}
XXXXXXX \= X \= X \= \kill
\> \\
\> $\rho$ \> = vigilance parameter with ($0 \leq \rho \leq 1.$) : field1\\
\> $\rho^a$ \> = initial vigilance parameter for the ART$^a$ part of the net\\ 
\> \> \> with (0 $< \rho^a <$ 1) : field1\\
\> $\rho^b$ \> = vigilance parameter for ART$^b$ with ($0 < \rho^b < 1$) : field2\\
\>  $\rho^c$ \> = Inter-ART-Reset control with ($0 < \rho^b < 1$) : field3\\
\> $a$ \> = strength of the influence of the lower level in F1 by the middle\\
\> \> \> level with ($a$ $>$ 0) : field2\\
\> $b$ \> = strength of the influence of the middle level in F1 by the upper\\
\> \> \> level with ($b$ $>$ 0) : field3\\
\> $c$ \> = part of the length of vector p with (0 $<$ c $<$ 1) : field4\\
\> $\Theta$ \> = Kind of threshold with ($0 \leq \Theta < 1.$) : field5\\
\> c \> = number of units : field1\\
\> n \> = iteration parameter : field1\\
\> x \> = number of selected neurons\\
\> \\
\end {tabbing}

Field1..Field5 are the positions in the control panel. For a more
detailed description of ART-parameters see section 9.12: ART Models in SNNS

Now here is a description of the steps the various update functions perform, 
and of the way in which they differ.



\subsection* {ART1\_Stable}
\index{ART1\_Stable}
The {\tt ART1\_Stable} update function updates the neurons activation
and output values until a stable state is reached. In one propagation
step the activation of all non-input units is calculated and then the
calculation of the output of all neuron follows. The state is
considered stable if the 'classifiable' or the 'not classifiable'
neuron is selected. 'Classifiable' means that the input vector
(pattern) is recognized by the net. 'Not classifiable' means that that
there is no neuron in the recognition layer which would fit the input
pattern. The required parameter is $\rho$ in field1.





\subsection* {ART1\_Synchronous}
\index{ART1\_Synchronous}
The alogrithm of the {\tt ART1\_Synchronous} update function is the
ART1 equivalent to the algorithm of the Synchronous\_Order function.
The only difference is that the winner of the ART1 recognition layer
is identified. The required parameter is $\rho$ in field1.



\subsection* {ART2\_Stable}
\index{ART2\_Stable}
The first task of this algorithm is to initialize the activation of
all units. This is nescessary each time a new pattern is loaded to the
network. The ART2 net is initialized for a new pattern now. The output
and activation will be updated with synchronous propagations until a
stable state is reached. One synchronous propagation cycle means that
each neuron calculate its output and then its new activation. The
required parameters are $\rho$, $a$, $b$, $c$, $\Theta$ in field1,
field2, field3, field4, field5 respectively.
 



\subsection* {ART2\_Synchronous}
\index{ART2\_Synchronous}
This function is the ART2 equivalent to the {\tt Synchronous\_Order}
function. The only difference is that additionally the winner neuron
of the ART1 recognition layer is calculated. The required parameters
are $\rho$, a, b, c, $\Theta$ in field1, field2, field3, field4,
field5 respectively.







\subsection* {ARTMAP\_Stable}
\index{ARTMAP\_Stable}
{\tt ARTMAP\_Stable} updates all units until a stable state is
reached. The state is considered stable if the classified or
unclassified unit is 'on'. All neurons compute their output and
activation in one propagation step. The propagation step continues
until the stable state is reached. The required parameters are
$\rho^a, \rho^b, \rho^c$ in field1, field2, field3 respectively.



\subsection* {ARTMAP\_Synchronous}
\index{ARTMAP\_Synchronous}
The first step is to calculate the output value of the input units 
(input units of ARTa, ARTb). Now a complete propagation step takes place, 
i.e. all units calculate their output and activation value. The 
search for two recognition neuron with highest activation follows. The search
takes place in both ARTa and ARTb. The required parameters are 
$\rho^a, \rho^b, \rho^c$ in field1, field2, field3 respectively.


\subsection* {Auto\_Synchronous}
\index{Auto\_Synchronous}
First the {\tt Auto\_Synchronous} function calculates the activation
of all neurons. The next step is to calculate the output of all units.
The two steps will be repeated n times. For the iteration parameter n,
which has to be provided in field1, a value of 50 has shown to be very
suitable.

\subsection* {BAM\_Order}
\index{BAM\_Order}
The first step of this update function is to search for the first
hidden unit of the network. The current output is saved and a new
output is calculated for all neurons of the hidden and output layer.
Once this is accomplished the next progression of the hidden and
output units starts. Now for each neuron of the hidden and output
layer the new output is saved and the old saved output is restored.
With this older output the activation of all hidden and output neurons
is calculated. After this task is accomplished the new saved output
value of all hidden and output neurons is restored.


\subsection* {BBTT\_Order}
\index{BBTT\_Order}
The {\tt BBTT\_Order} algorithm performs an update on a recurrent
network. The recurrent net can be transformed into a regular
feedforward net with an input, multiple hidden and output layer. At
the beginning the update procedure checks if there is a zero-input
pattern in the input layer. Suppose there is such a pattern, then the
so called {\tt i\_act} value buffer is set to 0 for all neurons.  In
this case {\tt i\_act} can be seen as a buffer for the output value of
the hidden and output neurons. The next step is to copy the {\tt
  i\_act} value to the output of all hidden and output neurons. The
new activation of the hidden and output units will be calculated. Now
the new output for every neuron in the hidden and output layer will be
computed and stored in {\tt i\_act}.






\subsection* {CC\_Order}
\index{CC\_Order}
The {\tt CC\_Order} update function propagates a pattern through the net. 
This means all neurons calculate their new activation and output in a 
topological order. The {\tt CC\_Order} update function also handles the 
special units which represent the candidate units.  

\subsection* {CounterPropagation}
\index{CounterPropagation}
The {\tt CounterPropagation} update algorithm updates a net that
consists of a input, hidden and output layer. In this case the hidden
layer is called the Kohonen layer and the output layer is called the
Grossberg layer.  At the beginning of the algorithm the output of the
input neurons is equal to the input vector. The input vector is
normalized to the length of one.  Now the progression of the Kohonen
layer starts. This means that a neuron with the highest net input is
identified. The activation of this winner neuron is set to 1. The
activation of all other neurons in this layer is set to 0.  Now the
output of all output neurons is calculated. There is only one neuron
of the hidden layer with the activation and the output set to 1. This
and the fact that the activation and the output of all output neurons
is the weighted sum on the output of the hidden neurons implies that
the output of the output neurons is the weight of the link between the
winner neuron and the output neurons. This update function makes sense
only in combination with the CPN learning function.
 

\subsection* {Dynamic\_LVQ}
\index{Dynamic\_LVQ}
This update algorithm initializes the output and activation value of
all input neurons with the input vector. Now the progression of the
hidden neurons begins.  First the activation and output of each of the
hidden neurons is initialized with 0 and the new activation will be
calculated. The hidden neuron with the highest activation will be
identified. Note that the activation of this winner unit has to be $>$
-1. The class which the input pattern belongs to will be propagated to
the output neuron and stored as the neurons activation.  This update
function is sensible only in combination with the DLVQ learning
function.

 
\subsection* {Hopfield\_Fixed\_Act}
\index{Hopfield\_Fixed\_Act}
This update function selects x neurons with the highest netinputs and 
associates the activation value of those units with 1. The activation value
of all other units is associated with 0. Afterwards the output value of all
neurons will be calculated. The required parameter is x in field1.


\subsection* {Hopfield\_Synchronous}
\index{Hopfield\_Synchronous}
This update function calculates the output of all neurons first. This has 
to be done in order to propagate the pattern which is represented by the 
input vector. The activation update of all neurons which are not input 
neurons follows. The next step is to calculate the output value of those 
units. The input units are handled next. The activation of the input 
neurons is calculated and the next progression updates the output of all 
input units. 

 
\subsection* {JE\_Order}
\index{JE\_Order}
This update function propagates a pattern from the input layer to the
first hidden layer, then to the second hidden layer, etc. and finally
to the output layer. After this follows a synchronous update of all
context units. This function is makes sense only for JE-networks.



\subsection* {JE\_Special}
\index{JE\_Special}
Using the update function {\tt JE\_Special}, input patterns will
be generated dynamically. Let $n$ be the number of input units and $m$
the number of output units of the network.  {\tt JE\_Special} 
generates the new input vector with the output of the last $n-m$ input
units and the outputs of the $m$ output units. The usage of this
update function requires $n > m$. The propagation of the newly generated
pattern is done like using {\tt JE\_Update}.  The number of the actual
pattern in the control panel has no meaning for the input pattern when
using {\tt JE\_Special}. This update function is used to determine the 
prediction capabilities of a trained network.


\subsection* {Kohonen\_Order}
\index{Kohonen\_Order}
The {\tt Kohonen\_Order} function propagates neurons in a topological order.
 There are 2 propagation steps. The first step all input units are 
propagated, which means that the output of all neurons is calculated. The 
second step consists of the propagation of all hidden units. This 
propagation step calculates all hidden neuron's activation and output. Please
note that the activation and output are normally not required for the
Kohonen algorithm. The activation and output values are used for display and 
evaluation reasons internally. The {\tt Act\_Euclid} activation function for 
example, copies the Euclidian distance of the unit from the training pattern 
to the units activation.

\subsection* {RCC\_Order}
\index{RCC\_Order}
This function is similar to the {\tt CC\_Order} except it takes special care 
of the recurrent links. Use this function only for RCC-networks.


\subsection* {Random\_Order}
\index{Random\_Order}
The {\tt Random\_Order} update function selects a neuron and
calculates its activation and output value. The selection process is
absolutly random and will be repeated n times. The parameter n is the
number of existing neurons. One specific neuron can be selected more
than one time while other neurons may be left out. This kind of update
function is rarely used and is just a theoretical base to prove the
stability of Hopfield nets.



\subsection* {Random\_Permutation}
\index{Random\_Permutation}
This update function is similar to the {\tt Random\_Order} function. The only
difference is that a random permutation of all neurons is used to select 
the order of the units. This guarantees that each neuron will be selected 
exactly once to calculate the output and activation value. This procedure 
has two big disadvantages. The first disadvantage is that the computation
of the permutation is very time consuming and the second disadvantage is 
that it takes a long time until a stable output vector has been established. 

\subsection* {Serial\_Order}
\index{Serial\_Order}
The {\tt Serial\_Order} update function calculates the activation and output 
value for each unit. The progression of the neurons is serial which means the 
computation process starts at the first unit and proceeds to the last one.



\subsection* {Synchronous\_Order}
\index{Synchronous\_Order}
With the synchronous update function all neurons change their value at the same
time. All neurons calculate their activation in one single step. The output 
of all neurons will be calculated after the activation step. The difference to 
the {\tt serial\_order} update function  is that the calculation of the 
output and activation value requires two progressions of all neurons.
This kind of propagation is very useful for distributed systems (SIMD). 

 
\subsection* {TimeDelay\_Order}
\index{TimeDelay\_Order}
The update function TimeDelay\_Order is used to propagate patters
through a time delay network. Its behavior is analogous to the
Topological\_Order functions with recognition of logical links.


  
\subsection*  {Topological\_Order}
\index{Topological\_Order}
This mode is the most favorable mode for feedforward nets. The neurons
calculate their new activation in a topological order. The topological order is
given by the net-topology. This means that the first processed layer 
is the input layer. The next processed layer is the first hidden layer 
and the last layer is the output layer. A learning cycle is defined as a pass
through all neurons of the net. Shortcut-connections are allowed.




\section {Initialization Functions}
\index{Initialization Functions}

In order to work with various net-models and learning algorithms,
different initialization functions that initialize the components of a
net are required. Backpropagation for example will not as long as all
weights have the same output value, since there can be no gradient
computed. Click \fbox {\tt OPTIONS} in the control panel to select an
init function.

\pagebreak

The following initialization functions are available:

\begin{center}
\begin{tabular}{|l|l|}
\hline
ART1\_Weights \hspace{0.5cm}         & ART1 networks\\
ART2\_Weights \hspace{0.5cm}         & for ART2 networks\\
ARTMAP\_Weights \hspace{0.5cm}       & for ARTMAP networks\\
CC\_Weights \hspace{0.5cm}           &  for Cascade Correlation networks\\
ClippHebb \hspace{0.5cm}             & for Associative Memory networks\\
CPN\_Rand\_Pat \hspace{0.5cm}        & for Counterpropagation\\
CPN\_Weights\_v3.2 \hspace{0.5cm}    & for Counterpropagation\\
CPN\_Weights\_v3.3 \hspace{0.5cm}    & for Counterpropagation\\
DLVQ\_Weights \hspace{0.5cm}         & for Dynamic Learning Vector Quantization networks\\
Hebb \hspace{0.5cm}                  & for Associative Memory networks\\
Hebb\_Fixed\_Act \hspace{0.5cm}      & for Associative Memory networks\\
JE\_Weights \hspace{0.5cm}           & for Jordan or Elman networks\\
Kohonen\_Rand\_Pat \hspace{0.5cm}    & for Self-Organizing Maps (SOMS)\\
Kohonen\_Const \hspace{0.5cm}        & for Self-Organizing Maps (SOMS)\\
Kohonen\_Weights\_v3.2 \hspace{0.5cm}& for Self-Organizing Maps (SOMS)\\
Pseudoinv \hspace{0.5cm}             & for Associative Memory networks\\
Randomize\_Weights \hspace{0.5cm}    & for any network, except the ART-family\\
Random\_Weights\_Perc \hspace{0.5cm} & for Backpercolation\\ 
RBF\_Weights \hspace{0.5cm}          & for Radial Basis Functions (RBFs)\\
RBF\_Weights\_Kohonen \hspace{0.5cm} & for Radial Basis Functions (RBFs)\\
RBF\_Weights\_Redo \hspace{0.5cm}    & for Radial Basis Functions (RBFs)\\
RCC\_Weights \hspace{0.5cm}       & for recurrent Cascade-Correlation networks\\
RM\_Random\_Weights \hspace{0.5cm}   & for Autoassociative Memory Networks\\
\hline
\end {tabular}
\end {center}



All these functions receive their input from the five fields
in the initialization panel. See figure 4.16.

Here is a small decription of the different initialization functions:



\subsection* {ART1\_Weights} 
\index{ART1\_Weights}
ART1\_Weights is responsible to set the initial values of the trainable
links in an ART1 network. These links are the ones
from {F$_1$} to {F$_2$} and the ones from {F$_2$} to {F$_1$} respectively.
For more details see chapter 9.12.1.2.


\subsection* {ART2\_Weights}
\index{ART2\_Weights}
For an ART2 network the weights of the top-down-links
({F$_2$} $\rightarrow$ {F$_1$} links) are set to 0.0 according to the
theory (\cite{CG87b}). The choice of the initial bottom-up-weights is 
decribed in chapter 9.12.2.2.
 


\subsection* {ARTMAP\_Weights}
\index{ARTMAP\_Weights}
The trainable weights of an ARTMAP network are primarily the
ones of the two ART1 networks {ART$^a$} and {ART$^b$}, therefore the 
initialization process is similar. For more details see chapter 9.12.1.2 
and chapter 9.12.2.2.

\subsection* {CC\_Weights}
\index{CC\_Weights}
{\tt CC\_Weights} calls the {\tt Randomize\_Weights} function. 
See {\tt Randomize\_Weights}.

\subsection*{ClippHebb}
\index{ClippHebb}
The {\tt ClippHebb} algorithm is almost the same as the {\tt Hebb}
algorithm, the only difference is that all weights can only be set 
to 1 and 0. After the activation for the neurons is calculated, all 
weights $>$ 1 will be set to 1. 
As mentioned in \ref{Hebb} the {\tt ClippHebb} algorithm is a learning 
algorithm.


\subsubsection*{CPN\_Rand\_Pat}
\index{CPN\_Rand\_Pat}
This Counterpropagation initialization function initializes all weight
vectors of the Kohonen layer with random input patterns from the
training set. This guarantees that the Kohonen layer has no dead
neurons.

The weights of the Grossberg layer are all initialized to 1.


\subsubsection*{CPN\_Weights\_v3.2}
\index{CPN\_Weights\_v3.2}
This function generates random points in an n-dimensional hypercube
and later projects them onto the surface of an n-dimensional unit
hypersphere or onto one of its main diagonal sectors (main diagonal
quadrant for $n=2$, octant for $n=3$, ...).

First the interval, from which the Kohonen weights for the
initialization tasks are selected, is determined. Depending upon the
initialization parameters, which have to be provided in field1
and field2, the interval may be $[-1\,;+1]$, $[0\,;+1]$, or
$[-1\,;0]$.

Every component $w_{ij}$ of every Kohonen layer neuron j is then
assigned a random value from the above interval, yielding weight
vectors $\vec {w_{j}}$, which are random points within an
n-dimensional hypercube.  

The length of each vector $\vec {w_{j}}$ is then normalized to 1.

All weights of the neurons of the Grossberg layer are set to 1.  

Note that this initialization function does NOT produce weight vectors
with equal point density on the hypersphere, because with increasing
dimension of the hypercube in which the random dots are generated,
many more points are originally in the corners of the hypercube than
in the interior of the inscribed hypersphere.


\subsubsection*{CPN\_Weights\_v3.3}
\index{CPN\_Weights\_v3.3} This function generates random points in an
n-dimensional cube, throws out all vectors with length $>1$ and
projects the remaining onto the surface of an n-dimensional unit
hypersphere or onto one of its main diagonal sectors (main diagonal
quadrant for $n=2$, octant for $n=3$, ...).

First the interval, from which the Kohonen weights for the
initialization tasks are selected, is determined. Depending upon the
initialization parameters, which have to be provided in field1
and field2, the interval may be $[-1\,;+1]$, $[0\,;+1]$, or
$[-1\,;0]$.

Every component $w_{ij}$ of every Kohonen layer neuron j is then
assigned a random value from the above interval, yielding weight
vectors $\vec {w_{j}}$, which are random points within an
n-dimensional hypercube. If the weight vector $\vec {w_{j}}$ thus
generated is outside the unit hypersphere or hypersphere sector, a new
random vector is generated until eventually one is inside the
hypersphere or hypersphere sector.
Finally, the length of each vector $\vec {w_{j}}$ is normalized to 1.

The Grossberg layer weight vector components are all set to 1.

Note that this initialization function DOES produce weight vectors
with equal point density on the hypersphere. However, the fraction of
points from the hypercube which are inside the enscribed hypersphere
decreases exponentially with increasing vector dimension, thus
exponentially increasing the time to perform the initialization. This
method is thus only suitable for input dimensions up to 12 - 15.
(read Hecht-Nielsen: Neurocomputing, chapter 2.4, pp. 41 ff. for an 
interesting discussion on n-dim. geometry).


\subsection* {DLVQ\_Weights}
\index{DLVQ\_Weights}
{\tt DLVQ\_Weights} calls the {\tt Randomize\_Weights} function. 
See {\tt Randomize\_Weights}.


\subsection*{Hebb}
\index{Hebb}
\label{Hebb}
This procedure is similar to the Hebbian Learning Rule with a learning 
rate of 1. Additionally the bias of all input and output neurons is 
set with the parameters p1 and p2 which have to be provided in field1 
and field2. Please note that the {\tt Hebb}, {\tt ClippHebb}, {\tt HopFixAct}
and {\tt PseudoInv} initialization functions are actually learning functions.
The reason why those functions are called initialization functions, is the 
fact that there is no true training because all weights will be calculated 
directly. In case the values of the parameters p1 and p2 are 1 and -1
the bias of the input and output neurons will be set to ld (n) and ld (k).
Where n is the number of input neurons and k is the number of output neurons.
These settings are also the default settings for p1 and p2. 
In any other case the p1 and p2 represent the bias of the input and 
output neurons without any modification.


\subsection*{Hebb\_FixAct}
\index{Hebb\_FixAct}
This rule is neccessary in order to do 'one-step-recall' simulations.
For more informations see \cite {Amari89}. 
For the calculation the bias following facts are assumed:

\begin {itemize}
\item {The implemented net is an autoassociative net}\\ 
      The neurons of an autoassociative net have to be input and output 
      neurons at the same time. A Hopfield network would be an example for 
      such a net.

\item {Fixed number of 1s}\\
      The patterns which are to be saved have a fixed 
      number of 1s.
\end {itemize}

The parameter h1 and h2 are required. Where h1 is the number of ones per 
pattern and h2 is the propable degree of distortion in percent. The parameters 
have to be inserted in field1 and field2.
This initialization function should be used only in connection with the 
{\tt Hopfield\_Fixed\_Act} update function. As mentioned in section \ref{Hebb} 
the {\tt Hebb\_FixAct} algorithm is a learning algorithm.



\subsection* {JE\_Weights}
\index{JE\_Weights}
This network consists of two types of neurons. The regular neurons and 
the so called context neurons. In such networks all links leading to 
context units are considered recurrent links.
The initialization function {\tt JE\_Weights} requires the
specification of five parameters:
 

\begin{itemize}
 \item \makebox[0.7cm][l]{$\alpha$, $\beta$} : 
       The weights of the forward connections are randomly choosen
       from the interval $[\alpha\,;\beta]$. $\alpha, \beta$ have to be 
       provided in field1 and field2 of the init panel.  
 \item \makebox[0.7cm][l]{$\lambda$} : 
       Weights of self recurrent links from context units to themselves.
       Simple Elman networks use $\lambda = 0$. {$\lambda$} has to be 
       provided in field3 of the init panel. 
 \item \makebox[0.7cm][l]{$\gamma$} :
       Weights of other recurrent links to context units. This value
       is often set to $1.0$. {$\gamma$} has to be provided in field4 
       of the init panel.
 \item \makebox[0.7cm][l]{$\psi$}  : 
       Initial activation of all context units. {$\psi$} has to be provided
       in field5 of the init panel.
\end{itemize}

Please note that it is required that {$\alpha > \beta$}. If this
should not be the case, an error message will appear on the screen.
The context units will be initialized as decribed above. For all other
neurons the bias and all weights will be randomly choosen from the
interval $[\alpha\,;\beta]$.



\subsubsection*{Kohonen\_Weights\_v3.2} 
\index{Kohonen\_Weights\_v3.2} 
This initialization function is identical to CPN\_Weights\_v3.2 except
that it only initializes the Kohonen layer, because there is no second
Grossberg layer as in Counterpropagation.


\subsubsection*{Kohonen\_Const}
\index{Kohonen\_Const} 
Each component $w_{ij}$ of each Kohonen weight vector $w_{j}$ is set
to the value of $\frac{1}{\sqrt{n}}$, thus yielding all identical
weight vectors $w_{j}$ of length 1. This is no problem, because the
Kohonen algorithm will quickly pull weight vectors away from this
central dot and move them into the proper direction.


\subsubsection*{Kohonen\_Rand\_Pat}
\index{Kohonen\_Rand\_Pat}

This initialization function initializes all weight vectors of the
single Kohonen layer with random input patterns from the training
set. This guarantees that the Kohonen layer initially has no dead
neurons.


\subsection*{PseudoInv}
\index{PseudoInv}
The {\tt PseudoInv} initialization function computes all weights with 
the help of the pseudo inverse weight matrix  which is calculated with 
the algorithm of Greville. The formula for the weight calculation 
is: W = Q{$S^+$}. Where {$S^+$} is the 'Pseudoinverse' of the input 
vectors, Q are the output vectors and W are the desired weights of the net.
 The bias is not set and there are no parameters neccessary. 
Please note that the calculated weights are usually odd. 
As mentioned in \ref{Hebb} the {\tt PseudoInv} algorithm is a learning 
algorithm.


\subsection* {Randomize\_Weights}
\index{Randomize\_Weights}
This function initializes all weights and the bias with distributed 
random values. The values are choosen from the interval $[\alpha\,;\beta]$. 
$\alpha$ and $\beta$ have to be provided in field1 and field2 of the 
init panel. It is required that $\alpha > \beta$.
 
\subsection* {Random\_Weights\_Perc}
\index{Random\_Weights\_Perc}
The first task of this function is to calculate the number of 
incoming links of a unit.
Once this is acomplished, the range of possible weight-values 
will be determined. The range will be calulated with the $\alpha$
and $\beta$ parameters, which have to be provided with the help of 
the init panel in field1 and field2. 
If $\alpha = \beta$ all weights and the bias will be set to 
the value of the $\alpha$ parameter. If $\alpha <> \beta$ the links of 
all neurons will be initalized with random values selected from the interval
$[\alpha\,;\beta]$ and divided by the number of incoming links of every 
neuron. The bias will also be set to zero.


\subsection* {RBF\_Weights}
\index{RBF\_Weights}
This procedure first selects evenly distributed centers $\vec{t_j}$
from the loaded training patterns and assigns them to the links between 
input and hidden layer. Subsequently the bias of all neurons (parameter $p$)
inside the hidden layer is set to a value determined by the user and finally 
the links between hidden and output layer are computed. For more details
see chapter 9.10.2.2.



\subsection* {RBF\_Weights\_Kohonen}
\index{RBF\_Weights_Kohonen}
Using the self-organizing method of Kohonen feature maps, appropriate 
centers are generated on base of the teaching patterns. The computed 
centers are copied into the corresponding links. No other links and bias 
are changed. For more details see chapter 9.10.2.2.



\subsection* {RBF\_Weights\_Redo} 
\index{RBF\_Weights_Redo} 
In contrast to the preceding procedure only the links between hidden and 
output layer are computed. All other links and bias remain unchanged.
For more details see chapter 9.10.2.2.

\subsection* {RCC\_Weights}
\index{RCC\_Weights}
RCC\_Weights calls the {\tt Randomize\_Weights} function. 
See {\tt Randomize\_Weights}.


\subsection* {RM\_Random\_Weights}
\index{RM\_Random_Weights}
The {\tt RM\_Random\_Weights} function initializes the bias and all
weights of all units which are not input-units with a random value.
This value is selected from the interval $[\alpha\,;\beta]$. $\alpha$
and $\beta$ have to be provided in field1 and field2 of the init
panel. $\alpha > \beta$ has to hold.


\setcounter{chapter}{9}
\setcounter{section}{8}
\setcounter{subsection}{1}
\subsection{Pruned-Cascade-Correlation (PCC)}
\label{pcc}
\index{PCC}

\subsubsection{The Algorithm}

The aim of Pruned-Cascade-Correlation (PCC) is to minimize the {\it
expected} test set error, instead of the actual training error
\cite{Weh94}. PCC tries to determine the optimal number of hidden
units and to remove unneeded weights after a new hidden unit is
installed. As pointed out by Wehrfritz, selection criteria or a
hold-out set, as it is used in ``stopped-learning'', may be applied to
digest away unneeded weights. In this release of SNNS, however, only
selection criteria for linear models are implemented.

The algorithm works as follows (CC steps are printed italic):

\begin{enumerate}
\item{\it Train the connections to the output layer}
\item{Compute the selection criterion}
\item{\it Train the candidates}
\item{\it Install the new hidden neuron}
\item{Compute the selection criterion}
\item{Set each weight of the last inserted unit to zero and compute
    the selection criterion; if there exists a weight, whose removal
    would decrease the selection criterion, remove the link, which
    decreases the selection criterion {\bf most}. Goto step 5 until a
    further removal would increase the selection criterion.}
\item{Compute the selection criterion; if it is greater than the one,
    computed before inserting the new hidden unit, notify the user
    that the net is getting too big.}
\end{enumerate}

\subsubsection{Mathematical Background}

In this release of SNNS, three model selection criteria are
implemented: the Schwarz's Bayesian criterion (SBC), Akaikes information
criterion (AIC) and the conservative mean square error of prediction (CMSEP).
The {\it SBC}, the default criterion, is more conservative compared to
the {\it AIC}. Thus, pruning via the {\it SBC} will produce smaller networks
than pruning via the {\it AIC}.
Be aware that both {\it SBC} and {\it AIC} are selection criteria for {\it
linear} models, whereas the {\it CMSEP} does not rely on any statistical
theory, but happens to work pretty well in an application. These
selection criteria for linear model can sometimes directly be applied
to nonlinear models, if the sample size is large. 



\setcounter{section}{10}
\section{Dynamic Decay Adjustment for RBFs (RBF--DDA)}

\subsection{The Dynamic Decay Adjustment Algorithm}

The Dynamic Decay Adjustment (DDA--)Algorithm is an extension of the
RCE-Algorithm (see~\cite{Huda92,ReCE82}) and offers easy and
constructive training for Radial Basis Function Networks. RBFs
trained with the DDA-Algorithm often achieve classification accuracy
comparable to Multi Layer Perceptrons (MLPs)\footnote{ As usual the
  term MLP refers to a multilayer feedforward network using the scalar
  product as a propagation rule and sigmoids as transfer functions.  }
but training is significantly faster (\cite{BeDi95}).

An RBF trained with the DDA-Algorithm (RBF-DDA) is similar 
in structure to the common feedforward MLP with one hidden layer
and without shortcut connections:
\begin{itemize}
  \item The number of units in the {\bf input layer} represents
        the dimensionality of the input space.
  \item The {\bf hidden layer} contains the RBF units.
        Units are added in this layer during training.
        The input layer is fully connected to the hidden layer.
  \item Each unit in the {\bf output layer} represents one 
        possible class, resulting in an 1-of-n or binary coding.
        For classification a winner-takes-all approach is used,
        i.e. the output with the highest activation determines
        the class. Each hidden unit is connected to exactly one output unit.
\end{itemize}

The main differences to an MLP are the activation function and
propagation rule of the hidden layer: Instead of using a sigmoid or
another nonlinear squashing function, RBFs use localized functions,
radial Gaussians, as an activation function. In addition, a
computation of the Euclidian distance to an individual reference
vector replaces the scalar product used in MLPs:
\[
    R_i(\vec{x}) = exp(-\frac{||\vec{x}-\vec{r}_i||^2}{\sigma_i^2})
\]
If the network receives vector $\vec{x}$ as an input, $R_i$ indicates
the activation of one RBF unit with reference vector $\vec{r}_i$ and
standard deviation $\sigma_i$.

The output layer computes the output for each class as follows:
\[
    f(\vec{x})=\sum_{i=1}^{m}A_i*R_i(\vec{x}) 
\]
with $m$ indicating the number of RBFs belonging to the corresponding
class and $A_i$ being the weight for each RBF.

An example of a full RBF-DDA is shown in figure~\ref{rbfdda-net}.
Note that there do not exist any shortcut connections between input
and output units in an RBF-DDA.

\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.6}{nn_models/rbf-net.ps}
  \end{minipage}
  \caption{The structure of a Radial Basis Function Network.}
  \label{rbfdda-net}
\end{center}
\end{figure}


In this illustration the weight vector that connects all input units
to one hidden unit represents the centre of the Gaussian. The
Euclidian distance of the input vector to this reference vector (or
{\it prototype}) is used as an input to the Gaussian which leads to a
local response; if the input vector is {\it close} to the prototype,
the unit will have a high activation. In contrast the activation will
be close to zero for larger distances.  Each output unit simply
computes a weighted sum of all activations of the RBF units belonging
to the corresponding class.

The DDA-Algorithm introduces the idea of distinguishing between {\it
  matching} and {\it conflicting} neighbors in an {\it area of
  conflict}. Two thresholds $\theta^{+}$ and $\theta^{-}$ are
introduced as illustrated in figure~\ref{one-dda-rbf}.

\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.6}{nn_models/dda_unit.ps}
  \end{minipage}
  \caption{One RBF unit as used by the DDA-Algorithm. Two {\it thresholds}     
           are used to define an {\it area of conflict} where no        
           other prototype of a conflicting class is allowed to exist.
           In addition, each training pattern has to be in the inner
           circle of at least one prototype of the correct class.}
  \label{one-dda-rbf}
\end{center}
\end{figure}

Normally, $\theta^{+}$ is set to be greater than $\theta^{-}$ which
leads to a {\it area of conflict} where neither matching nor
conflicting training patterns are allowed to lie\footnote{The only
  exception to this rule is the case where a pattern of the same class
  lies in the area of conflict but is covered by another RBF (of the
  correct class) with a sufficiently high activation.}.  Using these
thresholds, the algorithm constructs the network dynamically and
adjusts the radii individually.

In short the main properties of the DDA-Algorithm are:
\begin{itemize}
\item {\bf constructive training}: new RBF nodes are added whenever necessary.
      The network is built from scratch, the number of required
      hidden units is determined during training. Individual radii 
      are adjusted dynamically during training.
\item {\bf fast training}: usually about five epochs are needed to complete
      training, due to the constructive nature of the algorithm. 
      End of training is clearly indicated.     
\item {\bf guaranteed convergence}: the algorithm can be proven to
      terminate.
\item {\bf two uncritical parameters}: only the two parameters $\theta^{+}$
      and $\theta^{-}$ have to
      be adjusted manually. Fortunately the values of these two
      thresholds are not critical to determine. For all tasks that have
      been used so far, $\theta^{+}=0.4$ and $\theta^{-}=0.2$ was
      a good choice.
\item {\bf guaranteed properties} of the network: it can be shown that
      after training has terminated, the network holds several conditions for
      all training patterns: wrong classifications are below a certain
      threshold ($\theta^{-}$) and correct classifications are above 
      another threshold ($\theta^{+}$).
\end{itemize}

The DDA-Algorithm is based on two steps. During training, whenever a
pattern is misclassified, either a new RBF unit with an initial weight
= 1 is introduced (called {\it commit}) or the weight of an existing
RBF (which covers the new pattern) is incremented. In both cases the
radii of conflicting RBFs (RBFs belonging to the wrong class) are
reduced (called {\it shrink}).  This guarantees that each of the
patterns in the training data is covered by an RBF of the correct
class and none of the RBFs of a conflicting class has an inappropriate
response.

Two parameters are introduced at this stage, a {\it positive threshold}
$\theta^{+}$ and a {\it negative threshold} $\theta^{-}$. To commit
a new prototype, none of the existing RBFs of the correct class has
an activation above $\theta^{+}$ and during shrinking no RBF of
a conflicting class is allowed to have an activation above $\theta^{-}$.
Figure~\ref{dda-exp} shows an example that illustrates the first
few training steps of the DDA-Algorithm.


\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{3.6cm}
    \screendump{0.4}{nn_models/dda_step1.ps}
  \end{minipage}
  \hfill
  \begin{minipage}{3.6cm}
    \screendump{0.4}{nn_models/dda_step2.ps}
  \end{minipage}
  \vspace{0.1cm}
  \centerline{(1)\hspace{3.7cm}(2)}
  \vspace{0.5cm}
  \begin{minipage}{3.6cm}
    \screendump{0.4}{nn_models/dda_step3.ps}
  \end{minipage}
  \hfill
  \begin{minipage}{3.6cm}
    \screendump{0.4}{nn_models/dda_step4.ps}
  \end{minipage}
  \vspace{0.1cm}
  \centerline{(3)\hspace{3.7cm}(4)}
  \vspace{0.1cm}
  \caption{An example of the DDA-Algorithm: (1) a pattern of class $A$
    is encountered and a new RBF is created; (2) a training pattern of
    class $B$ leads to a new prototype for class $B$ and shrinks the
    radius of the existing RBF of class $A$; (3) another pattern of
    class $B$ is classified correctly and shrinks again the prototype
    of class $A$; (4) a new pattern of class $A$ introduces another
    prototype of that class.}
  \label{dda-exp}
\end{center}
\end{figure}

After training is finished, two conditions are true for all input--output
pairs\footnote{In this case the term ``input--class pair'' would
  be more justified, since the DDA--Algorithm trains the network to
  classify rather than approximate an input--output mapping.}
$(\vec{x}, c)$ of the training data:
\begin{itemize}
\item at least one prototype of the correct class $c$ has an activation value
      greater or equal to $\theta^{+}$:
  \[
    \exists i : R_i^c(\vec{x})\geq \theta^{+}
  \]
\item all prototypes of conflicting classes have activations less or equal to
      $\theta^{-}$ ($m_k$ indicates the number of prototypes belonging
      to class $k$):
  \[
    \forall k\neq c, 1\leq j\leq m_k: R_j^k(\vec{x})\leq \theta^{-}
  \]
\end{itemize}

For all experiments conducted so far, the choice of $\theta^{+}$=0.4
and $\theta^{-}$=0.2 led to satisfactory results. In theory, those
parameters should be dependent on the dimensionality of the feature
space but in practice the values of the two thresholds seem to be
uncritical.  Much more important is that the input data is normalized.
Due to the radial nature of RBFs each attribute should be distributed
over an equivalent range. Usually normalization into [0,1] is
sufficient.


\subsection{Using RBF--DDA in SNNS}

The implementation of the DDA-Algorithm always uses the Gaussian
activation function {\tt Act\_RBF\_Gaussian} in the hidden layer.  All
other activation and output functions are set to {\tt Act\_Identity}
and {\tt Out\_Identity}, respectively.

The {\it Learning function} has to be set to {\tt RBF-DDA}. 
No {\it Initialization} or {\it Update} functions are needed.

The algorithm takes three arguments that are set in the first three
fields of the {\tt LEARN} row in the {\it control panel}. These are
$\theta^{+}$, $\theta^{-}$ ($0 < \theta^{-}\leq\theta^{+} < 1$) and
the maximum number of RBF units to be diplayed in one row.  This last
item allows the user to control the appearance of the network on the
screen and has no influence on the performance.  Specifying 0.0 leads
to the default values $\theta^{+}=0.4$, $\theta^{-}=0.2$ and to a
maximum number of 20 RBF units displayed in a row.

Training of an RBF starts with either:
\begin {itemize}
\item   an {\bf empty} network, i.e.~a network consisting only
        of input and output units. No connections between input and
        output units are required, hidden units will be added during
        training. This can easily be 
        generated with the tool {\tt BIGNET} (choice {\tt FEED FORWARD})
\item   a {\bf pretrained} network already containing RBF units generated
        in an earlier run of {\tt RBF-DDA} (all networks not complying with 
        the specification of an RBF-DDA will be rejected).
\end{itemize}

After having loaded a training pattern set, a {\it learning-epoch} can
be started by pressing the {\tt ALL} button in the {\it control panel}.
At the beginning of each epoch, the weights between the hidden and the
output layer are automatically set to zero.  Note that the resulting
RBF and the number of required learning epochs can vary slightly
depending on the order of the training patterns.  If you train using a
single pattern (by pressing the {\tt SINGLE} button) keep in mind that
every training step increments the weight between the RBF unit of the
correct class covering that pattern and its corresponding output unit.
The end of the training is reached when the network structure does not
change any more and the Mean Square Error (MSE) stays constant from
one epoch to another.

The first desired value in an output pattern that is greater than 0.0
will be assumed to represent the class this pattern belongs to; only
one output may be greater than 0.0.  If there is no such output,
training is still executed, but no new prototype for this pattern is
commited. All existing prototypes are shrunk to avoid coverage of this
pattern, however.  This can be an easy way to define an
``error''--class without trying to model the class itself.



\setcounter{chapter}{11}
\include{rpc_version}

\setcounter{chapter}{14}
\setcounter{section}{11}
\section{Snns2c}
\index{Tools!snns2c}

Synopsis: {\tt snns2c} $<$network$>$ $[<$C-filename$>$
$[<$function-name$>]$ $]$ 

where:

\vspace{1ex}
\begin{tabular}{l l}
  {\tt $<$network$>$} & is the name of the SNNS network file,\\
  {\tt $<$C-filename$>$ }& is the name of the output file\\
  {\tt $<$function-name$>$ }& is the name of the procedure in the
   application.
\end{tabular}

This tool compiles an SNNS network file into an executable C source.
It reads a network file {\tt $<$network.net$>$} and generates a C source
named {\tt $<$C-filename$>$}. The network can be called now as a function
named {\tt $<$function-name$>$}. If the parameter  {\tt
$<$function-name$>$} is missing, the name of {\tt $<$C-filename$>$} is
taken without the ending ``*.c''. If this parameter is also missing,
the name of the network file is chosen and fitted with a new
ending for the output file. This name without ending is also used for
the function name.

It is not possible to train the generated net, SNNS has to be used for
this purpose.  After completion of network training with SNNS, the
tool {\sc snns2c} is used to integrate the trained network as a C
function into a separate application.

This program is also an example how to use the SNNS kernel interface
for loading a net and changing the loaded net into another format.  All
data and all SNNS functions -- except the activation functions -- are
placed in a single C function. 

{\bf Note:\\} {\sc Snns2c} does not support sites. Any networks created
with SNNS that make use of the site feature can not be converted to C
source by this tool. Output functions are not supported, either.
\vspace{0.3cm}

\noindent
The program can translate the following network-types:
\begin{itemize}
\item Feedforward networks trained with Backpropagation and all 
      variants of it like Quickprop, RPROP etc. 
\item Radial Basis Functions
\item Partially-recurrent Elman and Jordan networks
\item Time Delay Neural Networks (TDNN)
\item Dynamic Learning Vector Quantisation (DLVQ)
\item Backpropagation Through Time (BPTT, QPTT, BBPTT)
\item Counterpropagation Networks
\end{itemize}

While the use of SNNS or any parts of it in commercial applications
requires a special agreement/licensing from the developers, the use
of trained networks generated with {\sc snns2c} is hereby granted without
any fees for any purpose, provided proper academic credit to the SNNS 
team is given in the documentation of the application.

\subsection{Program flow}

Because the compilation of very large nets may require some time, the
program outputs messages, indicating which state of compilation is
passed at the moment.

\begin{description}
\item[{ loading net...}] the network file is loaded with the function
  offered by the kernel user interface.

\item[{ dividing net into layers ...}] all units are grouped into layers
  where all units have the same type and the same activation function.
  There must not exist any dependencies between the units of the
  layers except the connections of SPECIAL HIDDEN units to themselves
  in Elman and Jordan networks or the links of the BPTT-networks.

\item[{ sorting layers...}] these layers are sorted in topological
  order, e.g. first the input layer, then the hidden layers followed
  by the output layers and at last the special hidden layers.  A layer
  which has sources in another layer of the same type is updated later
  as the source layer.

\item[{ writing net...}] selects the needed activation functions and writes
  them to the C-source file. After that, the procedure for pattern
  propagation is written.
\end{description}


\subsection{Including the Compiled Network in the Own Application}

\paragraph{Interfaces:}

All generated networks may be called as C functions. This functions
have the form:\\
\centerline{\tt int{\em function-name}(float *in, float *out, int init)} 
where {\tt in} and {\tt out} are pointers to the input and output
arrays of the network. The init flag is needed by some network types
and it's special meaning is explained in \ref{special}. The function
normally returns the value 0 (OK). Other return values are explained in
section \ref{special}.

The generated C-source can be compiled separately. To use the network
it's necessary to include the generated header file (*.h) which is
also written by {\sc snns2c}. This header file contains a
prototype of the generated function and a record which contains
the number of input and output units also.

\paragraph{Example:}

If a trained network, was saved as ``myNetwork.net'' and compiled
with 
\begin{center}
{\tt snns2c myNetwork.net} 
\end{center}
then the generated Network can be compiled with
\begin{center}
{\tt gcc -c myNetwork.c}
\end{center}

To include the network in your own application the header file must be
included. There should also two arrays being provided, one for the
input and one for the output of the network. The number of inputs and
outputs can be derived from a record in the header file. This struct
is named like the function which contains the compiled network and has
the suffix REC to mark the record. So the number of input units is
determined with {\tt myNetworkREC.NoOfInput} and the number of outputs
with {\tt myNetworkREC.NoOfOutput} in this example.  Hence, your own
application should contain:

$\vdots$
\begin{verbatim}
#include "myNetwork.h"
\end{verbatim}
$\vdots$
\begin{verbatim}
float *netInput, *netOutput; /* Input and Output arrays of the Network */
netInput  = malloc(myNetworkREC.NoOfInput  * sizeof(float));
netOutput = malloc(myNetworkREC.NoOfOutput * sizeof(float));
\end{verbatim}
$\vdots$
\begin{verbatim}
myNetwork(netInput, netOutput, 0)
\end{verbatim}
$\vdots$\\
Don't forget to link the object code of the network to your application

\subsection{Special Network Architectures}
\label{special}

Normally, the architecture of the network and  the numbers of the
units are kept. Therefore a dummy unit with the number 0 is inserted in
the array which contains the units. Some architectures are translated
with other special features.

\paragraph{TDNN:}
Generally, a layer in a time delay neural network consists of feature units and
their delay units. {\sc snns2c} generates code only containing
the feature units. The delay units are only additional activations in
the feature unit. This is possible because every delay unit has the
same link weights to it's corresponding source units as its feature
unit.

So the input layer consists only of its prototype units, too. Therefore
it's not possible to present the whole input pattern to the
network. This is not necessary because it can be presented step by
step to the inputs. This is useful for a real-time application,
with the newest feature units as inputs. To mark a new sequence the
{\em init}-flag (parameter of the function) can be set to 1. After
this, the delays are filled when the init flag is set to 0 again. 
To avoid meaningless outputs the function returns NOT\_VALID until the
delays are filled again.

There is a new variable in the record of the header file for TDNNs. It
is called ``MinDelay'' and is the minimum number of time steps which
are needed to get a valid output after the {\em init}-flag was set.

\paragraph{CPN:} Counterpropagation doesn't need the output layer. The
output is calculated as a weighted sum of the activations of the
hidden units. Because only one hidden unit has the activation 1 and
all others the activation 0, the output can be calculated with the
winner unit, using the weights from this unit to the output.

\paragraph{DLVQ:} Here no output units are needed either. The output is
calculated as the bias of the winner unit.

\paragraph{BPTT:}
If all inputs are set to zero, the net is not initialized. This
feature can be chosen by setting the {\em init}-flag.


\subsection{Activation Functions}

\paragraph{Supported Activation Functions:}
Following activation functions are implemented in {\sc snns2c}

\begin{center}
\begin{tabular}{| l | l |}
\hline
\multicolumn{2}{|c|}{\bf Supported Activation Functions}\\
\hline\hline
Act\_Logistic&              Act\_StepFunc\\
Act\_Elliott &              Act\_Identity\\            
Act\_BSB &                  Act\_IdentityPlusBias\\    
Act\_TanH&                  Act\_RBF\_Gaussian\\       
Act\_TanHPlusBias&          Act\_RBF\_MultiQuadratic\\ 
Act\_TanH\_Xdiv2 &          Act\_RBF\_ThinPlateSpline\\
Act\_Perceptron&            Act\_TD\_Logistic\\        
Act\_Signum&                Act\_TD\_Elliott\\         
Act\_Signum0 &\\
\hline                    
\end{tabular}
\end{center}

\paragraph{Including Own Activation Functions:}
The file "tools/sources/functions.h" contains two arrays: One array
with the function names (ACT\_FUNC\_NAMES) and one for the macros
which represent the function (ACT\_FUNCTIONS). These macros are
realized as character strings so they can be written to the generated
C-source.

The easiest way to include an own activation function is to write the
two necessary entries in the first position of the arrays. After that
the constant ``ActRbfNumber'' should be increased. If a new Radial Basis
function should be included, the entries should be appended at the end
without increasing ActRbfNumber. An empty string ("") should still be the last
entry of the array Act\_FUNC\_NAMES because this is the flag for the end
of the array.

\subsection{Error messages}

Here is the list of possible error messages and their brief
description.

\begin{description}
\item[{ not enough memory}:] a dynamic memory allocation failed.
\item[{ can't load file}:]  SNNS network-file wasn't found.
\item[{ can't open file}:] same as can't load or disk full.
\item[{ wrong parameters}:] wrong kind or number of parameters.
\item[{ net contains illegal cycles}:] there are several possibilities:
  \begin{itemize}
  \item a connection from a unit which is not a SPECIAL HIDDEN unit 
        to itself.
  \item two layers are connected to each other when not exactly
     one of them is SPECIAL HIDDEN.
  \item cycles over more than two layers which don't match the architecture
     of Jordan networks.
  \end{itemize}
  BPTT-networks have no restrictions concerning links.
\item[{ can't find the function {\em actfunc}}:] the activation function 
  {\em actfunc} is not supported.
\item[{ net is not a CounterPropagation network}:]
Counter\-pro\-paga\-tion-networks need a\\ special architecture: one input, one
output and one hidden layer which are fully connected.
\item[{ net is not a Time Delay Neural Network}:]  
The SNNS TDNNs have a very special architecture. They should be
generated by the {\sc bignet}-tool. In other cases there is no guaranty for
successful compilation. 
\item[{ not supported network type}:]  There are several network
types which can't be compiled with the {\sc snns2c}. The {\sc
  snns2c}-tool will be maintained and so new network types will be implemented.
\item[{ unspecified Error}:] should not occur, it's only a feature
for user defined updates.
\item[{ unknown error code : {\em number}}:] must not occur, please
  report to the  SNNS-Team.
\end{description} 

\bibliography{UserManual}

\end{document}


