\chapter{Neural Network Terminology}
\label{Connectionism}
\thispagestyle{empty}

\index{Connectionism}
{\em Connectionism}\/ is a current focus of research in a number of
disciplines, among them artificial intelligence (or more general
computer science), physics, psychology, linguistics, biology and
medicine. Connectionism represents a special kind of information
processing: Connectionist systems consist of many primitive cells
({\em units}\/) which are working in parallel and are connected via
directed links ({\em links, connections}\/).  The main processing
principle of these cells is the distribution of activation patterns
across the links similar to the basic mechanism of the human brain,
where information processing is based on the transfer of activation
from one group of neurons to others through synapses. This kind of
processing is also known as {\em parallel distributed processing
(PDP)}\/.

The high performance of the human brain in highly complex cognitive
tasks like visual and auditory pattern recognition was always a great
motivation for modeling the brain. For this historic motivation
\index{Neural Net}
connectionist models are also called {\em neural nets}\/.  However,
most current neural network architectures do not try to closely
imitate their biological model but rather can be regarded simply as a
class of parallel algorithms.

In these models, knowledge is usually distributed throughout the net
and is stored in the structure of the topology and the weights of the
links. The networks are organized by (automated) training methods,
which greatly simplify the development of specific applications.
Classical logic in ordinary AI systems is replaced by vague
conclusions and associative recall (exact match vs.\ best match).  This
is a big advantage in all situations where no clear set of logical
rules can be given. The inherent fault tolerance of connectionist
models is another advantage. Furthermore, neural nets can be made
tolerant against noise in the input: with increased noise, the
quality of the output usually degrades only slowly ({\em graceful
performance degradation}\/).


\section{Building Blocks of Neural Nets}
\label{neuralNets}

The following paragraph describes a generic model for those neural
nets that can be generated by the SNNS simulator. The basic principles
and the terminology used in dealing with the graphical interface are
also briefly introduced. A more general and more detailed introduction
to connectionism can, e.g., be found in \cite{rum861}. For readers
fluent in German, the most comprehensive and up to date book on neural
network learning algorithms, simulation systems and neural hardware is
probably \cite{Zell94}

\index{Unit}
\index{Link}
\index{Connection}
A network consists of {\em units}\/\footnote{In the following the more
common name ''units'' is used instead of ''cells''.} and directed,
weighted {\em links}\/ (connections) between them. In analogy to
activation passing in biological neurons, each unit receives a net
input that is computed from the weighted outputs of prior units with
connections leading to this unit. Picture~\ref{PicSimpleNet} shows a
small network.

\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.75}{intro_figs/intro_smallnet.ps}
  \end{minipage}
\caption{A small network with three layers of units}
\label{PicSimpleNet}
\end{center}
\end{figure}

The actual information processing within the units is modeled in the
\index{Activation Function}
\index{Output Function}
SNNS simulator with the {\em activation function}\/ and the {\em
output function}. The activation function first computes the net input
of the unit from the weighted output values of prior units. It then
computes the new activation from this net input (and possibly its
previous activation). The output function takes this result to
generate the output of the unit.\footnote{The term {\em transfer
function}\/ often denotes the combination of activation and output
function. To make matters worse, sometimes the term activation
function is also used to comprise activation and output function.}
These functions can be arbitrary C functions linked to the simulator
kernel and may be different for each unit.

Our simulator uses a discrete clock. Time is not modeled explicitly
(i.e.\ there is no propagation delay or explicit modeling of activation
functions varying over time). Rather, the net executes in
{\em update steps}\/, where $a(t+1)$ is the activation of a unit one step
after $a(t)$.

The SNNS simulator, just like the Rochester Connectionist Simulator
(RCS, \cite{god87}), offers the use of {\em sites}\/ as
\index{Site}
additional network element. Sites are a simple model of the dendrites
of a neuron which allow a  grouping and different treatment of the input
signals of a cell. Each site can have a different site function.
This selective treatment of incoming information allows more
powerful connectionist models. Figure~\ref{PicUnitsWithSites} shows one
unit with sites and one without. 

\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.75}{intro_figs/intro_unitsites.ps}
  \end{minipage}
\caption{One unit with sites and one without}
\label{PicUnitsWithSites}
\end{center}
\end{figure}

In the following all the various network elements are described in
detail. 

%\clearpage

\subsection{Units}

\index{Unit}
Depending on their function in the net, one can distinguish three
types of units: The units whose activations are the problem input for
the net are called {\em input units}\/; the units whose output
\index{Input Unit}
\index{Output Unit}
represent the output of the net {\em output units}. The remaining
units are called {\em hidden units}\/, because they are not visible
\index{Hidden Unit}
from the outside (see e.g.\ figure~\ref{PicSimpleNet}).

In most neural network models the type correlates with the topological
position of the unit in the net: If a unit does not have input
connections but only output connections, then it is an input unit. If
it lacks output connections but has input units, it is an output unit.
If it has both types of connections it is a hidden unit.

It can, however, be the case that the output of a topologically
internal unit is regarded as part of the output of the network. The
IO-type of a unit used in the SNNS simulator has to be understood in
this manner. That is, units can receive input or generate output even
if they are not at the fringe of the network.

Below, all {\em attributes}\/ of a unit are listed:
\index{Unit!Attribute}

\begin{itemize}
\item {\tt no}: For proper identification, every unit has a
number\footnote{This number can change after saving but remains
unambiguous. See also chapter~\ref{NetworkSaving}}
attached to it. This number defines the order in which the units are
stored in the simulator kernel.

\item {\tt name}: The name can be selected arbitrarily by the user. It
must not, however, contain blanks or special characters, and has to
start with a letter. It is useful to select a short name that
describes the task of the unit, since the name can be displayed with
the network.

\item {\tt io-type} or {\tt io}: The IO-type defines the function of
the unit within the net. The following alternatives are possible

\index{Unit ! Type}

\begin{itemize}
\item {\em input}\/: input unit
\item {\em output}\/: output unit 
\item {\em dual}\/: both input and output unit
\item {\em hidden}\/: internal, i.e.\ hidden unit
\item {\em special}: this type can be used in any way, depending upon
the application. In the standard version of the SNNS simulator, the
weights to such units are not adapted in the learning algorithm
(see paragraph~\ref{learning}).
\index{Special Unit}
\index{Unit ! Type ! Special}
\item {\em special input, special hidden, special output}\/: sometimes
it is necessary to to know where in the network a special unit is
located. These three types enable the correlation of the units to the
various layers of the network.
\end{itemize}

\item {\tt activation}: The activation value.

\item {\tt initial activation} or {\tt i\_act}: This variable contains
the initial activation value, present after the initial loading of the
net. This initial configuration can be reproduced by resetting
({\em reset}\/) the net, e.g.\ to get a defined starting state of the net.

\item {\tt output}: the output value.

\index{Bias}
\item {\tt bias}: In contrast to other network simulators where the
bias (threshold) of a unit is simulated by a link weight from a
special 'on'-unit, SNNS represents it as a unit parameter. In the
standard version of SNNS the bias determines where the activation
function has its steepest ascent. (see e.g.\ the activation function
{\tt Act\_logistic}). Learning procedures like backpropagation change
the bias of a unit like a weight during training.

\index{Activation Function!General Formula}
\item {\tt activation function} or {\tt actFunc}: A new activation is
computed from the output of preceding units, usually multiplied by
the weights connecting these predecessor units with the current unit,
the old activation of the unit and its bias. When sites are being
used, the network input is computed from the site values. The general
formula is:

\begin{displaymath}
        a_{j}(t+1) = f_{act}(net_{j}(t), a_{j}(t), \theta_{j})
\end{displaymath}

where:

\begin{tabular}{ll}
         $a_{j}(t)$   & activation of unit $j$ in step $t$\\
         $net_{j}(t)$ & net input in unit $j$ in step $t$\\
         $\theta_{j}$ & threshold (bias) of unit $j$\\
\end{tabular}                                                    

\index{Activation Function!Predefined!Logistic}
The SNNS default activation function {\em Act\_logistic}\/, for
example, computes the network input simply by summing over all
weighted activations and then squashing the result with the logistic
function $f_{act}(x) = 1 / (1+e^{-x})$.
The new activation at time $(t+1)$ lies in the
range $[0,1]$\footnote{ Mathematically correct would be $]0,1[$, but
the values 0 and 1 are reached due to arithmetic inaccuracy.}. The variable
$\theta_{j}$ is the threshold of unit $j$.

\index{Net Input}
The net input $net_{j}(t)$ is computed with
\begin{eqnarray*}
        net_{j}(t) & = & \sum_{i} w_{ij} o_{i}(t) \quad 
        \mbox{if unit $j$ has no sites } \\
        net_{j}(t) & = & \sum_{k} s_{jk}(t)       \quad 
        \mbox{if the unit $j$ has sites, with site values}\\
        s_{jk}(t) & = & \sum_{i} w_{ij} o_{i}(t) \nonumber\\
\end{eqnarray*}
This yields the well-known logistic activation function
\begin{displaymath}
        a_{j}(t+1) = \frac{1}{{1 + e^{- \left( \sum_{i} w_{ij} o_{i}(t) 
                                        - \theta_{j} \right) }}}
\end{displaymath}

where:

\begin{tabular}{ll}
         $a_{j}(t)$   & activation of unit $j$ in step $t$\\
         $net_{j}(t)$ & net input in unit $j$ in step $t$\\
         $o_{i}(t)$   & output of unit $i$ in step $t$\\
         $s_{jk}(t)$  & site value of site $k$ on unit $j$ in step $t$\\
         $j$          & index for some unit in the net\\
         $i$          & index of a predecessor of the unit $j$\\
         $k$          & index of a site of unit $j$\\
         $w_{ij}$     & weight of the link from unit $i$ to unit $j$\\
         $\theta_{j}$ & threshold (bias) of unit $j$\\
\end{tabular}                                                    

Activation functions in SNNS are relatively simple C functions which
are linked to the simulator kernel. The user may easily write his own
activation functions in C and compile and link them to the simulator
kernel. How this can be done is described later.

\index{Output Function!General Formula}
\item {\tt output function} or {\tt outFunc}: The output function
computes the output of every unit from the current activation of this
unit. The output function is in most cases the identity function
(SNNS: {\tt Out\_identity}). This is the default in SNNS\@. The output
function makes it possible to process the activation before an
output occurs.

\begin{displaymath}
o_{j}(t) = f_{out}(a_{j}(t))
\end{displaymath}

where:

\begin{tabular}{ll}
         $a_{j}(t)$ &    Activation of unit $j$ in step $t$\\
         $o_{j}(t)$ &    Output of unit $j$ in step $t$\\
         $j$        &    Index for all units of the net\\
\end{tabular}

Another predefined SNNS-standard function, {\em Out\_Clip01} clips the
output to the range of $[0..1]$ and is defined as follows:

\begin{eqnarray*}
o_{j}(t) = \left\{ 
        \begin{array}{ll}
        0 & \mbox{if $a_{j}(t) < 0$}\\
        1 & \mbox{if $a_{j}(t) > 1$}\\
        a_{j}(t) & \mbox{otherwise}\\
        \end{array}
        \right.
\end{eqnarray*}

Output functions are even simpler C functions than activation
functions and can be user-defined in a similar way.

\index{F-Type}
\item {\tt f-type}: The user can assign so called f-types
(functionality types, prototypes) to a unit. The unusual name is for
historical reasons. One may think of an f-type as a pointer to some
prototype unit where a number of parameters has already been defined:

\begin{itemize}
\item activation function and output function
\item whether sites are present and, if so, which ones
\end{itemize}

These types can be defined independently and are used for grouping
units into sets of units with the same functionality. All changes in
the definition of the f-type consequently affect also all units of that
type. Therefore a variety of changes becomes possible with minimum
effort.

\item {\tt position}\/: Every unit has a specific position
(coordinates in space) assigned to it. These positions consist of 3
integer coordinates in a 3D grid. For editing and 2D visualization
only the first two (x and y) coordinates are needed, for 3D
visualization of the networks the z coordinate is necessary.

\item {\tt subnet no}\/: Every unit is assigned to a subnet.
With the use of this variable, structured nets can be displayed more
clearly than would otherwise be possible in a 2D presentation.

\item {\tt layers}: Units can be visualized in 2D in up to 8
layers\footnote{Changing it to 16 layers can be done very easily in
the source code of the interface.}. Layers can be displayed selectively.
This technique is similar to a presentation with several
transparencies, where each transparency contains one aspect or part of
the picture, and some or all transparencies can be selected to be
stacked on top of each other in a random order. Only those units which
are in layers (transparencies) that are 'on' are displayed. This way
portions of the network can be selected to be displayed alone. It is also
possible to assign one unit to multiple layers.  Thereby it is
feasible to assign any combination of units to a layer that represents
an aspect of the network.

\item {\tt frozen}: This attribute flag specifies that activation
and output are frozen. This means that these values don't change
during the simulation.
\end{itemize}

All `important' unit parameters like activation, initial activation,
output etc.\ and all function results are computed as floats with nine
decimals accuracy.


\subsection{Connections (Links)}
\index{Connection}
\index{Link}

The direction of a connection shows the direction of the transfer of
activation. The unit from which the connection starts is called the {\em
source unit}\/, or {\em source} for short, while the other is called
\index{Unit!Source Unit}
\index{Source Unit}
\index{Unit!Target Unit}
\index{Target Unit}
the {\em target unit}\/, or {\em target}. Connections where source and
target are identical (recursive connections) are possible. Multiple
connections between one unit and the same input port of another unit are
redundant, and therefore prohibited. This is checked by SNNS\@.

\index{Connection!Weight}
Each connection has a {\em weight} (or strength) assigned to it. The
effect of the output of one unit on the successor unit is defined by
this value: if it is negative, then the connection is inhibitory,
i.e.\ decreasing the activity of the target unit; if it is positive,
it has an excitatory, i.e.\ activity enhancing, effect.

The most frequently used network architecture is built hierarchically
bottom-up. The input into a unit comes only from the units of
preceding layers. Because of the unidirectional flow of
information within the net they are also called feed-forward nets
(as example see the neural net classifier introduced in chapter
\ref{PatternMatcher}). In many models a full connectivity between all
units of adjoining levels is assumed.

Weights are represented as floats with nine decimal digits of precision.


\subsection{Sites}
\index{Site}

A unit with sites doesn't have a direct input any more. All incoming
links lead to different sites, where the arriving weighted output
signals of preceding units are processed with different
user-definable site functions (see picture~\ref{PicUnitsWithSites}).
The result of the site function is represented by the site value. The
activation function then takes this value of each site as network
input.

The SNNS simulator does not allow multiple connections from a unit to
the same input port of a target unit. Connections to different sites
of the same target units are allowed. Similarly, multiple connections
from one unit to different input sites of itself are allowed as well.


\section{Update Modes}
\label{Modes}
\index{Unit!Activation Update Mode}
\index{Update Mode}

To compute the new activation values of the units, the SNNS simulator
running on a sequential workstation processor has to visit all of them
in some sequential order. This order is defined by the {\em
Update Mode}\/. Five update modes for general use are implemented in
SNNS. The first is a synchronous mode, all other are asynchronous,
i.e.\ in these modes units see the new outputs of their predecessors
if these have fired before them.

\begin{enumerate}
\item {\em synchronous}\/: The units change their activation all together
after each step. To do this, the kernel first computes the new
activations of all units from their activation functions in some
arbitrary order. After all units have their new activation value
assigned, the new output of the units is computed. The outside
spectator gets the impression that all units have fired
simultaneously (in sync).

\item {\em random permutation}\/: The units compute their new
activation and output function sequentially. The order is defined
randomly, but each unit is selected exactly once in every step.

\item {\em random}\/: The order is defined by a random number
generator. Thus it is not guaranteed that all units are visited
exactly once in one update step, i.e.\ some units may be updated
several times, some not at all.

\item {\em serial}\/: The order is defined by ascending internal unit
number. If units are created with ascending unit numbers from input to
output units, this is the fastest mode. Note that the use of serial
mode is not advisable if the units of a network are not in ascending
order. 

\item {\em topological}\/: The kernel sorts the units by their
topology. This order corresponds to the natural propagation of
activity from input to output. In pure feed-forward nets the input
activation reaches the output especially fast with this mode, because
many units already have their final output which doesn't change later. 
\end{enumerate}

Additionally, there are 12 more update modes for special network
topologies implemented in SNNS.

\begin{enumerate}
\item{\em CPN}\/: For learning with counterpropagation.

\item{\em Time Delay}\/: This mode takes into account the special
connections of time delay networks. Connections have to be updated in
the order in which they become valid in the course of time.

\item{\em ART1\_Stable}, {\em ART2\_Stable} and {\em ARTMAP\_Stable}\/:
Three update modes for the three adaptive resonance theory network
models. They propagate a pattern through the network until a stable
state has been reached.

\item{\em ART1\_Synchronous}, {\em ART2\_Synchronous} and 
{\em ARTMAP\_Synchronous}\/: Three other update modes for the three adaptive
resonance theory network models. They perform just one propagation
step with each call.

\item{\em CC} and {\em RCC}\/: Special update modes for the cascade
correlation and recurrent cascade correlation meta algorithms.

\item{\em BPTT}\/: For recurrent networks, trained with
  `backpropagation through time'.

\item{\em RM\_Synchronous}\/: Special update mode for auto-associative
memory networks.
\end{enumerate}

Note, that all update modes only apply to the {\em forward
propagation phase}, the backward phase in learning procedures like
backpropagation is not affected at all.


\section{Learning in Neural Nets}
\label{learning}
\index{Learning}

An important focus of neural network research is the question of
how to adjust the weights of the links to get the desired system
behavior. This modification is very often based on the Hebbian rule,
which states that a link between two units is strengthened if both
units are active at the same time. The Hebbian rule in its general form
is:

\index{Hebbian Rule}
\begin{displaymath}
     \Delta w_{ij} = g(a_{j}(t), t_{j}) h(o_{i}(t),w_{ij})
\end{displaymath}

where:

\begin{tabular}{ll}
  $w_{ij}$     & weight of the link from unit $i$ to unit $j$\\
  $a_{j}(t)$   & activation of unit $j$ in step $t$\\
  $t_{j}$      & teaching input, in general the desired output of unit $j$\\ 
  $o_{i}(t)$   & output of unit $i$ at time $t$\\
\end{tabular}

\begin{tabular}{ll}
  $g(\dots)$   & function, depending on the activation of the 
                 unit and the teaching input\\
  $h(\dots)$   & function, depending on the output of the
                 preceding element and the current\\
               & weight of the link\\
\end{tabular}

\index{Learning!Supervised Learning}
Training a feed-forward neural network with supervised learning
consists of the following procedure: 

An input pattern is presented to the network. The input is then
propagated forward in the net until activation reaches the output
\index{Propagation!Forward Propagation}
\index{Forward Propagation}
layer. This constitutes the so called {\em forward propagation phase}.

The output of the output layer is then compared with the teaching
input. The error, i.e.\ the difference (delta) $\delta_{j}$ between the
output $o_{j}$ and the teaching input $t_{j}$ of a target output unit
$j$ is then used together with the output $o_{i}$ of the source unit
$i$ to compute the necessary changes of the link $w_{ij}$. To compute
the deltas of inner units for which no teaching input is available,
(units of hidden layers) the deltas of the following layer, which are
already computed, are used in a formula given below. In this way the
\index{Propagation!Backward Propagation}
\index{Backward Propagation}
errors (deltas) are propagated backward, so this phase is called {\em
backward propagation}.

\index{Learning!Online Learning}
\index{Online Learning}
In {\em online learning}, the weight changes $\Delta w_{ij}$ are
applied to the network after each training pattern, i.e.\ after each
\index{Learning!Offline Learning}
\index{Learning!Batch Learning}
\index{Offline Learning}
forward and backward pass. In {\em offline learning} or {\em batch
learning} the weight changes are cumulated for all patterns in the
training file and the sum of all changes is applied after one full
cycle (epoch) through the training pattern file.

\index{Backpropagation}
The most famous learning algorithm which works in the manner described
is currently backpropagation. In the backpropagation learning
algorithm online training is usually significantly faster than batch
training, especially in the case of large training sets with many
similar training examples. 

The backpropagation weight update rule, also called {\em generalized
delta-rule} reads as follows:
\index{Delta-Rule}

\vspace*{-5mm}
\begin{eqnarray*}
        \Delta w_{ij} & = & \eta \  \delta_{j} \  o_{i}\\
        \delta_{j}    & = & \left\{
        \begin{array}{ll}
        f'_{j}(net_{j})(t_{j} - o_{j}) &
        \mbox{if unit $j$ is a output-unit}\\
        f'_{j}(net_{j}) \sum_{k} \delta_{k} w_{jk} & 
         \mbox{if unit $j$ is a hidden-unit}\\
        \end{array}
        \right. 
\end{eqnarray*}

where:

\begin{tabular}{ll}
        $\eta$        & learning factor eta (a constant)\\
        $\delta_{j}$  & error (difference between the real output and
                        the teaching input) of unit $j$\\
        $t_{j}$       & teaching input of unit $j$\\
        $o_{i}$       & output of the preceding unit $i$\\
        $i$           & index of a predecessor to the current unit $j$
                        with link $w_{ij}$ from $i$ to $j$\\
        $j$           & index of the current unit\\
        $k$           & index of a successor to the current unit $j$
                        with link $w_{jk}$ from $j$ to $k$\\
\end{tabular}

There are several backpropagation algorithms supplied with SNNS: one
``vanilla backpropagation'' called {\tt Std\_Backpropagation}, one
with momentum term and flat spot elimination called {\tt
BackpropMomentum} and a batch version called {\tt BackpropBatch}. They
can be chosen from the {\bf control} panel with the button \fbox{\tt
OPTIONS} and the menu selection {\tt select learning function}.

In SNNS, one may either set the number of training cycles in advance
or train the network until it has reached a predefined error on
the training set.


\section{Generalization of Neural Networks}
\index{Generalization}

\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{\textwidth} 
    \screendump{0.6}{xgui_figs/overtraining.ps}
  \end{minipage}
\caption{Error development of a training and a validation set}
\label{overtraining}
\end{center}
\end{figure}

One of the major advantages of neural nets is their ability to
generalize. This means that a trained net could classify data from the
same class as the learning data that it has never seen before. In real world applications developers
normally have only a small part of all possible patterns for the
generation of a neural net. To reach the best generalization, the
dataset should be split into three parts:

\begin{itemize}
\item The {\bf training set} is used to train a neural net. The error
  of this dataset is minimized during training.
\item The {\bf validation set} is used to determine the performance of
  a neural network on patterns that are not trained during learning.
\item A {\bf test set} for finally checking the over all performance
  of a neural net.
\end{itemize}

Figure~\ref{overtraining} shows a typical error development of a
training set (lower curve) and a validation set (upper curve).

The learning should be stopped in the minimum of the validation set
error. At this point the net generalizes best. When learning is not
stopped, overtraining occurs and the performance of the net on the
whole data decreases, despite the fact that the error on the training
data still gets smaller. After finishing the learning phase, the net
should be finally checked with the third data set, the test set.

SNNS performs one validation cycle every n training cycles. Just like
training, validation is controlled from the control panel.

\section{An Example of a simple Network}
\label{PatternMatcher}
\index{Network!Example}
\index{Example!Simple Network}

\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.5}{xgui_figs/picPatternMatcher.ps}
  \end{minipage}
\caption{Example network of the letter classifier}
\label{picPatternMatcher}
\end{center}
\end{figure}

This paragraph describes a simple example network, a neural network
classifier for capital letters in a 5x7 matrix, which is ready for use
with the SNNS simulator. Note that this is a toy example which is not
suitable for real character recognition.

\begin{itemize}
\item Network-Files: {\tt letters0.net} (untrained), 
{\tt letters.net} (trained)
\item Pattern-File: {\tt letters.pat}
\end{itemize}

The network in figure~\ref{picPatternMatcher} is a feed-forward net
with three layers of units (two layers of weights) which can recognize
capital letters. The input is a 5x7 matrix, where one unit
is assigned to each pixel of the matrix. An activation of $+1.0$
corresponds to ``pixel set'', while an activation value of $0.0$
corresponds to ``pixel not set''. The output of the network consists of
exactly one unit for each capital letter of the alphabet.

The following activation function and output function are used by
default:

\begin{itemize}
\item Activation function: {\tt Act\_logistic}
\item Output function: {\tt Out\_identity}
\end{itemize}

The net has one input layer (5x7 units), one hidden layer (10 units)
and one output layer (26 units named 'A' \ldots 'Z'). The total of
$(35 \cdot 10 + 10 \cdot 26) = 610$~connections form the distributed
memory of the classifier.

On presentation of a pattern that resembles the uppercase letter
``A'', the net produces as output a rating of which letters are
probable. 




