\chapter{Pruning Algorithms}
\label{Pruning}
\index{Pruning Algorithms}
\thispagestyle{empty}

This chapter describes the four pruning functions which are available
in SNNS. The first section of this chapter introduces the common ideas
of pruning functions, the second takes a closer look at the theory of
the implemented algorithms and the last part gives guidance for the
use of the methods. Detailed description can be found in
\cite{Biedermann94} (for ``non-contributing units'') and
\cite{Schreiner94} (for the rest).

\section{Background of Pruning Algorithms}
\index{Pruning Algorithms!Theory}

Pruning algorithms try to make neural networks smaller by pruning
unnecessary links or units, for different reasons:

\begin{itemize}
\item It is possible to find a fitting architecture this way.

\item The cost of a net can be reduced (think of runtime, memory and
cost for hardware implementation).

\item The generalization\footnote{Generalization: ability of a neural
net to recognize unseen patterns (test set) after training} can (but
need not) be improved.

\item Unnecessary input units can be pruned in order to give evidence
of the relevance of input values.
\end{itemize}

Pruning algorithms can be rated according to two criterions:

\begin{itemize}
\item {\it What} will be pruned? We distinguish weight pruning and
node pruning. Special types of node pruning are input pruning and
hidden unit pruning.

\item {\it How} will be pruned? The most common possibilities are
{\it penalty term algorithms} (like Backpropagation with Weight Decay,
see section \ref{bpwd}) and {\it sensitivity algorithms} which are
described in this chapter.
\end{itemize}

Sensitivity algorithms perform training and pruning of a neural net
alternately, according to the algorithm in figure \ref{prsensalg}.

\begin{figure}[htbp]
\mbox{\ }\hrulefill\mbox{\ }

\begin{enumerate}
\item Choose a reasonable network architecture.
\item Train the net with backpropagation or any similar learning
function into a minimum of the network.
\item Compute the {\it saliency} (relevance for the performance of the
network) of each element (link or unit respectively).
\item Prune the element with the smallest saliency.
\item Retrain the net (into a minimum again).
\item If the net error is not too big, repeat the procedure from
step~3 on.
\item (optional) Recreate the last pruned element in order to achieve
a small net error again.
\end{enumerate}

\mbox{\ }\hrulefill\mbox{\ }

\caption{Algorithm for sensitivity algorithms}
\label{prsensalg}
\end{figure}

\section{Theory of the implemented algorithms}

There are different approaches to determine the saliency of an element
in the net. This section introduces the implemented sensitivity
pruning algorithms.

\subsection{Magnitude Based Pruning}
\index{Pruning Algorithms!Magnitude Based Pruning}
\index{Magnitude Based Pruning}

This is the simplest weight pruning algorithm. After each training,
the link with the smallest weight is removed. Thus the saliency of a
link is just the absolute size of its weight. Though this method is
very simple, it rarely yields worse results than the more
sophisticated algorithms.

\subsection{Optimal Brain Damage}
\index{Pruning Algorithms!Optimal Brain Damage}
\index{Optimal Brain Damage}

Optimal Brain Damage (OBD) approximates the change of the error
function when pruning a certain weight. A Taylor series is used for
the approximation:
\begin{eqnarray}
\delta E & = & E (W + \delta W) - E (W) \nonumber \\
 & = & \frac{\partial E(W)}{\partial W} \cdot \delta W +
\frac{1}{2} \frac{\partial^2 E(W)}{\partial W^2} +
R_2 (W + \delta W)
\label{prtaylorobd}
\end{eqnarray}
To simplify the computation, we assume that
\begin{itemize}
\item the net error function was driven into a minimum by training, so
that the first term on the right side of equation (\ref{prtaylorobd})
can be omitted;

\item the net error function is locally quadratic, so that the last
term in the equation can be left out;

\item the remaining second derivative {(\it Hesse-matrix)} consists
only of diagonal elements, which affects the second term in equation
(\ref{prtaylorobd}).
\end{itemize}

The result of all these simplifications reads as follows:
\begin{equation}
\delta E\ =\ \frac{1}{2} \sum_{1 \leq i,j \leq n_n} h_{(ij),(ij)} 
\cdot \delta w_{ij}^2
\end{equation}

Now it is necessary to compute the diagonal elements of the
Hesse-Matrix. For the description of this and to obtain further
information read \cite{LeCun+90}.

\subsection{Optimal Brain Surgeon}
\index{Pruning Algorithms!Optimal Brain Surgeon}
\index{Optimal Brain Surgeon}

Optimal Brain Surgeon (OBS, see \cite{Hassibi+93}) was a further
development of OBD. It computes the full Hesse-Matrix iteratively,
which leads to a more exact approximation of the error function:
\begin{equation}
\delta E\ =\ \frac{1}{2}\ \delta W^T \cdot H \cdot \delta W
\label{probstaylorred}
\end{equation}
From equation (\ref{probstaylorred}), we form a minimization problem
with the additional condition, that at least one weight must be set to
zero:
\begin{equation}
\min_{(ij)}\ \left\{\ 
\min_{\delta W}\ 
\left(\frac{1}{2}\ \delta W^T \cdot H \cdot \delta W \right)
\ |\ E_{(ij)}^T \cdot \delta W\ +\ w_{ij}\ =\ 0 
\ \right\}
\end{equation}
and deduce a Lagrangian from that:
\begin{equation}
L\ =\ \frac{1}{2}\ \delta W^T \cdot H \cdot \delta W
\ +\ \lambda\cdot (E_{(ij)}^T \cdot \delta W\ +\ w_{(ij)})
\end{equation}
where $\lambda$ is an Lagrangian multiplier. This leads to
\begin{eqnarray}
\delta W & = & - \frac{w_{ij}}{[H^{-1}]_{(ij),(ij)}} \cdot H^{-1} \cdot
E_{(ij)} \label{obsgewver} \\
L_{(ij)} & = & s_{ij}\ =\ \frac{1}{2} \frac{w_{(ij)}^2}{[H^{-1}]_{(ij),(ij)}}
\mbox{\rule{0mm}{8mm}}
\end{eqnarray}
Note that the weights of all links are updated.

The problem is, that the inverse of the Hesse-Matrix has to be
computed to deduce saliency and weight change for every link. A
sophisticated algorithm has been developed, but it is still very slow
and takes much memory, so that you will get in trouble for bigger
problems.

\subsection{Skeletonization}
\index{Pruning Algorithms!Skeletonization}
\index{Skeletonization}

Skeletonization (\cite{Mozer+89}) prunes units by estimating the
change of the error function when the unit is removed (like OBS and
OBD do for links). For each node, the {\it attentional strength} is
introduced which leads to a different formula for the net input:
\begin{equation}
net_j\ =\ \sum_i w_{ij} \cdot \alpha_i \cdot o_i
\end{equation}
Figure \ref{prskel} illustrates the use of the attentional strength.

\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{1}{pruning_figs/Skeletonization.ps}
  \end{minipage}
\caption{Neural network with attentional strength for each input and
hidden neuron}
\label{prskel}
\end{center}
\end{figure}

Defining the relevance of a unit as the change in the error function
while removing the unit we get
\begin{equation}
\rho_i\ =\ E_{\alpha_i = 0}\ -\ E_{\alpha_i = 1} \approx\ 
\left. \frac{\partial E}{\partial \alpha_i} \right|_{\alpha_i = 1}
\end{equation}
In order to compute the saliency, the linear error function is used:
\begin{equation}
E\ =\ E^l\ =\ \sum_j | t_{pj} - o_{pj} |
\end{equation}

\subsection{Non-contributing Units}
\index{Pruning Algorithms!Non-contributing Units}
\index{Non-contributing Units}

This method uses statistical means to find units that don't contribute
to the net's behavior. The net is subdivided into its layers, the
output of each neuron is observed for the whole pattern set and units
are removed that
\begin{itemize}
\item don't vary their output,
\item always show the same output as another unit of the same layer or
\item always show the opposite output of another unit of the same
layer.
\end{itemize}
This function is the first part of the method introduced by
\cite{Sietsma+91}. For further information about the implementation
read \cite{Biedermann94}.


\section{Pruning Nets in SNNS}
\index{Pruning Algorithms!Usage}

To use one of the pruning algorithms mentioned above, set the learning
function in the control panel (options menu, see section
\ref{LearningParameters}) to {\tt PruningFeedForward}. 

{\bf Note:} This information is saved with the net. Be sure to check
the learning function each time you reload the net.

The pruning algorithms can be customized by changing the parameters in
the pruning panel (see figure \ref{prpanel}) which can be invoked by
choosing {\tt PRUNING} in the main menu. The figure shows the default
values of the parameters.

\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.6}{pruning_figs/PruningPanel.ps}
  \end{minipage}
\caption{Pruning Panel}
\label{prpanel}
\end{center}
\end{figure}

The first block of the panel controls the pruning, the second the
embedded learning epochs and the last two specify parameters for
certain pruning algorithms. To save the changes to the parameters and
close the pruning panel, press the button \fbox{\tt DONE}.

The current pruning function is shown in the box ``{\tt General
  Parameters for Pruning}''. To select a different one, press the
button \fbox{\tt SELECT} which pops up a list of the available pruning
functions, analogous to the change of the learning function.

There are two criterions to stop the pruning: The error after
retraining must not exceed
\begin{itemize}
\item the error before the first pruning by more then a certain
percentage determined by the user in the field ``{\tt Maximum error
increase in \%:}'' and
\item the absolute value given in the field ``{\tt Accepted error}''
\end{itemize}

Normally, the state of the net before the last (obviously too
expensive) pruning is restored at the end. You can prevent this by
switching the radio buttons next to ``{\tt Recreate last pruned
element}'' to \fbox{\tt NO}.

The second box in the pruning panel (``{\tt General Parameters for
Learning}'') allows to select the (subordinate) learning function in
the same way as the pruning function. Only learning functions for
feed-forward-nets appear in the list. You can select the number of
epochs for the first training and each retraining separately. The
training, however, stops when the absolute error falls short of the
``{\tt Minimum error to stop}''. This prevents the net from
overtraining.

The parameter for OBS in the third box is the initial value of the
diagonal elements in the Hesse-Matrix. For the exact meaning of that
parameter (to which OBS is said to be not very sensible) see
\cite{Hassibi+93}.

The last box allows the user to choose which kind of neurons should be
pruned by the node pruning methods. Input or hidden unit pruning can
be selected by two sets of radio buttons.

Learning Parameters of the subordinate learning function have to be
typed in in the control panel, as if the training would be processed by
this function only. The field \fbox{\tt CYCLES} in the control panel
has no effect on pruning algorithms. To start pruning, press the
button \fbox{\tt DONE} in the pruning panel and the button \fbox{\tt
ALL} or \fbox{\tt SINGLE} respectively in the control panel.
