biafore@beowulf.ucsd.edu (Louis Steven Biafore) (03/08/89)
-----------------------------------------------------------------------
The following technical report is now available.
-----------------------------------------------------------------------
DYNAMIC NODE CREATION
IN
BACKPROPAGATION NETWORKS
Timur Ash
ash@ucsd.edu
Abstract
Large backpropagation (BP) networks are very difficult
to train. This fact complicates the process of iteratively
testing different sized networks (i.e., networks with dif-
ferent numbers of hidden layer units) to find one that pro-
vides a good mapping approximation. This paper introduces a
new method called Dynamic Node Creation (DNC) that attacks
both of these issues (training large networks and testing
networks with different numbers of hidden layer units). DNC
sequentially adds nodes one at a time to the hidden layer(s)
of the network until the desired approximation accuracy is
achieved. Simulation results for parity, symmetry, binary
addition, and the encoder problem are presented. The pro-
cedure was capable of finding known minimal topologies in
many cases, and was always within three nodes of the
minimum. Computational expense for finding the solutions was
comparable to training normal BP networks with the same
final topologies. Starting out with fewer nodes than needed
to solve the problem actually seems to help find a solution.
The method yielded a solution for every problem tried. BP
applied to the same large networks with randomized initial
weights was unable, after repeated attempts, to replicate
some minimum solutions found by DNC.
-----------------------------------------------------------------------
Requests for reprints should be sent to the Institute for Cognitive
Science, C-015; University of California, San Diego; La Jolla, CA 92093.
(ICS Report 8901)
-----------------------------------------------------------------------kadirkam@paul.rutgers.edu (Janardhanan Kadirkamanathan) (11/22/90)
Message forwarded from visakan@eng.cam.ac.uk follows:
---------------------------------------------------------
The following technical report is available:
f-Projections: A nonlinear recursive estimation
algorithm for neural networks
V.Kadirkamanathan, M.Niranjan & F.Fallside
Technical Report CUED/F-INFENG/TR.53
Cambridge University Engineering Department
Trumpington Street, Cambridge CB2 1PZ, England
Abstract
By addressing the problem of sequential learning in neural networks,
we develop a new principle, f-projection, as a general method of
choosing a posterior estimate of a function, from the new information
received and the prior estimate. The principle is based on function
approximation and the posterior so obtained is optimal in the least
L-2 norm sense. The principle strikes a parallel with minimum cross
entropy, which provides a method of choosing a posterior probability
density estimate. Some fundamental properties of the principle of
f-projection are given with formal proofs.
Based on the principle of f-projection, a recursive (sequential)
estimation method called the method of successive f-projections, is
proposed. Some convergence related properties for this method are
given with formal proofs. The method is extended for parameter
estimation and to a sequential training algorithm for neural networks.
The problem of combining two separately trained neural networks is
also discussed.
Please send requests to:
Visakan Kadirkamanathan
Speech Laboratory
Cambridge Unviersity Engineering Department
Trumpington Street
Cambridge CB2 1PZ
England
email: visakan@eng.cam.ac.uk
-----------------------------------------------------------------------kadirkam@paul.rutgers.edu (Janardhanan Kadirkamanathan) (11/22/90)
Message forwarded from visakan@eng.cam.ac.uk follows:
----------------------------------------------------------
The authors in the technical report
f-Projections: A nonlinear recursive estimation
algorithm for neural networks
should have been,
V.Kadirkamanathan & F.Fallside
Technical Report CUED/F-INFENG/TR.53
......
A draft of the paper to appear in NIPS*90 is also available.
Sequential Adaptation of Radial Basis Function Neural
Networks and its application to time-series prediction
V.Kadirkamanathan, M.Niranjan & F.Fallside
The complete version of the paper will appear as a technical report
shortly.
-------------------------------------------------------------------mathew@elroy.jpl.nasa.gov (Mathew Yeates) (03/13/91)
I the following technical report (JPL Publication) is available
for anonymous ftp from the neuroprose directory at
cheops.cis.ohio-state.edu. This is a short version of a previous
paper "An Architecture With Neural Network Characteristics for Least
Squares Problems" and has appeared in various forms at several
conferences.
There are two ideas that may be of interest:
1) By making the input layer of a single layer Perceptron fully
connected, the learning scheme approximates Newtons algorithm
instead of steepest descent.
2) By allowing local interactions between synapses the network can
handle time varying behavior. Specifically, the network can
implement the Kalman Filter for estimating the state of a linear
system.
get both yeates.pseudo-kalman.ps.Z and
yeates.pseudo-kalman-fig.ps.Z
A Neural Network for Computing the Pseudo-Inverse of a Matrix
and Applications to Kalman Filtering
Mathew C. Yeates
California Institute of Technology
Jet Propulsion Laboratory
ABSTRACT
A single layer linear neural network for associative memory is
described. The matrix which best maps a set of input keys to desired
output targets is computed recursively by the network using a parallel
implementation of Greville's algorithm. This model differs from the
Perceptron in that the input layer is fully interconnected leading
to a parallel approximation to Newtons algorithm. This is in contrast
to the steepest descent algorithm implemented by the Perceptron.
By further extending the model to allow synapse updates to interact
locally, a biologically plausible addition, the network implements
Kalman filtering for a single output system.