neuron-request@HPLMS2.HPL.HP.COM ("Neuron-Digest Moderator Peter Marvit") (06/01/90)
Neuron Digest Thursday, 31 May 1990
Volume 6 : Issue 37
Today's Topics:
Special Issue on Neural Networks
Neuron Digest Submission
UPDATED program info for: 6/16 NEURAL NETS FOR DEFENSE Conference
TIME-SENSITIVE - DoD small Business Innovation Research Program
Send submissions, questions, address maintenance and requests for old issues to
"neuron-request@hplabs.hp.com" or "{any backbone,uunet}!hplabs!neuron-request"
Use "ftp" to get old issues from hplpm.hpl.hp.com (15.255.176.205).
------------------------------------------------------------
Subject: Special Issue on Neural Networks
From: Alexander Linden <hplabs!gmdzi!al>
Date: Thu, 31 May 90 09:31:58 -0200
Special Issue on Neural Networks in Parallel Computing
(To appear in August)
This special issue focuses on the third generation of neural networks,
which can be characterized as being heterogeneous, modular and
asynchronous.
Contents:
H. Muehlenbein: Limitations of Multilayer Perceptrons:
Towards Genetic Neural Networks
F. Smieja: The Geometry of Multilayer Perceptron Solutions
H. Muehlenbein
J. Kindermann: Inversion of Neural Networks by Gradient Descent
A. Linden
T. E. Lange: Simulation of Heterogeneous Neural Networks on
Serial and Parallel Machines
A. Singer: Implementations of Artificial Neural Networks on the
Connection Machine
X. Zhang: The Backpropagation Algorithm on Grid and Hypercube
et al. Architectures
M. Witbrock: An Implementation of Backpropagation Learning on GF11,
M. Zagha a large SIMD Parallel Computers
D. Whitley: Genetic Algorithms and Neural Networks: Optimizing
et al. Connections and Connectivity
M. Tenorio: Topology Synthesis Networks: Self Organization of
Structure and Weight Adjustment as a Learning
Paradigm
K. Obermayer: Large Scale Simulations of Self-Organizing Neural
Networks on Parallel Computers: Application for
Biological Modelling
R. Kentridge: Neural Networks for Learning in the Real World:
Representation, Reinforcement and Dynamics
- -------------------------------------------------------------------------
HOW TO ORDER:
The publisher is offering a special service. Copies of this issue at a
price of $25 can be obtained from
Dr. F. van Drunen
Elsevier Science Publishers
Mathematics and Computer Science Section
P.O. BOX 103
1000 AC Amsterdam
The Netherlands
FAX: +31-10-5862-616
- -------------------------------------------------------------------------
Copies of the first three papers can be got from
GMD
c/o Sekretariat Z1.HLRZ
P.O. BOX 1240
D-5205 Sankt Augustin 1
West Germany
FAX +49 - 2241 - 142618
Heinz Muehlenbein
------------------------------
Subject: Neuron Digest Submission
From: erik%adams.llnl.gov@lll-lcc.llnl.gov (ERIK JOHANSSON)
Date: Thu, 31 May 90 09:11:42 -0700
I have subscribed to the digest for a couple of months now, but have not
submitted anything. Judging by the recent traffic, I feel now is the
time.
I am a research engineer at Lawrence Livermore National Laboratory,
working in the areas of signal and image processing and neural network
applications. We have a neural network working group of 10 - 12 people
which meets on a regular basis. Our applications and areas of interest
include: pattern recognition, parallel implementations of NNs using
systolic arrays, signal detection and estimation, image data analysis,
chemical data analysis (chemometrics), optimization methods, NN theory,
and architectures.
I am working on optimization methods and chemical data analysis. In
December of last year, I successfully applied the conjugate gradient
optimization method to backpropagation networks. The results were
excellent, and I'm almost finished with a paper comparing back
propagation with several different forms of the conjugate gradient method
(the paper would have been completed some time ago, but my work schedule
would not permit it). I should have it finished in about 1 - 2 weeks,
and as soon as I get a publication release from the lab, I will make
preprints available on the net. I am briefly summarizing the results in
this submission.
To begin with, I needed some kind of metric to use in comparing the
performance of conventional backpropagation (with momentum) against the
conjugate gradient method. I agree with recent statements in the digest
that the number of cycles or iterations is not a valid metric, since in
backpropagation the training set is passed through the network once per
iteration, whereas in the conjugate gradient method the training set can
be passed through several times in a single iteration. The majority of
the CPU time required to train a backpropagation network is spent
computing the error function and the gradient of the error function.
Computing the error function requires forward propagating the entire
training set through the network. Computing the gradient requires
backpropagating the outputs from each pattern in the training set
(computed during the forward propagation) through the network.
Conventional backpropagation requires one forward propagation and one
backpropagation per iteration. The conjugate gradient method will have
several forward and back propagations at each iteration. The number of
forward and back propagations may or may not be the same, depending on
the type of line search (1-D minimization) used. Some line searches use
both the function value and the gradient at each step, some use only the
function value, and some use a combinations of the two. It seems to me
that since most of the CPU time is spent computing the error function and
gradient, the logical metric for comparison is the total number of
function calculations AND the total number of gradient calculations
required to train a network.
Using this metric, I have compared backpropagation with the conjugate
gradient method. For the conjugate gradient method, I used a cubic
approximation line search which computes function values and gradient
values at each step. Therefore, the number of function and gradient
calculations are identical. Since this is also the case with
conventional backpropagation, I have reduced the original metric to just
'function evaluations' for simplicity.
I chose the parity problem as a benchmark test. I realize many people
feel the parity problem is somewhat trivial; however, the error surfaces
are very complex (I've looked at slices along search directions, and the
error functions are EXTREMELY nonlinear). This nonlinearity presents
quite a challenge to optimization techniques, and hence makes for a nice
benchmark. I tested both one and two hidden layer networks on 3, 4, and
5 bit parity problems (i.e. 3-3-1, 3-3-3-1, 4-4-1, 4-4-4-1, 5-5-1, and
5-5-5-1 architectures). I used a full training set for each test (2^nth
patterns), with a stopping criterion of 1.e-6 for the normalized system
error (the average of the pattern errors). In addition, the number of
function evaluations were limited to 50000 - the optimization terminated
if this limit was reached. The weights were initialized to random
numbers uniformly distributed between -0.5 and +0.5.
The backpropagation tests were performed using several combinations of
step size and momentum. I tested several versions of the conjugate
gradient method: Fletcher-Reeves, Polak-Ribiere, Hestenes-Steifel, and
Shanno's method. Each method was tested using several values of the
linesearch termination critrerion (too complex to explain here - the
paper will explain in detail). In addition, I ran the tests using pure
steepest descent with a linesearch. The results are summarized below;
only the best results for each test are shown.
We found the conjugate gradient method to be at least 15 times faster
than conventional backpropagation, and in most cases many times faster.
In addition, our experience involving problems of considerable size has
shown that the conjugate gradient method is much faster in general than
conventional backpropagation.
Test Results:
BP - Backpropagation
FR - Fletcher - Reeves
PR - Polak - Ribiere
HS - Hestenes - Steifel
SH - Shanno
SD - Steepest descent
3 bit parity, 1 hidden layer (3-3-1):
Method # Func. Evals Error speedup over BP
FR 264 9.48e-7 7.0
PR 123 8.29e-7 15.0
HS 121 7.02e-8 15.2
SH 191 5.45e-7 9.6
SD 616 9.91e-7 3.0
BP 1843 9.98e-7
(BP stepsize 0.9, momentum 0.9)
3 bit parity, 2 hidden layers (3-3-3-1)
Method # Func. Evals Error speedup over BP
FR 372 7.47e-7 8.3
PR 200 7.87e-7 15.4
HS 239 9.11e-7 12.9
SH 275 7.33e-7 11.2
SD 1148 9.69e-7 2.7
BP 3078 9.81e-7
(BP stepsize 0.9, momentum 0.9)
4 bit parity, 1 hidden layer (4-4-1)
Method # Func. Evals Error speedup over BP
FR 1617 9.55e-7 NA
PR 461 9.39e-7 NA
HS 306 8.59e-7 NA
SH 2079 8.71e-7 NA
SD 5505 9.99e-7 NA
BP 50000 1.60e-2 (did not converge)
4 bit parity, 2 hidden layers (4-4-4-1)
Method # Func. Evals Error speedup over BP
FR 737 9.88e-7 18.3
PR 401 7.01e-7 33.6
HS 429 9.09e-7 31.4
SH 560 9.96e-7 24.0
SD 1840 9.98e-7 7.3
BP 13462 9.90e-7
(BP stepsize 0.7, momentum 0.7)
5 bit parity, 1 hidden layer (5-5-1)
Method # Func. Evals Error speedup over BP
FR 2145 9.74e-7 11.7
PR 1966 7.76e-7 12.8
HS 3249 8.89e-7 7.7
SH 750 8.97e-7 33.4
SD 11894 9.99e-7 2.1
BP 25087 9.98e-7
(BP stepsize 0.9, momentum 0.9)
5 bit parity, 2 hidden layers (5-5-5-1)
Method # Func. Evals Error speedup over BP
FR 1561 9.42e-7 NA
PR 1343 9.91e-7 NA
HS 1165 9.56e-7 NA
SH 1890 9.86e-7 NA
SD 9500 9.99e-7 NA
BP 50000 8.00e-2 (did not converge)
In all of the above tests, some form of the conjugate gradient method was
always at least 15 times faster than backpropagation. In most cases,
most notably those where backprop did not converge, it is many times
faster.
The conjugate gradient method can get stuck in local minima; however, the
algorithm can be modified to detect this and use a simple pattern search
to get out of the minimum, and then continue with the conjugate gradient
method. In addition, my experience using the algorithm on pattern
recognition problems has shown that when the algorithm gets stuck, it is
usually due to a large "flat" plateau in the error surface where the
gradient becomes quite small, not a well defined local minimum. Again,
the use of a pattern search (a systematic search through the error space)
can resolve this problem.
In general, I find the conjugate gradient method to be quite superior to
conventional backpropagation. Indeed, from an optimization viewpoint, the
idea of using a fixed step size is not a good one: the move taken can
either be so small that it would take an exceedingly long time to
converge, or so large that the minimum is missed and the algorithm
oscillates about the minimum, converging very slowly. The linesearch in
the conjugate gradient method corrects this problem by finding the
minimum along a search direction at each iteration to an accuracy
specified by the user. As with many complex numerical algorithms, the
conjugate gradient method may require the use of double precision
variables (this is problem dependent), but the speedup is well worth the
small increase in computation.
The paper, which will be completed shortly, has a detailed tutorial
derivation of the conjugate gradient method, an explanationm of its
application to the backpropagation learning problem, and a complete
listing of all the test results.
I look forward to any comments or questions the digest readers may
have.
Sincerely,
Erik M. Johansson
Lawrence Livermore National Laboratory
PO Box 808, L-156
Livermore, CA 94550
erik@adams.llnl.gov
(415) 423-9255
Disclaimer: The opinions expressed herein are my own and do not
necessarily represent the views of Lawrence Livermore National
Laboratory, the University of California, or the U.S. Government.
------------------------------
Subject: UPDATED program info for: 6/16 NEURAL NETS FOR DEFENSE Conference
From: neuron-request@hplabs.hpl.hp.com
Date: Thu, 31 May 90 10:43:28 -0700
[[ Editor's Note: Comments directly relevant to the charter of this
Digest - artificial and natutal neural networks - are always welcome. A
reminder, however, that general pro-/anti- defense debates are
appropriate in a different forum. -PM ]]
UPDATED Program and Registration information for:
The Second Annual Conference on
NEURAL NETWORKS FOR DEFENSE
Conference Chair: Prof. Bernard Widrow
June 16th, 1990: ** The day before IJCNN **
San Diego Marriot, San Diego, CA
"Neural Networks for Defense" is organized to encourage and promote
the transfer of neural network technology to practical defense-
oriented applications.
PROGRAM (June 16th, 1990):
==========================
MORNING:__________________________________________________________________
8:15am-8:45am
Mark A. Gluck (Stanford University):
"Opening remarks"
Robert Kolesar (Deputy Director for Adv. Technology, Naval Ocean Systems
Center)
"Defense funding of neural networks:
A programmatic overview of 6.1 -> 6.3 efforts"
8:45-10:30 PANEL SYMPOSIUM: INTERNAL DOD LABORATORIES______________________
Steven Speidel (Naval Ocean Systems Center)
"A neural target locator"
Steven Anderson (Captain, USAF; Air Force Weapons Laboratory, KAFB,NM)
"Neural networks for signal processing and pattern recognition"
Steven K. Rogers (Major, USAF; Air Force Institute of Technology, WPAFB, OH)
"Artificial Neural Networks for Automatic Target Recognition"
David Andes (Director of Neural Network R&D, Naval Weapons Center, China Lake)
"Artificial neural computing at the Naval Weapons Center"
10:30-11:00am COFFEE BREAK
11:00-12:30 PANEL SYMPOSIUM: SBIR SUPPORT OF NN R&D:_________________________
Craig Will (Editor, Neural Network Review)
"An overview of neural network research in the SBIR program"
Vincent D. Schaper (Navy SBIR Manager)
"The DoD SBIR program"
Luis Lopez (US Army Strategic Defense Command)
"U.S. Army Strategic Defense Commands' SBIR Neural Network Programs"
Robert L. Dawes (President, Martingale Research Corporation)
"Observations on the SBIR program by a successful participant"
James Johnson (Regional Vice President, Netrologic)
"SBIRs: A contractor and government perspective"
12:30-2:00 AFTER-LUNCH SPEAKER: Bernard Widrow (Stanford University)
2:00-4:00
2:00-4:00 SESSION: PROGRESS IN DEFENSE APPLICATONS:__________________________
Edward Rosenfeld (Intelligence Newsletter)
"Overview of Industry efforts in Neural Networks for Defense"
David Hamilton (Senior Development Engineer, Raytheon Submarine Signal Div.)
"Neural Network Defense Applications within Raytheon"
Robert North (President, HNC, Inc.)
"Neural Network Defense Applications at HNC"
Rich Peer (Senior Manager, McDonnel Douglas)
"Neural Network Applications at McDonnell Douglas"
Donald F. Specht (Senior Scientist, Lockheed Research Laboratory)
"Hardware Implementation of Neural Networks"
Joseph Walkush (SAIC)
"Neural Networks for Defense and Security at SAIC"
4:00-4:30 -- COFFEE BREAK
4:30-5:30 PANEL SYMPOSIUM: FORGING TRANSITIONS BETWEEN UNIVERSITIES AND
FOR ADVANCED APPLICATIONS OF NN FOR DEFENSE_______________________
Thomas McKenna (Scientific Officer, Office of Naval Research"
"Navy Transition Paths from Basic to Applied Research"
James Anderson (Prof. of Cognitive Science, Brown University)
"Highs and lows: A case study"
Terrence Sejnowski (Institute for Neural Computation, UC, San Diego/Salk Institute)
"Case history of a successful university-industry cooperative venture"
REGISTRATION:
=============
This meeting is UNCLASSIFIED but limited to those with an
explicit "need-to-know" and a clear professional commitment to the
defense and security interests of the United States.
***** ATTENDANCE IS STRICTLY LIMITED TO U.S. CITIZENS ONLY. *********
NOTE: Special Registration Fee Discounts for DoD Employees &
University Scientists working on DoD 6.1 Research
For further information, or to register, contact:
-------------------------------------------------
Lynne Mariani, Registration Coordinator
Neural Networks for Defense
500 Howard St.
San Francisco, CA 94105
Phone: (415) 995-2471
FAX: (415) 995-2494
------------------------------
Subject: TIME-SENSITIVE - DoD small Business Innovation Research Program
From: will@ida.org (Craig Will)
Date: Thu, 31 May 90 16:29:36 -0400
Please get this out quickly because if its nature. Tks, Craig Will
Department of Defense
Small Business Innovation Research Program
The U. S. Department of Defense has announced an unusual mid-year
solicitation for proposals for the Small Business Innovation Research
(SBIR) program.
The list of topics included in these soliciations is of general
interest in that they reflect the increasing level of interest by mil-
itary agencies in neural network technology and their perception of
the kinds of applications they are interested in solving with the
technology.
In the current solicitation only the Army, Navy, and DARPA are
participating; the Air Force and SDIO are not. There are 11 topics
specifically targeting neural networks, and at least 2 more topics
that specifically mention neural networks as possible approaches that
might be used.
The program is in three Phases. Phase I awards are essentially
feasibility studies of 6 months and with a dollar amount of about
$50,000, intended for a one-half person-year effort. Phase I contrac-
tors compete for Phase II awards of 2 years in length and up to
$500,000, intended for 2 to 5 person-years of effort. Phase III is
the commercial application phase of the research.
Proposals must be no longer than 25 pages in length, including
the cover sheet, summary, cost proposal, resumes and any attachments.
Deadline for proposals is July 2, 1990.
Principal investigators must be employees (50% or more time) of
small business firms. The program encourages small businesses to make
use of university-based and other consultants when appropriate.
The topics listed below are those that are specifically targeted
to or mention neural networks specifically as possible approaches:
Many other topics describe application and problem areas that could
also be solved with these approaches.
A90-215 Neural Network Based Classification Demonstration of
Vehicle from Laser Radar and Infrared Data. (Exploratory Develop-
ment). Development of methods to classify military vehicles (e.g.,
distinguish a tank from a truck).using laser radar and infrared
images. Phase I involves development of a neural network vehicle
classifier based on laser rader data. Phase II involves integrating
laser radar and infrared imagery together ``to demonstrate a multi-
sensor classifier showing high probability of classification and low
false-alarm rate."
A90-227 Application of Neural Networks to Command and Control.
(Exploratory Development). Apply neural networks to ``information
processing and decision making in a command and control operating
environment." Neural networks are seen as having ``promise for pro-
viding significant improvements in reaction times by providing quantum
leaps in the ability to quickly process information and perform deci-
sion aid tasks". Phase I involves development of a plan and demons-
tration software. Phase II involves the ``development and testing of
a working system in a field environment."
A90-236 VHSIC Application to Neural Network Systems. (Explora-
tory Development). Investigate application of Very High Speed
Integrated Circuit technology to the problem of electronically imple-
menting neural networks that require very large connectivity. Phase I
involves ``identification of specific very large Neural Networks which
could be implemented using VHSIC technology, and development of a
demonstratable design". Phase II involves ``fabrication and test of a
hardware brassboard for one or more specific applications".
A90-246 Neural Network Sensor Fusion for Apache Escort Jammer
Countermeasure Systems. (Exploratory Development). Develop a system
using neural networks that can provide improved awareness of the
situation and management of electronic countermeasures for a pilot in
an electronic warfare environment. The system should provide improved
accuracy and/or increased processing speed. Phase II involves defin-
ing the network architecture and training, and possibly simulate and
test the network in a demonstration setting. Phase II involves simu-
lating the network ``in a manner than approaches real-time perfor-
mance," test it, embed the net in an Apache Escort system [that
manages countermeasures] and demonstrate its capabilities.
A90-428 Neural Network Software / Hardware for Directed and
Kinetic Energy Anti-satellite (ASAT) Weapons System. (Basic
Research). Development of ``new and innovating neural network algo-
rithms and architectures that will aid in developing a real-time,
economical and reliable kinetic and directed energy antisatellite
(ASAT) weapons system." Problems include ``weapons pointing, beam
control, acquisition, tracking, sensor focal planes, signal and data
processing, guidance and control algorithms, control of cryocoolers,
[and] array image processing". Phase I involves demonstration of con-
cept by simulation or prototype development. Phase II involves
``incorporating the principle developed in Phase I into a prototype"
or showing proof of feasibility for a demonstration phase.
N90-317 Neural Network Applications for Nondestructive Inspec-
tion of Aircraft. (Research). Develop new approaches to automatic
inspection of aircraft that is nondestructive. X-ray, ultra-sonic,
eddy-current, and acoustic sensors are typically used to detect such
flaws as cracks, voids, porosity, incudisions, delaminations, pits,
corrosion, etc. Neural networks are seen as a way of integrating this
sensor data so as to recognize specific flaws. Neural network
approaches for ``robotic sensor placement" are also of interest.
N90-351 Artificial Intelligence and Neural Network Technologies
for Mission Planning and Execution Applications. (Exploratory
Development). Application of AI and neural network techniques to help
automate or assist in the planning and execution of missions. Goal is
to develop techniques that can result in a fielded system within
``five to fifteen years". Phase I involves assessment of mission
planning and identification of promising technologies. Phase II
involves building a demonstration system using the concepts developed
in Phase I.
N90-372 Neural Network Applications to Flight Control (Explora-
tory Development). Investigate potential for using neural networks to
help stabilize a high-performance aircraft in flight that is subject
to changing environmental conditions as well as instabilities result-
ing from its own dynamics. Phase I involves development and demons-
tration by simulation of a network architecture that can stabilize an
aircraft. Phase II expands to ``provide robust control and stabiliza-
tion features in a distributed neural network having excellent sur-
vivability and fault tolerant properaties."
N90-384 LSI (Large System Integrated) Neural Networks for Asso-
ciative Memory Arrays. (Advanced Development). Investigate neural
network architectures and hardware implementation techniques for
contstruction of ``an associative memory array" based on artificial
neural networks to ultimately achieve 10 ** 11 and 10 ** 12 intercon-
nects / second" for application to video and audio ``matching" prob-
lems. Phase I involves investigating materials, devices, architec-
tures, and modeling. Phase II involves ``a technology demonstration
illustrating the several orders of magnitude improvement offered by
the physical use of VLSI associative memory arrays based on ANNs."
DARPA 90-115 Unique Applications for Artificial Neural Networks.
(Exploratory Development). Identification and development of applica-
tions that can show ``outstanding potential to demonstrate particular
advantages of artificial neural networks...in systems that perform
challenging tasks that are at or beyond the limits of capability of
conventional information processing systems." Applications that help
discover ``important unusual and under-recognized `niches'" for neural
networks are particularly sought. Phase I involves a providing a con-
ceptual design and laboratory demonstration. Phase I involves
building a compact prototype system.
DARPA 90-124 Artificial Neural Network Target Recognition
Demonstration. (Basic Research). Develop hardware for implementing a
specific neural network algorithm that has been developed by the Army
for object extracting-classifying pixels in an image into candidate
regions suggesting objects. Details of the algorithm will be fur-
nished by DARPA ``as required". Phase I involves design and demons-
tration of a candidate hardware approach that shows scalability and
real-time operation. Phase II involves building a full-scale, real-
time hardware system that can process real images as a laboratory
demonstration. ``
Topics A90-430 and A90-473 also specifically mention the possi-
blity of using neural network approaches, while many other topics are
also presumably candidates.
For more details on the July, 1990 soliciation obtain a copy of
the SBIR Program Solicitation book (229 pages in length) from the
Defense Technical Information Center: Attn: DTIC/SBIR, Building 5,
Cameron Station, Alexandria, Virginia 22304-6145. Telephone: Toll-
free, (800) 368-5211. For Virginia, Alaska, Hawaii: (202) 274-6902.
Craig A. Will
Computer and Software Enginering Division
Institute for Defense Analyses
Alexandria, VA
will@ida.org
------------------------------
End of Neuron Digest [Volume 6 Issue 37]
****************************************