neuron-request@HPLABS.HP.COM ("Neuron-Digest Moderator Peter Marvit") (08/05/89)
Neuron Digest Friday, 4 Aug 1989
Volume 5 : Issue 33
Today's Topics:
TR available: Optimum Supervised Learning
Connection Science Journal
CRG-TR-89-3
TR available: Contribution Analysis
wanted: guest researcher
Machine Learning Mailing List.
Neural Network Session at SME Conference AUTOFACT
Postdoc position at Bellcore
Preprint Available: Configural-cue Network Model of Animal& Human Learning
Ph.D. thesis available
call for papers
Send submissions, questions, address maintenance and requests for old issues to
"neuron-request@hplabs.hp.com" or "{any backbone,uunet}!hplabs!neuron-request"
Use "ftp" to get old issues from hplpm.hpl.hp.com (15.255.176.205).
------------------------------------------------------------
Subject: TR available: Optimum Supervised Learning
From: Manoel Fernando Tenorio <tenorio@ee.ecn.purdue.edu>
Date: Tue, 06 Jun 89 17:05:00 -0500
The Tech Report below will be available by June, 15. Please do not reply to
this posting. Send all you requests to jld@ee.ecn.purdue.edu
Self Organizing Neural Network for Optimum Supervised Learning
Manoel Fernando Tenorio Wei-Tsih Lee
School of Electrical Engineering School of Electrical Engineering
Purdue University Purdue University
W. Lafayette, IN. 47907 W. Lafayette, IN. 47907
tenorio@ee.ecn.purdue.edu lwt@ed.ecn.purdue.edu
Summary
Current neural network algorithms can be classified by the following
characteristics: the architecture of the network, the error criteria used,
the neuron transfer function, and the algorithm used during learning. For
example: in the case of back propagation, one would classify the algorithm
as a fixed architecture (feedforward in most cases), using a MSE criteria,
and a sigmoid function on a weighted sum of the input, with the Generalized
Delta Rule performing a gradient descent in the weight space. This
characterization is important in order to assess the power of such
algorithms from a modeling viewpoint. The expressive power of a network is
intimately related with these four features.
In this paper, we will discuss a neural network algorithm with noticeably
different characteristics from current networks. The Self Organizing Neural
Network (SONN) [TeLe88] is an algorithm that through a search process
creates the network necessary and optimum in the sense of performance and
complexity. SONN can be classified as follows. The network architecture is
constructed through a search using Simulated Annealing (SA),and it is
optimum in that sense. The error criteria used is a modification of the
Minimum Description Length Criteria called the Structure Estimation Criteria
(SEC); it takes into account both the performance of the algorithm and the
complexity of the structure generated. The neuron transfer function is
individually chosen from a pool of functions, and the weights are adjusted
during the neuron creation. This function pool can be selected with a priori
knowledge of the problem, or simply use a class of non-linearities shown to
be general enough for a wide variety of problems.
Although the algorithm is stochastic in nature (SA), we show that its
performance is extremely high both in comparative and absolute terms. In
[TeLe88], we have used SONN as an algorithm to identify and predict chaotic
series, particularly the Mackey-Glass equation [LaFa87, Mood88] was used.
For comparison, the experiments of using Back Propagation for this problem
were replicated under the same computational environment. The results
indicated that for about 10% of the computational effort, the SONN delivered
a 2.11 times better model (normalized RMSE). Some inherited aspects of the
algorithm are even more interesting: there were 3.75 times less weights, 15
times less connections, 6.51 times less epochs over the data set, and only
1/5 of the data was fed to the algorithm. Furthermore, the algorithm
generates a symbolic representation of the network which can be used to
substitute it, or be used for the analysis of the problem.
******************************************************************************
We have further developed the algorithm, and although not part of the report
above, it will be part of a paper submitted to NIPS'89. There, some major
improvements on the algorithm are reported. The same chaotic series problem
can now run with 26.4 less epochs over the data set that BP, and have
generated the same model in about 18.5 seconds of computer time. (This is
down from 2 CPU hours in a Gould NP1 Powernode 9080).
Performance on a Sun 3-60 was sightly over 1 minute. These performance
figures include the use of an 8 times larger function pool; the final
performance now independs of the size of the pool.
Other aspects of the algorithm are also important considering. Because of
its stochastic nature, no two runs of the algorithm should be the same. This
can become a hindrance if a suboptimal solution is desired, since at every
run the set of suboptimal models can be different. A report on modifications
of the original SONN to run on an A* search are presented. Since the
algorithm generates partial structures at each iteration, the learning
process is only optimized for the structure presently generated. If such
substructure is used as a part of a larger structure, then no provision is
made to readjust its weights making the final model slightly stiff. A
provision for melting the structure (parametric readjustment) is also
discussed. Finally, the combination of symbolic processing with this
numerical method can lead to construction of AI-NN based methods for
supervised and unsupervised learning. The ability of SONN to take symbolic
constraints and produce symbolic information can make such a system
possible. Implications of this design are also explored.
[LaFa87] - Alans Lapedes and Robert Farber, How Neural Networks Work, TR
LA-UR-88-418, Los Alamos, 1987.
[Mood88] - J. Moody, Fast Learning in Multi-Resolution Hierarchies, Advances
in Neural Information Processing Systems, D. Touresky, Ed., Morgan Kaufmann,
1989 (NIPS88).
[TeLe88] - M. F. Tenorio and W-T Lee, Self Organizing Neural Networks for
the Identification Problem, Advances in Neural Information Processing
Systems, D. Touresky, Ed., Morgan Kaufmann, 1989 (NIPS88).
------------------------------
Subject: Connection Science Journal
From: Lyn Shackleton <lyn@CS.EXETER.AC.UK>
Date: Wed, 07 Jun 89 13:36:25 -0000
ANNOUNCEMENT
Issue 1. of the new journal CONNECTION SCIENCE has just gone to press and
Issue 2. will follow shortly. The editors are very pleased with the response
they have received and would welcome more high quality submissions or
theoretical notes.
VOLUME 1 NUMBER 1 CONTENTS
Michael C Mozer & Paul Smolensky
'Using Relevance to Reduce Network Size Automatically'
James Hendler
'The Design and Implementation of Symbolic Marker-Passing Systems'
Eduardo R Caianello, Patrik E Eklund & Aldo G S Ventre
'Implementations of the C-Calculus'
Charles P Dolan & Paul Smolensky
'Tensor Product Production System: A Modular Architecture and
Representation'
Christopher J Thornton
'Learning Mechanisms which Construct Neighbourhood Representations'
Ronald J Williams & David Zipser
'Experimental Analysis of the Real-Time Recurrent Learning
Algorithm'
Editor: Dr NOEL E SHARKEY, Centre for Connection Science, Dept of Computer
Science, University of Exeter, UK
Associate Editors:
Andy CLARK (University of Sussex, Brighton, UK)
Gary COTTRELL (University of California, San Diego, USA)
James A HENDLER (University of Maryland, USA)
Ronan REILLY (St Patrick's College, Dublin, Ireland)
Richard SUTTON (GTE Laboratories, Waltham, MA, USA)
FORTHCOMING IN VOLUMES 1 & 2
Special Issue on Natural Language, edited by Ronan Reilly & Noel Sharkey
Special Issue on Hybrid Symbolic/Connectionist Systems, edited by
James Hendler
For further details please contact.
lyn shackleton
(assistant editor)
Centre for Connection Science JANET: lyn@uk.ac.exeter.cs
Dept. Computer Science
University of Exeter UUCP: !ukc!expya!lyn
Exeter EX4 4PT
Devon BITNET: lyn@cs.exeter.ac.uk.UKACRL
U.K.
------------------------------
Subject: CRG-TR-89-3
From: Carol Plathan <carol@ai.toronto.edu>
Date: Fri, 16 Jun 89 15:33:15 -0400
The Technical Report CRG-TR-89-3 by Hinton and Shallice (May 1989) can be
obtained by sending me your full mailing address. An abstract of this Report
follows:
LESIONING A CONNECTIONIST NETWORK: INVESTIGATIONS OF ACQUIRED DYSLEXIA
-----------------------------------------------------------------------
Geoffrey E. Hinton Tim Shallice
Department of Computer Science MRC Applied Psychology Unit
University of Toronto Cambridge, UK
ABSTRACT:
- --------
A connectionist network which had been trained to map orthographic
representation into semantic ones was systematically 'lesioned'. Wherever
it was lesioned it produced more Visual, Semantic, and Mixed visual and
semantic errors than would be expected by chance. With more severe lesions
it showed relatively spared categorical discrimination when item
identification was not possible. Both phenomena are qualitatively similar
to those observed in neurological patients. The error pattern is that
characteristically found in deep dyslexia. The spared categorical
discrimination is observed in semantic access dyslexia and also in a form of
pure alexia. It is concluded that the lesioning of connectionist networks
may produce phenomena which mimic non-transparent aspects of the behaviour
of neurological patients.
------------------------------
Subject: TR available: Contribution Analysis
From: Dennis Sanger <sanger@boulder.Colorado.EDU>
Date: Thu, 22 Jun 89 09:56:59 -0600
University of Colorado at Boulder Technical Report CU-CS-435-89 is
now available:
Contribution Analysis: A Technique for Assigning
Responsibilities to Hidden Units in Connectionist Networks
Dennis Sanger
AT&T Bell Laboratories and the University of Colorado at Boulder
ABSTRACT: Contributions, the products of hidden unit
activations and weights, are presented as a valuable tool
for investigating the inner workings of neural nets. Using
a scaled-down version of NETtalk, a fully automated method
for summarizing in a compact form both local and distributed
hidden-unit responsibilities is demonstrated. Contributions
are shown to be more useful for ascertaining hidden-unit
responsibilities than either weights or hidden-unit
activations. Among the results yielded by contribution
analysis: for the example net, redundant output units are
handled by identical patterns of hidden units, and the
amount of responsibility a hidden unit takes on is inversely
proportional to the number of hidden units.
Please send requests to conn_tech_report@boulder.colorado.edu.
------------------------------
Subject: wanted: guest researcher
From: unido!gmdzi!zsv!joerg@uunet.UU.NET (Joerg Kindermann)
Date: Fri, 23 Jun 89 16:20:55 +0200
If you are a postgraduate student of scientist with a strong background in
neural networks, we are interested to get in touch with you:
We are a small team (5 scientists plus students) doing research in neural
networks here at the GMD. Currently we are applying to get funding for
several positions of guest researchers. But: we need strong arguments (i.e.
good people who are interested in a stay) to actually get the money.
Our research interests are both theoretical and application oriented. The
main focus is on temporal computation (time series analysis) by neural
networks. We are using multi-layer recurrent networks and gradient learning
algorithms (backpropagation, reinforcement). Applications are speech
recognition, analysis of medical data (ECG, ...), and navigation tasks for
autonomous vehicles (2-D simulation only). A second research direction is
the optimization of neural networks by means of genetic algorithms. We are
using both SUN3s and a parallel Computer (64 cpu transputer-based).
So, if you are interested, please write a letter, indicating your background
in neural networks and preferred dates for your stay.
Dr. Joerg Kindermann
Gesellschaft fuer Mathematik und Datenverarbeitung mbH (GMD)
Postfach 1240 email: joerg@gmdzi.uucp
D-5205 St. Augustin 1, FRG phone: (+49 02241) 142437
------------------------------
Subject: Machine Learning Mailing List.
From: Michael Pazzani <pazzani@ICS.UCI.EDU>
Date: Fri, 21 Jul 89 11:49:40 -0700
Could you include this message in the next neuron mailing?
Machine Learning Mailing List
A new mailing list, ML-LIST is being formed. ML-LIST will be devoted to
discussions of issues concerning machine learning and announcements of
interest to the machine learning community, including neural network
learning algorithms. The list is moderated by Michael Pazzani of University
of California, Irvine. The motivation for the list is to promote informal
discussions of research topics and methodology. The list will be mailed out
approximately once a week.
To reduce the volume of information and to maintain a forum for meaningful
interaction, many topics not appropriate to the scientific treatment of
computational approaches to learning will be avoided. These include:
* Questions on programming languages
* Extended philosophical debates
* Flames
* Why are all the machine learning faculty at Irvine so short?
* Messages with many spelling and grammar errors
Messages will be encouraged on topics such as:
* Discussions of research results
* Discussions of research methodology
* Brief announcements of the availability of Tech Reports
* Abbreviated Calls for Papers & Conference Announcements
In order to avoid unnecessary mail traffic, local redistribution of the
ML-LIST, is encouraged. To subscribe to ML-LIST send mail including an
e-mail address to:
ml-request@ics.uci.edu
Before rushing to subscribe, talk to your System Administrator about your
local facilities for mail redistribution.
Submissions to ML-LIST are e mailed to:
ml@ics.uci.edu
Thanks
Michael Pazzani
------------------------------
Subject: Neural Network Session at SME Conference AUTOFACT
From: David Kanecki <kanecki@vacs.uwp.wisc.edu>
Date: Tue, 25 Jul 89 16:46:18 -0500
I received a flyer from the Society of Manufacturing Engineers(SME) about
the AUTOFACT conference being held in Detroit, Michigan on October 30th thru
November 2nd.
Session 29 on Neural Networks includes talks as "Neural Network Basics", "An
Artificial Neural System for Reading Printed Credit Card Numbers", "Optical
Neural Networks for Process Control" and "Neurocomputing: Applications for
an Important New Data Processing Technology"
The theme for the conference is partnership for integration which in my last
paper to the neuron digest was the basis for the paper I sent.
To receive more information contact:
AUTOFACT Conferene
SME
Conference Regiostrar
Technical Activities
One SME Drive
P.O. Box 930
Dearborn, MI 48121-0930
Phone: (313)-271-1080
Fax: (313)-271-2861
------------------------------
Subject: Postdoc position at Bellcore
From: Joshua Alspector <josh@flash.bellcore.com>
Date: Fri, 28 Jul 89 16:41:32 -0400
POSTDOCTORAL POSITION IN NEURAL NET RESEARCH
The Adaptive Systems Research Group at Bellcore, Morristown, NJ is looking
for a postdoctoral researcher for a period of 1 - 2 years starting
approximately November, 1989. Bellcore has a stimulating research
environment for neural computation with active programs in neural network
theory and analysis, in applications such as speech recognition and expert
systems, and in optical and electronic implementation. This is an excellent
opportunity for a researcher to be exposed to this environment while
contributing to the effort in VLSI implementation.
A test chip that implements a neural network based learning algorithm
related to the Boltzmann machine has been designed, fabricated and tested.
The next step is to implement a useful, multiple-chip system that can learn
to solve difficult artificial intelligence problems. We will extend our
study of electronic implementation issues to large scale systems using a
three-pronged approach:
1) Further development of learning algorithms and architectures suitable for
modular VLSI implementation. To be useful, algorithms must be implementable
because learning by example takes too long using serial computer
simulations. Therefore, the algorithms should take into account the
constraints imposed by VLSI.
2) Functional simulation of large scale hardware systems using benchmark
test problems. We will build a computer-based development system for
testing algorithms in software. This will be composed of software modules,
some of which eventually will be replaced by hardware learning modules. A
computation module may be run on a remote parallel machine. This will serve
as a platform for algorithm development, will perform functional simulation
of a hardware system before design, and also will be the front end for
testing the chips and boards after fabrication.
3) Design and fabrication of prototype chips suitable for inclusion in such
systems. As a first step in the development of large scale, modular VLSI
systems, our learning test chip will be expanded to contain more neurons and
synapses and to enable construction of a multichip system. This system
would be taken to board-level design and fabrication. Evaluation will
involve a speed comparison using a variety of benchmarks of three neural
network implementations: software on a serial machine, software on a general
purpose parallel machine, and special purpose neural hardware using the
board level system we build. Chip and board design will be carried out using
a combination of sophisticated VLSI CAD tools.
The successful candidate should be involved in many aspects of this work
including the design of algorithms and architectures for VLSI neural
implementation, computer programming to simulate and test the existing and
proposed neural architectures, and the design of analog and digital chips to
implement them. He or she should be capable of doing independent publishable
research in neural network learning theory, in parallel software simulation,
in applications of neural information processing, or in VLSI implementations
of neural network learning models.
Please enclose a resume, a copy of a recent paper, and the names, addresses,
and phone numbers of three references. Send applications to:
Joshua Alspector
Bellcore, MRE 2E-378
445 South St.
Morristown, NJ 07960-1910
------------------------------
Subject: Preprint Available: Configural-cue Network Model of Animal& Human Learning
From: gluck@psych.Stanford.EDU (Mark Gluck)
Date: Mon, 31 Jul 89 09:59:01 -0700
Pre-print of short conference paper available:
Eleventh Annual Conference of the Cognitive Science Society,
Ann Arbor, MI. August 16-19, 1989.
A CONFIGURAL-CUE NETWORK MODEL OF
ANIMAL AND HUMAN ASSOCIATIVE LEARNING
Mark A. Gluck
Gordon H. Bower
Michael R. Hee
Dept. of Psychology
Stanford University
Jordan Hall; Bldg. 420
Stanford, CA 94305
(415) 725-2434
Email: gluck@psych.stanford.edu
ABSTRACT
We test a configural-cue network model of human classification and
recognition learning based on Rescorla & Wagner's (1972) model of classical
conditioning. The model extends the stimulus representation assumptions
from our earlier one-layer network model (Gluck & Bower, 1988b) to include
pair-wise conjunctions of features as unique cues. Like the exemplar
context model of Medin & Schaffer (1978), the representational assumptions
of the configural-cue network model embody an implicit exponential decay
relationship between stimulus similarity and and psychological (Hamming)
distance, a relationship which has received substan- tial independent
empirical and theoretical support (Shepard, 1957, 1987). In addition to
results from animal learning, the model accounts for several aspects of
complex human category learning, including the relationship between category
similarity and linear separability in determining classification difficulty
(Medin & Schwanenflugel, 1981), the relationship between classif- ication
and recognition memory for instances (Hayes-Roth & Hayes-Roth, 1977), and
the impact of correlated attributes on classification (Medin, Altom,
Edelson, & Freko, 1982).
------------------------------
Subject: Ph.D. thesis available
From: swain@cs.rochester.edu
Date: Tue, 01 Aug 89 12:03:00 -0400
[[ Editor's Note: Please see the end of this annoucnement that this
publication costs $7.50! -PM]]
The following Ph.D. thesis now available:
PARALLEL OBJECT RECOGNITION FROM STRUCTURE
(THE TINKERTOY PROJECT)
Paul R. Cooper
Department of Computer Science
University of Rochester
Technical Report 301
July 1989
Abstract:
This thesis examines the problem of recognizing structurally composed
objects. The task is the recognition of Tinkertoys --- objects whose
identity is defined solely by the spatial relationships between simple
parts. Ultimately, a massively parallel framework incorporating a principled
treatment of uncertainty and domain dependence is developed to address the
problem.
The basic architecture of the solution is formed by posing structure
matching as a part-wise correspondence problem in a labelling framework,
then applying the unit/value principle. The solution is developed
incrementally. Complexity and correctness analyses and implementation
experiments are provided at each phase.
In the first phase, a special purpose network implementing discrete
connectionist relaxation is used to topologically discriminate between
objects. In the second step, the algorithm is generalized to a massively
parallel formulation of constraint satisfaction, yielding an arc consistency
algorithm with the fastest known time complexity. At this stage the
formulation of the application problem is also generalized, so geometric
discrimination can be achieved. Developing an implementation required
defining a method for the domain specific optimization of the parallel arc
consistency algorithm. The optimization method is applicable to arbitrary
domains.
In the final phase, the solution is generalized to handle uncertain input
information and statistical domain dependence. Segmentation and recognition
are computed simultaneously by a coupled Markov Random Field. Both problems
are posed as labelling problems within a unified high-level MRF
architecture. In the segmentation subnet, evidence from the image is
combined with clique potentials expressing both qualitative {\em a priori}
constraints and learnable domain dependent knowledge. Matching constraints
and coupling constraints complete the definition of the field. The
effectiveness of the framework is demonstrated in experiments involving the
traditionally difficult problems of occlusion and accidental alignment.
============
TO ORDER, send requests to tr@cs.rochester.edu or physical mail to:
Technical Reports Librarian, Department of Computer Science, University of
Rochester, Rochester, NY 14627. The cost is $7.25. Make checks payable to
the University of Rochester.
------------------------------
Subject: call for papers
From: <OOMIDVAR%UDCVAX.BITNET@CORNELLC.cit.cornell.edu>
Date: Fri, 04 Aug 89 11:18:00 -0400
Would you please post the following call for papers:
Call For Papers
International Journal of Computer Aided VLSI Design
The above journal will publish a special issue on Neural Networks. Authors
are invited to submit original manuscripts describing the recent advances in
modelling, design, development, and VLSI Implementation of neural networks.
Other suggested topics are neurocomputers, and their applications in the
areas of vision, speech, self organization and control. Submit five copies
of full-length paper fifteen to twenty pages long by the end of August 1989
to Dr. Omid M. Omidvar, Department of Computer Science, University of the
District of Columbia, 4200 Connecticut Avenue, N.W., Washington, D.C. 20008.
Phone (202) 282-7345. Fax: (202) 282-3677. email: oomidvar@udcvax.bitnet.
------------------------------
End of Neurons Digest
*********************