ethem@ICSI.Berkeley.EDU (Ethem Alpaydin) (05/22/91)
learning. The following TR is available by anonymous net access at icsi-ftp.berkeley.edu (128.32.201.55) in postscript. Instructions to ftp and uncompress follow text. Hard copies may be requested by writing to either of the addresses below: ethem@icsi.berkeley.edu Ethem Alpaydin ICSI 1947 Center St. Suite 600 Berkeley CA 94704-1105 USA ------------------------------------------------------------------------------ GAL: Networks that grow when they learn and shrink when they forget Ethem Alpaydin International Computer Science Institute Berkeley, CA TR 91-032 Abstract Learning when limited to modification of some parameters has a limited scope; the capability to modify the system structure is also needed to get a wider range of the learnable. In the case of artificial neural networks, learning by iterative adjustment of synaptic weights can only succeed if the network designer predefines an appropriate network structure, i.e., number of hidden layers, units, and the size and shape of their receptive and projective fields. This paper advocates the view that the network structure should not, as usually done, be determined by trial-and-error but should be computed by the learning algorithm. Incremental learning algorithms can modify the network structure by addition and/or removal of units and/or links. A survey of current connectionist literature is given on this line of thought. ``Grow and Learn'' (GAL) is a new algorithm that learns an association at one-shot due to being incremental and using a local representation. During the so-called ``sleep'' phase, units that were previously stored but which are no longer necessary due to recent modifications are removed to minimize network complexity. The incrementally constructed network can later be finetuned off-line to improve performance. Another method proposed that greatly increases recognition accuracy is to train a number of networks and vote over their responses. The algorithm and its variants are tested on recognition of handwritten numerals and seem promising especially in terms of learning speed. This makes the algorithm attractive for on-line learning tasks, e.g., in robotics. The biological plausibility of incremental learning is also discussed briefly. Keywords Incremental learning, supervised learning, classification, pruning, destructive methods, growth, constructive methods, nearest neighbor. -------------------------------------------------------------------------- Instructions to ftp the above-mentioned TR (Assuming you are under UNIX and have a postscript printer --- messages in parantheses indicate system's responses): ftp 128.32.201.55 (Connected to 128.32.201.55. 220 icsi-ftp (icsic) FTP server (Version 5.60 local) ready. Name (128.32.201.55:ethem):)anonymous (331 Guest login ok, send ident as password. Password:)(your email address) (230 Guest login Ok, access restrictions apply. ftp>)cd pub/techreports (250 CWD command successful. ftp>)bin (200 Type set to I. ftp>)get tr-91-032.ps.Z (200 PORT command successful. 150 Opening BINARY mode data connection for tr-91-032.ps.Z (153915 bytes). 226 Transfer complete. local: tr-91-032.ps.Z remote: tr-91-032.ps.Z 153915 bytes received in 0.62 seconds (2.4e+02 Kbytes/s) ftp>)quit (221 Goodbye.) (back to Unix) uncompress tr-91-032.ps.Z lpr tr-91-032.ps Happy reading, I hope you'll enjoy it. +-----------------------------------------------------------------------------+ | Ethem Alpaydin | DON'T | | International Computer Science Institute | WORRY | | 1947 Center St. #600 | THE | | Berkeley CA 94704-1105 | HAPPY | | (415)643-9153 | | +-----------------------------------------------------------------------------+