WHITELEY-J@osu-20.ircc.ohio-state.edu (J. Whiteley) (10/02/90)
I'm looking for a reference which shows that two layers are sufficient to approximate any function using a BP network. All I've dug up so far is: Cybenko, G. Approximation by superpositions of a sigmoidal function. Unpublished manuscript, Oct. 1988. Can someone point me to some other references. Also, is Cybenko's (or any other of the references I'm looking for) proof limited to networks which use the sigmoid or is it more general similar to Kolmogorov's proof for three layers? Many thanks for any help!!! --Rob Whiteley (whiteley-j@osu-20.ircc.ohio-state.edu)
minsky@media-lab.MEDIA.MIT.EDU (Marvin Minsky) (10/03/90)
WHITELEY-J@osu-20.ircc.ohio-state.edu (J. Whiteley) asks > I'm looking for a reference which shows that two layers are > sufficient to approximate any function using a BP network. All > I've dug up so far is ... Any boolean function has a disjunctive (and a conjunctive) normal form hence can be expressed exactly by two layers, one with threshold 1 and the other with threshold N, for the number of inputs. However, if the fan-out is less than N, then the question is much more serious, and you have to read "Perceptrons", etc.
guansy@cs.tamu.edu (Sheng-Yih Guan) (10/03/90)
It has been proved (see, for instance, Funahashi, 1989; Moore and Poggio, 1988) that a network with two layers of hidden sigmoidal units can approximate arbitrarily well any continuous function. It is interesting to notice that this statement still holds true if there is just one hidden layer (Carrol and Dickinson, 1989; Cybenko, 1989; Funahashi, 1989). S. M. Carrol and B. W. Dickinson. Construction of neural nets using the Radon transform. In Proceedings of the International Joint Conference on Neural Networks, pages I-607-I-611, Washinton D. C., June 1989. IEEE TAB Neural Network Committee. K. Funahashi. On the approximate realization of continuous mappings by neural networks. Neural Networks, 2:183-192, 1989. -Stanley _ _ _ ___________ | \ /_| / / Visualization Lab /____ ____/ \ \ // / / Computer Science Dept / / _ _ _ _ | | // / / Texas A&M University / / / | | \ / | | | || | |// / / College Station, TX 77843-3112/ / / /| | \//|| | | || / / / /____ Tel: (409)845-0531 / / / -|| | |\/ || | !_!| !__/ /______/ stanley@visual1.tamu.edu /_/ /_/ || !_! || !____!
smagt@fwi.uva.nl (Patrick van der Smagt) (10/03/90)
In article <12626564934017@osu-20.ircc.ohio-state.edu> WHITELEY-J@osu-20.ircc.ohio-state.edu (J. Whiteley) writes: >I'm looking for a reference which shows that two layers are sufficient >to approximate any function using a BP network. All I've dug up so >far is: > >Cybenko, G. Approximation by superpositions of a sigmoidal function. >Unpublished manuscript, Oct. 1988. > There are three articles proving this. The first one is the most complete: %A K. Hornik %A M. Stinchcombe %A H. White %T Multilayer feedforward networks are universal approximators %J Neural Networks %V 2 %N 5 %D 1989 %P 359--366 %A G. Cybenko %T Approximation by superpositions of a sigmoidal function %J Mathematics of Control, Signals, and Systems %V 2 %N 4 %D 1989 %P 303--314 %A K.-I. Funahashi %T On the approximate realization of continuous mappings by neural networks %J Neural Networks %V 2 %N 3 %D 1989 %P 193--192 Furthermore, the next paper proves something similar but now for hidden units with a Gaussian activation function (very, very useful for function approximation; I experience faster learning and greater accuracy, although be careful with initial weight values and learning rates): %A E. J. Hartman %A J. D. Keeler %A J. M. Kowalski %T Layered Neural Networks with Gaussian Hidden Units as Universal Approximations %J Neural Computations %V 2 %N 2 %D Summer 1990 %P 210-215 Send me an email if you want a proof for binary networks (from Perceptrons). Patrick van der Smagt /\/\ \ / Organization: Faculty of Mathematics & Computer Science / \ University of Amsterdam, Kruislaan 409, _ \/\/ _ NL-1098 SJ Amsterdam, The Netherlands | | | | Phone: +31 20 525 7466 | | /\/\ | | Telex: 10262 hef nl | | \ / | | Fax: +31 20 592 5155 | | / \ | | email: smagt@fwi.uva.nl | | \/\/ | | | \______/ | \________/ /\/\ \ / / \ \/\/ ``The opinions expressed herein are the author's only and do not necessarily reflect those of the University of Amsterdam.''