[comp.ai.neural-nets] Reference for sufficiency of two layers

WHITELEY-J@osu-20.ircc.ohio-state.edu (J. Whiteley) (10/02/90)

I'm looking for a reference which shows that two layers are sufficient
to approximate any function using a BP network.  All I've dug up so
far is:

Cybenko, G.  Approximation by superpositions of a sigmoidal function.
Unpublished manuscript, Oct. 1988.

Can someone point me to some other references.  Also, is Cybenko's (or 
any other of the references I'm looking for) proof limited to networks
which use the sigmoid or is it more general similar to Kolmogorov's
proof for three layers?

Many thanks for any help!!!

--Rob Whiteley
  (whiteley-j@osu-20.ircc.ohio-state.edu)

minsky@media-lab.MEDIA.MIT.EDU (Marvin Minsky) (10/03/90)

WHITELEY-J@osu-20.ircc.ohio-state.edu (J. Whiteley) asks

 > I'm looking for a reference which shows that two layers are
 > sufficient to approximate any function using a BP network.  All
 > I've dug up so far is ...

Any boolean function has a disjunctive (and a conjunctive) normal form
hence can be expressed exactly by two layers, one with threshold 1 and
the other with threshold N, for the number of inputs.  However, if the
fan-out is less than N, then the question is much more serious, and
you have to read "Perceptrons", etc.

guansy@cs.tamu.edu (Sheng-Yih Guan) (10/03/90)

It has been proved (see, for instance, Funahashi, 1989; Moore and Poggio, 1988)
that a network with two layers of hidden sigmoidal units can approximate
arbitrarily well any continuous function.  It is interesting to notice that 
this statement still holds true if there is just one hidden layer (Carrol and
Dickinson, 1989; Cybenko, 1989; Funahashi, 1989).

S. M. Carrol and B. W. Dickinson.  Construction of neural nets using the 
Radon transform.  In Proceedings of the International Joint Conference
on Neural Networks, pages I-607-I-611, Washinton D. C., June 1989.
IEEE TAB Neural Network Committee.


K. Funahashi.  On the approximate realization of continuous mappings by 
neural networks.  Neural Networks, 2:183-192, 1989.

-Stanley
 _       _   _                            ___________          
|  \    /_| / /    Visualization Lab     /____  ____/         
 \  \  //  / /   Computer Science Dept       / /  _   _    _   _   
  | | //  / /     Texas A&M University      / /  / | | \  / | | | ||
  | |//  / / College Station, TX 77843-3112/ /  / /| |  \//|| | | ||
  /  /  / /____    Tel: (409)845-0531     / /  / -|| | |\/ || | !_!| 
 !__/  /______/ stanley@visual1.tamu.edu /_/  /_/ || !_!   || !____!

smagt@fwi.uva.nl (Patrick van der Smagt) (10/03/90)

In article <12626564934017@osu-20.ircc.ohio-state.edu> WHITELEY-J@osu-20.ircc.ohio-state.edu (J. Whiteley) writes:
>I'm looking for a reference which shows that two layers are sufficient
>to approximate any function using a BP network.  All I've dug up so
>far is:
>
>Cybenko, G.  Approximation by superpositions of a sigmoidal function.
>Unpublished manuscript, Oct. 1988.
>
There are three articles proving this.  The first one is the most
complete:

	%A K. Hornik
	%A M. Stinchcombe
	%A H. White
	%T Multilayer feedforward networks are universal approximators
	%J Neural Networks
	%V 2
	%N 5
	%D 1989
	%P 359--366

	%A G. Cybenko
	%T Approximation by superpositions of a sigmoidal function
	%J Mathematics of Control, Signals, and Systems
	%V 2
	%N 4
	%D 1989
	%P 303--314

	%A K.-I. Funahashi
	%T On the approximate realization of continuous mappings
	by neural networks
	%J Neural Networks
	%V 2
	%N 3
	%D 1989
	%P 193--192

Furthermore, the next paper proves something similar but now
for hidden units with a Gaussian activation function (very, very useful
for function approximation; I experience faster learning and greater
accuracy, although be careful with initial weight values and
learning rates):

	%A E. J. Hartman
	%A J. D. Keeler
	%A J. M. Kowalski
	%T Layered Neural Networks with Gaussian Hidden Units
	as Universal Approximations
	%J Neural Computations
	%V 2
	%N 2
	%D Summer 1990
	%P 210-215

Send me an email if you want a proof for binary networks (from Perceptrons).


Patrick van der Smagt                                               /\/\
                                                                    \  /
Organization: Faculty of Mathematics & Computer Science             /  \
              University of Amsterdam, Kruislaan 409,            _  \/\/  _
              NL-1098 SJ  Amsterdam, The Netherlands            | |      | |
Phone:        +31 20  525 7466                                  | | /\/\ | |
Telex:        10262 hef nl                                      | | \  / | |
Fax:          +31 20  592 5155                                  | | /  \ | |
email:        smagt@fwi.uva.nl                                  | | \/\/ | |
                                                                | \______/ |
                                                                 \________/

								    /\/\
                                                                    \  /
                                                                    /  \
                                                                    \/\/
``The opinions expressed herein are the author's only and do
not necessarily reflect those of the University of Amsterdam.''