ck@rex.cs.tulane.edu (Cris Koutsougeras) (08/23/89)
>I am trying to teach a network to accept as input a step function >and return as output the same step function or a scaled version of the >same. I have tried single and multiple inputs and outputs with >multiple hidden layers for my network..but am having no luck >whatsoever...I am using back propogation learning and my hidden >layers use the sigmoidal function. > >if anybody has any suggestions to make please post reply on net >or send mail to above address. > >Neural Network Group >Chemical Engineering Department >University of Texas at Austin > With the back prop only an analytic (continous differentiable) can be learned. The perfect step function is not therefore it will not be learned. A close approximation can be learned if your training set has enough samples around the point of the jump and of course quite enough coverage of the rest of the total interval. Look at it from an other point of view. No matter how many units (nodes) you put in your net, you are never going to have enough non-linearity to match the perfect step function. If you want I can send you a paper on some related experiments from our group at Tulane U. C. Koutsougeras
heirich@beowulf.ucsd.edu (Alan Heirich) (08/24/89)
In article <1060@rex.cs.tulane.edu> ck@rex.cs.tulane.edu (Cris Koutsougeras) writes: >With the back prop only an analytic (continous differentiable) can be learned. >The perfect step function is not therefore it will not be learned. A close >approximation can be learned if your training set has enough samples around >the point of the jump and of course quite enough coverage of the rest of the >total interval. Look at it from an other point of view. No matter how many >units (nodes) you put in your net, you are never going to have enough >non-linearity to match the perfect step function. If you want I can send you >a paper on some related experiments from our group at Tulane U. This statement makes me uncomfortable. It has been proven (White) that a two-layer feed forward network with a signmoid activation function (such as you find in standard back propagation networks) can approximate any Borel measurable function to an arbitrary degree of precision. So, for all intents and purposes, such a network can match a perfect step function to any discernible precision. On the other hand, I have not seen any literature proving bounds on learnability. This is obviously a very important area, but if people are still quibbling about the prevelance of local minima in error landscapes, I can't imagine that anyone has proven such a result about which functions are learnable! ------------------------- Alan Heirich Comp. Sci. & Eng., Cognitive Science C-014 University of California, San Diego 92093 heirich@cs.ucsd.edu aheirich@ucsd.bitnet