svissag@hubcap.clemson.edu (Steve L Vissage II) (11/14/90)
The Subject says it all. Could someone please give me a brief, general description of the concept of back-propagation? Steve L Vissage II svissag@hubcap.clemson.edu
greenba@gambia.crd.ge.com (ben a green) (11/14/90)
Backpropagation is nothing more than the application of the chain rule of differentiation to the task of calculating the gradient with respect to weights and biases of a cost function for a layered, feedforward net. It's amazing that it took so many years after Minsky and Paepert's denunciation of the perceptron for people to think of BP as a solution to the training problem. (Maybe the discovery really was the use of differentiable node functions instead of flipflops, not backpropagation.) See _Parallel Distributed Processing_, vol. 1, chapter 8. It's a book by Rumelhart, McClelland, et al. from MIT Press, 1988. -- Ben A. Green, Jr. greenba@gambia.crd.ge.com "Nature abhors a physicist."
guansy@cs.tamu.edu (Sheng-Yih (Stanley) Guan) (11/15/90)
>The Subject says it all. Could someone please give me a brief, general >description of the concept of back-propagation? > >Steve L Vissage II A problem that once plagued error-correction learning Artificial Neural Systems was their inability to extend learning beyond a two-layer ANS. Specifically, the amount of error each hidden layer processing element should be credited for the ouput processing elements' errors was not defined. Fortunately this problem, known as the credit assignment ^^^^^^^^^^^^^^^^ problem has been solved by using backpropagation algorithm. ^^^^^^^ Quoted from Artificial Neural Systems by P. K. Simpson _ _ _ ___________ | \ /_| / / Visualization Lab /____ ____/ \ \ // / / Computer Science Dept / / _ _ _ _ | | // / / Texas A&M University / / / | | \ / | | | || | |// / / College Station, TX 77843-3112/ / / /| | \//|| | | || / / / /____ Tel: (409)845-0531 / / / -|| | |\/ || | !_!| !__/ /______/ stanley@visual1.tamu.edu /_/ /_/ || !_! || !____!
aam9n@hudson.acc.Virginia.EDU (Ali Minai) (11/15/90)
In article <10087@helios.TAMU.EDU> guansy@cs.tamu.edu (Sheng-Yih (Stanley) Guan) writes: > >A problem that once plagued error-correction learning Artificial Neural >Systems was their inability to extend learning beyond a two-layer ANS. >Specifically, the amount of error each hidden layer processing element >should be credited for the ouput processing elements' errors was not >defined. Fortunately this problem, known as the credit assignment > ^^^^^^^^^^^^^^^^ >problem has been solved by using backpropagation algorithm. >^^^^^^^ > >Quoted from Artificial Neural Systems by P. K. Simpson Well, I wouldn't go so far as to say that the credit assignment problem is "solved" by back-propagation. As someone pointed out earlier, back-propagation is just a neat application of the chain rule to calculate derivatives in the network. The gradient values returned by the algorithm are, of course, a strictly *local* and *numerical* estimate of the "true" gradient --- though this notion is itself somewhat shaky. All in all, back-propagation is beautiful; I love it; I spend many hours a day working with it; but the final solution it ain't. P.K. Simpson is obviously indulging in what Hecht-Nielsen rather politely describes as "hype". One point that I seldom see noted in connection with back-propagation is that Werbos' original procedure, which he called "dynamic feedback", is a very general method, and is directly applicable to all sorts of optimization problems (see, for example, Werbos' papers in recent issues of Neural Networks and IEEE Trans. on Systems, Man, and Cybernetics). Numerous researchers (including Mozer & Smolensky, Le Cun & Becker, Chauvin etc.) have utilized the generality of the algorithm to back-propagate various kinds of gradient information as part of neural net learning algorithms. Ciao, Ali Minai
guansy@cs.tamu.edu (Sheng-Yih (Stanley) Guan) (11/16/90)
In article <1990Nov15.042207.29026@murdoch.acc.Virginia.EDU> aam9n@hudson.acc.Virginia.EDU (Ali Minai) writes: >>A problem that once plagued error-correction learning Artificial Neural >>Systems was their inability to extend learning beyond a two-layer ANS. >>Specifically, the amount of error each hidden layer processing element >>should be credited for the ouput processing elements' errors was not >>defined. Fortunately this problem, known as the credit assignment > ^^^^^^^^^^^^^^^^ >>problem has been solved by using backpropagation algorithm. >>^^^^^^^ >> >>Quoted from Artificial Neural Systems by P. K. Simpson Minai>Well, I wouldn't go so far as to say that the credit assignment problem Minai>is "solved" by back-propagation. (the rest deleted) My apology for quoting the text of P. K. Simpson out of context which leads the conclusion from A. Minai saying that the credit assignment problem is not really solved by back-propagation. My intention of making the quote is to explain the backpropagation from another point of view. Indeed, backpropagation was introduced to solve the credit assignment problem which the ultimate purpose is to try to assign the weights associated with links appropriately. As also noted in Simpson's book, backpropagation is not quaranteed to find the global error minimum during training, only the local error minimum. Even that, there are numerous applications of backpropagation which all of them have claimed at least they have achieved moderate success in using backpropagation to do the training. When I talk about backpropagation here, I mean the original backpropagation alg. and all its variants. Minai>P.K. Simpson is obviously indulging in what Hecht-Nielsen rather politely Minai>describes as "hype". Well, I wouldn't go so far as to say that. Minai>One point that I seldom see noted in connection with back-propagation is Minai>that Werbos' original procedure, which he called "dynamic feedback", is Minai>a very general method, and is directly applicable to all sorts of Minai>optimization problems Coincidently, in Simpson's book there is a note of Werbos's dynamic procedure in the section 5.4.3 Backpropagation. "Werbos (1974) independently discovered the backpropagation algorithm and several variants - calling the algorithm dynamic feedback - while working on his thesis in statistics." The nice thing about Simpson's book is that he has tried to compare and summarize different Artificial Neural Systems using similar notations and criteria which is itself not an easy job (the reader should be cautioned that there are some errors in his book.) Ciao, Stanley Guan