drt@chinet.chi.il.us (Donald Tveter) (09/02/90)
I just sent some C source files to comp.sources.misc that will do fairly fast back-propagation. I assume they will be posted shortly. If you are interested, look for them. The four parts total around 135K. The programs were developed on a UNIX (tm of AT&T Bell Labs) PC running UNIX System V, version 3.5 of the software. I assume the programs will run on most other systems with little or no change. (I cannot look into bugs on any system other than a UNIX PC!!! If you find bugs or have other problems on some other system, I will be happy to collect the bugs and your fixes but I cannot test them myself.) I produced these files in order to learn back-propagation, to study the effects of various speed-up techniques and to do problems that will appear in a textbook on AI I've written, called _The Basis of Artificial Intelligence_. The book should be published by Computer Science Press. (I say should, because whereas I have a publication agreement with them, the new management (W.H. Freeman, the book publishing arm of Scientific American, Inc.) doesn't seem to take contracts too seriously.) Four different versions can be made using these files. First, there is the standard form of back-prop using 64-bit floating point weights. Second there is a faster version using 16-bit integer weights. Third and fourth, there are integer and floating point versions that produce symmetric two-layer networks of the Hopfield/Boltzman machine variety. On a UNIX PC with a 10Mhz 68010 the integer version can do up to around 45,000 feedforward connections per second and around 25,000 weight changes per second. These values depend on the exact problem. On a 20MHz 386 machine (chinet) the integer version may be around 5 times faster, but I have never tested the programs extensively on this machine. Some of the other features are: * approximating the sigmoid with a piece-wise linear function * the differential step size method * Fahlman's derivative term * the delta-bar-delta update method * changing the sharpness of the sigmoid * continuous and periodic updates * skipping learned patterns * saving weights in text and binary formats * adding hidden units to evade local minimas * recurrent networks Some of these methods and their effectiveness, especially the use of integer weights will be described in an article scheduled to be published by _AI Expert_ in their November issue. I will probably write another article for them describing some of the other techniques as well. I have copyrighted this code but I am giving permission for casual users and educational users to use, modify and redistribute the code on a not-for-profit basis. Commercial users are welcome to test the code to see if its useful, but if you want to do more than play with it, you must contact me and I will charge you a small fee. Don Tveter 5228 N. Nashville Ave. Chicago, Illinois 60656 drt@chinet.chi.il.us