yunshu@eng.umd.edu (Yun-Shu Peter Chiou) (01/10/91)
Does anyone out there have any references or have done any works
on the effects of finite word length arithmetic on Back-Propagation.
Any information is appreciated.
Thanks in advance.
==============================================================================
Yun-Shu Peter Chiou +-------+ | --
/--- ------ -- | --+-- / ---+--- / \
yunshu@eng.umd.edu | | / ---- +--+--+ | / ----- \
+---+- | / / | + | +--+--+/ \
19902 Halfpenny Place | | | \ \ +--+--+ | + | +--+ |
Gaithersburg, MD 20879 | | | \ \ ----+--- +--+--+ | | | |
| | | | / | | | +--+ | |
Tel: (301)417-9474 +---+- |---/ /------------/ ---+--- +--+ | |
| | | | _|
==============================================================================kooijman@duteca (Richard Kooijman) (01/10/91)
yunshu@eng.umd.edu (Yun-Shu Peter Chiou) writes: >Does anyone out there have any references or have done any works >on the effects of finite word length arithmetic on Back-Propagation. I don't have references, but I have done it myself. I wrote a BP simulator for IBM compatibles and wanted some more speed. So I replaced all floating-point arithmetic by fixed-point arithmetic. The speed gain was > 3.5. The convergence appeared to evolve somewhat less smoothly, but I can't say there were any more problems. Everything appeared to work OK and a lot faster. The only important thing to do is to limit the value of the weights. If a neuron has many inputs and they have large values for their weights and they are all 'on', then the weighted sum of the inputs may overflow for the fixed-point type you choose for the activation of the neuron. Another solution is to check the values you add to the activation. If the value and activation are positive and the sum is negative then there was an overflow (mind you, there are no automatic overflow checks on integer types on a IBM compatible). One more important thing: I worked with numbers accurate to four decimals. This seemed to be enough and real live neurons have also been reported to be not so accurate. >8-) Richard.
tom@asi.com (Tom Baker) (01/11/91)
Yun-Shu Peter Chiou (yunshu@eng.umd.edu) writes: > Does anyone out there have any references or have done any works > on the effects of finite word length arithmetic on Back-Propagation. I have done a lot of work with BP using limited precision calculations. My masters thesis was on the subject, and last summer Jordan Holt worked with us to run a lot of benchmark data on our limited precision simulator. We are submitting a paper on Jordan's results to IJCNN '91 in Seattle. We use 16 bit weights, and 8 bit inputs and outputs. We have found that this representation does as well as floating point for most of the data sets that we have tried. I have also seen several other papers where 16 bit weights were used successfully. I am also trying to collect a bibliography on limited precision. I would like to see the references that you get. I do not have all of the references that I have in a form that can be sent out. I will post them soon. I would like to keep in touch with the people that are doing research in this area. Thomas Baker INTERNET: tom@asi.com Adaptive Solutions, Inc. UUCP: (uunet,ogicse)!adaptive!tom 1400 N.W. Compton Drive, Suite 340 Beaverton, Oregon 97006