rgg@cs.mu.oz.au (Rupert G. Goldie) (07/18/90)
Profiling a backpropagation or quickprop net shows a large cost in computing the activation function (I was using a sigmoid). What methods are people using to reduce this cost ? Approximating the function with a polynomial, or rolling your own exponent function seem like possibilities, but does this increase the learning time or have other side-effects ? Thanks, Rupert. ---- Rupert G. Goldie rgg@munmurra.cs.mu.OZ.AU Computer Science Honours Student, University of Melbourne "Nobody expects the Spanish Inquisition"
yegerleh@vivaldi.ecn.purdue.edu (James D Yegerlehner) (07/18/90)
rgg@cs.mu.oz.au(Rupert G. Goldie) wrote: >Profiling a backpropagation or quickprop net shows a large cost in computing >the activation function (I was using a sigmoid). What methods are people >using to reduce this cost ? Approximating the function with a polynomial, or >rolling your own exponent function seem like possibilities, but does this >increase the learning time or have other side-effects ? >Thanks, > > Rupert. >---- >Rupert G. Goldie rgg@munmurra.cs.mu.OZ.AU >Computer Science Honours Student, University of Melbourne > >"Nobody expects the Spanish Inquisition" I'm writing a neural net application, and am using integer math; once I have accumulated the inputs to a neuron, I mask off the most significant byte, and use it as an index into a 256 entry look-up table approximation of a sigmoid-function. I also have a look up table for the slope of the sigmoid function. With the remaining 24 bits of the accumulated neuron input (I'm doing this on a motorola 68000; a 16 bit state X a 16 bit weight yields a 32 bit result using the 68000's single instruction multiply MULS) I drop the LSB and multiply by the slope of the sigmoid. In this way, I realize a piecewise linear approximation of a sigmoid function, which is MUCH FASTER than its floating point equivalent. Of course there are some obvious permutations on this, like using a 64K look up table, and thereby getting a perfect signoid within 16 bits of resolution, or using the 256 entry lookup table, and dispensing with the linear approximation in-between. Jim Yegerlehner yegerleh@mn.ecn.purdue.edu Controls and Measurements, Mechanical Engineering, Purdue Univ.