camargo@cs.columbia.edu (Francisco Camargo) (10/20/89)
I received a reply for my last message that I'd like to share with you and also clarify what I'm trying to accomplish. (Please David, don't be mad at me for making your message public. I do appreciate your help) ----------------------------------------------------------------------------- >> Hi. i saw your posting on comp.ai.neural-nets. i have set up a small >> net (2in, 4 hidden, 1out) which was able to multiply two inputs (x, y) >> producing k*x*y. i think i trained the net on the input range >> [0.2..0.8] x [0.2..0.8] at 0.1 steps. this seemed to give good results >> as the network correctly interpolated the function on non-training >> points. however, it did quite poorly on points outside the training >> domain. >> >> i would like to say that the way it worked was that the hidden >> units somehow computed the log of the inputs. the output unit >> then summed the logs and exponentiated the result yielding the >> product. unfortunately, when i looked at the individual units, >> it was hard to figure out how the net was doing the computation! >> >> i can send you more details if you like? >> >> david >> blackman@hodgkin.med.upenn.edu ----------------------------------------------------------------------------- Here goes my reply to David. ----------------------------------------------------------------------------- Hi David, I did the same experiment, and got a net with 2 inputs, 5 hidden, 3 hidden and 1 output to do the same thing, although I was quit unhappy with the precision that the net achieved (Error < 0.001 after ~15000 iterations and not so good ability to 'extrapolate' products outside the trainning set). I guess that we could live with low precision if this computations were to be used in a closed loop form of control, or whatever that might be. What I'm trying to accomplish is to set a equivalency of higher order nets and first order net. For the discrete case this is easy, since that one can substitute the multiplication by the AND operation, and therefore transform for instance, a 2nd order recurrent net with N neurons and N^3 weights using N+3*(N^2)/2 1st order neurons and N^3 adaptive weights + (N^2) fixed weights. (I can send you details if you want). This result generalizes very easily for even higher order nets. The really interesting case though is the continuous case, because it would allow for a method of "building" arbitrary polynomial expressions using only first order networks (although, with large number of neurons). I believe that one could build a ``recurrent'' and very fast small net to compute the products, and therefore transform any high order continuous net into first order. Another interesting question: "is this transformation important at all ?" Since that we are increasing the number of neurons by very large quantities, one might argue that this is an uninteresting problem. In any case, I'd like to throw this problem to the NNet community, and hear what they have to say. Thanks for your comments. camargo@cs.columbia.edu