ross@psych.uq.oz.au (Ross Gayler) (04/13/91)
[Sorry if you have already seen this. We have had problems posting news. RG]
I am doing some work that involves higher order units. I am looking for a
good intuitive-level description of the range of response surfaces that can
be computed by a single higher order unit.
First some definition:
A single first order unit (your every day, common or garden variety unit) forms
its output as the weighted sum of its inputs (I am ignoring thresholds and
sqashing functions). A unit like this divides its input space with a
hyperplane. Considering only whether the output is positive or negative, the
space defined by the N inputs to the unit is divided into two regions
separated by a flat boundary. The weights applied to the inputs determine
the location and orientation of the boundary.
A higher order unit forms its output as the weighted sum of products of its
inputs. For example, a second order unit of three inputs {i1,i2,i3} would
have: o = w1*i1*i2 + w2*i1*i3 + w3*i2*i3 + w0. A higher order unit can also
be polynomial by summing terms of different orders. The response pattern of
a higher order unit is a hyperplane in the space defined by the product terms
but is a more complex pattern in the space defined by the inputs.
I am interested in the case where the input signals range over the closed
interval [-1,+1]. I am also (mostly) interested only in the cases where
the inputs are at their extreme values, and in the sign of the output.
Now the question:
Is there a simple (intuitive level) characterisation of the range of allowable
response surfaces that can be formed by a higher order unit?
R.J. Williams (Chapter 10 of the PDP book, Vol 1) showed that a higher order
unit could implement any boolean function of its inputs (equivalent at the
extremes of the inputs). His proofs were for units with inputs in the range
[0,1]. I presume that the same holds true for inputs in the range [-1,+1]
even though some of the monotonicity properties he refers to don't hold any
more.
Assuming that Williams' result still holds, I can know the result
intellectually, but I still don't have a strong intuitive feel for what a
higher order unit can do. I guess the part that I feel uncomfortable with
is that a higher order unit can implement any boolean function *if* there
are enough terms. But what sort of response surface can it produce under
more restricted conditions where there may be a large number of inputs but
only low order terms?
I am hoping that someone on the net will be able to give me an interesting
and brilliantly insightful characterisation of the possible response surfaces.
No heavy maths please, I did statistics :-).
Ross Gayler
ross@psych.psy.uq.oz.au