GA.CJJ@forsythe.stanford.edu (Clifford Johnson) (08/15/89)
CJ> [Neural nets] are limited in the patterns that they recognize, CJ> and are stumped by change. LS> *flame bit set* LS> Go read about Adaptive Resonance Theory (ART) before making sweeping LS> and false generalisations of this nature! CJ> I would have thought stochastic convergence theory more relevant CJ> than resonance theory. LS> I refer to "stumped by change", which admittedly is rather LS> inexact in itself. I am not familiar with "stochastic convergence", LS> although perhaps there is another name for it? In my original message I did clarify this somewhat. The point is that neural nets in essence automate Bayesian types of induction algorithms. In adapting to change, they only do so according to statistical/numerical rules that are bounded by their (implicit or explicit) preprogrammed characterizations and parameterizations of their inputs. Thus, a change in the basic *type* of pattern is beyond their cognition. Second, a change in the parameters of patterns they can adaptively recognize is only implemented over the time it takes for them to make enough mistakes that the earlier statistics are in effect overwritten. Third, I do not dispute that the characterizations/parameterizations of neural nets are complex enough to provide for "differences" (which could be called "changes") in individual sets of observations (e.g. differently shaped letters in optical readers). Stochastic convergence addresses the rate at which statistical induction algorithims converge to their asymptotic solutions, if and when they do. Given stable input distributions (i.e. no changes!), convergence has been shown for all but the most pathalogical kinds of input - but the process nevertheless takes many observations in many cases. Adaption does not even begin until after the first mistaken classification. Thanks for the reference. The kind of stochastic convergence that applies to generic neural net methodologies was worked on by J. Van Ryzin in the 1960s. Incidentally, the asymptotic result is not an approach to certainty of pattern recognition, but an approach to the minimum attainable probability of misclassification. See, e.g., Repetitive Play In Finite Statistical Games With Unknown Distributions by Van Ryzin, Annals of Mathematical Statistics, Vol. 37, No. 4, Aug. 1966.
jk3k+@andrew.cmu.edu (Joe Keane) (08/15/89)
In article <4331@lindy.Stanford.EDU> GA.CJJ@forsythe.stanford.edu (Clifford Johnson) writes: >In my original message I did clarify this somewhat. The point is >that neural nets in essence automate Bayesian types of induction >algorithms. In adapting to change, they only do so according to >statistical/numerical rules that are bounded by their (implicit >or explicit) preprogrammed characterizations and >parameterizations of their inputs. Some neural networks have carefully hand-crafted topologies. But if you use a standard topology and training algorithm in a new domain, where is the ``preprogramming''? Similarly, with a standard topology, you aren't giving it any ``parameterization''; it learns them all by itself. >Thus, a change in the basic >*type* of pattern is beyond their cognition. This doesn't follow. It may seem intuitive to you, but i think it's false. Fill in some more steps and i'll tell you where i think the problem is. >Second, a change in >the parameters of patterns they can adaptively recognize is only >implemented over the time it takes for them to make enough >mistakes that the earlier statistics are in effect overwritten. What's this about mistakes? You can train simply by reinforcement on good examples. But it is often better to use corrective training. To do this, you _force_ the net to make a mistake, to get it well trained. A neural net can have some characteristics which are strongly selected and some which are easily changed. So it can learn a new part of the input space without much changing its performance on part it's already learned. This behavior can come out of the simplest nets.
GA.CJJ@forsythe.stanford.edu (Clifford Johnson) (08/16/89)
In article <sYtuVDy00V4G81l_xE@andrew.cmu.edu>, jk3k+@andrew.cmu.edu (Joe Keane) writes: >In article (Clifford Johnson) writes: >> In adapting to change, they [NNs] only do so according to >>statistical/numerical rules that are bounded by their (implicit >>or explicit) preprogrammed characterizations and >>parameterizations of their inputs. > >Some neural networks have carefully hand-crafted topologies. But if you use a >standard topology and training algorithm in a new domain, where is the >``preprogramming''? That's why I was careful to state "implicit or explicit" re the "preprogramming." Whatever the topology, a definite set of distribution functions is implied. True, convergence of recognition outputs to fit a very wide range of inputs may be engineered, but convergence takes time. It proceeds in steps determined by the topology, and assumes a constant sampling space. The lack of constancy, i.e. change, is what stumps it. >Similarly, with a standard topology, you aren't giving it >any ``parameterization''; it learns them all by itself. Yes and no. The parameter-space is basically bounded by the topology. You can't, for example, have more degrees of freedom learned by the system than exist in its topology. And again, a change in the external or real parameters is only relearned over time, which is my main point. >>Thus, a change in the basic >>*type* of pattern is beyond their cognition. > >This doesn't follow. It may seem intuitive to you, but i think it's false. >Fill in some more steps and i'll tell you where i think the problem is. If a neural-net optical character reader is suddenly confronted with chinese characters, it isn't going to learn to read them, if it's only classification choices are arabic. Continued training might result in a systematic many-to-one translation of chinese characters into their "closest" arabic equivalents, closest being dependent on the aforesaid net's topological design. Yes, a better net might be built to include the capability to develop further classifications (again this takes time), but it wouldn't have a clue as to what the new patterns "mean" in terms of decision-making that was originally defined only in arabic terms.
jl3j+@andrew.cmu.edu (John Robert Leavitt) (08/16/89)
Clifford Johnson writes: [stuff about chinese characters in an arabic NN] I see a problem here (I may be wrong, but it feels right). Clifford: If I asked you to figured something out for me and gave you a bunch of data and you figured it out... and then I asked you to figure it again for a little more data (and you agreed)... and then I handed you a few sheets of Chinese characters and told you it was the data set... I'll bet you COULD classify the characters...There's be the squiggly looking ones... the ones with the three crossbars, the ones that look like tress, etc. You could even (given time) come up with a separate little niche in your mind for each and every character (ah, yes, the squiggly tree shaped one with the puddle shape at the left...). But, you probably would not be able to infer their meaning simply by being shown them... Does this mean you are limited by the type of data you can receive? How about an NN which decides which part of the spectrum a signal belongs to... Then you give it all sorts of colors... and it learns the stuff and all is fine and dandy... and then you shine white light at it... I'll bet it gets confused... Now, suppose I teach you as a child to know the colors and black and white and all that... then I shine some radio waves at you... I'll be you don't even notice the signal... See the problem... Neural nets can be limited, but they don't have to be... -John. PS: Those of you who know what you are talking about, back me up if I'm right.. Be merciful if I blew it... +-----------------------------------+--------------------------------+--------+ | US-Snail: 5715 Elsworth Ave. D-2 / You're the fastest runnner, / _ | | Pittsburgh, PA 15232 / but you're not allowed to win. / / \ | | E-Mail: jl3j@andrew.cmu.edu / -Howard Jones / /- -\ | | Phone: (412) 441-7724 +--------------+-----------------+ __| |__ | +------------------------------+ / sigh / / \_/ \ | | All these moments will be lost in time... +-----------------+--------------+ | like tears in the rain. -Batty / I speak only for myself... hah! | +------------------------------------------+----------------------------------+
andrew@berlioz (Lord Snooty @ The Giant Poisoned Electric Head ) (08/17/89)
Robert Leavitt writes: > Be merciful if I blew it... I don't think you did (but maybe I will!). An existence proof is that Arabs are known to have learnt Chinese. Will that do? -- ........................................................................... Andrew Palfreyman There's a good time coming, be it ever so far away, andrew@berlioz.nsc.com That's what I says to myself, says I, time sucks jolly good luck, hooray!