spoffojj@hq.af.mil (Jason Spofford) (08/02/90)
I would like to hear some reactions to the following generalizations: 1. Currently there is no unified theory of neural networks. 2. Several training algorithms have been developed. Each algorithm works on a small subset of NN architectures, usually with a particular neuron model. The training algorithms, even when all combined, only use a small percentage of the possible NN architectures. 3. It is almost like an art trying to solve NN problems, frequently requiring a variety of interconnected NN architectures and a good bit of trail and error. 4. Little is known in theory about how many neurons in it takes to solve a problem, and typically guesses are made to the number of neurons required to solve problems. Thus efficient use of NN machinery is not assured. 5. Each training algorithm can only solve a narrow class of problems. 6. Many algorithms scale up very poorly, taking an extremely long time to train NNs. I'm not knocking the field of NN's, I just want to be sure my assumptions about the current state of NNs are correct. These are all comments that I have heard since I've been working with neural networks. Please give your opinions!! As you may of gathered from my previous post, I am working on applying the genetic algorithm to solving NN problems. My hope, and that is what it is at this point, is to develop a GA that makes no assumptions or restrictions on NN architectures and that can solve a wide class of problems. I'd like to think I'm attacking the metaproblem of NN's, artificially developing NN's in a way not too unlike biological systems. Thanks. -- ---------------------------------------------------------- ) Jason Spofford <((((((> spoffojj.hq.af.mil ( ) LAN Manager George Mason Univ. Grad. Stud. ( ----------------------------------------------------------
bill@hooey.unm.edu (william horne) (08/04/90)
In article <spoffojj.649607641@lgn> spoffojj@hq.af.mil (Jason Spofford) writes: >I would like to hear some reactions to the following generalizations: > ....... >2. Several training algorithms have been developed. Each algorithm >works on a small subset of NN architectures, usually with a particular >neuron model. The training algorithms, even when all combined, only >use a small percentage of the possible NN architectures. > ....... >5. Each training algorithm can only solve a narrow class of problems. > >As you may of gathered from my previous post, I am working on applying >the genetic algorithm to solving NN problems. My hope, and that is >what it is at this point, is to develop a GA that makes no assumptions >or restrictions on NN architectures and that can solve a wide class of >problems. I'd like to think I'm attacking the metaproblem of NN's, >artificially developing NN's in a way not too unlike biological >systems. > Here's my $0.02..... I don't think GAs have much to offer for learning techniques in networks which have a good gradient search technique for learning (i.e. MLPs, recurrent networks, etc...), and especially when these networks use floating point weight representations. The learning algorithm for these networks can always be cast in terms of minimizing some criterion function, and as a result can be viewed as a search of an error surface in the weight space. My experience with GAs have been that they are terrible at searching the bizarre error surfaces associated with something like MLPs, in fact they are no better than a completely random search. This seems to be due to the fact that the bits in floating point representations are highly correlated with each other. There are things you can do to avoid this, like Grey coding and not allowing crossovers in the middle of a 32-bit word, etc... These algorithms seem to improve the performance of the GA, but not to the point where they are competitive with a simple gradient search. I always thought GAs were fine if your search space consisted of attributes which are binary and not highly correlated. I don't see them as particularly appropriate for learning algorithms for these types of networks. Maybe they are good for other types of networks I haven't considered closely. In any case I don't think they are the global solution to NN learning. Feel free to flame this... -Bill
spoffojj@hq.af.mil (Jason Spofford) (08/05/90)
bill@hooey.unm.edu (william horne) writes: >I don't think GAs have much to offer for learning techniques in networks >which have a good gradient search technique for learning (i.e. >MLPs, recurrent networks, etc...), and especially when these networks >use floating point weight representations. GA's are general purpose algorithms and can't hope to compete with specifically tailored training algorithms. The problem is that a specific training algorithm can only solve a small subset of problems. The GA fits in by solving many problems equally well (or bad .. depending on your perspective). > My experience with >GAs have been that they are terrible at searching the bizarre error surfaces >associated with something like MLPs, in fact they are no better than a >completely random search. This seems to be due to the fact that the bits >in floating point representations are highly correlated with each other. >There are things you can do to avoid this, like Grey coding and not allowing >crossovers in the middle of a 32-bit word, etc... These algorithms seem >to improve the performance of the GA, but not to the point where they are >competitive with a simple gradient search. I am not sure exactly how you are defining random search. Do you mean hoping around in the search space totally at random, keeping the best individual to date? Obviously, GA's are biologically inspired. If one is not a creationist, than one can only attribute the development of NN solutions in living creatures to this process. So there is motivation and justification for researching the topic. You touched upon the most important aspect of research of GA/NN's and that is the genetic representation. The one I use is not the answer but it does provide positive results! My solution to the 9x9 37 character recognition problem took just over one trial per bit of genetic code to solve the problem (much better than random). The nature of the search space the GA must traverse is directly related to the genetic representation. But once a good genetic representation is found, than a myriad of problems can be solved simply by changing the fitness function. Training algorithms can't compete with that type of flexibility, although they may solve their subclass of problems efficiently. -- ---------------------------------------------------------- ) Jason Spofford The LAN Manager ( ) spoffojj.hq.af.mil George Mason Univ. Grad. Stud. ( ----------------------------------------------------------
pluto@zaius.ucsd.edu (Mark Plutowski) (08/05/90)
In article <1990Aug3.175023.28210@ariel.unm.edu> bill@hooey.unm.edu (william horne) writes: >In article <spoffojj.649607641@lgn> spoffojj@hq.af.mil (Jason Spofford) writes: SPOFFORD: I would like to hear some reactions to the following generalizations: ... SPOFFORD: I'd like to think I'm attacking the metaproblem of NN's, SPOFFORD: artificially developing NN's in a way not too unlike biological SPOFFORD: systems. HOOEY Writes: Here's my $0.02..... I don't think GAs have much to offer for learning techniques in networks ... which have a good gradient search technique for learning [or] use floating point weight representations. ... My experience with GAs have been that they are terrible at searching the bizarre error surfaces ... in fact they are no better than a completely random search. This seems to be due to the fact that the bits in floating point representations are highly correlated with each other. There are things you can do to avoid this, ...... but not to the point where they are competitive with a simple gradient search. I don't see [GAs] as particularly appropriate for learning algorithms ... ... In any case I don't think they are the global solution to NN learning. > >Feel free to flame this... > >-Bill Well, OK, if you insist! Actually, this is not much of a flame, but more of a memo that GAs can help a great deal, with that one aspect of network learning we all know and love: The Restart Method! Yes, you too have used it, if you've done any network training at all. Now, what's wrong with utilizing a bit of knowledge (or, a byte or two even) from past restarts to guide the settings of parameters and initial weights for the next one? GAs can do this. Now if they could only get the population counts down...
bill@wayback.unm.edu (william horne) (08/07/90)
In article <12173@sdcc6.ucsd.edu> pluto@zaius.ucsd.edu (Mark Plutowski) writes: (In response to my flame on the usefullness of GAs in NNs) > >Well, OK, if you insist! Actually, this is not much of a flame, but more of >a memo that GAs can help a great deal, with that one aspect of network >learning we all know and love: The Restart Method! Yes, you too have >used it, if you've done any network training at all. Now, what's wrong >with utilizing a bit of knowledge (or, a byte or two even) from past >restarts to guide the settings of parameters and initial weights for the >next one? GAs can do this. > This may be possible assuming that your knowledge from previous starting points could be used to direct your next guess at a starting point. How would this knowledge help except to not make a guess near your old starting point? As far as I know there is no type of known regularity concerning local minima which is particularly suitable towards GAs. As far a setting of parameters, I prefer something like delta-bar-delta which uses a local measure of the surface to guide setting of parameters. One interesting application of GAs seems to be trying to find the best topology, i.e. number of nodes, layers etc... I could see how you could combine the best of two networks maybe.... -Bill
schraudo@beowulf.ucsd.edu (Nici Schraudolph) (08/19/90)
bill@wayback.unm.edu (william horne) writes: [on using GAs for searching initial weight space:] >This may be possible assuming that your knowledge from previous starting >points could be used to direct your next guess at a starting point. >How would this knowledge help except to not make a guess near your old >starting point? As far as I know there is no type of known regularity >concerning local minima which is particularly suitable towards GAs. I'd phrase it the other way round: most of the GA/NN research is aimed at finding a GA (specifically, a genetic representation of NNs) for which the recombination operator exploits some regularity concerning the basins of attraction for NN gradient descent. The two main questions are: 1) Are there any such regularities in the first place, aside from simple invariances such as flipping the sign of all weights? 2) Can we find genetic encodings and/or recombination operators that exploit them? -- Nicol N. Schraudolph, C-014 nici%cs@ucsd.edu University of California, San Diego nici%cs@ucsd.bitnet La Jolla, CA 92093-0114 ...!ucsd!cs!nici
kingsley@hpwrce.HP.COM (Kingsley Morse) (08/21/90)
Nicol N. Schraudolph writes: > most of the GA/NN research is aimed at >finding a GA (specifically, a genetic representation of NNs) for which >the recombination operator exploits some regularity concerning the basins >of attraction for NN gradient descent. The two main questions are: >1) Are there any such regularities in the first place, aside from simple > invariances such as flipping the sign of all weights? > >2) Can we find genetic encodings and/or recombination operators that > exploit them? I agree with Nicol. I encoded a neural net in a GA and fine tuned it with gradient descent, but as the net evolved to be larger and larger, it wouldn't learn more and more. The aggregate of brain material became computationally "stiff", in that the elements were too tightly coupled. We know that GAs can evolve true intelligence, because we've evolved to our present human intellect. But just knowing that GAs CAN work isn't enough. The question now is: What genetic encoding will allow a large network to stay flexible and be trained with many patterns?
eabg020@orion.oac.uci.edu (Donald Doherty) (08/21/90)
>... I encoded a neural net in a GA and fine tuned it >with gradient descent, but as the net evolved to be larger and larger, it >wouldn't learn more and more. The aggregate of brain material became >computationally "stiff", in that the elements were too tightly >coupled. > >[...] > > What genetic encoding will allow a large network to stay flexible and >be trained with many patterns? Interesting Kingsley... you have seen a principle exhibited by your electronic network that I would have probably predicted given our knowledge of biological systems. At the risk of being overly simplistic, "primitive" or relatively simple organisms tend to be "hard wired" through genetic determination. There are many instances of studies on insect nervous systems, for example, that demonstrate genetically predetermined progenitor cells and *relatively* straitforward mechanisms that lead to specific connectivity (Bently, for instance, studies such a system in the legs of grasshoppers at UC Berkeley.) On the other hand, it is evident that even in relatively "simple" mammals (i.e. mice and rats) genetic factors in connectivity and nervous function are interweaved in a highly complex manner with "epigenetic" factors. For instance, activity dependent changes in genomic expression is a robust and probably widespread phenomenon in the brain (For instance, see papers by Chris Gall on activity dependent changes in expression of various peptides etc. in the hippocampus and dentate gyrus.) These changes almost certainly result in changes in the activity and processing going on in these structures. Your network exhibits "behaviors" stemming strictly from the genetic algorithm giving what could only be relatively unflexible results. It is evident that in biological systems many orders of magnitude more degrees of freedom result from dynamics not coded for in their DNA.
dmb@odin.icd.ab.com (David Babuder) (08/21/90)
In article <3430008@hpwrce.HP.COM> kingsley@hpwrce.HP.COM (Kingsley Morse) writes: >We know that GAs can evolve true intelligence, because we've evolved to >our present human intellect. Seems like an assumption.... particularly in view of the next section > But just knowing that GAs CAN work isn't enough. The question now is: > What genetic encoding will allow a large network to stay flexible and >be trained with many patterns? Is there a change in the genetic encoding of humans who are learning disabled? More specifically, are the learning disabilities attributed to genetic problems? I do not have an answer, but the answers may help you to determine how much error is induced by the above assumption. Dave Babuder Allen-Bradley Company (ICCG) A Rockwell International Company
usenet@nlm.nih.gov (usenet news poster) (08/22/90)
kingsley@hpwrce.HP.COM (Kingsley Morse) ("km>") writes: Nicol N. Schraudolph ("ns>") writes: ns> most of the GA/NN research is aimed at ns> finding a GA (specifically, a genetic representation of NNs) for which ns> the recombination operator exploits some regularity concerning the basins ns> of attraction for NN gradient descent. The two main questions are: ns> 1) Are there any such regularities in the first place, aside from simple ns> invariances such as flipping the sign of all weights? ns> 2) Can we find genetic encodings and/or recombination operators that ns> exploit them? [...] km> We know that GAs can evolve true intelligence, because we've evolved to km> our present human intellect. I am not sure that this is strictly correct. Biological genetics evolved a neural structure capable of being trained, but I am not aware of any evidence for genetic type algorithms actually playing a role in biological learning. Specifically, somatic cells, such as neurons, do not undergo recombination. There are periods of large scale cell death in neurological development so a "selection of the fittest" parallel algorithm might be invoked, but that is not a recombination operator. km> But just knowing that GAs CAN work isn't enough. The question now is: km> What genetic encoding will allow a large network to stay flexible and km> be trained with many patterns? Maybe a better question is: what network structures are trainable? Food for thought: how hard can you expect training to be? The brain has on the order of 10^10 neurons each with 10^2 to 10^4 synapses and fires ~10 times/second. That adds up to 10^13 - 10^14 connections/second. Training a human still takes a few years. David States
kingsley@hpwrce.HP.COM (Kingsley Morse) (08/23/90)
DS> I am not aware of any evidence for genetic type algorithms actually DS> playing a role in biological learning. My understanding is that twin studies have shown that intelligence is inherited. (Twin studies measure twins who were separated at birth, to distinguish between inherited and environmental effects.) I'm assuming that if intelligence is inherited, then it's encoded in chromosomes, and thus operated on by GAs. DS> Specifically, somatic cells, such as neurons, do not undergo DS> recombination. Can you be more specific? Do you mean that the components of individual cells don't undergo recombination? Do you mean that entire cells aren't recombined with other cells? What references are you using? And by the by, what was that "x" in your computational complexity table under the "Really Smart Systems" notestring?
loren@tristan.llnl.gov (Loren Petrich) (08/23/90)
In article <3430010@hpwrce.HP.COM> kingsley@hpwrce.HP.COM (Kingsley Morse) writes: >DS> I am not aware of any evidence for genetic type algorithms actually >DS> playing a role in biological learning. > >My understanding is that twin studies have shown that intelligence is >inherited. (Twin studies measure twins who were separated at birth, to >distinguish between inherited and environmental effects.) I'm assuming that >if intelligence is inherited, then it's encoded in chromosomes, and thus >operated on by GAs. I think that the issue of the inheritance of human intelligence is more complicated than that. First of all, in studying this question, one must separate actual inheritance from environmental effects. Twins separated at birth are close to ideal; although they might both have been influenced by having been in the same womb together. Taking an egg cell, letting it divide a number of times, splitting up the result, and implanting all these cells in several different women who all live in different cultural environments would be ideal. Stephen Jay Gould in _The Mismeasure of Man_ and Richard(?) Lewontin and Stephen(?) Kamin in _Not In Our Genes_ all give a severely critical look at the question of the inheritance of intelligence, and find the evidence somewhat lacking. Consider the fate of one of the leading advocates -- Sir Cyril Burt, who had published his results in a journal that he had edited. There were several twin studies whose correlation coefficient had remained unchanged -- despite a change in the (alleged) sample size. Two women who had supposedly worked on some of these papers were nowhere to be found. When these issues were raised in the late 1970's, Kamin(?) and others concluded that Burt had faked some of his data. His defenders, like Richard Herrnstein, denied the charge of fakery, but attributed Burt's "results" to carelessness. But on a more fundamental level, the potential for being able to do what we do is certainly inherited. What the twin studies would reveal would be any genetic component in _variation_ between individuals. If all people had the same "inborn" intelligence, then genetic studies would not reveal any difference between one individual and another. And there may well be several kinds of "intelligence." For example, understanding language and generating language appear to be two separate activities; tests on people with brain damage sometimes reveals that only one of these capabilities is present. I think this result (if it holds up) is very importaint for Artificial Intelligence work -- it would imply that doing one thing well may not translate easily into doing another thing well. Thus, a system that is good at vision may not be good at language comprehension or coordination of a robot arm. Loren Petrich, the Master Blaster: loren@sunlight.llnl.gov Since this nodename is not widely known, you may have to try: loren%sunlight.llnl.gov@star.stanford.edu
usenet@nlm.nih.gov (usenet news poster) (08/23/90)
kingsley@hpwrce.HP.COM (Kingsley Morse "km>") writes:
states@tech.NLM.NIH.GOV (David States "ds>")
ds> I am not aware of any evidence for genetic type algorithms actually
ds> playing a role in biological learning.
km> My understanding is that twin studies have shown that intelligence is
km> inherited. (Twin studies measure twins who were separated at birth, to
km> distinguish between inherited and environmental effects.) I'm assuming that
km> if intelligence is inherited, then it's encoded in chromosomes, and thus
km> operated on by GAs.
The issue of genetics and IQ is contentious to say the least. First,
you should be aware that the "classic" twin studies by Cyril Burt are
now known to have been totally fabricated and are a classic example of
scientific fraud.
Leaving the specifics of IQ and twins aside, genetics determines the
underlying structure of an organism (you are not a kangaroo because
your development was not governed by a kangaroo's genes). Intelligence,
to me, refers to adaptability and learning ability. The fact that you
sneeze when your nose is tickled may be genetically determined, but it
is not particularly evidence of learning. In the sense that humans are
generally regarded as being more intelligent than kangaroos, I suppose
that genetics has a role in intelligence.
ds> Specifically, somatic cells, such as neurons, do not undergo recombination.
km> Can you be more specific? Do you mean that the components of individual
km> cells don't undergo recombination? Do you mean that entire cells aren't
km> recombined with other cells? What references are you using?
Biological neural systems are composed of "somatic" cells like the most of
the rest of the body. This means that these cells have lost the ability
to undergo sexual recombination with exchange of genetic information (meiosis)
and instead can only undergo asexual division (mitosis). In the adult brain,
neurons have generally lost the ability to divide completely. Therefore,
during the training of a biological neural system, individual neurons have
no way of recombining genetic information.
See:
Watson et al, Molecular Biology of the Gene
Darnel, Molecular Cell Biology
km> And by the by, what was that "x" in your computational complexity table
km> under the "Really Smart Systems" notestring?
Just an arbitrary empirically determined number so C = x^N implies that C
grows exponentially with N without saying exactly how fast.
David Statesusenet@nlm.nih.gov (usenet news poster) (08/24/90)
kingsley@hpwrce.HP.COM (Kingsley Morse "km>") writes:
states@tech.NLM.NIH.GOV (David States "ds>") writes
ds> I am not aware of any evidence for genetic type algorithms actually
ds> playing a role in biological learning.
km> My understanding is that twin studies have shown that intelligence is
km> inherited. (Twin studies measure twins who were separated at birth, to
km> distinguish between inherited and environmental effects.) I'm assuming that
km> if intelligence is inherited, then it's encoded in chromosomes, and thus
km> operated on by GAs.
The whole area of genetics and intelligence is highly contentious, to say
the least. First, you should be aware that the "classic" studies on IQ
and twins by Cyril Burt are now known to have been fabricated are a
classic example of scientific fraud.
Genetics determines the structure of a biological system. You are not
a kangaroo because your development was not governed by a kangaroo's genes.
Intelligence and learning, to me, imply adaptability, not a fixed structure.
The fact that you sneeze when your nose is tickled may be genetically
determined, but it is not evidence of intelligence. In the sense that
humans are generally regarded as being more intelligent than kangaroos,
I suppose it can be said that genetics has a role in determining
intelligence.
ds> Specifically, somatic cells, such as neurons, do not undergo
ds> recombination.
km> Can you be more specific? Do you mean that the components of individual
km> cells don't undergo recombination? Do you mean that entire cells aren't
km> recombined with other cells? What references are you using?
Neurons are "somatic" cells meaning that they have lost the ability to
undergo sexual division and that they cannot exchange genetic information.
In the adult brain most neurons have lost the ability to divide altogether.
Therefore, at least at the level of whole neurons or collections of neurons,
there is no way to recombine optimal segments of previously trained data,
the basis of a "genetic algortithm".
See: SW Kuffler and JG Nicholls
From Neuron to Brain
Sinauer Associates, 1977
km> What was that "x" in your computational complexity table under
km> the "Really Smart Systems" notestring?
An arbitrary base for the exponent that is empirically determined so
C = x^N implies that C grows exponentially with N but does not necessarily
say how fast
David Statesal@gtx.com (Alan Filipski) (08/25/90)
In article <3430008@hpwrce.HP.COM> kingsley@hpwrce.HP.COM (Kingsley Morse) writes: >I agree with Nicol. I encoded a neural net in a GA and fine tuned it >with gradient descent, but as the net evolved to be larger and larger, it >wouldn't learn more and more. The aggregate of brain material became >computationally "stiff", in that the elements were too tightly >coupled. > >We know that GAs can evolve true intelligence, because we've evolved to >our present human intellect. But just knowing that GAs CAN work isn't >enough. The question now is: > > What genetic encoding will allow a large network to stay flexible and >be trained with many patterns? Complex biological (not just nervous) systems use strategies of hierarchical and sequential decomposition. Our brains, for example, are not randomly connected nets but have thousands of recognizable discrete structures, from tiny nuclei to huge cortical sheets. Many of these are repeated and some are found within others. Hierarchy and repetition seem to correspond to context-free and regular languages, respectively. I don't know much about GA's, but is seems to me that a grammatical encoding might be appropriate to get around that scale barrier. A "mutation" might involve not only tweaking some weights or connections, but repeating whole structures or putting them together in different ways. Just a half-baked idea. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ( Alan Filipski, GTX Corp, 8836 N. 23rd Avenue, Phoenix, Arizona 85021, USA ) ( {decvax,hplabs,uunet!amdahl,nsc}!sun!sunburn!gtx!al (602)870-1696 ) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
rabani@sdbio2.ucsd.edu (Ely Rabani/D Smith) (08/25/90)
While I am somewhat ignorant about the area of NN and GA models (for the present,) biology happens to be my field. Now, (1) while it is true that MOST somatic cells, including neurons, do not undergo chromosomal recombination of the meiotic sort there is the exception of lymphoid recombination by which high degree of variability in the immune response is generated; (2) there is the phenomenon of somatic cell crossing over, but it is a rarity so far as it is observed; (3) somatic cells are capable of certain kinds of DNA repair that involve mechanisms similar to those of chromosomal recombination. A difficulty occurs to me at the level of molecular biology. Unless there is some kind of relatively high level descriptor language, the symbols of which are themselsves strung together on the chromosomes derrived from each parent, and it is some sort of recombination between these two (linearly orderred) sets, i.e. positionally contexted cis-acting elements, that is being imputed, I don't quite understand what this supposed recombination accomplishes (if we are strictly concerned with learning by an organism, by this mechanism.) The impression that I have of this model flies in the face of the one gene one protein model of inheritence, where (in general) on which of a pair of homologous chromosomes a gene (allele) resides does not influence the nature (though not necessarily quantitative expression) of its product. On the other hand this kind of recombination (in particular the specifics of the mechanism and what we have to recombine to begin with) is entirely significant evolutionarily at the level of a population. It may be purely prejudicial, but I suspect that this kind of model would be most applicable to specific and narrow behaviours (and perhaps the KINDS of learning they entail) than to more generally applicable and abstract kinds of learning. This is not to say that the latter may not employ the former over domains or sets of domains of learning (in particular, the kinds of assertions made by Chomskians in contrast to empiricist models of language acquisition.) Nonetheless, I find the recombination model, insofar as it implies the recombination of genetic material, most pertinent to things like prey selectivity among crawley things (by some kind of quantitative heritability. ----- Now, if I haven't bored of scared you all away yet, I would like to attend to the second reason for my post: Presently a problem that concerns me is that of character recognition, both printed and handwritten. So far as I have been able to determine, the problem has not been entirely cracked, but there is evidence of progress over both domains. Can anyone supply me with references on the topic, information on the current state of the art, or related general apprehensions? Please e-mail response to: rabani@sdbio2.ucsd.edu Thank you, Ely
kingsley@hpwrce.HP.COM (Kingsley Morse) (08/26/90)
/ hpwrce:comp.ai.neural-nets / al@gtx.com (Alan Filipski) / 1:01 pm Aug 24, 1990 / In article <3430008@hpwrce.HP.COM> kingsley@hpwrce.HP.COM (Kingsley Morse) writes: >I agree with Nicol. I encoded a neural net in a GA and fine tuned it >with gradient descent, but as the net evolved to be larger and larger, it >wouldn't learn more and more. The aggregate of brain material became >computationally "stiff", in that the elements were too tightly >coupled. > >We know that GAs can evolve true intelligence, because we've evolved to >our present human intellect. But just knowing that GAs CAN work isn't >enough. The question now is: > > What genetic encoding will allow a large network to stay flexible and >be trained with many patterns? Complex biological (not just nervous) systems use strategies of hierarchical and sequential decomposition. Our brains, for example, are not randomly connected nets but have thousands of recognizable discrete structures, from tiny nuclei to huge cortical sheets. Many of these are repeated and some are found within others. Hierarchy and repetition seem to correspond to context-free and regular languages, respectively. I don't know much about GA's, but is seems to me that a grammatical encoding might be appropriate to get around that scale barrier. A "mutation" might involve not only tweaking some weights or connections, but repeating whole structures or putting them together in different ways. Just a half-baked idea. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ( Alan Filipski, GTX Corp, 8836 N. 23rd Avenue, Phoenix, Arizona 85021, USA ) ( {decvax,hplabs,uunet!amdahl,nsc}!sun!sunburn!gtx!al (602)870-1696 ) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ----------
kingsley@hpwrce.HP.COM (Kingsley Morse) (08/26/90)
/ hpwrce:comp.ai.neural-nets / kingsley@hpwrce.HP.COM (Kingsley Morse) / 12:57 pm Aug 25, 1990 / / hpwrce:comp.ai.neural-nets / al@gtx.com (Alan Filipski) / 1:01 pm Aug 24, 1990 / In article <3430008@hpwrce.HP.COM> kingsley@hpwrce.HP.COM (Kingsley Morse) writes: >I agree with Nicol. I encoded a neural net in a GA and fine tuned it >with gradient descent, but as the net evolved to be larger and larger, it >wouldn't learn more and more. The aggregate of brain material became >computationally "stiff", in that the elements were too tightly >coupled. > >We know that GAs can evolve true intelligence, because we've evolved to >our present human intellect. But just knowing that GAs CAN work isn't >enough. The question now is: > > What genetic encoding will allow a large network to stay flexible and >be trained with many patterns? Complex biological (not just nervous) systems use strategies of hierarchical and sequential decomposition. Our brains, for example, are not randomly connected nets but have thousands of recognizable discrete structures, from tiny nuclei to huge cortical sheets. Many of these are repeated and some are found within others. Hierarchy and repetition seem to correspond to context-free and regular languages, respectively. I don't know much about GA's, but is seems to me that a grammatical encoding might be appropriate to get around that scale barrier. A "mutation" might involve not only tweaking some weights or connections, but repeating whole structures or putting them together in different ways. Just a half-baked idea. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ( Alan Filipski, GTX Corp, 8836 N. 23rd Avenue, Phoenix, Arizona 85021, USA ) ( {decvax,hplabs,uunet!amdahl,nsc}!sun!sunburn!gtx!al (602)870-1696 ) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ---------- ----------
kingsley@hpwrce.HP.COM (Kingsley Morse) (08/26/90)
Alan Filipski writes: >Complex biological (not just nervous) systems use strategies of >hierarchical and sequential decomposition. >Our brains, for example, are not randomly connected nets but have >thousands of recognizable discrete structures, from tiny nuclei to huge >cortical sheets. Many of these are repeated and some are found within >others. >Hierarchy and repetition seem to correspond to context-free and regular >languages, respectively. I don't know much about GA's, but is seems to >me that a grammatical encoding might be appropriate to get around that >scale barrier. A "mutation" might involve not only tweaking some >weights or connections, but repeating whole structures or putting them > together in different ways. I like the term "scale barrier". By the way, I wrote the GA so it would randomly duplicate parts of the genetic code, in hopes of "repeating whole structures or putting them together in different ways". Biological GAs call it duplication mutation. I may have done it wrong though, because the resultingly "stiff" brain was still thwarted by the "scale barrier". Would it make sense that cortical columns are the repeated and independent brain structures that are scalable? if so, should we hard code these in our GAs? Kingsley
spoffojj@hq.af.mil (Jason Spofford) (08/28/90)
In <3430013@hpwrce.HP.COM> kingsley@hpwrce.HP.COM (Kingsley Morse) writes: >I like the term "scale barrier". By the way, I wrote the GA so it would >randomly duplicate parts of the genetic code, in hopes of "repeating whole >structures or putting them together in different ways". Biological GAs >call it duplication mutation. I may have done it wrong though, because >the resultingly "stiff" brain was still thwarted by the "scale barrier". I was hoping you could expand a little on the GA/NN process you implemented. Do you evolve NN structures that are then subjected to a specific training algorithm, or do you evolve weights for a fixed structure or do you evolve both the weights and the structure? I have not reached the "scale barrier" as you mentioned. Perhaps it is because my problems are too small. What size/typeproblems are you attempting to solve? It could be that this scale barrier is a product of the scalability of the training algorithm (assuming you use one). On another note, I find the discussion on the brain's architecture very interesting. It's my OPINION that there are many types of neurons in biological systems. Some lend themselves to fixed behaviors and are relatively untrainable. For instance, neurons that control reflex actions. The genetic code must contain specific behaviors and connections for these neurons. On the other hand, some neurons are somehow influenced by training processes. They have very non-specific connections and behaviors. They are designed to be influenced by the environment and possibly by "Instructor" neurons. The genetic code in this instance does not directly encode behavior, it just sets the stage by which the behavior can be set or unset. Any other opinions? -- ---------------------------------------------------------- ) Jason Spofford The AF LAN Manager ( ) spoffojj@hq.af.mil George Mason Univ. Grad. Stud. ( ----------------------------------------------------------
usenet@nlm.nih.gov (usenet news poster) (09/16/90)
In article <schraudo.651012988@beowulf> schraudo@beowulf.ucsd.edu (Nici Schraudolph) writes: >I'd phrase it the other way round: most of the GA/NN research is aimed at >finding a GA (specifically, a genetic representation of NNs) for which >the recombination operator exploits some regularity concerning the basins >of attraction for NN gradient descent. The two main questions are: > >1) Are there any such regularities in the first place, aside from simple > invariances such as flipping the sign of all weights? Sure, suppose you want to train a net to perform multiple discrete tasks such as recgonize English and Spanish speech and you structure the net so that a group of nodes is devoted to each task. In addition, each task specific block of nodes must recognize when its domain is relevant. You then have forced a regularity on the net which gaurantees the existence of a recombindation operator (exchanging task specific nodes enblock). >2) Can we find genetic encodings and/or recombination operators that > exploit them? Yes, but in a sense it is trivial. If you knew the problems were separate, then why make life complicated by combining them in the first place? The existence of task specific groups of neurons in biological networks is obvious (the retina vs visual cortex vs ...). Recombination operators in the training of these nets is less obvious. >Nicol N. Schraudolph, C-014 nici%cs@ucsd.edu David States