own@castle.ed.ac.uk (O Morgan) (04/18/91)
Greetings,
I'm about to start work on a project involving neural networks,
and would like some indication of the feasibility of the project,
as I only have abstract notions of NN's rather than practical
experience. This is therefore to see if I can attract any
helpful comments from this group.
The project involves trying to structure an existing database
into some sort of PDP system. The reasons for this are to
provide the model with some of the qualities of NN, such as
tolerance to noisy input, associative memory and the ability to
induce the model from a data set.
The database is a socio-economic database of 200 farmers, with up
to 500 items of information relating to each farmer. (NB: all
references below to fields refer to database fields 8-).
1) My first question is whether a problem of this scale is likely
to succeed or not. In terms of current work, does this pass as a
small, medium or large application?
2) So far, having browsed a few books, I can see some
possibilities: One is the back propagation type (BP), another the
interactive competition (IAC) model, as they are described in the
Rumelhart and McClelland book. Hopfield nets also look very
interesting. As I see it, the relative merits/disadvantages of
each approach are:
The advantage of IAC over BP (as I see it) is that the model
would be a very flexible associative memory. The user could
provide values for any set of fields, and the model would provide
a guess for all the missing values. In the case of BP, the Input
values and the Output values are set prior to the learning stage,
so the model can only be used "in one direction".
The disadvantage of IAC (as I've understood it) is that every
database record is explicitly recorded in the model. Ie: to
incorporate farm X in the model, a "variables" 'farm_X' and
'_farm_X' are created in the representation. The number of these
ID variables will increase in proportion to the size of the
database. On the other hand, in BP, the learning process
amalgamates all the records so that with the introduction of a
new record, the size of the model is constant, only the (some)
weights are changed to take account of the new data.
Here I would ask a question: is there a representation of NN
that can combine both these advantages, namely flexibility at
run-time in the choice of inputs and outputs, and the compactness
of the BP model. Hopfield nets look as if they might have this
quality, but none of the books I've seen (3) give any idea as to
how you train such networks.
3) Currently I am reading up various references and intend using
the McClelland+Rumelhart software to experiment options. Is this
a practical vehicle? Does anyone have any suggestion w.r.t. other
environments that might prove useful. In this sort of project,
do you eventually end up writing your own code?
4) The data base I am using contains large amounts of symbolic
information. Is this sort of data readily processed by Neural
networks (assuming it is given a numeric representation).
Thanks in advance,
Olly Morgan
--
----------------------------------------------------------------------------
Olly Morgan @ Scottish Agricultural College, Edinburgh EH9 2HH, Scotland
Tel: (+44 31) 662 4395 E.Mail: O.Morgan@ed.ac.uk
----------------------------------------------------------------------------slehar@park.bu.edu (Steve Lehar) (04/24/91)
The prime consideration in whether or not to use a neural paradigm for a database implementation is the nature of the information to be stored. If the information is essentially deterministic and reliable, and the operations performed on such data explicit and well defined, then you are better off using a conventional database program. The neural approach holds promise only for those kinds of applications where a conventional database is no good, i.e. where the information stored or the retrieval operations are in some sense fuzzy or imprecise. For instance, if your data represents the color of a farmer's fields as detected by satellite sensors at various bands, and the information you would like to retrieve is the viability of his crops, this information is full of ambiguity. First of all, the color of the fields does not fall into discrete categories, but represents a continuum. Secondly, the mapping from color to viability is by no means explicit and well defined, and depends on a host of other considerations, such as geographic location, current season, recent weather, etc. No conventional database package could retrieve this kind of information without an explicit algorithm for how to compute viability from color. A neural algorithm however can be trained to correlate viability and color from known data, and can sometimes generalize this to new data (*note). On the other hand, if your data represents names, addresses, incomes, crops, etc. and you would like to retrieve the names of all corn farmers in Idaho with income exceeding 50,000 and with names between A and K, then a conventional database is the only way to go. The translation from fuzzy to hard has already been made implicitly in the gathering of the data - (Did the income figure include that swap of the old pick-up with his brother-in-law's tractor, and how much was that old tractor REALLY worth, and so forth...) so that now you would like to retrieve only those farmers whose REPORTED income exceeds 50,000, and you would consider it an error if the program infered from other sources that this figure was wrong and therefore included other names in the list. In other words, you would like the program to be the perfect bureaucrat, "Hey, I'm just doin' my job- those are the rules, I'm not paid to think...". If you do choose to go the neural route, you should be aware that many of the paradigms in the literature are famous for their historical importance or the interesting issues they raise, but are not ripe for practical application. For example the so-called "Hopfield net" (Actually the Grossberg Additive model [1], see [2] for explaination) introduces the interesting notion of auto-associative feedback networks. In their raw form however these networks have been shown to have a very limited storage capacity, and an "unfriendly" behavior when over-loaded, i.e. they don't complain, they just start giving wrong answers. I am less familiar with the Interactive Competition model (IAC) of Rummelhart and McClelland, except that I know that it was very influential in introducing connectionist concepts to linguists and psychologists, and is much referenced in those fields. I am not familiar with the practical implementations of the model, but I would suspect that these have been of a prototype nature. Such multi-level feedback models are fraught with dynamic subtleties that are often difficult to balance, and I would be interested to hear if someone had made practical use of this paradigm. [1] Grossberg, Math. Biosci. 4, 201 (1969); Grossberg and Pepe, J. Stat. Phys. 3, 95 (1971) [2] Carpenter, Cohen, & Grossberg "COMPUTING WITH NEURAL NETWORKS" (1987) Science, 235, 1226-1227. *note to "AI-vs-NN flamesters: this sentence is not to be construed as a claim that the neural model cannot be implemented with conventional algorithms, nor indeed be expressed in more conventional mathematical terms. Of course it can. Sometimes however it is more convenient to express these operations as a "neural" paradigm. -- (O)((O))(((O)))((((O))))(((((O)))))(((((O)))))((((O))))(((O)))((O))(O) (O)((O))((( slehar@park.bu.edu )))((O))(O) (O)((O))((( Steve Lehar Boston University Boston MA )))((O))(O) (O)((O))((( (617) 424-7035 (H) (617) 353-6741 (W) )))((O))(O) (O)((O))(((O)))((((O))))(((((O)))))(((((O)))))((((O))))(((O)))((O))(O)
slehar@park.bu.edu (Steve Lehar) (04/25/91)
The prime consideration in whether or not to use a neural paradigm for a database implementation is the nature of the information to be stored. If the information is essentially deterministic and reliable, and the operations performed on such data explicit and well defined, then you are better off using a conventional database program. The neural approach holds promise only for those kinds of applications where a conventional database is no good, i.e. where the information stored or the retrieval operations are in some sense fuzzy or imprecise. For instance, if your data represents the color of a farmer's fields as detected by satellite sensors at various bands, and the information you would like to retrieve is the viability of his crops, this information is full of ambiguity. First of all, the color of the fields does not fall into discrete categories, but represents a continuum. Secondly, the mapping from color to viability is by no means explicit and well defined, and depends on a host of other considerations, such as geographic location, current season, recent weather, etc. No conventional database package could retrieve this kind of information without an explicit algorithm for how to compute viability from color. A neural algorithm however can be trained to correlate viability and color from known data, and can sometimes generalize this to new data (*note). On the other hand, if your data represents names, addresses, incomes, crops, etc. and you would like to retrieve the names of all corn farmers in Idaho with income exceeding 50,000 and with names between A and K, then a conventional database is the best way to go. The translation from fuzzy to hard has already been made implicitly in the gathering of the data - (Did the income figure include that swap of the old pick-up with his brother-in-law's tractor, and how much was that old tractor REALLY worth, and so forth...) so that now you would like to retrieve only those farmers whose REPORTED income exceeds 50,000, and you would consider it an error if the program infered from other sources that this figure was wrong and therefore included other names in the list. In other words, you would like the program to be the perfect bureaucrat, "Hey, I'm just doin' my job- those are the rules, I'm not paid to think...". If you do choose to go the neural route, you should be aware that many of the paradigms in the literature are famous for their historical importance or the interesting issues they raise, but are not ripe for practical application. For example the so-called "Hopfield net" (Actually the Grossberg Additive model [1], see [2] for explaination) introduces the interesting notion of auto-associative feedback networks. In their raw form however these networks have been shown to have a very limited storage capacity, and an "unfriendly" behavior when over-loaded, i.e. they don't complain, they just start giving wrong answers. I am less familiar with the Interactive Competition model (IAC) of Rummelhart and McClelland, except that I know that it was very influential in introducing connectionist concepts to linguists and psychologists, and is much referenced in those fields. I am not familiar with the practical implementations of the model, but I would suspect that these have been of a prototype nature. Such multi-level feedback models are fraught with dynamic subtleties that are often difficult to balance, and I would be interested to hear if someone had made practical use of this paradigm. [1] Grossberg, Math. Biosci. 4, 201 (1969); Grossberg and Pepe, J. Stat. Phys. 3, 95 (1971) [2] Carpenter, Cohen, & Grossberg "COMPUTING WITH NEURAL NETWORKS" (1987) Science, 235, 1226-1227. *note to "AI-vs-NN flamesters: this sentence is not to be construed as a claim that the neural model cannot be implemented with conventional algorithms, nor indeed be expressed in more conventional mathematical terms. Of course it can. Sometimes however it is more convenient to express these operations as a "neural" paradigm. -- (O)((O))(((O)))((((O))))(((((O)))))(((((O)))))((((O))))(((O)))((O))(O) (O)((O))((( slehar@park.bu.edu )))((O))(O) (O)((O))((( Steve Lehar Boston University Boston MA )))((O))(O) (O)((O))((( (617) 424-7035 (H) (617) 353-6741 (W) )))((O))(O) (O)((O))(((O)))((((O))))(((((O)))))(((((O)))))((((O))))(((O)))((O))(O)