markh@csd4.csd.uwm.edu (Mark William Hopkins) (09/19/90)
This is something you might want to try out if you have the resources and
time. A first attempt at implementation using a backpropagation net led to a
4 page program (with the neural net contained in a 200k data file).
Description:
Basically, this natural language generator is nothing more than a
sophisticated random number generator. What it does is basically learn a
set of pairs (C, x): where C is an N-symbol context, and x the symbol expected
to follow C.
It generates natural language by looking at the N most recently generated
symbols and then using the context to determine what should come next. It
them prints out the symbol, updates the context, and continues...
Symbols can be ASCII characters, or a restricted subset, or entire words,
or phonemes. Use anything that's appropriate.
The neural architecture can be any of your favorite architectures if you
think it'll get the job done.
Data structure used:
C ... A queue consisting of up to N symbols,
used to store the most recent N-symbol context.
Basic neural net primitives:
STORE(C, x); ... 'learn' the pair (C, x).
x = RECALL(C); ... produce the symbol x, based on context C
and what has been previously learned.
Learning phase:
Clear the queue C;
while (there is input to process) {
read a symbol x;
STORE the pair (C, x);
Add x to the queue C;
}
Output phase:
Initialize the queue C with 'seed' output.
while (there is request for more output) {
x = RECALL(C);
write the symbol x;
Add x to the queue C;
}
For those of you who decide to try it out, I'd be interested to know
of what kind of convergence properties result, how you encoded symbols,
and what kind of architecture you used to achieve the result.
Unless you introduce some 'noise' into the net, you're going to have a
finite state machine (though probably a huge one), which will eventually cycle.
My attempt was using a backpropagation neural net with one hidden layer and
ASCII-based I/O. It made a heroic effort at converging, especially when passed
through some highly repetitive program diasgnostic output, but couldn't quite
seem to get the hang of it. Since it was ASCII based, it had to make up for
a whole lot of lost ground, and was only able to get as far as anticipating
where capitals, small letters, and digits would occur...