akkh@mullian.ee.mu.OZ.AU (A. Hui) (09/17/90)
Hi all netters,
It is me again! Let me stir up some discussions this time.
The following is a summary I produced on the issue of efficiency and
Implementation of NN. Any comments?? Any admenment you experts
feel like to add?? What I want is eventually built up a comprehensive list
on the subject of efficiency and implementation issues on NN.
Thanks in advance for any suggestion!!
I will summarize to the net.
1. Efficiency Issues
1.1. Training time
-data and initial weights dependent
-don't use zero as input, otherwise no learning will be produced
-local minima problem
- dimensionity of the inputs : Networks converge faster for high dimension
input data because high dimensional input data can be distinguished easier :
example 16x16 images is easier to be distinguished than 8x8 images.
1.2 Various parameters
-learning rate
-to remenber as much previous info, the cell should change as little as possible when learning sth. new
-large does not mean faster, too large can cause weight instability
-optimal when inversely proportional to the no. of hidden neurons and the first derivatives of the activation function.
-same learning rate vs different learning rate for different node in the network :
distribute learning evenly thru the network vs encouraging competition.
1.3 Techniques
-various techniques : line search, use of momentum term, GRU, local interaction
heuristic,
-related : Conj. Grad., Newtons' 2nd
2. Implementation Issues
2.1. Network size
-3 layers is enough for everything
-if input is irregular, no. of hidden neurons required = no. of training
patterns otherwise, dependent on the type of regularity inherited
2.2 Generalization
- use noisy input to train a network instead of a clean pattern
- fewer hidden neurons generalize better
2.3 Various techniques : network pruning, net growing
-network pruning
- start with a large network
- once trained, the net was analysis for any reduntance neurons
- all relunctance neurons were pruned.
-growing a net
- grow the network as required until the network learn all the input patterns
===============================================================================
Alvaro Hui |ACSnet akkh@mullian.oz
4th Year B.E.\ B.Sc. |Internet & akkh@mullian.ee.mu.OZ.AU
University of Melbourne |Arpanet rcoahk@koel.co.rmit.OZ.AU