Lecture 6

Associative memory networks

Back propagation
Multi-layer perceptrons
(Local) minimization of cost functions
Radial basis functions (RBFs)
Cascade–correlation architecture
Higher order networks

The perceptron

• http://en.wikipedia.org/wiki/Perceptron
• Rosenblatt, F. (1958) The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65, 386–408.
• Duda, Richard O., Hart, Peter E., and Stork, David (2001) Pattern Classification. Second Edition. New York: Wiley.
  • Sections 5.1-5.8: Linear discriminant functions, pp. 215-249 http://en.wikipedia.org/wiki/Linear_discriminant_analysis

Back propagation and cost functions

• http://en.wikipedia.org/wiki/Multilayer_perceptron
• http://en.wikipedia.org/wiki/Back_propagation
• http://en.wikipedia.org/wiki/Cost_function
• Duda, Richard O., Hart, Peter E., and Stork, David (2001) Pattern Classification. Second Edition. New York: Wiley.
  • Sections 6.1-6.8: Multilayer neural networks, pp. 282-318.
• Rumelhart, David E., Hinton, Geoffrey E., & Williams, Ronald J. (1986) Learning internal representations by error propagation. Part 2 Part 3 Part 4 Part 5 Part 6. In David E. Rumelhart and James L. McClelland (Eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition I. Cambridge, Mass.: MIT Press. pp. 318–362.

Radial basis functions (RBFs)

• http://en.wikipedia.org/wiki/Radial_basis_function
• Moody, John, and Darken, Christian J. (1989) Fast learning in networks of locally–tuned processing units. Neural Computation, 1, 281–294.
• Lowe, David (2003) Radial basis function networks. In Arbib, Michael A. (2003) The Handbook of Brain Theory and Neural Networks, Second Edition. Cambridge, Mass.: MIT Press. pp. 937-940.
• Duda, Richard O., Hart, Peter E., and Stork, David (2001) Pattern Classification. Second Edition. New York: Wiley.
  • Section 6.10.1: Radial basis function Networks (RBFs), pp. 324-325

Cascade–correlation network

• http://en.wikipedia.org/wiki/Cascade_correlation
• Fahlman, Scott E., and Lebiere, Christian (1990) The cascade–correlation learning architecture. In David S. Touretzky (Ed.) Neural Information Processing Systems 2, Proceedings of the NIPS Conference, Denver, 1989, San Mateo, Calif.: Morgan Kaufmann Publishers. pp. 524–532. http://www.cs.iastate.edu/~honavar/fahlman.pdf
• Duda, Richard O., Hart, Peter E., and Stork, David (2001) Pattern Classification. Second Edition. New York: Wiley.
  • Section 6.10.6: Cascade-correlation, pp. 329-330.

Higher order networks

• Giles, C. Lee, and Maxwell, Thomas (1987) Learning, invariance, and generalization in high–order neural networks. Applied Optics, 26, 4972–4978.