Lecture 2

Supervised learning methods

Memory-based algorithms: K-nearest neighbors (K-NN)
Approaching supervised learning problems fairly and systematically
Training, testing, validation, and cross-validation
ROC curves and the c-index
Resampling: bootstrapping, boosting, bagging
Combining systems: mixing models and voting
Data preparation: component analysis
Brief introduction to statistical pattern recognition and Bayesian estimation

Memory-based algorithms

• Duda, Richard O., Hart, Peter E., and Stork, David (2001) Pattern Classification. Second Edition. New York: Wiley.
  • Section 4.1-4.6: Nonparametric techniques, pp. 161-192
    
• http://en.wikipedia.org/wiki/KNN

Training, testing, validation, and cross-validation

• Duda, Richard O., Hart, Peter E., and Stork, David (2001) Pattern Classification. Second Edition. New York: Wiley.
  • Section 9.6.2: Cross-validation , pp. 483-485.

C-index and ROC curves

• http://en.wikipedia.org/wiki/Roc_curve
• Duda, Richard O., Hart, Peter E., and Stork, David (2001) Pattern Classification. Second Edition. New York: Wiley.
  • Section 2.8.3: Signal detection theory and operating characteristics, pp. 48-51.

Resampling: Bootstrapping, boosting, bagging

• http://en.wikipedia.org/wiki/Resampling_%28statistics%29
• Duda, Richard O., Hart, Peter E., and Stork, David (2001) Pattern Classification. Second Edition. New York: Wiley.
  • Section 9.4: Resampling for estimating statistics, pp. 471-475.
    • 9.4.1 Jackknife http://en.wikipedia.org/wiki/Resampling_%28statistics%29#Jackknife
    • 9.4.2 Bootstrap http://en.wikipedia.org/wiki/Bootstrapping_%28statistics%29
  • Section 9.5: Resampling for classifier design, pp. 475-482
    • 9.5.1 Bagging http://en.wikipedia.org/wiki/Bootstrap_aggregating
    • 9.5.2 Boosting http://en.wikipedia.org/wiki/Boosting

Mixing models and voting

• Duda, Richard O., Hart, Peter E., and Stork, David (2001) Pattern Classification. Second Edition. New York: Wiley.
  • Section 9.7: Combining classifiers, pp. 495-499.
• Carpenter, Gail A., Grossberg, Stephen, Markuzon, Natalya, Reynolds, John H., and Rosen, David B. (1992) Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps. IEEE Transactions on Neural Networks, 3, 698–713.

Component analysis

• Duda, Richard O., Hart, Peter E., and Stork, David (2001) Pattern Classification. Second Edition. New York: Wiley.
  • Section 3.8: Component analysis and discriminants, pp. 114-124
    • 3.8.1 Principal component analysis (PCA) http://en.wikipedia.org/wiki/Principal_Component_Analysis
    • 3.8.2 Fisher linear discriminant http://en.wikipedia.org/wiki/Fisher_linear_discriminant
    • 3.8.3 Multiple discriminant analysis

Maximum-likelihood and Bayesian parameter estimation

• http://en.wikipedia.org/wiki/Thomas_Bayes
• Duda, Richard O., Hart, Peter E., and Stork, David (2001) Pattern Classification. Second Edition. New York: Wiley, pp. 84-97
  • 3.1 Introduction
  • 3.2 Maximum-likelihood estimation http://en.wikipedia.org/wiki/Maximum_likelihood
  • 3.3 Bayesian estimation http://en.wikipedia.org/wiki/Bayes%27_theorem
  • 3.4 Bayesian parameter estimation: Gaussian case