Do the ski gloves fit?

Stuart Dreyfus presents excerpts from Bellman, of dynamic programming fame, in RICHARD BELLMAN ON THE BIRTH OF DYNAMIC PROGRAMMING.

There’s a bit of research-related therapeutic talk here, and at least one provocative idea.

On logical scientific progress:

“Scientific developments can always be made logical and rational with sufficient hindsight. It is amazing, however, how clouded the crystal ball looks beforehand. We all wear such intellectual blinders and make such inexplicable blunders that it is amazing that any progress is made at all.
…
I strongly recommend the interesting study of these and related matters by Jacques Hadamard, the great French mathematician, in his book The Psychology of Invention in the Mathematical Field (Dover Publications, New York, 1945: paperback)”

On choosing your problem:

“Similarly, there are many questions that are difficult to answer, but hardly worth asking. The well-trained mathematician does not measure the value of a problem solely by its intractability. The challenge is there, but even very small boys do not accept all dares.”

…

“It is usually, if not always, impossible to predict where a theoretical investigation will end once started. But what one can be certain of is that the investigation of a meaningful scientific area will lead to meaningful mathematics. Inevitably, as soon as one pursues the basic theme of obtaining numerical answers to numerical questions, one will be led to all kinds of interesting and significant problems in pure mathematics”

1. Tools
   • NLTK
   • Python
   • HTK
   • CSLUK
   • Matlab Auditory Demonstrations
   • Applied Parallel Computing (SMA 5505) Spring 2005
   • Pervasive Human Centric Computing (SMA 5508) Spring 2006 (Python, device programming)
   • Automata, Computability, and Complexity Spring 2005
   • Computation Structures Fall 2002
2. Artificial Intelligence & Knowledge Representation
   • AI Fall 2006
   • Probabilistic Systems and Applied Prob
   • Techniques in AI
   • 6.034 Artificial Intelligence Spring 2005
   • Minds and Machines Spring 2007
   • Theory of Knowledge Fall 2003
   • Decisions, Games, and Rational Choice Spring 2008
   • Logic I Fall 2005
   • Logic II Spring 2004
3. Acoustics
   • Acoustics of Speech and Hearing
   • Speech Communication
   • The Human Intelligence Enterprise Spring 2002
   • The Human Intelligence Enterprise Spring 2006
   • Computational Cognitive Science Fall 2004
4. Computational Linguistics
   • Introduction to Philosophy of Language Spring 2006
   • Introduction to Philosophy of Language Spring 2005
   • Advanced Natural Language Processing
   • Natural Language and the Computer Representation of Knowledge
   • Language Processing Fall 2004
   • Language Acquisition I, Spring 2002
   • Introduction to Syntax Fall 2003
   • Advanced Syntax Spring 2007
   • Argument Structure and Syntax Spring 2003
   • Pragmatics in Linguistic Theory Fall 2006
   • Syntactic Models, Spring 2006
   • Advanced Semantics Spring 2005
   • Topics in Semantics Fall 2002
   • Psycholinguistics Spring 2005
   • The Linguistic Study of Bilingualism
   • Introduction to Linguistics Spring 2005
   • Introduction to Linguistics Spring 2004
   • Language and its Structure I: Phonology Fall 2002
   • Language and its Structure II: Syntax Fall 2003
   • Language and its Structure III: Semantics and Pragmatics
5. Experimental Linguistics & Psycholinguistics
   • Transcribing Prosodic Structure of Spoken Utterances with ToBI
   • Linguistic Phonetics Fall 2005
   • Introduction to Phonology Fall 2002
   • Topics in Linguistic Theory: Laboratory Phonology Spring 2007
   • Topics in Linguistics Theory Spring 2003
6. Philosophy
   • Topics in Philosophy of Language: Vagueness Fall 2005
   • Topics in Theory of Knowledge: A Priori Knowledge Spring 2005
   • Classification, Natural Kinds, and Conceptual Change: Race as a Case Study Spring 2004
7. Language-based Linguistics (as opposed to Concept-based)
   • Language Acquisition Fall 2001
   • opics in Linguistics: Creole Languages and Caribbean Identities Spring 2004
   • Linguistic Theory and the Japanese Language Fall 2004
   • Special Topics in Linguistics: Genericity Spring 2007
   • Grammar of a Less Familiar Language (Kurdish) Spring 2003
8. Compilers/PL
   • Multithreaded parallelism Languages and Compilers
   • Building Programming Experience: A Lead-In to 6.001
   • Computer Language Engineering

capacity of channel with given restriction on power and distortion

XOR distortion(x, y, xhat) = x (xor) y (xor) xhat
Determine capacity for various values of distortion

Power allocation P=P_1+P_2 for two Gaussian sources (variance sigma_i) conveyed over channels with AWGN (variance n_i).

EM and ML
McLaughlin
Rabiner: Tutorial on HMMs
Dempster: EM and applications
CR Rao: Maximum Likelihood
Erich Lehmann: Statistical Inference

Problem sets

ch 6-14
Kernel Trick from PRML
F-statistic
Lagrange multipliers
Reproducing kernel Hilbert spaces
Geometry and algebra related to projections (linear algebra)
canonical correlation
IEEE splines
calculus of variations
approximate inference
projection pursuit
OCW: basic inequalities and statistical tests
Fisher info
finance project

ICA, boosted trees, random forests,

monkey typewriter
pdfs

(Really useless without the author’s color illustrations.)

Exhibits for a superficial introduction to K-means clustering, extended to Gaussian mixture modeling, followed by a description of the Gap Statistic and multidimensional scaling.

Appropriate feature selection and weighting is crucial for clustering algorithms to successfully handle multi-dimensional data. A feature is relevant when it is correlated with the classification, mutually independent of other features, but possibly correlated with other features. For any feature, these characteristics, and hence the weighting, can be determined using information theoretic quantities, e.g., mutual information with other features and the veridical cluster assignment available from training data. An application of the technique to feature weighting in a speech separation task is presented.

(more…)

Hastie, Tibshirani & Friedman, Chapter 12.1-12.3, (without the authors’ color illustrations)

An introduction to support vector classifiers developed as formalization of notion of optimal separating hyperplane, leading to support vector machines by basis expansion of data. The kernel trick for easy adoption of arbitrary bases expansion (even when basis functions themselves are not known), and the use of SVM for regression is also presented.

Featured for keeping me busy this semester.

• Choose clear and spare expression, without repetition, and with pauses that let concepts sink into the students’ minds. This (besides mastery of the subject matter) lends the understated feel to lectures of masters like Naveen Garg and Uri Eden.
• Use slides only as exhibits to illustrate a point, not as the central tool for the lecture. Do not write more than two lines of text on a slide — it is supposed to anchor interest, not absorb all attention — and more text makes people read the words and gloss over the meaning.
• Ask questions about basic concepts being used, not so much to test, as much as to get students talking and making them link the new stuff to what they already know. Making students answer even trivial questions — that come along in regular business of the lecture — revives their interest.
• Prepare a clear lesson plan for every lecture. Going in with only understanding — however thorough — of the material but no lesson plan is a recipe for a shameful disaster.
• Prepare a clear plan for using the board (or learn to use it properly). Scribbling around, erasing useful equations, writing in gaps, overwriting, all drain away students’ attention. Besides the plan, practice using the board offline till you can use it coherently in class.
• Don’t gainsay yourself, or get ahead of yourself in lecturing. Saying “Oh, I said that, but I meant that only for sufficiently large n” is a good way to make students lose track of everything. A good way to avoid this is to have a clear lesson plan. Student attention is like a river, not like hypertext, it needs coherent flow.
• Avoid silly jokes (they have a talent for sounding funny when they creep in, but that’s just because you’re talking for more than an hour — everybody else will hate them, and you). Puns, visual jokes, and other word play are an annoyance to mature audience, and a distraction for everybody including you, the lecturer.
• Leave personal baggage outside the class. The connection that you are facilitating is between students and the subject; not the connection between you and the students, or you and the subject.
• Do not use adjectives to describe the class, and never ever use a negative adjective for the lecture/subject or any part of it; and show your enthusiasm about the subject by your excellence in teaching it instead.
• Explain the intent of the lecture: is it to convey a thorough understanding, or is it to give a flavor? Leaving this out makes students feel out of place or dumb, when they assume they’re understanding less than they are supposed to be able to.
• Motivate every idea, and keep the motivation visible at all times. To focus the lecture, it’s helpful to remind students of the motivating idea by pointing it out on the board.
• Do not present a solution without specifying the problem. For example, “We’re going to do additive models” is not a valid beginning, and it doesn’t help if the next thing you discuss is how backfitting, EM or your favorite algorithm does a good job of computing the parameters. “We got some data from ____, and it looks like this. To figure out why this could be the case, we model it as a ____. This is a tough problem, so we make simplifying assumptions ______ which lead us to additive models which we’ll talk about today.”
• Avoid references to new material that’s no longer on the board. Either keep it on the board, or point it out in a book, or don’t mention it. It’s not good enough to say “On the last slide we saw…”, “This Q is the cost function for the formula we just derived,” If they’re writing it all down, this might work, but this encourages students to tune out.
• Do not drop references to material you’re not discussing. It makes the referred material sound tough or daunting than it actually ever can be.

Links for learning R, clone of the statistical processing language S from Bell Labs, roughly along the abscissa of the learning curve.

• R Tutorial: Using R A quick introduction
• Statistical Computing with R: A tutorial An introduction with graphics — a big draw for R
• R Course A course, self-contained — maybe it should be at the top and be the only link here.
• R Tutorial A course-like tutorial to the basics of R