Archive for the ‘information theory’ Category

EC517/Info Theory Reminders

Monday, May 19th, 2008

capacity of channel with given restriction on power and distortion

XOR distortion(x, y, xhat) = x (xor) y (xor) xhat
Determine capacity for various values of distortion

Power allocation P=P_1+P_2 for two Gaussian sources (variance sigma_i) conveyed over channels with AWGN (variance n_i).

Information Theoretic Feature Selection for Clustering

Saturday, May 3rd, 2008

Appropriate feature selection and weighting is crucial for clustering algorithms to successfully handle multi-dimensional data. A feature is relevant when it is correlated with the classification, mutually independent of other features, but possibly correlated with other features. For any feature, these characteristics, and hence the weighting, can be determined using information theoretic quantities, e.g., mutual information with other features and the veridical cluster assignment available from training data. An application of the technique to feature weighting in a speech separation task is presented.


(more…)