Lecture 14 Dimensionality Reduction II: Feature Extraction short version STAT 479: Machine Learning, Fall 2018 Sebastian Raschka http://stat.wisc.edu/~sraschka/teaching/stat479-fs2018/ Sebastian Raschka STAT 479: Machine Learning FS 2018 1 Dimensionality Reduction Feature Selection Feature Extraction Today Sebastian Raschka STAT 479: Machine Learning FS 2018 2 Dimensionality Reduction Feature Selection Feature Extraction Linear Methods Principal Component Analysis (PCA) Independent Component Analysis (ICA) Autoencoders (linear act. func.) Singular Vector Decomposition (SVD) Linear Discriminant Analysis (LDA) (Supervised) ... Nonlinear Methods Sebastian Raschka t-Distr. Stochastic Neigh. Emb. (t-SNE) Uniform Manifold Approx. & Proj. (UMAP) Kernel PCA Spectral Clustering Autoencoders (non-linear act. func.) ... STAT 479: Machine Learning FS 2018 3 Goals of Dimensionality Reduction Reduce Curse of Dimensionality problems Increase storage and computational efficiency Visualize Data in 2D or 3D Sebastian Raschka STAT 479: Machine Learning FS 2018 4 Principal Component Analysis (PCA) 1) Find directions of maximum variance x2 PC2 PC 2 PC1 PC2 x2 x1 PC1 Sebastian Raschka STAT 479: Machine Learning FS 2018 5 Principal Component Analysis (PCA) 2) Transform features onto directions of maximum variance x2 PC2 x2 PC1 PC2 x2 PC 2 PC 2 PC1 PC2 x2 PC 1 PC2 x1 x1 PC1 Sebastian Raschka STAT 479: Machine Learning PC 1 PC1 FS 2018 6 Principal Component Analysis (PCA) 3) Usually consider a subset of vectors of most variance (dimensionality reduction) x2 PC 2 PC2 PC2 PC 1 PC1 PC1 x2 x1 PC1 Sebastian Raschka STAT 479: Machine Learning FS 2018 7 Principal Component Analysis (PCA) (in a nutshell) Given design matrix X ∈ ℝn×m find vector αi with maximum variance repeat: find αi+1 with maximum variance uncorrelated with αi (repeat k times, where k is the desired number of dimensio