Trang 1
Lecture 14 Dimensionality Reduction II: Feature Extraction short version STAT 479: Machine Learning, Fall 2018 Sebastian Raschka http://stat.wisc.edu/~sraschka/teaching/stat479-fs2018/ Sebastian Raschka STAT 479: Machine Learning FS 2018 1
Trang 2
Dimensionality Reduction Feature Selection Feature Extraction Today Sebastian Raschka STAT 479: Machine Learning FS 2018 2
Trang 3
Dimensionality Reduction Feature Selection Feature Extraction • Principal Component Analysis (PCA) Linear • Independent Component Analysis (ICA) • Autoencoders (linear act. func.) Methods • Singular Vector Decomposition (SVD) • Linear Discriminant Analysis (LDA) (Supervised) • ... Nonlinear • t-Distr. Stochastic Neigh. Emb. (t-SNE) • Uniform Manifold Approx. & Proj. (UMAP) Methods • Kernel PCA • Spectral Clustering • Autoencoders (non-linear act. func.) • ... Sebastian Raschka STAT 479: Machine Learning FS 2018 3
Trang 4
Goals of Dimensionality Reduction • Reduce Curse of Dimensionality problems • Increase storage and computational efficiency • Visualize Data in 2D or 3D Sebastian Raschka STAT 479: Machine Learning FS 2018 4
Trang 5
Principal Component Analysis (PCA) 1) Find directions of maximum variance x PC2 2 PC1 PC2 x PC2 2 x 1 PC1 Sebastian Raschka STAT 479: Machine Learning FS 2018 5
Trang 6
Principal Component Analysis (PCA) 2) Transform features onto directions of maximum variance x PC2 2 x PC2 2 PC1 PC2 PC1 PC2 PC1 PC1 x PC2 2 x PC2 2 x 1 x 1 PC1 PC1 Sebastian Raschka STAT 479: Machine Learning FS 2018 6
Trang 7
Principal Component Analysis (PCA) 3) Usually consider a subset of vectors of most variance (dimensionality reduction) x PC2 2 PC1 PC1 PC2 PC2 PC1 x2 x 1 PC1 Sebastian Raschka STAT 479: Machine Learning FS 2018 7
Trang 8
Principal Component Analysis (PCA) (in a nutshell) Given design matrix X ∈ ℝn×m find vector αi with maximum variance repeat: find αi+1 with maximum variance uncorrelated with αi (repeat k times, where k is the desired number of dimensions; k ≤ m ) Sebastian Raschka STAT 479: Machine Learning FS 2018 8
Trang 9
Principal Component Analysis (PCA) Two approaches to solve PCA (on standardized data): 1. Constrained maximization (e.g., Lagrange multipliers) 2. Eigen-decomposition of covariance matrix directly Sebastian Raschka STAT 479: Machine Learning FS 2018 9
Trang 10
Principal Component Analysis (PCA) (in a nutshell) Collect vectors αi in a projection matrix A ∈ ℝm×k (Sorted from highest to lowest associated eigenvalue) Compute projected data points: Z = XA Sebastian Raschka STAT 479: Machine Learning FS 2018 10
Trang 11
Principal Component Analysis (PCA) Usually useful to plot the explained variance (normalized eigenvalues) Sebastian Raschka STAT 479: Machine Learning FS 2018 11
Trang 12
Principal Component Analysis (PCA) Keep in mind that PCA is unsupervised! Sebastian Raschka STAT 479: Machine Learning FS 2018 12
Trang 13
PCA Factor Loadings • The loadings are the unstandardized values of the eigenvectors • We can interpret the loadings as the covariances (or correlation in case we standardized the input features) between the input features and the and the principal components (or eigenvectors), which have been scaled to unit length Sebastian Raschka STAT 479: Machine Learning FS 2018 13
Trang 15
Mirrored Results in PCA • Not due to an error; reason for this difference is that, depending on the eigensolver, eigenvectors can have either negative or positive signs For instance, if v is an eigenvector of a matrix Σ , we have Σv = λv, where λ is the eigenvalue then −λ is also an eigenvalue of the same value since Σ(−v) = − Σv = − λv = λ(−v) Sebastian Raschka STAT 479: Machine Learning FS 2018 15
Trang 16
t-Distributed Stochastic Neighbor Embedding (t-SNE) VAN DER MAATEN AND HINTON (t-SNE is only meant for visualization not for preparing datasets!) 0 1 2 3 4 5 Note that MNIST has 6 7 28 x 28 = 784 dimensions 8 9 (a) Visualization by t-SNE. Shown are 6000 images from MNIST projected in 2D Maaten, L. V. D., & Hinton, G. (2008). Visualizing data using t-SNE. Journal of machine learning research, 9(Nov), 2579-2605. Sebastian Raschka STAT 479: Machine Learning FS 2018 16
Trang 17
Stochastic Nearest Neighbor Embeddings (SNE) Given high-dimensional datapoints, x ,...,x ∈ ℝm 1 n represent intrinsic structure of the data in 1D, 2D, or 3D (for visualization) How? 1) Model neighboring datapoint pairs based on the distance of those points in the high-dimensional space 2) Find a probability distribution of the pairwise distances in the low dimensional space that is as close as possible as the original probability distribution Main Idea: Map points near on a manifold to a near position in low-dimensional space Sebastian Raschka STAT 479: Machine Learning FS 2018 17
Trang 18
Stochastic Nearest Neighbor Embeddings (SNE) pi|j + pj|i Based on probability of selecting neighboring points pij = 2n (this is a modification to make the entropy [later slides] symmetric) where the conditional probability, pj|i, that xi would pick xj as its neighbor if neighbors were picked in proportion to their probability density under a Gaussian centered at xi exp( − ||x − x ||2 /2σ2) σ p = i j i neighborhood size is controlled by i j|i ∑ exp( − ||x − x ||2 /2σ2) (in turn controlled by perplexity parameter) k≠i i k i Denominator makes sure that similarity is independent of the point's density For reference, note that the normal distribution is defined as 1 (x − μ)2 p(x|μ,σ) = exp( − ) 2πσ2 2σ2 Sebastian Raschka STAT 479: Machine Learning FS 2018 18
Trang 19
t-Distributed Stochastic Neighbor Embedding (t-SNE) (1 + ||z − z ||2)−1 q = i j in t-SNE, modeled with Student's t-distribution ij ∑ (1 + ||z − z ||2)−1 k≠i i k to prevent crowding problem where zi and zj are the points in the low-dimensional space • t-distribution has "fatter" tails (more scale invariant to points far away) • t-distribution avoids crowding problem • (minor point: is faster for computing the density; no exponential) Sebastian Raschka STAT 479: Machine Learning FS 2018 19
Trang 20
t-Distribution With 1 degree of freedom same as Cauchy distribution Sebastian Raschka STAT 479: Machine Learning FS 2018 20
Hiển thị 20 trang đầu. Tải tài liệu để xem đầy đủ.
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!