Lecture 07 Ensemble Methods STAT 479: Machine Learning, Fall 2018 Sebastian Raschka http://stat.wisc.edu/~sraschka/teaching/stat479-fs2018/ Sebastian Raschka STAT 479: Machine Learning FS 2018 1 Overview Majority Voting Bagging Ensemble Methods Boosting Random Forests Stacking Sebastian Raschka STAT 479: Machine Learning FS 2018 2 Majority Voting Sebastian Raschka STAT 479: Machine Learning FS 2018 3 Unanimity Majority Plurality Sebastian Raschka STAT 479: Machine Learning FS 2018 4 Majority Vote Classifier Training set h1 h2 ... hn Predictions y1 y2 ... yn New data Classification models Voting Final prediction yf yf̂ = mode{h1(x), h2(x), . . . hn(x)} where hi(x) = yî Sebastian Raschka STAT 479: Machine Learning FS 2018 5 Why Majority Vote? assume n independent classifiers with a base error rate ϵ here, independent means that the errors are uncorrelated assume a binary classification task assume the error rate is better than random guessing (i.e., lower than 0.5 for binary classification) ∀ϵi ∈ {ϵ1, ϵ2, . . . , ϵn}, ϵi < 0.5 Sebastian Raschka STAT 479: Machine Learning FS 2018 Why Majority Vote? assume n independent classifiers with a base error rate ϵ here, independent means that the errors are uncorrelated assume a binary classification task assume the error rate is better than random guessing (i.e., lower than 0.5 for binary classification) ∀ϵi ∈ {ϵ1, ϵ2, . . . , ϵn}, ϵi < 0.5 The probability that we make a wrong prediction via the ensemble if k classifiers predict the same class label n k n−k P(k) = ϵ (1 − ϵ) (k) Sebastian Raschka STAT 479: Machine Learning k > ⌈n/2⌉ FS 2018 Why Majority Vote? The probability that we make a wrong prediction via the ensemble if k classifiers predict the same class label n k P(k) = ϵ (1 − ϵ)n−k (k) k > ⌈n/2⌉ Ensemble error: n k ϵens = ϵ (1 − ϵ)n−k ∑ (k) k n 11 ϵens = 0.25k(1 − 0.25)11−k = 0.034 ∑( k ) k=6 11 Sebastian Raschka STAT 479: Machine Lea