Page 1
Lecture 07 Ensemble Methods STAT 479: Machine Learning, Fall 2018 Sebastian Raschka http://stat.wisc.edu/~sraschka/teaching/stat479-fs2018/ Sebastian Raschka STAT 479: Machine Learning FS 2018 1
Page 2
Overview Majority Voting Bagging Ensemble Methods Boosting Random Forests Stacking Sebastian Raschka STAT 479: Machine Learning FS 2018 2
Page 5
Majority Vote Classifier Training set New data Classification h h . . . h models 1 2 n Predictions y y . . . y 1 2 n Voting Final prediction y yf̂ = mode{h1(x),h2(x),...hn(x)} f where hi(x) = yî Sebastian Raschka STAT 479: Machine Learning FS 2018 5
Page 6
Why Majority Vote? • assume n independent classifiers with a base error rate ϵ • here, independent means that the errors are uncorrelated • assume a binary classification task • assume the error rate is better than random guessing (i.e., lower than 0.5 for binary classification) ∀ϵi ∈ {ϵ1,ϵ2,...,ϵn},ϵi < 0.5 Sebastian Raschka STAT 479: Machine Learning FS 2018
Page 7
Why Majority Vote? • assume n independent classifiers with a base error rate ϵ • here, independent means that the errors are uncorrelated • assume a binary classification task • assume the error rate is better than random guessing (i.e., lower than 0.5 for binary classification) ∀ϵi ∈ {ϵ1,ϵ2,...,ϵn},ϵi < 0.5 The probability that we make a wrong prediction via the ensemble if k classifiers predict the same class label n P(k) = ϵk(1 − ϵ)n−k k > ⌈n/2⌉ (k) Sebastian Raschka STAT 479: Machine Learning FS 2018
Page 8
Why Majority Vote? The probability that we make a wrong prediction via the ensemble if k classifiers predict the same class label n P(k) = ϵk(1 − ϵ)n−k k > ⌈n/2⌉ (k) Ensemble error: n n ϵ = ϵk(1 − ϵ)n−k ens ∑(k) k 11 11 ϵ = 0.25k(1 − 0.25)11−k = 0.034 ens ∑( k ) k=6 Sebastian Raschka STAT 479: Machine Learning FS 2018
Page 10
"Soft" Voting n ŷ = argmax∑wipi,j j i=1 pi,j predicted class membership : probability of the ith classifier for class label j wj: optional weighting parameter, default wi = 1/n,∀wi ∈ {w1,...,wn} 10 Sebastian Raschka STAT 479: Machine Learning FS 2018
Page 11
"Soft" Voting Use only for well-calibrated classifiers! n ŷ = argmax∑wipi,j j i=1 pi,j predicted class membership : probability of the ith classifier for class label j wj: optional weighting parameter, default wi = 1/n,∀wi ∈ {w1,...,wn} 11 Sebastian Raschka STAT 479: Machine Learning FS 2018
Page 12
"Soft" Voting n ŷ = argmax∑wipi,j j i=1 Binary classification example j ∈ {0,1} hi(i ∈ {1,2,3}) h1(x) → [0.9,0.1] h2(x) → [0.8,0.2] h3(x) → [0.4,0.6] 12 Sebastian Raschka STAT 479: Machine Learning FS 2018
Page 13
n "Soft" Voting ŷ = argmax∑wipi,j j i=1 Binary classification example j ∈ {0,1} hi(i ∈ {1,2,3}) h1(x) → [0.9,0.1] h2(x) → [0.8,0.2] h3(x) → [0.4,0.6] p(j = 0|x) = 0.2 ⋅ 0.9 + 0.2 ⋅ 0.8 + 0.6 ⋅ 0.4 = 0.58 p(j = 1|x) = 0.2 ⋅ 0.1 + 0.2 ⋅ 0.2 + 0.6 ⋅ 0.6 = 0.42 ŷ = argmax{p(j = 0|x), p(j = 1|x)} j 13 Sebastian Raschka STAT 479: Machine Learning FS 2018
Page 14
Bagging (Bootstrap Aggregating) Breiman, L. (1996). Bagging predictors. Machine learning, 24(2), 123-140. Sebastian Raschka STAT 479: Machine Learning FS 2018 14
Page 15
Algorithm 1 Bagging 1: Let n be the number of bootstrap samples 2: 3: for i=1 to n do 4: Draw bootstrap sample of size m, Di 5: Train base classifier hi on Di 6: yˆ = mode{h1(x), ..., hn(x)} Sebastian Raschka STAT 479: Machine Learning FS 2018 15
Page 16
Bootstrap Sampling Original Dataset x x x x x x x x x x 1 2 3 4 5 6 7 8 9 10 Bootstrap 1 x x x x x x x x x x x x x 8 6 2 9 5 8 1 4 8 2 3 7 10 Bootstrap 2 x x x x x x x x x x x x 10 1 3 5 1 7 4 2 1 8 6 9 Bootstrap 3 x x x x x x x x x x x x x x 6 5 4 1 2 4 2 6 9 2 3 7 8 10 Training Sets Test Sets Sebastian Raschka STAT 479: Machine Learning FS 2018 16
Page 17
Bootstrap Sampling 1 n P(not chosen) = (1 − ) , n 1 ≈ 0.368, n → ∞. e Sebastian Raschka STAT 479: Machine Learning FS 2018 17
Page 18
1 n P(not chosen) = (1 − ) , n 1 ≈ 0.368, n → ∞. e 1 n P(chosen) = 1 − (1 − ) ≈ 0.632 n Sebastian Raschka STAT 479: Machine Learning FS 2018 18
Page 19
Bootstrap Sampling Training Bagging Bagging example round 1 round 2 … indices 1 2 7 … 2 2 3 … 3 1 2 … 4 3 1 … 5 7 1 … 6 2 7 … 7 4 7 … h1 h2 hn Sebastian Raschka STAT 479: Machine Learning FS 2018 19
Page 20
Bagging Classifier Training set Bootstrap T T samples 1 n TT22 . . . New data Classification h h . . . h models 1 2 n Predictions y1 y . . . yn 2 Voting Final prediction yf ŷ = mode{h (x),h (x),...h (x)} f 1 2 n where hi(x) = yî Sebastian Raschka STAT 479: Machine Learning FS 2018 20
Affichage des 20 premières pages. Téléchargez le document pour le voir en entier.
Commentaires (0)
Aucun commentaire pour le moment. Soyez le premier !