Trang 1
Lecture 11 Model Evaluation 4: Algorithm Comparisons STAT 479: Machine Learning, Fall 2018 Sebastian Raschka http://stat.wisc.edu/~sraschka/teaching/stat479-fs2018/ Sebastian Raschka STAT 479: Machine Learning FS 2018 1
Trang 2
Overview Bias and Variance Overfitting and Underfitting Basics Holdout method Confidence Intervals Repeated holdout Resampling methods Model Eval Lectures Empirical confidence intervals Hyperparameter tuning Cross-Validation Model selection This Lecture Algorithm Selection Statistical Tests Evaluation Metrics Sebastian Raschka STAT 479: Machine Learning FS 2018 2
Trang 3
Overview, (my) "recommendations" ▪ 2-way holdout method (train/test split) Large dataset ▪ Confidence interval via normal approximation Performance estimation ▪ (Repeated) k-fold cross-validation without independent test set Small dataset ▪ Leave-one-out cross-validation without independent test set ▪ Confidence interval via 0.632(+) bootstrap Large dataset ▪ 3-way holdout method (train/validation/test split) Model selection (hyperparameter optimization) and performance estimation ▪ (Repeated) k-fold cross-validation with independent test set Small dataset ▪ Leave-one-out cross-validation with independent test set ▪ Disjoint training sets + test set (algorithm comparison, AC) Large dataset ▪ McNemar test (model comparison, MC) Model & algorithm ▪ Cochran’s Q + McNemar test comparison (MC) ▪ Combined 5x2cv F test (AC) Small dataset ▪ Nested cross-validation (AC) Sebastian Raschka STAT 479: Machine Learning FS 2018 3
Trang 4
Comparing two machine learning classifiers -- McNemar's Test McNemar's test, introduced by Quinn McNemar in 1947 [1], is a non-parametric statistical test for paired comparisons that can be applied to compare the performance of two machine learning classifiers: Task Gaussian data … Paired nominal data Compare a group Binomial test to a reference value Compare a McNemar’s test pair of groups Compare two test, unpaired groups Fisher’s exact test This work by Sebastian Raschka is licensed under a Creative Commons Attribution 4.0 International License. [1] McNemar, Quinn. "Note on the sampling error of the difference between correlated proportions or percentages." Psychometrika 12.2 (1947): 153-157. Sebastian Raschka STAT 479: Machine Learning FS 2018 4
Trang 5
Comparing two machine learning classifiers -- McNemar's Test • Also referred to as "within-subjects chi-squared test" • Applied to paired nominal data based on a version of a 2x2 confusion matrix • Compares the predictions of two models to each other rather than listing false positive, true positive, false negative, and true negative counts of a single model • The layout of the 2x2 confusion matrix suitable for McNemar's test is shown in the following figure: Model 2 Model 2 correct wrong A B Model 1correct C D Model 1wrong This work by Sebastian Raschka is licensed under a Creative Commons Attribution 4.0 International License. Sebastian Raschka STAT 479: Machine Learning FS 2018 5
Trang 6
Comparing two machine learning classifiers -- McNemar's Test • Given such a 2x2 confusion matrix as shown in the previous figure, we can compute the accuracy of a Model 1 via (A+B) / (A+B+C+D) • Similarly, we can compute the accuracy of Model 2 as (A+C) / N • Cells B and C (the off-diagonal entries) tell us how the models differ Model 2 Model 2 correct wrong A B Model 1correct C D Model 1wrong This work by Sebastian Raschka is licensed under a Creative Commons Attribution 4.0 International License. Sebastian Raschka STAT 479: Machine Learning FS 2018 6
Trang 7
Comparing two machine learning classifiers -- McNemar's Test • Let's take a look at the following example: A B Model 2 Model 2 Model 2 Model 2 correct wrong correct wrong 9959 11 9945 25 Model 1correct Model 1correct 1 29 15 15 Model 1wrong Model 1wrong This work by Sebastian Raschka is licensed under a • Creative Commons Attribution 4.0 International License. What is the prediction accuracy of models 1 and 2? Sebastian Raschka STAT 479: Machine Learning FS 2018 7
Trang 8
Comparing two machine learning classifiers -- McNemar's Test • What is the prediction accuracy of models 1 and 2? A B Model 2 Model 2 Model 2 Model 2 correct wrong correct wrong 9959 11 9945 25 Model 1correct Model 1correct 1 29 15 15 Model 1wrong Model 1wrong This work by Sebastian Raschka is licensed under a In both subpanel A and B, the accuracy ofCreative Commons Attribution 4.0 International License. Model 1 and Model 2 are ???% and ???%, respectively. • Model 1 accuracy subpanel A: (???)/10000 × 100% = ???% • Model 1 accuracy subpanel B: (???)/10000 × 100% = ???% • Model 2 accuracy subpanel A: (???)/10000 × 100% = ???% • Model 2 accuracy subpanel B: (???)/10000 × 100% = ???% Sebastian Raschka STAT 479: Machine Learning FS 2018 8
Trang 9
Comparing two machine learning classifiers -- McNemar's Test • What is the prediction accuracy of models 1 and 2? A B Model 2 Model 2 Model 2 Model 2 correct wrong correct wrong 9959 11 9945 25 Model 1correct Model 1correct 1 29 15 15 Model 1wrong Model 1wrong This work by Sebastian Raschka is licensed under a In both subpanel A and B, the accuracy ofCreative Commons Attribution 4.0 International License. Model 1 and Model 2 are 99.7% and 99.6%, respectively. • Model 1 accuracy subpanel A: (9959 + 11)/10000 × 100% = 99.7% • Model 1 accuracy subpanel B: (9945 + 25)/10000 × 100% = 99.7% • Model 2 accuracy subpanel A: (9959 + 1)/10000 × 100% = 99.6% • Model 2 accuracy subpanel B: (9945 + 15)/10000 × 100% = 99.6% Sebastian Raschka STAT 479: Machine Learning FS 2018 9
Trang 10
Comparing two machine learning classifiers -- McNemar's Test In both subpanel A and B, the accuracy of Model 1 and Model 2 are 99.7% and 99.6%, respectively. A B Model 2 Model 2 Model 2 Model 2 correct wrong correct wrong 9959 11 9945 25 Model 1correct Model 1correct 1 29 15 15 Model 1wrong Model 1wrong This work by Sebastian Raschka is licensed under a In subpanel A: Creative Commons Attribution 4.0 International License. • Model 1 got 11 predictions right that Model 2 got wrong • Model 2 got 1 prediction right that Model 1 got wrong • Based on this 11:1 ratio (based on our intuition), does Model 1 perform substantially better than Model 2? In subpanel B: • The Model 1:Model 2 ratio is 25:15 • This is less conclusive about which model is the better one to choose. Sebastian Raschka STAT 479: Machine Learning FS 2018 10
Trang 11
Comparing two machine learning classifiers -- McNemar's Test In both subpanel A and B, the accuracy of Model 1 and Model 2 are 99.7% and 99.6%, respectively. A B Model 2 Model 2 Model 2 Model 2 Model 2 Model 2 correct wrong correct wrong correct wrong A B 9959 11 9945 25 Model 1correct Model 1correct Model 1correct C D 1 29 15 15 Model 1wrong Model 1wrong Model 1wrong This work by Sebastian Raschka is licensed under a This work by Sebastian Raschka is licensed under a Creative Commons Attribution 4.0 International License. Creative Commons Attribution 4.0 International License. In McNemar's Test, we formulate the • null hypothesis: the probabilities p(B) and p(C) are the same • alternative hypothesis: the performances of the two models are not equal Sebastian Raschka STAT 479: Machine Learning FS 2018 11
Trang 12
Comparing two machine learning classifiers -- McNemar's Test In both subpanel A and B, the accuracy of Model 1 and Model 2 are 99.7% and 99.6%, respectively. A B Model 2 Model 2 Model 2 Model 2 Model 2 Model 2 correct wrong correct wrong correct wrong A B 9959 11 9945 25 Model 1correct Model 1correct Model 1correct C D 1 29 15 15 Model 1wrong Model 1wrong Model 1wrong This work by Sebastian Raschka is licensed under a This work by Sebastian Raschka is licensed under a Creative Commons Attribution 4.0 International License. Creative Commons Attribution 4.0 International License. In McNemar's Test, we formulate the • null hypothesis: the probabilities p(B) and p(C) are the same • alternative hypothesis: the performances of the two models are not equal The McNemar test statistic ("chi-squared") can be computed as follows: (B − C)2 χ2 = B + C Sebastian Raschka STAT 479: Machine Learning FS 2018 12
Trang 13
Comparing two machine learning classifiers -- McNemar's Test The McNemar test statistic ("chi-squared") can be computed as follows: (B − C)2 χ2 = B + C • Set a significance threshold, for example, α = 0.05 • Compute the p-value -- assuming that the null hypothesis is true, the p-value is the probability of observing the given empirical (or a larger) chi-squared value (chi^2 distribution with 1 degree of freedom, and relatively large numbers in cells B and C, say > 25) • If the p-value is lower than our chosen significance level, we can reject the null hypothesis that the two model's performances are equal Sebastian Raschka STAT 479: Machine Learning FS 2018 13
Trang 14
Comparing two machine learning classifiers -- McNemar's Test A B Model 2 Model 2 Model 2 Model 2 correct wrong correct wrong 9959 11 9945 25 Model 1correct Model 1correct 1 29 15 15 Model 1wrong Model 1wrong This work by Sebastian Raschka is licensed under a Creative Commons Attribution 4.0 International License. • If we did this for scenario B in the previous figure (chi^2=2.5), we would obtain a p-value of 0.1138, which is larger than our significance threshold, and thus, we cannot reject the null hypothesis. • If we computed the p-value for scenario A (chi^2=8.3), we would obtain a p-value of 0.0039, which is below the set significance threshold (alpha=0.05) and leads to the rejection of the null hypothesis; we can conclude that the models' performances are different (for instance, Model 1 performs better than Model 2). Sebastian Raschka STAT 479: Machine Learning FS 2018 14
Trang 15
Comparing two machine learning classifiers -- McNemar's Test Continuity Correction Approximately 1 year after Quinn McNemar published the McNemar Test (McNemar 1947), Allen L. Edwards [1] proposed a continuity corrected version, which is the more commonly used variant today: |B − C| − 1 2 2 ( ) χ = . B + C "This correction will have the obvious result of reducing the absolute value of the difference, [B - C], by unity." [1] [1] Edwards, Allen L. "Note on the “correction for continuity” in testing the significance of the difference between correlated proportions." Psychometrika 13.3 (1948): 185-187. Sebastian Raschka STAT 479: Machine Learning FS 2018 15
Trang 16
Comparing two machine learning classifiers -- McNemar's Test Exact p-values via the Binomial test • McNemar's test approximates the p-values reasonably well if the values in cells B and C are larger than 50 • But it makes sense to use a computationally more expensive binomial test to compute the exact p-values (esp. if B and C are relatively small) -- since the chi- squared value from McNemar's test may not be well-approximated by the chi- squared distribution Sebastian Raschka STAT 479: Machine Learning FS 2018 16
Trang 17
Comparing two machine learning classifiers -- McNemar's Test Exact p-values via the Binomial test • McNemar's test approximates the p-values reasonably well if the values in cells B and C are larger than 50 • But it makes sense to use a computationally more expensive binomial test to compute the exact p-values (esp. if B and C are relatively small) -- since the chi- squared value from McNemar's test may not be well-approximated by the chi- squared distribution The exact p-value can be computed as follows: n n p = 2 0.5i(1 − 0.5)n−i, ∑(i) i=B where n=b+c, and the factor 2 is used to compute the two-sided p-value. Sebastian Raschka STAT 479: Machine Learning FS 2018 17
Trang 18
Comparing two machine learning classifiers -- McNemar's Test Exact p-values via the Binomial test • The following heat map illustrates the differences between the McNemar approximation of the chi-squared value (with and without Edward's continuity correction) to the exact p-values computed via the binomial test: Uncorrected vs. exact Corrected vs. exact 0 0 0.0200 0.6 absolute p-value di 0.0175 absolute p-value di 20 20 0.5 0.0150 40 0.4 40 0.0125 C C 0.0100 0.3 60 ff 60 ff erence 0.0075 erence 0.2 0.0050 80 80 0.1 0.0025 100 0.0 100 0.0000 0 20 40 60 80 100 0 20 40 60 80 100 B B (As we can see in this heat map, the p-values from the continuity-corrected version of McNemar's test are almost identical to the p-valuesThis work by Sebastian Raschka is licensed under a from a binomial test if both B and C are larger than 50.) Creative Commons Attribution 4.0 International License. Sebastian Raschka STAT 479: Machine Learning FS 2018 18
Trang 19
Multiple Hypothesis Testing Issue 1. Conduct an omnibus test under the null hypothesis that there is no difference between the classification accuracies 2. If the omnibus test led to the rejection of the null hypothesis, conduct pairwise post hoc tests, with adjustments for multiple comparisons, to determine where the differences between the model performances occurred Sebastian Raschka STAT 479: Machine Learning FS 2018 19
Trang 20
Multiple Hypothesis Testing Issue 1. Conduct an omnibus test under the null hypothesis that there is no difference between the classification accuracies (Cochran's Q test would be a good choice, which is a generalized version of McNemar's test for three or more models) 2. If the omnibus test led to the rejection of the null hypothesis, conduct pairwise post hoc tests, with adjustments for multiple comparisons, to determine where the differences between the model performances occurred (McNemar's Test would be a candidate here) Sebastian Raschka STAT 479: Machine Learning FS 2018 20
Hiển thị 20 trang đầu. Tải tài liệu để xem đầy đủ.
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!