Lecture 11 Model Evaluation 4: Algorithm Comparisons STAT 479: Machine Learning, Fall 2018 Sebastian Raschka http://stat.wisc.edu/~sraschka/teaching/stat479-fs2018/ Sebastian Raschka STAT 479: Machine Learning FS 2018 1 Overview Bias and Variance Basics Overfitting and Underfitting Holdout method Confidence Intervals Model Eval Lectures Repeated holdout Resampling methods Empirical confidence intervals Hyperparameter tuning Cross-Validation Model selection This Lecture Algorithm Selection Statistical Tests Evaluation Metrics Sebastian Raschka STAT 479: Machine Learning FS 2018 2 Overview, (my) "recommendations" Large dataset Performance estimation Small dataset Large dataset Model selection (hyperparameter optimization) and performance estimation Small dataset Large dataset Model & algorithm comparison Small dataset Sebastian Raschka 2-way holdout method (train/test split) Confidence interval via normal approximation (Repeated) k-fold cross-validation without independent test set Leave-one-out cross-validation without independent test set Confidence interval via 0.632(+) bootstrap 3-way holdout method (train/validation/test split) (Repeated) k-fold cross-validation with independent test set Leave-one-out cross-validation with independent test set Disjoint training sets + test set (algorithm comparison, AC) McNemar test (model comparison, MC) Cochran’s Q + McNemar test (MC) Combined 5x2cv F test (AC) Nested cross-validation (AC) STAT 479: Machine Learning FS 2018 3 Comparing two machine learning classifiers -- McNemar's Test McNemar's test, introduced by Quinn McNemar in 1947 [1], is a non-parametric statistical test for paired comparisons that can be applied to compare the performance of two machine learning classifiers: Task Gaussian data Compare a group to a reference value … Paired nominal data Binomial test Compare a pair of groups McNemar’s test Compare two unpaired grou