Lecture 09 Model Evaluation 2: Confidence Intervals STAT 479: Machine Learning, Fall 2018 Sebastian Raschka http://stat.wisc.edu/~sraschka/teaching/stat479-fs2018/ Sebastian Raschka STAT 479: Machine Learning FS 2018 1 Overview Bias and Variance Basics Overfitting and Underfitting This Lecture Holdout method Confidence Intervals Model Eval Lectures Repeated holdout Resampling methods Empirical confidence intervals Hyperparameter tuning Cross-Validation Model selection Algorithm Selection Statistical Tests Evaluation Metrics Sebastian Raschka STAT 479: Machine Learning FS 2018 2 Concepts first (More) Code at the end of the lecture Sebastian Raschka STAT 479: Machine Learning FS 2018 3 Main points why we evaluate the predictive performance of a model: 1. Want to estimate the generalization performance, the predictive performance of our model on future (unseen) data. 2. Want to increase the predictive performance by tweaking the learning algorithm and selecting the best performing model from a given hypothesis space. 3. Want to identify the ML algorithm that is best-suited for the problem at hand; thus, we want to compare different algorithms, selecting the best-performing one as well as the best performing model from the algorithm’s hypothesis space. Sebastian Raschka STAT 479: Machine Learning FS 2018 4 Training set error is an optimistically biased estimator of the generalization error Test set error is an unbiased estimator of the generalization error (test sample and hypothesis chosen independently) (in practice, it is actually pessimistically biased; why?) Sebastian Raschka STAT 479: Machine Learning FS 2018 5 Training Evaluation 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 Holdout Method 2-Fold Cross-Validation 1 2 3 4 5 6 7 8 9 10 Sebastian Raschka STAT 479: Machine Learning FS 2018 6 Often using the holdout method is not a good idea ... Sebastian Raschka STAT 479: Machine Learning FS 2018 7 Often