Lecture 08 Model Evaluation 1: Introduction to Overfitting and Underfitting STAT 479: Machine Learning, Fall 2018 Sebastian Raschka http://stat.wisc.edu/~sraschka/teaching/stat479-fs2018/ Sebastian Raschka STAT 479: Machine Learning FS 2018 1 Overview Bias and Variance Basics Overfitting and Underfitting Holdout method Confidence Intervals Model Eval Lectures Resampling methods Repeated holdout Empirical confidence intervals Hyperparameter tuning Cross-Validation Model selection Algorithm Selection Statistical Tests Evaluation Metrics Sebastian Raschka STAT 479: Machine Learning FS 2018 2 Overfitting and Underfitting Sebastian Raschka STAT 479: Machine Learning FS 2018 3 Overfitting and Underfitting "Generalization Performance" Want a model to "generalize" well to unseen data ("high generalization accuracy" or "low generalization error") Sebastian Raschka STAT 479: Machine Learning FS 2018 4 Overfitting and Underfitting Assumptions i.i.d. assumption: inputs are independent, and training and test examples are identically distributed (drawn from the same probability distribution) For some random model that has not been fitted to the training set, we expect both the training and test error to be equal The training error or accuracy provides an (optimistically) biased estimate of the generalization performance Sebastian Raschka STAT 479: Machine Learning FS 2018 5 Overfitting and Underfitting Model Capacity Underfitting: both training and test error are large Overfitting: gap between training and test error (where test error is higher) Large hypothesis space being searched by a learning algorithm > high tendency to overfit Sebastian Raschka STAT 479: Machine Learning FS 2018 6 Overfitting and Underfitting Sebastian Raschka STAT 479: Machine Learning FS 2018 7 "[...] model has high bias/variance" -- What does that mean? Sebastian Raschka STAT 479: Machine Learning FS 2018 8 "[...] model has high