STAT 479: Machine Learning Lecture Notes Sebastian Raschka Department of Statistics University of Wisconsin–Madison http://stat.wisc.edu/∼sraschka/teaching/stat479-fs2018/ Fall 2018 Contents 8 Model Evaluation 1: Overfitting and Underfitting 1 8.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 8.2 Overfitting and Underfitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 8.3 Bias and Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 8.4 Bias-Variance Decomposition of the Squared Loss . . . . . . . . . . . . . . . . 6 8.5 Bias-Variance Decomposition of the 0-1 Loss . . . . . . . . . . . . . . . . . . 7 8.6 Conclusion 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . STAT 479: Machine Learning Lecture Notes Sebastian Raschka Department of Statistics University of Wisconsin–Madison http://stat.wisc.edu/∼sraschka/teaching/stat479-fs2018/ Fall 2018 8 Model Evaluation 1: Overfitting and Underfitting 8.1 Overview In this lecture, we discuss some of the basic terms and machine learning fundamentals that are relevant for model evaluation, namely, bias and variance, and overfitting and underfitting. Figure 1: Overview of topics being covered in this lecture in the context of topics related to model evaluation that we will cover at a later point in time. 8.2 Overfitting and Underfitting The overall goal in machine learning is to obtain a model/hypothesis that generalizes well to new, unseen data. Sebastian Raschka STAT479 FS18. L01: Intro to Machine Learning Page 2 In other words, we want a model that generalizes well to unseen data, which we can measure, for example, by using an independent test set – while it sounds like this should be very straightforward, there are some pitfalls which we will discuss in the next lecture. Some of the evaluation metrics we can use to measure the performance on the test set are the prediction ac