STAT 479: Machine Learning Lecture Notes Sebastian Raschka Department of Statistics University of Wisconsin–Madison http://stat.wisc.edu/∼sraschka/teaching/stat479-fs2018/ Fall 2018 Contents 9 Model Evaluation 2: Confidence Intervals and Resampling Techniques 9.1 Introduction: Essential Model Evaluation Terms and Techniques . . . . . . . 9.1.1 1 1 Performance Estimation: Generalization Performance Vs. Model Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Assumptions and Terminology . . . . . . . . . . . . . . . . . . . . . . 2 Resubstitution Validation and the Holdout Method . . . . . . . . . . . . . . . 5 9.2.1 Stratification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Holdout Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 9.3.1 Pessimistic Bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 9.4 Confidence Intervals via Normal Approximation . . . . . . . . . . . . . . . . . 8 9.5 Overview of the Next Sections on Resampling . . . . . . . . . . . . . . . . . . 10 9.6 Resampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 9.7 Repeated Holdout Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 9.8 The Bootstrap Method and Empirical Confidence Intervals . . . . . . . . . . 14 9.9 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 9.1.2 9.2 9.3 STAT 479: Machine Learning Lecture Notes Sebastian Raschka Department of Statistics University of Wisconsin–Madison http://stat.wisc.edu/∼sraschka/teaching/stat479-fs2018/ Fall 2018 9 Model Evaluation 2: Confidence Intervals and Resampling Techniques 9.1 Introduction: Essential Model Evaluation Terms and Techniques Machine learning has become a central part of our life – as consumers, customers, and hopefully as researchers and practitioners. Whether we are applying predictive modeling techniques to o