STAT 479: Machine Learning Lecture Notes Sebastian Raschka Department of Statistics University of Wisconsin–Madison http://stat.wisc.edu/∼sraschka/teaching/stat479-fs2018/ Fall 2018 Contents 10 Model Evaluation 3: Model Selection and Cross-Validation 1 10.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 10.2 About Hyperparameters and Model Selection . . . . . . . . . . . . . . . . . . 1 10.3 The Three-Way Holdout Method for Hyperparameter Tuning . . . . . . . . . 3 10.4 Introduction to K-Fold Cross-Validation . . . . . . . . . . . . . . . . . . . . . 6 10.5 Special Cases: 2-Fold and Leave-One-Out Cross-Validation . . . . . . . . . . 7 10.6 K-fold Cross-Validation and the Bias-Variance Trade-off . . . . . . . . . . . . 10 10.7 Model Selection via K-fold Cross-Validation . . . . . . . . . . . . . . . . . . . 12 10.8 A Note About Model Selection and Large Datasets . . . . . . . . . . . . . . . 14 10.9 A Note About Feature Selection During Model Selection . . . . . . . . . . . . 14 10.10The Law of Parsimony . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 10.11Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 STAT 479: Machine Learning Lecture Notes Sebastian Raschka Department of Statistics University of Wisconsin–Madison http://stat.wisc.edu/∼sraschka/teaching/stat479-fs2018/ Fall 2018 10 Model Evaluation 3: Model Selection and CrossValidation 10.1 Overview Almost every machine learning algorithm comes with a large number of settings that we, the machine learning researchers and practitioners, need to specify. These tuning knobs, the so-called hyperparameters, help us control the behavior of machine learning algorithms when optimizing for performance, finding the right balance between bias and variance. Hyperparameter tuning for performance optimization is an art in itself, and there are no hard-andfast rules that guarantee best performance on a given datas