Lecture 10 Model Evaluation 3: Cross Validation STAT 479: Machine Learning, Fall 2018 Sebastian Raschka http://stat.wisc.edu/~sraschka/teaching/stat479-fs2018/ Sebastian Raschka STAT 479: Machine Learning FS 2018 1 Overview Bias and Variance Basics Overfitting and Underfitting Holdout method Confidence Intervals Model Eval Lectures Repeated holdout Resampling methods Empirical confidence intervals Hyperparameter tuning Cross-Validation Model selection This Lecture Algorithm Selection Statistical Tests Evaluation Metrics Sebastian Raschka STAT 479: Machine Learning FS 2018 2 Hyperparameters Sebastian Raschka STAT 479: Machine Learning FS 2018 3 Hyperparameters feature 2 nonparametric model: k-nearest neighbors ? k=1 k=1 kk=3 =3 k=5 Hyperparameter k=5 feature 1 Sebastian Raschka STAT 479: Machine Learning FS 2018 4 Hyperparameters parametric model: logistic regression Hyperparameter Sebastian Raschka STAT 479: Machine Learning FS 2018 5 3-Way Holdout 4 instead of "regular" holdout to avoid "data leakage" during hyperparameter optimization Training Data Validation Data Training Labels Validation Labels Test Data 5 Best Hyperparameter Values Learning Algorithm Model Prediction Performance Model Test Labels Training Data Training Labels 1 Data Validation Data Labels Validation Labels 6 Best Hyperparameter Values Data Learning Algorithm Labels Test Data Final Model Test Labels Hyperparameter values 2 Training Data Hyperparameter values Training Labels Model Learning Algorithm Model Hyperparameter values Validation Data Prediction Model Validation Labels Model Performance 3 Validation Data Prediction Model Validation Labels Validation Data Prediction Model Validation Labels Best Hyperparameter values Performance Best Model Performance Sebastian Raschka STAT 479: Machine Learning FS 2018 6 Main points why we evaluate the predictive performance of a model: 1. Wan