第 1 页
Lecture 10 Model Evaluation 3: Cross Validation STAT 479: Machine Learning, Fall 2018 Sebastian Raschka http://stat.wisc.edu/~sraschka/teaching/stat479-fs2018/ Sebastian Raschka STAT 479: Machine Learning FS 2018 1
第 2 页
Overview Bias and Variance Overfitting and Underfitting Basics Holdout method Confidence Intervals Repeated holdout Resampling methods Model Eval Lectures Empirical confidence intervals Hyperparameter tuning Cross-Validation Model selection This Lecture Algorithm Selection Statistical Tests Evaluation Metrics Sebastian Raschka STAT 479: Machine Learning FS 2018 2
第 4 页
Hyperparameters nonparametric model: k-nearest neighbors ? feature 2 k =k=11 k =k=33 Hyperparameter k =k=55 feature 1 Sebastian Raschka STAT 479: Machine Learning FS 2018 4
第 5 页
Hyperparameters parametric model: logistic regression Hyperparameter Sebastian Raschka STAT 479: Machine Learning FS 2018 5
第 6 页
3-Way Holdout Best Hyperparameter Training Data Validation Values 4 Data Model Training Labels Validation Learning Labels Algorithm instead of "regular" holdout to avoid "data Test Data Prediction leakage" during hyperparameter 5 Performance Model optimization Test Labels Training Data Training Labels Best Hyperparameter Validation 1 Data Data 6 Data Values Final Labels Validation Labels Learning Model Labels Algorithm Test Data Test Labels Hyperparameter Model values 2 Training Data Learning Hyperparameter Model values Algorithm Training Labels Model Hyperparameter values Validation Data Prediction Performance Model Validation Labels Best Hyperparameter values Validation Data Prediction 3 Best Performance Model Validation Model Labels Validation Data Prediction Performance Model Validation Labels Sebastian Raschka STAT 479: Machine Learning FS 2018 6
第 7 页
Main points why we evaluate the predictive performance of a model: 1. Want to estimate the generalization performance, the predictive performance of our model on future (unseen) data. 2. Want to increase the predictive performance by tweaking the learning algorithm and selecting the best performing model from a given hypothesis space. 3. Want to identify the ML algorithm that is best-suited for the problem at hand; thus, we want to compare different algorithms, selecting the best-performing one as well as the best performing model from the algorithm’s hypothesis space. Sebastian Raschka STAT 479: Machine Learning FS 2018 7
第 8 页
k-Fold Cross-Validation Part 1 Model Evaluation Sebastian Raschka STAT 479: Machine Learning FS 2018 8
第 9 页
k-Fold Cross-Validation A Validation Training Fold Fold 1st Performance1 2nd Performance 2 3rd Performance 3 Performance ∑5 = 1 5 Performancei 4th Performance 4 i =1 K Iterations (K-Folds) 5th Performance 5 B C Training Fold Data Prediction Validation Training Fold Labels Fold Data Performance Model Hyperparameter Values Validation Fold Labels Learning Algorithm Model Sebastian Raschka STAT 479: Machine Learning FS 2018 9
第 10 页
k-Fold Cross-Validation - non-overlapping test folds; utilizes all data for testing - overlapping training folds - some variance estimate from different training sets, (but no unbiased estimate) - more pessimistic for small k because we withhold data from fitting A Validation Training Fold Fold 1st Performance1 2nd Performance 2 3rd Performance 3 Performance ∑5 = 1 5 Performance 4th Performance i 4 i =1 K Iterations (K-Folds) 5th Performance 5 B C Training Fold Data Prediction Validation Training Fold Labels Fold Data Performance Model Hyperparameter Values Validation Fold Labels Learning Algorithm Model Sebastian Raschka STAT 479: Machine Learning FS 2018 10
第 11 页
k-Fold CV special cases: k=2 & k=n Training Evaluation 1 2 3 4 5 6 7 8 9 10 Holdout Method 1 2 3 4 5 6 7 8 9 10 2-Fold Cross-Validation 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 Repeated Holdout 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 ... Sebastian Raschka STAT 479: Machine Learning FS 2018 11
第 12 页
k-Fold CV special cases: k=2 & k=n Training Evaluation Training evaluation 1 2 3 4 5 6 7 8 9 10 Holdout Method 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 2-Fold Cross-Validation 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 Repeated Holdout 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 ... Sebastian Raschka STAT 479: Machine Learning FS 2018 12
第 13 页
k-Fold Cross-Validation "[...] where available sample sizes are modest, holding back compounds for model testing is ill-advised. This fragmentation of the sample harms the calibration and does not give a trustworthy assessment of fit anyway. It is better to use all data for the calibration step and check the fit by cross-validation, making sure that the cross- validation is carried out correctly. [...] The only motivation to rely on the holdout sample rather than cross-validation would be if there was reason to think the cross-validation not trustworthy -- biased or highly variable. But neither theoretical results nor the empiric results sketched here give any reason to disbelieve the cross-validation results." [1] 1. Hawkins, D. M., Basak, S. C., & Mills, D. (2003). Assessing model fit by cross-validation. Journal of chemical information and computer sciences, 43(2), 579-586. Sebastian Raschka STAT 479: Machine Learning FS 2018 13
第 14 页
Sebastian RaschkaLOOCV vs HoldoutSTAT479 FS18. L10: Cross Validation Page 10 Table 1: Summary of the findings from the LOOCV vs. holdout comparison study conducted by Hawkins and others Douglas M Hawkins, Subhash C Basak, and Denise Mills. “Assessing model fit by cross-validation”. In: Journal of Chemical Information and Computer Sciences 43.2 (2003), pp. 579–586. See text for details. Experiment Mean Standard deviation True R2 — q2 0.010 0.149 True R2 — hold 50 0.028 0.184 True R2 — hold 20 0.055 0.305 True R2 — hold 10 0.123 0.504 1. Hawkins, D. M., Basak, S. C., & Mills, D. (2003). Assessing model fit by cross-validation. Journal of chemical information and computer sciences, 43(2), 579-586. sets and averaging the results. In rows 2-4, the researchers used the holdout method for fitting models to the 100-example training sets, and they evaluated the performances on holdout sets of sizes 10, 20, and 50 samples. Each experiment was repeated 75 times, and the mean column shows the average di↵erence between the estimated R2 and the true R2 The reported "mean" refers to the averaged divalues. As we can see, the estimates obtained via LOOCV (fference between the true coeq2) are the closest to the trueffiRcients of2 determination (R2 ) and the coeon average. The estimates obtained from the 50-example test set via the holdout methodfficients obtained via LOOCV (here called q2) after repeating are also passable, though. Based on these particular experiments, we may agree with the this procedure on diresearchers’ conclusion:fferent 100-example training Taking the third of these points, if you have 150 or more compounds available, then you can certainly make a random split into 100 for calibration and 50 or more for testing. However it is hard to see why you would want to do this. Douglas M Hawkins, Subhash C Basak, and Denise Mills. “Assessing model fit by cross-validation”. In: Journal of Chemical Information and Computer Sciences 43.2 (2003), pp. 579–586 One reason why we may prefer the holdout method may be concerns about computational eciency, if the dataset is suSebastian Raschka ciently large.STAT 479: Machine LearningAs a rule of thumb, we can say that theFS 2018 14
第 15 页
Sebastian RaschkaLOOCV vs HoldoutSTAT479 FS18. L10: Cross Validation Page 10 Table 1: Summary of the findings from the LOOCV vs. holdout comparison study conducted by Hawkins and others Douglas M Hawkins, Subhash C Basak, and Denise Mills. “Assessing model fit by cross-validation”. In: Journal of Chemical Information and Computer Sciences 43.2 (2003), pp. 579–586. See text for details. Experiment Mean Standard deviation True R2 — q2 0.010 0.149 True R2 — hold 50 0.028 0.184 True R2 — hold 20 0.055 0.305 True R2 — hold 10 0.123 0.504 1. Hawkins, D. M., Basak, S. C., & Mills, D. (2003). Assessing model fit by cross-validation. Journal of chemical information and computer sciences, 43(2), 579-586. sets and averaging the results. In rows 2-4, the researchers used the holdout method for fitting models to the 100-example training sets, and they evaluated the performances on holdout sets of sizes 10, 20, and 50 samples. Each experiment was repeated 75 times, and the mean column shows the average di↵erence between the estimated R2 and the true R2 The reported "mean" refers to the averaged divalues. As we can see, the estimates obtained via LOOCV (fference between the trueq2) are the closest to the true R2 coefficients of determination (R2 ) and the coeon average. The estimates obtained from the 50-example test set via the holdout methodfficients obtained via LOOCV (here are also passable, though. Based on these particular experiments, we may agree with the called q2) after repeating this procedure on diresearchers’ conclusion: fferent 100-example training In rows 2-4, the researchers used the holdout method for fitting models to the 100-Taking the third of these points, if you have 150 or more compounds available, example training sets, and they evaluated the performances on holdout sets of sizesthen you can certainly make a random split into 100 for calibration and 50 or 10, 20, and 50 samples. Each experiment was repeated 75 times, and the meanmore for testing. However it is hard to see why you would want to do this. column shows the average diDouglas M Hawkins, Subhash C Basak, and Denise Mills.fference between the estimated R2“Assessing modeland the true R2 values. fit by cross-validation”. In: Journal of Chemical Information and Computer Sciences 43.2 (2003), pp. 579–586 One reason why we may prefer the holdout method may be concerns about computational eciency, if the dataset is suSebastian Raschka ciently large.STAT 479: Machine LearningAs a rule of thumb, we can say that theFS 2018 15
第 16 页
Problems with LOOCV for Classification • While LOOCV is almost unbiased, one downside of using LOOCV over k-fold cross-validation with k < n is the large variance of the LOOCV estimate. • LOOCV is "defect" when using a discontinuous loss-function such as the 0-1 loss in classification or even in continuous loss functions such as the mean-squared-error. • LOOCV has high variance because the test set only contains one sample Sebastian Raschka STAT 479: Machine Learning FS 2018 16
第 17 页
Problems with LOOCV for Classification "With k=n, the cross-validation estimator is approximately unbiased for the true (expected) prediction error, but can have high variance because the n "training sets" are so similar to one another." [1] [1] Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1, No. 10). New York, NY, USA:: Springer series in statistics. Sebastian Raschka STAT 479: Machine Learning FS 2018 17
第 18 页
Problems with LOOCV for Classification n n n n Var ∑Xi = ∑∑Cov(Xi, Xj) = ∑Var(Xi) + 2 ∑ Cov(Xi, Xj) ( ) i=1 i=1 j=1 i=1 1≤i<j≤n For correlated variables, the variance of their sum is the sum of their covariances Or in other words, we can attribute the high variance to the fact that the mean of highly correlated variables has a higher variance than the mean of variables that are not highly correlated? Sebastian Raschka STAT 479: Machine Learning FS 2018 18
第 19 页
Empirical Study and Recommendation % acc 75 % acc std dev C4.5 Soybean Vehicle 70 75 Estimated 7 Soybean Soybean 6 65 70 Soybean Vehicle 5 Rand 60 Vehicle65 Breast Rand 4 55 60 Vehicle 3 50 Rand55 2 50 Rand Hypo 45 folds 1 Chess 2 5 10 20 -5 -2 -1 Mushroom 45 samples 0 folds % acc 1 2 5 10 20 50 100 2 5 10 20 -5 -2 -1 100 std dev % acc Naive-Bayes 99.5 Mushroom 100 7 99 99.5 Mushroom Vehicle 98.5 Hypo 6 Rand Chess99 Soybean 98 Hypo 5 98.5 97.5 Chess 4 Breast 98 97 97.5 3 96.5 97 2 96 96.5 folds Chess 2 5 10 20 -5 -2 -1 1 Hypo Mushroom 96 samples 0 folds 1 2 5 10 20 50 100 2 5 10 20 -5 -2 -1 Sebastian Raschka STAT 479: Machine Learning FS 2018 19
第 20 页
Summarizing k-Fold CV for Model Evaluation What happens if we increase k? • The bias of the performance estimator decreases (more accurate) • The variance of the performance estimators increases (more variability) • The computational cost increases (more iterations, larger training sets during fitting) • Exception: decreasing the value of k in k-fold cross-validation to small values (for example, 2 or 3) also increases the variance on small datasets due to random sampling effects. Sebastian Raschka STAT 479: Machine Learning FS 2018 20
评论 (0)
暂无评论。快来抢沙发吧！