Lecture 08 Model Evaluation 1: Introduction to Overfitting and Underfitting STAT 479: Machine Learning, Fall 2018 Sebastian Raschka http://stat.wisc.edu/~sraschka/teaching/stat479-fs2018/ Sebastian Raschka STAT 479: Machine Learning FS 2018 1
Overview Bias and Variance Overfitting and Underfitting Basics Holdout method Confidence Intervals Repeated holdout Resampling methods Model Eval Lectures Empirical confidence intervals Hyperparameter tuning Cross-Validation Model selection Algorithm Selection Statistical Tests Evaluation Metrics Sebastian Raschka STAT 479: Machine Learning FS 2018 2
Overfitting and Underfitting "Generalization Performance" • Want a model to "generalize" well to unseen data ("high generalization accuracy" or "low generalization error") Sebastian Raschka STAT 479: Machine Learning FS 2018 4
Overfitting and Underfitting Assumptions • i.i.d. assumption: inputs are independent, and training and test examples are identically distributed (drawn from the same probability distribution) • For some random model that has not been fitted to the training set, we expect both the training and test error to be equal • The training error or accuracy provides an (optimistically) biased estimate of the generalization performance Sebastian Raschka STAT 479: Machine Learning FS 2018 5
Overfitting and Underfitting Model Capacity • Underfitting: both training and test error are large • Overfitting: gap between training and test error (where test error is higher) • Large hypothesis space being searched by a learning algorithm -> high tendency to overfit Sebastian Raschka STAT 479: Machine Learning FS 2018 6
"[...] model has high bias/variance" -- What does that mean? Sebastian Raschka STAT 479: Machine Learning FS 2018 8
"[...] model has high bias/variance" -- What does that mean? Originally formulated for regression Sebastian Raschka STAT 479: Machine Learning FS 2018 9
Bias-Variance Decomposition • Decomposition of the loss into bias and variance help us understand learning algorithms, concepts are correlated to underfitting and overfitting • Helps explain why ensemble methods (last lecture) might perform better than single models Sebastian Raschka STAT 479: Machine Learning FS 2018 11
Bias-Variance Intuition Low Variance High Variance (Precise) (Not Precise) Low Bias(Accurate) High Bias (Not Accurate) Sebastian Raschka STAT 479: Machine Learning FS 2018 12
Bias and Variance Intuition High bias (There are two points where the bias is zero) Sebastian Raschka STAT 479: Machine Learning FS 2018 17
Bias and Variance Intuition High Variance Why? (here, I fit an unpruned decision tree) Sebastian Raschka STAT 479: Machine Learning FS 2018 18
Bias and Variance Example where f(x) is some true (target) function suppose we have multiple training sets Sebastian Raschka STAT 479: Machine Learning FS 2018 19
Hiển thị 19 trang đầu. Tải tài liệu để xem đầy đủ.
Chưa có bình luận nào. Hãy là người đầu tiên!
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!