Trang 1
Lecture 09 Model Evaluation 2: Confidence Intervals STAT 479: Machine Learning, Fall 2018 Sebastian Raschka http://stat.wisc.edu/~sraschka/teaching/stat479-fs2018/ Sebastian Raschka STAT 479: Machine Learning FS 2018 1
Trang 2
Overview Bias and Variance Overfitting and Underfitting This Lecture Basics Holdout method Confidence Intervals Repeated holdout Resampling methods Model Eval Lectures Empirical confidence intervals Hyperparameter tuning Cross-Validation Model selection Algorithm Selection Statistical Tests Evaluation Metrics Sebastian Raschka STAT 479: Machine Learning FS 2018 2
Trang 3
• Concepts first • (More) Code at the end of the lecture Sebastian Raschka STAT 479: Machine Learning FS 2018 3
Trang 4
Main points why we evaluate the predictive performance of a model: 1. Want to estimate the generalization performance, the predictive performance of our model on future (unseen) data. 2. Want to increase the predictive performance by tweaking the learning algorithm and selecting the best performing model from a given hypothesis space. 3. Want to identify the ML algorithm that is best-suited for the problem at hand; thus, we want to compare different algorithms, selecting the best-performing one as well as the best performing model from the algorithm’s hypothesis space. Sebastian Raschka STAT 479: Machine Learning FS 2018 4
Trang 5
• Training set error is an optimistically biased estimator of the generalization error • Test set error is an unbiased estimator of the generalization error (test sample and hypothesis chosen independently) • (in practice, it is actually pessimistically biased; why?) Sebastian Raschka STAT 479: Machine Learning FS 2018 5
Trang 6
Training Evaluation 1 2 3 4 5 6 7 8 9 10 Holdout Method 1 2 3 4 5 6 7 8 9 10 2-Fold Cross-Validation 1 2 3 4 5 6 7 8 9 10 Sebastian Raschka STAT 479: Machine Learning FS 2018 6
Trang 7
Often using the holdout method is not a good idea ... Sebastian Raschka STAT 479: Machine Learning FS 2018 7
Trang 8
Often using the holdout method is not a good idea ... Test set error as generalization error estimator is pessimistically biased (not so bad) But it does not account for variance in the training data (bad) Sebastian Raschka STAT 479: Machine Learning FS 2018 8
Trang 9
Why is pessimistic bias not "so bad"? Suppose we have the following ranking based on accuracy: h2: 75% > h1: 70% > h3: 65%, we would still rank them the same way if we add a 10% pessimistic bias: h2: 65% > h1: 60% > h3: 55%. Sebastian Raschka STAT 479: Machine Learning FS 2018 9
Trang 10
Often using the holdout method is not a good idea ... • Test set error as generalization error estimator is pessimistically biased (not so bad) • Does not account for variance in the training data (bad) Sebastian Raschka STAT 479: Machine Learning FS 2018 10
Trang 11
We have to keep in mind that a dataset represents a random sample drawn from a probability distribution, and we typically assume that this sample is representative of the true population – more Issues with Subsampling (Independence violation)or less. Now, further subsampling without replacement alters the statistic (mean, proportion, and variance) of the sample. The degree to which subsampling without replacement affects the statistic of a sample is inversely proportional to the size of the sample. Let us have a look at an example using the Iris dataset 1, which we randomly divide into 2/3 training data and 1/3 test data as illustrated in Figure 1. (The source code for generating this graphic is available on GitHub2.) All samples (n = 150) Training samples (n = 100) Test samples (n = 50) The Iris dataset consists of 50 Setosa, 50 Versicolor, and 50 Virginica flowers; the flower species are distributed uniformly:Figure 1: Distribution of Iris flower classes upon random subsampling into training and test sets. • 33.3% Setosa • 1https://archive.ics.uci.edu/ml/datasets/iris 33.3% Versicolor 2https://github.com/rasbt/model-eval-article-supplementary/blob/master/code/iris-random-dist.ipynb • 33.3% Virginia If our random function assigns 2/3 of the flowers (100) to the training set and 1/3 of the flowers (50) to the test set, it may yield6 the following: • training set → 38 x Setosa, 28 x Versicolor, 34 x Virginica • test set → 12 x Setosa, 22 x Versicolor, 16 x Virginica Sebastian Raschka STAT 479: Machine Learning FS 2018 11
Trang 12
Holdout evaluation Training Data Training Labels Data 1 Labels Test Data Test Labels Hyperparameter 2 Training Data Values Model Training Labels Learning Algorithm Test Data Prediction 3 Performance Model Test Labels Hyperparameter Data Values Final 4 Model Labels Learning Algorithm Sebastian Raschka STAT 479: Machine Learning FS 2018 12
Trang 13
Can we use the holdout method for model selection? Sebastian Raschka STAT 479: Machine Learning FS 2018 13
Trang 14
Holdout validation Training Data Training Labels (hyperparam. tuning) 1 Data Validation Data Labels Validation Labels Test Data Test Labels Hyperparameter Model values 2 Training Data Learning Hyperparameter Model values Algorithm Training Labels Model Hyperparameter values Validation Data Prediction Performance Model Validation Labels Best Hyperparameter values Validation Data Prediction 3 Best Performance Model Validation Model Labels Validation Data Prediction Performance Model Validation Labels Best Hyperparameter Training Data Validation Values 4 Data Model Training Labels Validation Learning Labels Algorithm Test Data Prediction 5 Performance Model Test Labels Best Hyperparameter 6 Data Values Final Labels Learning Model Algorithm Sebastian Raschka STAT 479: Machine Learning FS 2018 14
Trang 15
Holdout validation (hyperparam. tuning) Training Data Training Labels Validation 1 Data Data Labels Validation Labels Test Data Test Labels Hyperparameter Model values 2 Training Data Learning Hyperparameter Model values Algorithm Training Labels Model Hyperparameter values Validation Data Prediction Performance Model Validation Labels Best Hyperparameter values Validation Sebastian RaschkaData STAT 479: Machine LearningPrediction FS 2018 15
Trang 16
Model Hyperparameter values Holdout validation (hyperparam. tuning) Validation Data Prediction Performance Model Validation Labels Best Hyperparameter values Validation Data Prediction 3 Best Performance Model Validation Model Labels Validation Data Prediction Performance Model Validation Labels Best Hyperparameter Training Data Validation Values 4 Data Model Training Labels Validation Learning Labels Algorithm Test Data Prediction 5 Performance Model Test Labels Sebastian Raschka STAT 479: Machine Learning FS 2018 16
Trang 17
Performance Model Validation Labels Holdout validation (hyperparam. tuning) Best Hyperparameter Training Data Validation Values 4 Data Model Training Labels Validation Learning Labels Algorithm Test Data Prediction 5 Performance Model Test Labels Best Hyperparameter 6 Data Values Final Labels Learning Model Algorithm Sebastian Raschka STAT 479: Machine Learning FS 2018 17
Trang 18
Cross-Validation is generally better ... but ... Bengio, Y., & Grandvalet, Y. (2004). No unbiased estimator of the variance of k-fold cross-validation. Journal of machine learning research, 5(Sep), 1089-1105. Sebastian Raschka STAT 479: Machine Learning FS 2018 18
Trang 19
Bias of Estimators Example Normal Distribution: 𝒩(μ,σ2) 1 1 (x[i] − μ)2 Probability density f(x[i];μ,σ2) = exp − function: 2πσ2 ( 2 σ2 ) Is the sample mean an unbiased estimator of the mean of the Gaussian? 1 μ̂ = x[i] n ∑ i Sebastian Raschka STAT 479: Machine Learning FS 2018 19
Trang 20
Bias of Estimators Example Is the sample mean an unbiased estimator of the mean of the Gaussian? 1 Bias(μ)̂ = E[μ]̂ − μ μ̂ = x[i] n ∑ 1 i = E[ x[i]] − μ n ∑ i 1 = E[x[i]] − μ n ∑ i 1 = ∑μ − μ n i = μ − μ = 0 Sebastian Raschka STAT 479: Machine Learning FS 2018 20
Hiển thị 20 trang đầu. Tải tài liệu để xem đầy đủ.
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!