1 ページ
Lecture 09 Model Evaluation 2: Confidence Intervals STAT 479: Machine Learning, Fall 2018 Sebastian Raschka http://stat.wisc.edu/~sraschka/teaching/stat479-fs2018/ Sebastian Raschka STAT 479: Machine Learning FS 2018 1
2 ページ
Overview Bias and Variance Overfitting and Underfitting This Lecture Basics Holdout method Confidence Intervals Repeated holdout Resampling methods Model Eval Lectures Empirical confidence intervals Hyperparameter tuning Cross-Validation Model selection Algorithm Selection Statistical Tests Evaluation Metrics Sebastian Raschka STAT 479: Machine Learning FS 2018 2
3 ページ
• Concepts first • (More) Code at the end of the lecture Sebastian Raschka STAT 479: Machine Learning FS 2018 3
4 ページ
Main points why we evaluate the predictive performance of a model: 1. Want to estimate the generalization performance, the predictive performance of our model on future (unseen) data. 2. Want to increase the predictive performance by tweaking the learning algorithm and selecting the best performing model from a given hypothesis space. 3. Want to identify the ML algorithm that is best-suited for the problem at hand; thus, we want to compare different algorithms, selecting the best-performing one as well as the best performing model from the algorithm’s hypothesis space. Sebastian Raschka STAT 479: Machine Learning FS 2018 4
5 ページ
• Training set error is an optimistically biased estimator of the generalization error • Test set error is an unbiased estimator of the generalization error (test sample and hypothesis chosen independently) • (in practice, it is actually pessimistically biased; why?) Sebastian Raschka STAT 479: Machine Learning FS 2018 5
6 ページ
Training Evaluation 1 2 3 4 5 6 7 8 9 10 Holdout Method 1 2 3 4 5 6 7 8 9 10 2-Fold Cross-Validation 1 2 3 4 5 6 7 8 9 10 Sebastian Raschka STAT 479: Machine Learning FS 2018 6
7 ページ
Often using the holdout method is not a good idea ... Sebastian Raschka STAT 479: Machine Learning FS 2018 7
8 ページ
Often using the holdout method is not a good idea ... Test set error as generalization error estimator is pessimistically biased (not so bad) But it does not account for variance in the training data (bad) Sebastian Raschka STAT 479: Machine Learning FS 2018 8
9 ページ
Why is pessimistic bias not "so bad"? Suppose we have the following ranking based on accuracy: h2: 75% > h1: 70% > h3: 65%, we would still rank them the same way if we add a 10% pessimistic bias: h2: 65% > h1: 60% > h3: 55%. Sebastian Raschka STAT 479: Machine Learning FS 2018 9
10 ページ
Often using the holdout method is not a good idea ... • Test set error as generalization error estimator is pessimistically biased (not so bad) • Does not account for variance in the training data (bad) Sebastian Raschka STAT 479: Machine Learning FS 2018 10
11 ページ
We have to keep in mind that a dataset represents a random sample drawn from a probability distribution, and we typically assume that this sample is representative of the true population – more Issues with Subsampling (Independence violation)or less. Now, further subsampling without replacement alters the statistic (mean, proportion, and variance) of the sample. The degree to which subsampling without replacement affects the statistic of a sample is inversely proportional to the size of the sample. Let us have a look at an example using the Iris dataset 1, which we randomly divide into 2/3 training data and 1/3 test data as illustrated in Figure 1. (The source code for generating this graphic is available on GitHub2.) All samples (n = 150) Training samples (n = 100) Test samples (n = 50) The Iris dataset consists of 50 Setosa, 50 Versicolor, and 50 Virginica flowers; the flower species are distributed uniformly:Figure 1: Distribution of Iris flower classes upon random subsampling into training and test sets. • 33.3% Setosa • 1https://archive.ics.uci.edu/ml/datasets/iris 33.3% Versicolor 2https://github.com/rasbt/model-eval-article-supplementary/blob/master/code/iris-random-dist.ipynb • 33.3% Virginia If our random function assigns 2/3 of the flowers (100) to the training set and 1/3 of the flowers (50) to the test set, it may yield6 the following: • training set → 38 x Setosa, 28 x Versicolor, 34 x Virginica • test set → 12 x Setosa, 22 x Versicolor, 16 x Virginica Sebastian Raschka STAT 479: Machine Learning FS 2018 11
12 ページ
Holdout evaluation Training Data Training Labels Data 1 Labels Test Data Test Labels Hyperparameter 2 Training Data Values Model Training Labels Learning Algorithm Test Data Prediction 3 Performance Model Test Labels Hyperparameter Data Values Final 4 Model Labels Learning Algorithm Sebastian Raschka STAT 479: Machine Learning FS 2018 12
13 ページ
Can we use the holdout method for model selection? Sebastian Raschka STAT 479: Machine Learning FS 2018 13
14 ページ
Holdout validation Training Data Training Labels (hyperparam. tuning) 1 Data Validation Data Labels Validation Labels Test Data Test Labels Hyperparameter Model values 2 Training Data Learning Hyperparameter Model values Algorithm Training Labels Model Hyperparameter values Validation Data Prediction Performance Model Validation Labels Best Hyperparameter values Validation Data Prediction 3 Best Performance Model Validation Model Labels Validation Data Prediction Performance Model Validation Labels Best Hyperparameter Training Data Validation Values 4 Data Model Training Labels Validation Learning Labels Algorithm Test Data Prediction 5 Performance Model Test Labels Best Hyperparameter 6 Data Values Final Labels Learning Model Algorithm Sebastian Raschka STAT 479: Machine Learning FS 2018 14
15 ページ
Holdout validation (hyperparam. tuning) Training Data Training Labels Validation 1 Data Data Labels Validation Labels Test Data Test Labels Hyperparameter Model values 2 Training Data Learning Hyperparameter Model values Algorithm Training Labels Model Hyperparameter values Validation Data Prediction Performance Model Validation Labels Best Hyperparameter values Validation Sebastian RaschkaData STAT 479: Machine LearningPrediction FS 2018 15
16 ページ
Model Hyperparameter values Holdout validation (hyperparam. tuning) Validation Data Prediction Performance Model Validation Labels Best Hyperparameter values Validation Data Prediction 3 Best Performance Model Validation Model Labels Validation Data Prediction Performance Model Validation Labels Best Hyperparameter Training Data Validation Values 4 Data Model Training Labels Validation Learning Labels Algorithm Test Data Prediction 5 Performance Model Test Labels Sebastian Raschka STAT 479: Machine Learning FS 2018 16
17 ページ
Performance Model Validation Labels Holdout validation (hyperparam. tuning) Best Hyperparameter Training Data Validation Values 4 Data Model Training Labels Validation Learning Labels Algorithm Test Data Prediction 5 Performance Model Test Labels Best Hyperparameter 6 Data Values Final Labels Learning Model Algorithm Sebastian Raschka STAT 479: Machine Learning FS 2018 17
18 ページ
Cross-Validation is generally better ... but ... Bengio, Y., & Grandvalet, Y. (2004). No unbiased estimator of the variance of k-fold cross-validation. Journal of machine learning research, 5(Sep), 1089-1105. Sebastian Raschka STAT 479: Machine Learning FS 2018 18
19 ページ
Bias of Estimators Example Normal Distribution: 𝒩(μ,σ2) 1 1 (x[i] − μ)2 Probability density f(x[i];μ,σ2) = exp − function: 2πσ2 ( 2 σ2 ) Is the sample mean an unbiased estimator of the mean of the Gaussian? 1 μ̂ = x[i] n ∑ i Sebastian Raschka STAT 479: Machine Learning FS 2018 19
20 ページ
Bias of Estimators Example Is the sample mean an unbiased estimator of the mean of the Gaussian? 1 Bias(μ)̂ = E[μ]̂ − μ μ̂ = x[i] n ∑ 1 i = E[ x[i]] − μ n ∑ i 1 = E[x[i]] − μ n ∑ i 1 = ∑μ − μ n i = μ − μ = 0 Sebastian Raschka STAT 479: Machine Learning FS 2018 20
コメント (0)
まだコメントはありません。最初のコメントを書きましょう！