Eval algo slides (11) (Đánh giá mô hình, so sánh thuật toán và kiểm định McNemar) - Sebastian Raschka
Generating preview...
Slide bài giảng số 11 về đánh giá mô hình, tập trung so sánh thuật toán và kiểm định McNemar.
Description
Lecture 11 Model Evaluation 4: Algorithm Comparisons STAT 479: Machine Learning, Fall 2018 Sebastian Raschka http://stat.wisc.edu/~sraschka/teaching/stat479-fs2018/ Sebastian Raschka STAT 479: Machine Learning FS 2018 1 Overview Bias and Variance Basics Overfitting and Underfitting Holdout method Confidence Intervals Model Eval Lectures Repeated holdout Resampling methods Empirical confidence intervals Hyperparameter tuning Cross-Validation Model selection This Lecture Algorithm Selection Statistical Tests Evaluation Metrics Sebastian Raschka STAT 479: Machine Learning FS 2018 2 Overview, (my) "recommendations" Large dataset Performance estimation Small dataset Large dataset Model selection (hyperparameter optimization) and performance estimation Small dataset Large dataset Model & algorithm comparison Small dataset Sebastian Raschka 2-way holdout method (train/test split) Confidence interval via normal approximation (Repeated) k-fold cross-validation without independent test set Leave-one-out cross-validation without independent test set Confidence interval via 0.632(+) bootstrap 3-way holdout method (train/validation/test split) (Repeated) k-fold cross-validation with independent test set Leave-one-out cross-validation with independent test set Disjoint training sets + test set (algorithm comparison, AC) McNemar test (model comparison, MC) Cochran’s Q + McNemar test (MC) Combined 5x2cv F test (AC) Nested cross-validation (AC) STAT 479: Machine Learning FS 2018 3 Comparing two machine learning classifiers -- McNemar's Test McNemar's test, introduced by Quinn McNemar in 1947 [1], is a non-parametric statistical test for paired comparisons that can be applied to compare the performance of two machine learning classifiers: Task Gaussian data Compare a group to a reference value … Paired nominal data Binomial test Compare a pair of groups McNemar’s test Compare two unpaired grou
AI summary
- Document name
- Eval algo slides (11) (Đánh giá mô hình, so sánh thuật toán và kiểm định McNemar) - Sebastian Raschka
- School / Course
- University Wisconsin-Madison · Machine learning
- Content
- Bài giảng này giới thiệu các phương pháp đánh giá và so sánh thuật toán học máy, tập trung vào McNemar's Test để so sánh hai mô hình dựa trên ma trận nhầm lẫn.
- Table of contents
- Lecture 11
- Model Evaluation 4:
- Algorithm Comparisons
- STAT 479: Machine Learning, Fall 2018
- Sebastian Raschka
- Overview
- Bias and Variance
- Basics
- Overfitting and Underfitting
- Holdout method
- Confidence Intervals
- Model Eval Lectures
- Repeated holdout
- Resampling methods
- Empirical confidence intervals
- Hyperparameter tuning
- Cross-Validation
- Model selection
- This Lecture
- Algorithm Selection
- Statistical Tests
- Evaluation Metrics
- Overview, (my) "recommendations"
- Large dataset
- Performance estimation
- Small dataset
- Model selection (hyperparameter optimization) and performance estimation
- Model & algorithm comparison
- 2-way holdout method (train/test split)
- Confidence interval via normal approximation
- (Repeated) k-fold cross-validation without independent test set
- Leave-one-out cross-validation without independent test set
- Confidence interval via 0.632(+) bootstrap
- 3-way holdout method (train/validation/test split)
- (Repeated) k-fold cross-validation with independent test set
- Leave-one-out cross-validation with independent test set
- Disjoint training sets + test set (algorithm comparison, AC)
- McNemar test (model comparison, MC)
- Cochran’s Q + McNemar test (MC)
- Combined 5x2cv F test (AC)
- Nested cross-validation (AC)
- Comparing two machine learning classifiers -- McNemar's Test
- McNemar's test, introduced by Quinn McNemar in 1947 [1], is a non-parametric statistical test for paired comparisons that can be applied to compare the performance of two machine learning classifiers:
- Task
- Gaussian data
- Compare a group to a reference value
- Paired nominal data
- Binomial test
- Compare a pair of groups
- McNemar’s test
- Compare two unpaired groups
- test, Fisher’s exact test
- Comparing two machine learning classifiers -- McNemar's Test
- Also referred to as "within-subjects chi-squared test"
- Applied to paired nominal data based on a version of a 2x2 confusion matrix
- Compares the predictions of two models to each other rather than listing false positive, true positive, false negative, and true negative counts of a single model
- The layout of the 2x2 confusion matrix suitable for McNemar's test is shown in the following figure:
- Model 2 correct
- Model 2 wrong
- Model 1 correct
- Pages
- 42 pages
- Uploaded by
- Uni24h
Frequently asked questions
Is this document free?
Yes. “Eval algo slides (11) (Đánh giá mô hình, so sánh thuật toán và kiểm định McNemar) - Sebastian Raschka” is free — just sign in and click Download to get the original file.
How many pages is this document?
The document has 42 pages, for the course Machine learning. You can preview it online before downloading.
Can I preview before downloading?
Yes. You can preview this document right on this page with the online reader, then decide whether to download.
Eval algo slides (11) (Đánh giá mô hình, so sánh thuật toán và kiểm định McNemar) - Sebastian Raschka
Generating preview...
Lecture 11 Model Evaluation 4: Algorithm Comparisons STAT 479: Machine Learning, Fall 2018 Sebastian Raschka http://stat.wisc.edu/~sraschka/teaching/stat479-fs2018/ Sebastian Raschka STAT 479: Machine Learning FS 2018 1 Overview Bias and Variance Basics Overfitting and Underfitting Holdout method Confidence Intervals Model Eval Lectures Repeated holdout Resampling methods Empirical confidence intervals Hyperparameter tuning Cross-Validation Model selection This Lecture Algorithm Selection Statistical Tests Evaluation Metrics Sebastian Raschka STAT 479: Machine Learning FS 2018 2 Overview, (my) "recommendations" Large dataset Performance estimation Small dataset Large dataset Model selection (hyperparameter optimization) and performance estimation Small dataset Large dataset Model & algorithm comparison Small dataset Sebastian Raschka 2-way holdout method (train/test split) Confidence interval via normal approximation (Repeated) k-fold cross-validation without independent test set Leave-one-out cross-validation without independent test set Confidence interval via 0.632(+) bootstrap 3-way holdout method (train/validation/test split) (Repeated) k-fold cross-validation with independent test set Leave-one-out cross-validation with independent test set Disjoint training sets + test set (algorithm comparison, AC) McNemar test (model comparison, MC) Cochran’s Q + McNemar test (MC) Combined 5x2cv F test (AC) Nested cross-validation (AC) STAT 479: Machine Learning FS 2018 3 Comparing two machine learning classifiers -- McNemar's Test McNemar's test, introduced by Quinn McNemar in 1947 [1], is a non-parametric statistical test for paired comparisons that can be applied to compare the performance of two machine learning classifiers: Task Gaussian data Compare a group to a reference value … Paired nominal data Binomial test Compare a pair of groups McNemar’s test Compare two unpaired grou
Read full document
- Document name
- Eval algo slides (11) (Đánh giá mô hình, so sánh thuật toán và kiểm định McNemar) - Sebastian Raschka
- School / Course
- University Wisconsin-Madison · Machine learning
- Content
- Bài giảng này giới thiệu các phương pháp đánh giá và so sánh thuật toán học máy, tập trung vào McNemar's Test để so sánh hai mô hình dựa trên ma trận nhầm lẫn.
- Table of contents
- Lecture 11
- Model Evaluation 4:
- Algorithm Comparisons
- STAT 479: Machine Learning, Fall 2018
- Sebastian Raschka
- Overview
- Bias and Variance
- Basics
- Overfitting and Underfitting
- Holdout method
- Confidence Intervals
- Model Eval Lectures
- Repeated holdout
- Resampling methods
- Empirical confidence intervals
- Hyperparameter tuning
- Cross-Validation
- Model selection
- This Lecture
- Algorithm Selection
- Statistical Tests
- Evaluation Metrics
- Overview, (my) "recommendations"
- Large dataset
- Performance estimation
- Small dataset
- Model selection (hyperparameter optimization) and performance estimation
- Model & algorithm comparison
- 2-way holdout method (train/test split)
- Confidence interval via normal approximation
- (Repeated) k-fold cross-validation without independent test set
- Leave-one-out cross-validation without independent test set
- Confidence interval via 0.632(+) bootstrap
- 3-way holdout method (train/validation/test split)
- (Repeated) k-fold cross-validation with independent test set
- Leave-one-out cross-validation with independent test set
- Disjoint training sets + test set (algorithm comparison, AC)
- McNemar test (model comparison, MC)
- Cochran’s Q + McNemar test (MC)
- Combined 5x2cv F test (AC)
- Nested cross-validation (AC)
- Comparing two machine learning classifiers -- McNemar's Test
- McNemar's test, introduced by Quinn McNemar in 1947 [1], is a non-parametric statistical test for paired comparisons that can be applied to compare the performance of two machine learning classifiers:
- Task
- Gaussian data
- Compare a group to a reference value
- Paired nominal data
- Binomial test
- Compare a pair of groups
- McNemar’s test
- Compare two unpaired groups
- test, Fisher’s exact test
- Comparing two machine learning classifiers -- McNemar's Test
- Also referred to as "within-subjects chi-squared test"
- Applied to paired nominal data based on a version of a 2x2 confusion matrix
- Compares the predictions of two models to each other rather than listing false positive, true positive, false negative, and true negative counts of a single model
- The layout of the 2x2 confusion matrix suitable for McNemar's test is shown in the following figure:
- Model 2 correct
- Model 2 wrong
- Model 1 correct
- Pages
- 42 pages
- Uploaded by
- Uni24h
Comments (0)
No comments yet. Be the first!
Eval algo notes (11) (Các kiểm định thống kê và so sánh thuật toán) - Sebastian Raschka
Refresher probabilities statistics (Xác suất và thống kê cho Machine Learning) - Afshine Amidi and Shervine Amidi
Ensembles notes (07) (Các phương pháp học ensemble) - Sebastian Raschka
Super cheatsheet machine learning (Khái niệm và công thức chính trong học máy) - Afshine Amidi and Shervine Amidi
Cheatsheet Deep Learning (Khái niệm chính về học sâu) - Afshine Amidi and Shervine Amidi
Chương 7.Cơ học lượng tử - Vật lý đại cương 3 - TS.Nguyễn Thị Trang
Chương 6.Quang học lượng tử - Vật lý đại cương 3 - TS.Nguyễn Thị Trang
Chương 5.Thuyết tương đối - Vật lý đại cương 3 - TS.Nguyễn Thị Trang
Chương 4. Tán xạ ánh sáng - Vật lý đại cương 3 - TS.Nguyễn Thị Trang
Chương 3.Phân cực ánh sáng - Vật lý đại cương 3 - TS.Nguyễn Thị Trang
Comments (0)
No comments yet. Be the first!