STAT 479: Machine Learning Lecture Notes Sebastian Raschka Department of Statistics University of Wisconsin–Madison http://stat.wisc.edu/∼sraschka/teaching/stat479-fs2018/ Fall 2018 Contents 11 Statistical Tests and Algorithm Comparison 1 11.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 11.2 Testing the Difference of Proportions . . . . . . . . . . . . . . . . . . . . . . . 1 11.3 Comparing Two Models with the McNemar Test . . . . . . . . . . . . . . . . 3 11.4 Exact p-Values via the Binomial Test . . . . . . . . . . . . . . . . . . . . . . . 5 11.5 Multiple Hypotheses Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 11.6 Cochran’s Q Test for Comparing the Performance of Multiple Classifiers . . . 7 11.7 The F -test for Comparing Multiple Classifiers . . . . . . . . . . . . . . . . . . 10 11.8 Comparing Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 11.9 Resampled Paired t-Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 11.10k-fold Cross-validated Paired t-Test . . . . . . . . . . . . . . . . . . . . . . . . 13 11.11Dietterich’s 5x2-Fold Cross-Validated Paired t-Test . . . . . . . . . . . . . . . 13 11.12Alpaydin’s Combined 5x2cv F -test . . . . . . . . . . . . . . . . . . . . . . . . 14 11.13Effect size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 11.14Nested Cross-Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 11.15Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 STAT 479: Machine Learning Lecture Notes Sebastian Raschka Department of Statistics University of Wisconsin–Madison http://stat.wisc.edu/∼sraschka/teaching/stat479-fs2018/ Fall 2018 11 Statistical Tests and Algorithm Comparison 11.1 Overview This fourth lecture in the model evaluation part of this machine learning class provides overviews of several statistical hypothesis testin

Đọc toàn bộ tài liệu