Page 1
Lecture 12 Model Evaluation 5: Performance Metrics STAT 479: Machine Learning, Fall 2018 Sebastian Raschka http://stat.wisc.edu/~sraschka/teaching/stat479-fs2018/ Sebastian Raschka STAT 479: Machine Learning FS 2018 1
Page 2
Recap (Step 1) (Step 2) Class labels Training set Test set pipeline.fit(…) pipeline.predict(…) Pipeline Scaling .fit(…) & .transform(…) .transform(…) Dimensionality Reduction .fit(…) & .transform(…) Learning Algorithm .transform(…) .fit(…) Predictive Model Class labels .predict(…) Source: Python Machine Learning, 2nd Edition Sebastian Raschka STAT 479: Machine Learning FS 2018 2
Page 3
Recap High bias High variance 1.0 Accuracy Accuracy Number of training samples Number of training samples Good bias-variance trade-off Training accuracy Validation accuracy Desired accuracy Accuracy Number of training samples Source: Python Machine Learning, 2nd Edition Sebastian Raschka STAT 479: Machine Learning FS 2018 3
Page 4
P Positivesmance of a classifier for binary classification tasks. This square matrix consisNegatives (TP)of columns and rows that list the number of instances as absolute or relativ2x2 Confusion Matrix(FN) Actual ”actual class” vs. ”predicted class” ratios. Class Let P be the label of class 1 and N be the label of a second class or the lab Falseof all classes that areTrue not class 1 in a multi-class setting. N Positives Negatives (FP) (TN) Predicted class P N The following equations are based onTrue An introduction to ROC analysisFalse by Tom Fawcett [1]. P Positives Negatives (TP) (FN) Actual 2 Prediction Error and AccuracyClass False True N Positives Negatives Both the prediction error (ERR) and accuracy (ACC) provide general informa- tion about how many samples are misclassified. The(FP) (TN) error can be understood as the sum of all false predictions divided by the number of total predications, and the the accuracy is calculated as the sum of correct predictions divided by the total number of predictions, respectively. The following equations are based onF P + F N An introduction to ROC analysis by To Fawcett [1].ERR = = 1 ACC (1) F P + F N + T P + T N T P + T N ACC = = 1 ERR (2) F P + F N + T P + T N 2 Prediction Error and Accuracy Both the prediction error (ERR) and accuracy (ACC) provide general inform tion about how many samples are misclassified. TheSebastian Raschka STAT 479: Machine Learning FS 2018 error can be understoo4
Page 5
Confusion Matrix for Multi-Class Settings Predicted Labels Class 0 Class 1 Class 2 Class 0 T(0,0) Class 1 T(1,1) True Labels Class 2 T(2,2) Confusions matrices are traditionally for binary class problems but we can easily generalize it to multi-class settings Sebastian Raschka STAT 479: Machine Learning FS 2018 5
Page 6
ages that were incorrectly classified as spam (False Positives): A situation e a person misses an important message is considered as ”worse” than a sit-False Positive Rate and False Negative Rate n where a person ends up with a few spam messages in his e-mail inbox. In ast to the FPR, the True Positive Rate provides useful information about action of positive (or relevant) samples that were correctly identified out total pool of Positives. F P F P F P R = = (3) N F P + T N T P T P T P R = = (4) P F N + T P ecision, Recall, and the F1-Score ision (PRE)•andThink of it in a spam classification problem (what are true positives, and ifRecall (REC) are metrics that are more commonly used in mation Technologyyou had to pick one at the expense of the other: would you ratherand related to the False and True Prositive Rates. In Recall is synonymous to thedecrease the FPR or increase the TPR?)True Positive Rate and also sometimes called itivity. The F1-Score can be understood as a combination of both Precision ecall [2]. recision (PRE) and Recall (REC) are metrics that are more commonly used formation Technology and related to the False and True Prositive Rates. t, Recall is synonymous to the True Positive Rate and is sometimes also Sensitivity. The F1-Score can be understood as a combination of both ision and Recall [2]. Sebastian Raschka STAT 479: Machine Learning FS 2018 6
Page 8
fact, Recall is synonymous to the True Positive Rate and also sometimes called Sensitivity. The F1-Score can be understood as a combination of both Precision and Recall [2]. Precision, Recall, and F1 Score Precision (PRE) and Recall (REC) are metrics that are more commonly used in Information Technology and related to the False and True Prositive Rates. In fact, Recall is synonymous to the True Positive Rate and is sometimes also called Sensitivity. The F1-Score can be understood as a combination of both Precision and Recall [2]. T P P RE = (5) T P + F P T P T P REC = T P R = = (6) P F N + T P P RE · REC F1 = 2 · (7) P RE + REC • Terms that are more popular in Information Technology • Recall is actually just another term for True Positive Rate (or "sensitivity") 3 Sebastian Raschka STAT 479: Machine Learning FS 2018 8
Page 9
Precision and Recall https://en.wikipedia.org/wiki/Precision_and_recall Sebastian Raschka STAT 479: Machine Learning FS 2018 9
Page 10
Sensitivity and Specificity 5 Sensitivity and Specificity Sensitivity (SEN) is synonymous to Recall and the True Positive Rate whereas Specificity (SPC) is synonymous to the True Negative Rate — Sensitivity mea- sures the recovery rate of the Positives and complimentary, the Specificity mea- sures the recovery rate of the Negatives. T P T P SEN = T P R = REC = = (8) P F N + T P T N T N SP C = T N R = = (9) N F P + T N 6 Matthews correlation coecient Sensitivity measures the recovery rate of the Positives and complimentary, the Specificity Matthews correlation coemeasures the recovery rate of the Negatives.cient (MCC) was first formulated by Brian W. Matthews [3] in 1975 to assess the performance of protein secondary structure predictions. The MCC can be understood as a specific case of a linear correlation coecient (Pearson r) for a binary classification setting and is considered as especially use- ful in unbalanced class settings. The previous metrics take values in the range between 0 (worst) and 1 (best), whereas the MCC is bounded between the range 1 (perfect correlation between ground truth and predicted outcome) and -1 (inverse or negative correlation) — a value of 0 denotes a random prediction. Sebastian Raschka STAT 479: Machine Learning FS 2018 10
Page 11
P F N + T P Matthew's Correlation CoeT N T N fficient SP C = T N R = = (9) N F P + T N 6 Matthews correlation coecient • Matthews correlation coefficient (MCC) was first formulated by Brian W. Matthews [1] in 1975 to assess the performance of protein secondary structure predictions • Matthews correlation coecient (MCC) was first formulated by Brian W. Matthews The MCC can be understood as a specific case of a linear correlation coefficient (Pearson r) for a binary classification setting[3] in 1975 to assess the performance of protein secondary structure predictions. The MCC can be understood as a specific case of a linear correlation coecient • Considered as especially useful in unbalanced class settings.(Pearson r) for a binary classification setting and is considered as especially use- • The previous metrics take values in the range between 0 (worst) and 1 (best)ful in unbalanced class settings. The previous metrics take values in the range • The MCC is bounded between the range 1 (perfect correlation between ground truth and predictedbetween 0 (worst) and 1 (best), whereas the MCC is bounded between the outcome) and -1 (inverse or negative correlation) — a value of 0 denotes a random prediction.range 1 (perfect correlation between ground truth and predicted outcome) and -1 (inverse or negative correlation) — a value of 0 denotes a random prediction. T P · T N F P · F N M CC = p (10) (T P + F P)(T P + F N)(T N + F P)(T N + F N) [1] Brian W Matthews. Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochimica et7 Receiver Operator Characteristic (ROC) Biophysica Acta (BBA)- Protein Structure, 405(2):442–451, 1975. Receiver Operator Characteristics (ROC) graphs are useful tools to select clas- sification models based on their performance with respect to the False Positive and True Positive rates. The diagonal of a ROC graph can be interpreted as random guessing and classification models that fall below the diagonal are considered as worse thanSebastian Raschka STAT 479: Machine Learning FS 2018 11
Page 12
Balanced Accuracy / Average Per-Class Accuracy Predicted Labels Predicted Labels Class 0 Class 1 Class 2 Class 0 Class 1 Class 2 Class 0 T(0,0) Class 0 3 0 0 Class 1 T(1,1) Class 1 7 50 12 True Labels True LabelsClass 2 T(2,2) Class 2 0 0 18 T 3 + 50 + 18 ACC = ACC = ≈ 0.79 n 90 83/90 + 71/90 + 78/90 APC ACC = ≈ 0.86 3 Sebastian Raschka STAT 479: Machine Learning FS 2018
Page 13
Balanced Accuracy / Average Per-Class Accuracy Predicted Labels Class 0 Neg Class Class 0 3 0 Predicted Labels Neg Class 0 Class 1 Class 2 Class 7 80 True Labels Class 0 3 0 0 Predicted Labels Neg Class 1 7 50 12 Class 1 True Labels Class Class 2 0 0 18 Class 1 50 19 True LabelsNeg Class 0 21 Predicted Labels Class 2 Neg Class 83/90 + 71/90 + 78/90 APC ACC = ≈ 0.86 Class 2 18 0 3 True LabelsNeg Class 12 60 Sebastian Raschka STAT 479: Machine Learning FS 2018
Page 14
Receiver Operating Characteristic curve (ROC curve) • Trade-off between True Positive Rate and False Positive Rate • ROC can be plotted by changing the prediction threshold • ROC term comes from "Radar Receiver Operators" (analysis of radar [RAdio Direction And Ranging] images) Sebastian Raschka STAT 479: Machine Learning FS 2018 14
Page 15
Receiver Operating Characteristic curve (ROC curve) • ?.? = Perfect Prediction • ?.? = Random Prediction Sebastian Raschka STAT 479: Machine Learning FS 2018 15
Page 18
Binary Classifiers and One-vs-Rest (OvR) / One-vs-All (OvA) Big O: O( ? ) Choose the class with the highest confidence score Sebastian Raschka STAT 479: Machine Learning FS 2018 18
Page 19
Binary Classifiers and One-vs-One (OvO) / All-vs-All (AvA) num_classes x (num_classes - 1) / 2 Big O: O( ? ) Select the class by majority vote (and use confidence score in case of ties) Sebastian Raschka STAT 479: Machine Learning FS 2018 19
Page 20
Macro and Micro Averaging TP1 + … + TPc PREmicro = TP1 + … + TPc + FP1 + … + FPc PRE1 + … + PREc PREmacro = c Micro-averaging is useful if we want to weight each instance or prediction equally, whereas macro-averaging weights all classes equally to evaluate the overall performance of a classifier with regard to the most frequent class labels. Sebastian Raschka STAT 479: Machine Learning FS 2018 20
Affichage des 20 premières pages. Téléchargez le document pour le voir en entier.
Commentaires (0)
Aucun commentaire pour le moment. Soyez le premier !