Lecture 12 Model Evaluation 5: Performance Metrics STAT 479: Machine Learning, Fall 2018 Sebastian Raschka http://stat.wisc.edu/~sraschka/teaching/stat479-fs2018/ Sebastian Raschka STAT 479: Machine Learning FS 2018 1 Recap Class labels (Step 1) (Step 2) Test set Training set pipeline.fit(…) pipeline.predict(…) Pipeline .fit(…) & .transform(…) .fit(…) & .transform(…) Scaling .transform(…) Dimensionality Reduction Learning Algorithm .fit(…) .transform(…) Predictive Model Class labels .predict(…) Source: Python Machine Learning, 2nd Edition Sebastian Raschka STAT 479: Machine Learning FS 2018 2 Recap High bias High variance Accuracy Accuracy Number of training samples Number of training samples Good bias-variance trade-off Training accuracy Validation accuracy Desired accuracy Accuracy 1.0 Number of training samples Source: Python Machine Learning, 2nd Edition Sebastian Raschka STAT 479: Machine Learning FS 2018 3 P Actual Class N mance of aNegatives classifier for binary classification tasks. This square matrix consis Positives (TP) of columns(FN) and rows that list the number of instances as absolute or relativ 2x2 Confusion Matrix ”actual class” vs. ”predicted class” ratios. Let P be the label of class 1 and N be the label of a second class or the lab True False of all classes that are not class 1 in a multi-class setting. Positives (FP) Negatives (TN) Predicted class P N True False to ROC analysis by Tom The following equations are based on An introduction Positives Negatives Fawcett [1]. P (TP) (FN) Actual Class 2 Prediction Error and Accuracy True False Negatives Positives Both the prediction errorN (ERR) and accuracy (ACC) provide general informa(FP) tion about how many samples are misclassified.(TN) The error can be understood as the sum of all false predictions divided by the number of total predications, and the the accuracy is calculated as the sum of correct predictions divided by the total number of pred