Machine Learning Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2017-11-24 Disclaimer: Big Data software is constantly updated, code samples may be outdated.
Introduction Methodology Classification Regression Clustering Association Rule Mining Meta-Learning Summary Outline 1 Introduction 2 Methodology 3 Classification 4 Regression 5 Clustering 6 Association Rule Mining 7 Meta-Learning 8 Summary Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 2 / 49
Introduction Methodology Classification Regression Clustering Association Rule Mining Meta-Learning Summary Data Mining (Knowledge Discovery) [1,35] Definition Data mining: process of discovering patterns in large data sets (Semi-)Automatic analysis of large data to identify interesting patterns Using artificial intelligence, machine learning, statistics and databases Tasks / Problems for data mining Classification: predict the category of samples Regression: find a function to model numeric data with the least error Anomaly detection: identify unusual data (relevant or error) Association rule learning: identify relationships between variables Clustering: discover and classify similar data into structures and groups Summarization: find a compact representation of the data Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 3 / 49
Introduction Methodology Classification Regression Clustering Association Rule Mining Meta-Learning Summary Terminology for Input Data [1, 40] Sample: instances (subset) of the unit of observation Feature: measurable property of a phenomenon (explanatory variable) The set of features is usually written as vector (f1, ..., fn) Label/response: outcome/property of interest for analysis/prediction Dependent variable Discrete in classification, continuous in regression Forms of features/labels Numeric: a (potentially discrete) number characterizes the property e.g., age of people Categorical/nominal: a set of classes e.g., eye color Dichotomous (binary) variable: contains only two classes (Male: Yes/No) Ordinal: an ordered set of classes e.g., babies, teens, adults, elderly Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 4 / 49
Introduction Methodology Classification Regression Clustering Association Rule Mining Meta-Learning Summary Example Data Imagine we have data about alumni from the university Field of study Gender Age Succ. exams Fail. exams Avg. grade∗ Graduate Dur. studies CS M 24 21 1 2.0 Yes 10 CS M 22 5 2 1.7 Enrolled 2 Physics F 23 20 1 1.3 Enrolled 6 Physics M 25 8 10 3.0 No 10 Categorical: field of study, gender, graduate, (favourite colour) Numeric: age, successful/failed exams, duration of studies Numeric: average grade; Ordinal: very good, good, average, failed Our goal defines the machine learning problem Predict if a student will graduate ⇒ classification Prescriptive analysis: we may want to support these students better Predict the duration (in semesters) for the study ⇒ regression Clustering to see if there are interesting classes of students We could label these, e.g., the prodigies, the lazy, ... Probably not too helpful for the listed features Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 5 / 49
Introduction Methodology Classification Regression Clustering Association Rule Mining Meta-Learning Summary Terminology for Learning [40] Online learning: update the model constantly while it is applied Offline (batch) learning: learn from data (training phase), then apply Supervised learning: feature and label are provided in the training Unsupervised learning: no labels provided, relevant structures must be identified by the algorithms, i.e., descriptive task of pattern discovery Reinforcement learning: algorithm tries to perform a goal while interacting with the environment Humans use reinforcement, (semi)-supervised and unsupervised learning Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 6 / 49
Introduction Methodology Classification Regression Clustering Association Rule Mining Meta-Learning Summary Overview of Machine Learning Algorithms (Excerpt) Classification k-Nearest neighbor Naive bayes Pattern detection Decision trees Association rules Classification rule learners k-means clustering density-based clustering Regression/Numeric prediction model-based clustering Linear regression Regression trees Meta-learning algorithms Model trees Bagging Boosting Regression & classification Random forests Neuronal networks Support vector machines Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 7 / 49
Introduction Methodology Classification Regression Clustering Association Rule Mining Meta-Learning Summary Machine Learning in Practice [1] Process / Phases 1 Data collection: combining data into a single source 2 Data exploration and preparation: inspection and data cleanup 3 Model training: depending on machine learning task choose algorithm 4 Model evaluaton: check accuracy of the model 5 Model improvement: if necessary try to improve accuracy by utilizing advanced methods or providing additional input Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 8 / 49
Introduction Methodology Classification Regression Clustering Association Rule Mining Meta-Learning Summary Cross Industry Standard Process for Data Mining [39] CRISP-DM is a commonly used methodology from data mining experts Phases Business understanding: business objectives, requirements, constraints; converting the problem to a data mining problem Data understanding: collecting initial data, exploration, assessing data quality, identify interesting subsets Data preparation: creation of derived data from the raw data (data munging) Modeling: modeling techniques are selected and applied to create models, assess model quality/validation Evaluation (wrt business): check business requirements, review construction of the model(s), decide use Source: Kenneth Deployment: applying the model for knowledge extraction; Jensen [38] creating a report, implementing repeatable data mining process Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 9 / 49
Introduction Methodology Classification Regression Clustering Association Rule Mining Meta-Learning Summary 1 Introduction 2 Methodology 3 Classification 4 Regression 5 Clustering 6 Association Rule Mining 7 Meta-Learning 8 Summary Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 10 / 49
Introduction Methodology Classification Regression Clustering Association Rule Mining Meta-Learning Summary Normalization of Data [1, p. 72] Several algorithms require that numeric variables are normalized The numbers of the feature vector are treaded identically Example: in the features (age, income, is_male), income is >> age Treatment: scale features similar, e.g., all values between 0 and 1 Min-Max normalization X = X−min(X) new max(X)−min(X) Z-Score standardization X = X−mean(X) new StdDev(X) Especially useful for normal distributed data Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 11 / 49
Introduction Methodology Classification Regression Clustering Association Rule Mining Meta-Learning Summary Dummy Coding [1] Problem: distance is not defined for categorical data Regression does not make sense for categorical data Dummy coding transforms N classes into N-1 dummy (proxy) variables 0 indicates instance is of given class 1 indicates use other class The last class is the reference class Dummy coding works well for features Independent prediction of several “feature” classes must be resolved, i.e., more than one class is predicted as 1 Example Color: Red, blue, green Dummy variables: color_red, color_blue, color_green Color green could be omitted and be the reference Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 12 / 49
Introduction Methodology Classification Regression Clustering Association Rule Mining Meta-Learning Summary Treating Missing Data [32, 33, 1, p.300] Problem: a feature is not available for an example Alternatives Deletion: remove examples with missing (N/A) data Problem: we may have many features of which many examples miss one Imputation: replace N/A with substitution values Hot-deck imputation: replace value with a random value from similiar entity Last observation carried forward: simply use last observed value Replace with median, mean (of similar entities) Interpolation (or Kriging) Apply a regression model Statistic regression: replace with mean + random variance Replacing too many instances may complicate analysis/exploration Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 13 / 49
Introduction Methodology Classification Regression Clustering Association Rule Mining Meta-Learning Summary Strategy for Learning [40] Goal: Learn properties of the population from a sample Data quality is usually suboptimal Erroneous samples (random noise, ambivalent data) Overfitting: a model describes noise in the sample instead of population properties Underfitting: a model ignores small but important patterns Robust algorithms reduce the chance of fitting noise How accurate is a specific model on the population? Should we train a model on our data and check its accuracy on the same? As the model is trained on the data, it should be able to be accurate A lookup table might reproduce the data perfectly but is not useful Resubstitution error: training/testing with the same data Shows how well the model can fit A bad fit can be an indicator for ambivalent/erroneous data A bad fit can also show that the method is not appropriate for the data Personally, I always do check model quality first on the resubstitution error Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 14 / 49
Introduction Methodology Classification Regression Clustering Association Rule Mining Meta-Learning Summary Systematic Testing with the Holdout Method Split data into training (50%), test (25%) and validation (25%) set Training set: build/train model from this data sample Validation set: check model quality and refine the models Test set: check final model accuracy on this set (expected accuracy) Once the best model is identified, train it on complete data set Holdout method. The figure is based on [1, p.337]. Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 15 / 49
Introduction Methodology Classification Regression Clustering Association Rule Mining Meta-Learning Summary Supplementary Strategies Problems Sometimes we have not sufficient training samples Suboptimal selection of training samples may cause problems Classification: some classes may have only a few training samples k-fold cross validation Prevents cases in which we partition data suboptimally See next slide Leave-one-out cross validation Builds model with all elements except one Compute model accuracy on the last (test) element Repeat the process for each element Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 16 / 49
Introduction Methodology Classification Regression Clustering Association Rule Mining Meta-Learning Summary k-fold cross validation 1 Split data into k sets 2 For all permutations: train from k-1 sets, validate with remaining set 3 Compute average error metrics Example with the iris data set 1 library(cvTools) 2 set.seed(123) # initialize random seed generator 3 4 data(iris) 5 # create 10 folds Creating only one training set 6 f = cvFolds(nrow(iris), K=10, R=1, type="random") 7 8 # retrieve all sets 1 # create two classes, train and validation set 9 for (set in 1:10){ 2 mask = sample(2, nrow(iris), repl=T, prob=c(0.9,0.1)) 10 validation = iris[ f$subsets[f$which == set] ,] # 135 elements 3 validation = iris[mask==1, ] 11 training = iris[ f$subsets[f$which != set], ] # 15 elements 4 training = iris[mask==2, ] 12 13 # TODO Now build your model with training data and validate it 14 # TODO Build error metrics for this repeat 15 } 16 17 # Output aggregated error metrics for all repeats 18 19 # Some packages perform the k-cross validation for you Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 17 / 49
Introduction Methodology Classification Regression Clustering Association Rule Mining Meta-Learning Summary Stratified sampling [11] Stratification: dividing the population into homogeneous subgroups before sampling e.g., for clinical trials: people (not) having a disease and smokers, 4 groups Draw the same number of random samples from each group If we have the data already: Split the observed samples into classes and distribute these instances across traing/test/validation set Alternatively: Draw the same number of elements from each class Example problem: class imbalance problem [1, p. 312] Consider we have test A for a disease We know that 990 people are healthy and 10 people have the disease Assume the test always reports “healthy” Is this a good test? It is correct in 99% of cases! ⇒ A careful assessment of model performance is needed Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 18 / 49
Introduction Methodology Classification Regression Clustering Association Rule Mining Meta-Learning Summary Evaluating Model Performance Idea: compare true value with predicted “value” on the training data Algorithms return the predicted class/numeric value Classification returns the class (e.g., color, healthy: yes/no) Regression the numeric value Algorithms may return a probability of the prediction Likelihood that the value was correct on the training/test set Sometimes the choice is tight, i.e., 49% class A vs. 51% class B We may skip such results and say we cannot determine the class! There are different metrics to assess the quality of the model Metrics depend on the problem: classification vs. regression Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 19 / 49
Introduction Methodology Classification Regression Clustering Association Rule Mining Meta-Learning Summary Assessing Correctness of Classification Models Confusion matrix Visualizes the performance of the classification Shows count in observation (row) and prediction class (column) Class A Class B Class C Class A AA AB AC Class B BA BB BC Class C CA CB CC Often one class is of interest (e.g., class A) True positive (TP): observation is true, predicted as true (AA) False positive (FP): observation is false, prediction is true (BA, CA) True negative (TN): observation is false, predicted as false (BB, CC) False negative (FN): observation is true, prediction is false (else) There are useful metrics defined on these values Accuracy, error rate, sensitivity, specifity, precision, recall Kappa statistic: correctness vs. random correctness F-measure (F-score): weights precision and recall equally Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 20 / 49
Showing the first 20 pages. Download the document to see it all.
No comments yet. Be the first!
Comments (0)
No comments yet. Be the first!