Statistics: A Primer Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2016-10-28 Disclaimer: Big Data software is constantly updated, code samples may be outdated.
Descriptive Statistics Distribution of Values Inductive Statistics Summary Outline 1 Descriptive Statistics 2 Distribution of Values 3 Inductive Statistics 4 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 2 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Statistics: Overview Statistics is the study of the collection, analysis, interpretation, presentation, and organization of data [21] Either describe properties of a sample or infer properties of a population Important terms [10] Unit of observation: the entity described by the data Unit of analysis: the major entity that is being analyzed Example: Observe income of each person, analyse differences of countries Statistical population: Complete set of items that share at least one property that is subject of analysis Subpopulation share additional properties Sample: (sub)set of data collected and/or selected from a population If chosen properly, they can represent the population There are many sampling methods, we can never capture ALL items Independence: one observation does not effect another Example: select two people living in Germany randomly Dependent: select one household and pick a married couple Julian M. Kunkel Lecture BigData Analytics, 2016 3 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Statistics: Variables Dependent variable: represents the output/effect Example: word count of a Wikipedia article; income of people Independent variable: assumed input/cause/explanation Example: number of sentences; age, educational level Characterization Univariate analysis: characterize a single variable Bivariate analysis: describes/analyze relationships between two vars Multivariate statistics: analyze/observe multiple dependent variables Example: chemicals in the blood stream of people or chance for cancers, Independent variables are personal information/habits Julian M. Kunkel Lecture BigData Analytics, 2016 4 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Descriptive Statistics [10] The discipline of quantitatively describing main features of sampled data Summarize observations/selected samples Exploratory data analysis (EDA): approach for inspecting data Using different chart types, e.g., box plots, histograms, scatter plot Methods for univariate analysis Distribution of values, e.g., mean, variance, quantiles Probability distribution and density t-test (e.g., check if data is t-distributed) Methods for bivariate analysis Correlation coefficient1 describes linear relationship Rank correlation2: extent by which one variable increases with another var Methods for multivariate analysis Principal component analysis (PCA) converts correlated variables into linearly uncorrelated variables called principal components 1Pearson’s product-moment coefficient 2By Spearman or Kendall Julian M. Kunkel Lecture BigData Analytics, 2016 5 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Example Dataset: Iris Flower Data Set Contains information about iris flower Three species: Iris Setosa, Iris Virginica, Iris Versicolor Data: Sepal.length, Sepal.width, Petal.length, Petal.width 2.0 3.0 4.0 0.5 1.5 2.5 l l l llll l l llll lllll llll l lllll llll l 7.5 ll l l l l R example l llllll lllllll ll lllll l lllllllll llllllllll llllll llll Sepal.Length llllllllllll lll lllllllll lll lllllllll lllllll ll llll lllllllll llll llllllllll lll ll lllllllll l ll lllllll ll l lll lllllllll 6.0 lllllllll l l llll lllll l lll l llll l llllllllllll l l ll llllllllllll ll llllllllll ll lllll l llllll l lllllll llllll llll lll l lllll l lll l llll l llll l l ll llllllllllllllll llllllllllllllll lll llllll llllllllll lll ll l lllll lll lllllllll l l lllllllll l l llllll llllll llllll 1 > data(iris) # load iris data l lll llll llll 4.5 l l l 2 > summary(iris) l l l 3 Sepal.Length Sepal.Width Petal.Length l l ll ll 4.0 ll l lll lll lllllll ll lllllll ll lllllll ll 4 Min. :4.300 Min. :2.000 Min. :1.000 lllllllll l lllllllll l llllllll l l l llllllllll lllll ll Sepal.Width lllllllllll ll lllll lllllllllll ll lllll 5 1st Qu.:5.100 1st Qu.:2.800 1st Qu.:1.600 lllllllll l llllllllllllll lllllllll lllllllllllllll lllllllll lllll llllllll ll 3.0lll llll llllllllllll lllllllllllllllll lllllll l llllllllllllll l lllllllllllll lllllll lllllllllllllllllllllllllllll l llllllllllll llllllll ll lllllllllllllll llllll l l lllllllllllllllllllllll 6 Median :5.800 Median :3.000 Median :4.350 ll lllllllll l l l lllllllll ll l llllllllllll l l ll lll l l llllll l llllll 7 Mean :5.843 Mean :3.057 Mean :3.758 lll l l l l ll 2.0 l l l 8 3rd Qu.:6.400 3rd Qu.:3.300 3rd Qu.:5.100 7 llll lll l llll 9 Max. :7.900 Max. :4.400 Max. :6.900 l llllll lllll ll llll lll llllllllllllll lll lllllllllll llll llllll llll 10 lll llllll l llllllll lll llllll 5 lllllllllllllllllllll l lllllllllll l llllllll lllllllllllllllllll ll l llllllll llllllllll llllllllllllllllllll ll llllllllllllllllll 11 Petal.Width Species lllllllllll llllllll llllllllllllllllll Petal.Length llllllllllllllllll l l ll l ll l l ll l lll lll lll 3 12 Min. :0.100 setosa :50 13 1st Qu.:0.300 versicolor:50 llllllllllllll lllllllll llll ll ll lll lllllll lllll l l lllll llllllll lllll lllllllll l l ll lllllllllllllllllllllllllllllll llllllllllllll llllllllll llllllllll 1 ll l l ll 14 Median :1.300 virginica :50 15 Mean :1.199 2.5 l ll ll l l lllll l lllll llllllll lll llllllllll l lllllllll ll ll lllllll lll l llllllllll l lllllllll lll 16 3rd Qu.:1.800 lllllllllll lll lll llllllll lllllllll lllllllllllllllll l lll l l ll ll ll lllll l 17 Max. :2.500 1.5 llllllllllllllllllll ll llllllllllllllllll lllllllllllllllllll l Petal.Width llllllllllllll llll lllllllllllll lllll lll lllllllllllllll 18 llll ll llll llllllllll llllllllll 19 # Draw a matrix of all variables ll ll ll 0.5 ll l llllll lllll l l lllllllllll l llllllllllllll lllll l llllllllllllllllllllll llllll l l llllllllllllllllllllllllllllllll llllllllllllllllllllllllllllllllll 20 > plot(iris[,1:4], col=iris$Species) 4.5 6.0 7.5 1 3 5 7 R plot of the iris data Julian M. Kunkel Lecture BigData Analytics, 2016 6 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Distribution of Values: Histograms [10] Distribution: frequency of outcomes (values) in a sample Example: Species in the Iris data set setosa: 50 versicolor: 50 virginica: 50 Histogram: graphical representation of the distribution Partition observed values into bins Count number of occurrences in each bin It is an estimate for the probability distribution R example 20 30 15 20 1 # nclass specifies the number of bins 10 2 # by default, hist uses equidistant bins Frequency Frequency 3 hist(iris$Petal.Length, nclass=10, main="") 10 5 4 5 hist(iris$Petal.Length, nclass=25, main="") 0 0 1 2 3 4 5 6 7 1 2 3 4 5 6 7 iris$Petal.Length iris$Petal.Length Histograms with 10 and 25 bins Julian M. Kunkel Lecture BigData Analytics, 2016 7 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Distribution of Values: Density [10] Probability density function (density): Likelihood for a continuous variable to take on a given value Kernel density estimation (KDE) approximates the density R example 1 # The kernel density estimator moves a function (kernel) in a window across samples 2 # With bw="SJ" or "nrd" it automatically determines the bandwidth, i.e., window size 3 d = density(iris$Petal.Length, bw="SJ", kernel="gaussian") 4 plot(d, main="") 0.5 0.4 0.3 Density 0.2 0.1 0.0 1 2 3 4 5 6 7 N = 150 Bandwidth = 0.192 Density estimation of Petal.Length Julian M. Kunkel Lecture BigData Analytics, 2016 8 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Distribution of Values: Quantiles [10] Percentile: value below which a given percentage of observations fall q-Quantiles: values that partition a ranked set into q equal sized subsets Quartiles: 3 data points that split a ranked set into 4 equal points Q1=P(25), Q2=median=P(50), Q3=P(75), interquartile range iqr=Q3-Q1 Five-number summary: (min, Q1, Q2, Q3, max) Boxplot: shows quartiles (Q1,Q2,Q3) and whiskers Whiskers extend to values up to 1.5 iqr from Q1 and Q3 Outliers are outside of whiskers R example 8 1 > boxplot(iris, range=1.5) # 1.5 interquartile range 2 > d = iris$Sepal.Width 3 > quantile(d) 6 4 0% 25% 50% 75% 100% 5 2.0 2.8 3.0 3.3 4.4 setosasetosa l 4 setosa ll 6 > q3 = quantile(d,0.75) # pick value below which are 75% 7 > q1 = quantile(d,0.25) 8 > irq = (q3 - q1) 2 versicolor l 9 # identify all outliers based on the interquartile range 10 > mask = d < (q1 - 1.5*irq) | d > (q3 + 1.5*irq) 0 11 # pick outlier selection from full data set Sepal.Length Sepal.Width Petal.Length Petal.Width Species 12 > o = iris[mask,] 13 # draw the species name of the outliers on the boxplot Boxplot 14 > text(rep(1.5,nrow(o)), o$Sepal.Width, o$Species, ↪→ col=as.numeric(o$Species)) Julian M. Kunkel Lecture BigData Analytics, 2016 9 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Density Plot Including Summary 1 d = density(iris$Petal.Length, bw="SJ", ↪→ kernel="gaussian") 2 3 # add space for two axes 4 par(mar=c(5, 4, 4, 6) + 0.1) 5 plot(d, main="") l l 1.0 ll l l 0.5 ll 6 # draw lines for Q1, Q2, Q3 ll l 7 q = quantile(iris$Petal.Length) l ll 8 q = c(q, mean(iris$Petal.Length)) ll 0.8 0.4 9 abline(v=q[1], lty=2, col="green", lwd=2) l l l 10 abline(v=q[2], lty=3, col="blue", lwd=2) l l 0.6 11 abline(v=q[3], lty=3, col="red", lwd=3) l 0.3 l 12 abline(v=q[4], lty=3, col="blue", lwd=2) ll Density l l 13 abline(v=q[5], lty=2, col="green", lwd=2) l 0.4 14 abline(v=q[6], lty=4, col="black", lwd=2) 0.2 l llll l l l 15 # Add titles l l 16 text(q, rep(-0.01, 5), c("min", "Q1", "median", 0.1 0.2 Cummulative distribution function l ↪→ "Q3", "max", "mean")) 17 l l 18 # identify x limits 0.0 minll Q1 mean median Q3 max 0.0 19 xlim = par("usr")[1:2] 1 2 3 4 5 6 7 20 par(new=TRUE) 21 N = 150 Bandwidth = 0.192 22 # Empirical cummulative distribution function 23 e = ecdf(iris$Petal.Length) Density estimation with 5-number summary and 24 plot(e, col="blue", axes=FALSE, xlim=xlim, ylab="", cumulative density function ↪→ xlab="", main="") 25 26 axis(4, ylim=c(0,1.0), col="blue") 27 mtext("Cummulative distribution function", side=4, ↪→ line=2.5) Julian M. Kunkel Lecture BigData Analytics, 2016 10 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Correlation Coefficients Measures (linear) correlation between two variables Result is between -1 and +1 >0.7: strong positive correlation >0.2: weak positive correlation 0: no correlation, < 0: negative correlation R example 1 library(corrplot) 2 d = iris 3 d$Species = as.numeric(d$Species) # It is normally not Sepal.LengthSepal.WidthPetal.LengthPetal.WidthSpecies ↪→ 1 adviced to convert categorial data to numeric 4 corrplot(cor(d), method = "circle") # the right plot Sepal.Length l 0.8 5 0.6 6 mplot = function(x,y, name){ 7 pdf(name,width=5,height=5) # plot into a PDF Sepal.Width l 0.4 8 p = cor(x,y, method="pearson") # compute correlation 9 k = cor(x,y, method="spearman") 0.2 10 plot(x,y, xlab=sprintf("x\n cor. coeff: %.2f rank coef.: Petal.Length 0 ↪→ %.2f", p, k)) 11 dev.off() −0.2 12 } Petal.Width −0.4 13 14 mplot(iris$Petal.Length, iris$Petal.Width, "iris-corr.pdf") −0.6 15 # cor. coeff: 0.96 rank coef.: 0.94 Species −0.8 16 17 x = 1:10; y = c(1,3,2,5,4,7,6,9,8,10) −1 18 mplot(x,y, "linear.pdf") # cor. coeff: 0.95 rank coef.: 0.95 19 Corrplot 20 mplot(x, x*x*x , "x3.pdf") # cor. coeff: 0.93 rank coef.: 1 Julian M. Kunkel Lecture BigData Analytics, 2016 11 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Example Correlations for Plots of Two Variables (X, Y) Correlations for x,y plots; Source: [22] Julian M. Kunkel Lecture BigData Analytics, 2016 12 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary 1 Descriptive Statistics 2 Distribution of Values 3 Inductive Statistics 4 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 13 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Distribution of Values Data exploration helps to understand distribution of values for a variable To illustrate data distribution, statistics uses: Probability density function (PDF): value and probability; P(x) Cumulative distribution function (CDF): sum from −∞ to value; P(X ≤ x) Many standard distributions exists that can be models for processes Discrete probability functions are drawn from a limited set of values Continuous probability functions are ∈ R When we explore data, we may identify these common distributions Discrete probability functions are derived from useful processes Julian M. Kunkel Lecture BigData Analytics, 2016 14 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary (Continuous) Uniform Distribution Notation: unif(a, b) Mean: a+b 2 (b−a)2 Variance: 12 (a) Source: IkamusumeFan, PDF of the (b) Source: IkamusumeFan, CDF of the uniform probability distribution (10) uniform probability distribution (10) Julian M. Kunkel Lecture BigData Analytics, 2016 15 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Normal Distribution Notation: N(μ, σ2) Mean: μ Variance: σ2 Source: Dan Kernler, For the normal distribution, the values less than one standard deviation away from the mean account for 68.27% of the set; while two standard deviations from the mean account for 95.45%; and three standard deviations account for 99.73%. [10] (a) Source: Inductiveload, Probability density (b) Source: Inductiveload, Cumulative function for the normal distribution (10) distribution function for the normal distribution (10) Julian M. Kunkel Lecture BigData Analytics, 2016 16 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Exponential Distribution Notation: Exponential(λ) Models inter-arrival time of λ (rate) > 0 a Poisson process Mean: λ−1 Memoryless, useful to Variance: λ−2 model failure rates (c) Source: Skbkekas, Probability density (d) Source: Skbkekas, Cumulative distribution function (10) (10) Julian M. Kunkel Lecture BigData Analytics, 2016 17 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Gamma Distribution Notation: Gamma(α, β) α (shape), β (rate) > 0 Mean: α β Variance: α β2 (e) MarkSweep and Cburnett, Probability (f) MarkSweep and Cburnett, Cumulative density plots of gamma distributions (10) distribution plots of gamma distributions (10) Julian M. Kunkel Lecture BigData Analytics, 2016 18 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Real Data Observations can be based on several (unknown) processes Fraction of people of a given age that died in 1999 Mark Bailin, Mortality by age [10] Julian M. Kunkel Lecture BigData Analytics, 2016 19 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Real Data (2) Performance of reading 1 MiB of data from two different storage systems Lustre parallel file systems vs. pooled memory of the XPD Julian M. Kunkel Lecture BigData Analytics, 2016 20 / 34
显示前 20 页。下载资料查看完整内容。
暂无评论。快来抢沙发吧！
评论 (0)
暂无评论。快来抢沙发吧！