Statistics: A Primer Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2017-11-27 Disclaimer: Big Data software is constantly updated, code samples may be outdated.
Descriptive Statistics Distribution of Values Inductive Statistics Summary Outline 1 Descriptive Statistics 2 Distribution of Values 3 Inductive Statistics 4 Summary Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 2 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Statistics: Overview Statistics is the study of the collection, analysis, interpretation, presentation, and organization of data [21] Either describe properties of a sample or infer properties of a population Important terms [10] Unit of observation: the entity described by the data, e.g., people Unit of analysis: the major entity that is being analyzed Example: Observe income of each person, analyse differences of countries Statistical population: Complete set of items that share at least one property that is subject of analysis Subpopulation share additional properties, e.g., gender of people Sample: (sub)set of data collected and/or selected from a population If chosen properly, they can represent the population There are many sampling methods, we can never capture ALL items Independence: one observation does not effect another Example: select two people living in Germany randomly Dependent: select one household and pick a married couple Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 3 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Statistics: Variables Dependent variable: represents the output/effect Example: word count of a Wikipedia article; income of people Independent variable: assumed input/cause/explanation Example: number of sentences; age, educational level Characterization Univariate analysis: characterize a single variable Bivariate analysis: describes/analyze relationships between two vars Multivariate statistics: analyze/observe multiple dependent variables Example: chemicals in the blood stream of people or chance for cancers, Independent variables are personal information/habits Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 4 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Descriptive Statistics [10] The discipline of quantitatively describing main features of sampled data Summarize observations/selected samples Exploratory data analysis (EDA): approach for inspecting data Using different chart types, e.g., box plots, histograms, scatter plot Methods for univariate analysis Distribution of values, e.g., mean, variance, quantiles Probability distribution and density t-test, e.g., check if data is t-distributed Methods for bivariate analysis Correlation coefficient1 describes linear relationship Rank correlation2: extent by which one variable increases with another var Methods for multivariate analysis Principal component analysis (PCA) converts correlated variables into linearly uncorrelated variables called principal components 1Pearson’s product-moment coefficient 2By Spearman or Kendall Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 5 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Example Dataset: Iris Flower Data Set Contains information about iris flower Three species: Iris Setosa, Iris Virginica, Iris Versicolor Data: Sepal.length, Sepal.width, Petal.length, Petal.width 2.0 3.0 4.0 0.5 1.5 2.5 l l l llll l l llll lllll llll l lllll llll l 7.5 ll l l l l R example l llllll lllllll ll lllll l lllllllll llllllllll llllll llll Sepal.Length llllllllllll lll lllllllll lll lllllllll lllllll ll llll lllllllll llll llllllllll lll ll lllllllll l ll lllllll ll l lll lllllllll 6.0 lllllllll l l llll lllll l lll l llll l llllllllllll l l ll llllllllllll ll llllllllll ll lllll l llllll l lllllll llllll llll lll l lllll l lll l llll l llll l l ll llllllllllllllll llllllllllllllll lll llllll llllllllll lll ll l lllll lll lllllllll l l lllllllll l l llllll llllll llllll 1 > data(iris) # load iris data l lll llll llll 4.5 l l l 2 > summary(iris) l l l 3 Sepal.Length Sepal.Width Petal.Length l l ll ll 4.0 ll l lll lll lllllll ll lllllll ll lllllll ll 4 Min. :4.300 Min. :2.000 Min. :1.000 lllllllll l lllllllll l llllllll l l l llllllllll lllll ll Sepal.Width lllllllllll ll lllll lllllllllll ll lllll 5 1st Qu.:5.100 1st Qu.:2.800 1st Qu.:1.600 lllllllll l llllllllllllll lllllllll lllllllllllllll lllllllll lllll llllllll ll 3.0lll llll llllllllllll lllllllllllllllll lllllll l llllllllllllll l lllllllllllll lllllll lllllllllllllllllllllllllllll l llllllllllll llllllll ll lllllllllllllll llllll l l lllllllllllllllllllllll 6 Median :5.800 Median :3.000 Median :4.350 ll lllllllll l l l lllllllll ll l llllllllllll l l ll lll l l llllll l llllll 7 Mean :5.843 Mean :3.057 Mean :3.758 lll l l l l ll 2.0 l l l 8 3rd Qu.:6.400 3rd Qu.:3.300 3rd Qu.:5.100 7 llll lll l llll 9 Max. :7.900 Max. :4.400 Max. :6.900 l llllll lllll ll llll lll llllllllllllll lll lllllllllll llll llllll llll 10 lll llllll l llllllll lll llllll 5 lllllllllllllllllllll l lllllllllll l llllllll lllllllllllllllllll ll l llllllll llllllllll llllllllllllllllllll ll llllllllllllllllll 11 Petal.Width Species lllllllllll llllllll llllllllllllllllll Petal.Length llllllllllllllllll l l ll l ll l l ll l lll lll lll 3 12 Min. :0.100 setosa :50 13 1st Qu.:0.300 versicolor:50 llllllllllllll lllllllll llll ll ll lll lllllll lllll l l lllll llllllll lllll lllllllll l l ll lllllllllllllllllllllllllllllll llllllllllllll llllllllll llllllllll 1 ll l l ll 14 Median :1.300 virginica :50 15 Mean :1.199 2.5 l ll ll l l lllll l lllll llllllll lll llllllllll l lllllllll ll ll lllllll lll l llllllllll l lllllllll lll 16 3rd Qu.:1.800 lllllllllll lll lll llllllll lllllllll lllllllllllllllll l lll l l ll ll ll lllll l 17 Max. :2.500 1.5 llllllllllllllllllll ll llllllllllllllllll lllllllllllllllllll l Petal.Width llllllllllllll llll lllllllllllll lllll lll lllllllllllllll 18 llll ll llll llllllllll llllllllll 19 # Draw a matrix of all variables ll ll ll 0.5 ll l llllll lllll l l lllllllllll l llllllllllllll lllll l llllllllllllllllllllll llllll l l llllllllllllllllllllllllllllllll llllllllllllllllllllllllllllllllll 20 > plot(iris[,1:4], col=iris$Species) 4.5 6.0 7.5 1 3 5 7 R plot of the iris data Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 6 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Distribution of Values: Histograms [10] Distribution: frequency of outcomes (values) in a sample Example: Species in the Iris data set setosa: 50 versicolor: 50 virginica: 50 Histogram: graphical representation of the distribution Partition observed values into bins Count number of occurrences in each bin It is an estimate for the probability distribution R example 20 30 15 20 1 # nclass specifies the number of bins 10 2 # by default, hist uses equidistant bins Frequency Frequency 3 hist(iris$Petal.Length, nclass=10, main="") 10 5 4 5 hist(iris$Petal.Length, nclass=25, main="") 0 0 1 2 3 4 5 6 7 1 2 3 4 5 6 7 iris$Petal.Length iris$Petal.Length Histograms with 10 and 25 bins Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 7 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Distribution of Values: Density [10] Probability density function (density): Likelihood for a continuous variable to take on a given value Kernel density estimation (KDE) approximates the density R example 1 # The kernel density estimator moves a function (kernel) in a window across samples 2 # With bw="SJ" or "nrd" it automatically determines the bandwidth, i.e., window size 3 d = density(iris$Petal.Length, bw="SJ", kernel="gaussian") 4 plot(d, main="") 0.5 0.4 0.3 Density 0.2 0.1 0.0 1 2 3 4 5 6 7 N = 150 Bandwidth = 0.192 Density estimation of Petal.Length Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 8 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Distribution of Values: Quantiles [10] Percentile: value below which a given percentage of observations fall q-Quantiles: values that partition a ranked set into q equal sized subsets Quartiles: 3 data points that split a ranked set into 4 equal points Q1=P(25), Q2=median=P(50), Q3=P(75), interquartile range iqr=Q3-Q1 Five-number summary: (min, Q1, Q2, Q3, max) Boxplot: shows quartiles (Q1,Q2,Q3) and whiskers Whiskers extend to values up to 1.5 iqr from Q1 and Q3 Outliers are outside of whiskers R example 8 1 > boxplot(iris, range=1.5) # 1.5 interquartile range 2 > d = iris$Sepal.Width 3 > quantile(d) 6 4 0% 25% 50% 75% 100% 5 2.0 2.8 3.0 3.3 4.4 setosasetosa l 4 setosa ll 6 > q3 = quantile(d,0.75) # pick value below which are 75% 7 > q1 = quantile(d,0.25) 8 > irq = (q3 - q1) 2 versicolor l 9 # identify all outliers based on the interquartile range 10 > mask = d < (q1 - 1.5*irq) | d > (q3 + 1.5*irq) 0 11 # pick outlier selection from full data set Sepal.Length Sepal.Width Petal.Length Petal.Width Species 12 > o = iris[mask,] 13 # draw the species name of the outliers on the boxplot Boxplot 14 > text(rep(1.5,nrow(o)), o$Sepal.Width, o$Species, ↪→ col=as.numeric(o$Species)) Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 9 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Density Plot Including Summary 1 d = density(iris$Petal.Length, bw="SJ", ↪→ kernel="gaussian") 2 3 # add space for two axes 4 par(mar=c(5, 4, 4, 6) + 0.1) 5 plot(d, main="") l l 1.0 ll l l 0.5 ll 6 # draw lines for Q1, Q2, Q3 ll l 7 q = quantile(iris$Petal.Length) l ll 8 q = c(q, mean(iris$Petal.Length)) ll 0.8 0.4 9 abline(v=q[1], lty=2, col="green", lwd=2) l l l 10 abline(v=q[2], lty=3, col="blue", lwd=2) l l 0.6 11 abline(v=q[3], lty=3, col="red", lwd=3) l 0.3 l 12 abline(v=q[4], lty=3, col="blue", lwd=2) ll Density l l 13 abline(v=q[5], lty=2, col="green", lwd=2) l 0.4 14 abline(v=q[6], lty=4, col="black", lwd=2) 0.2 l llll l l l 15 # Add titles l l 16 text(q, rep(-0.01, 5), c("min", "Q1", "median", 0.1 0.2 Cummulative distribution function l ↪→ "Q3", "max", "mean")) 17 l l 18 # identify x limits 0.0 minll Q1 mean median Q3 max 0.0 19 xlim = par("usr")[1:2] 1 2 3 4 5 6 7 20 par(new=TRUE) 21 N = 150 Bandwidth = 0.192 22 # Empirical cummulative distribution function 23 e = ecdf(iris$Petal.Length) Density estimation with 5-number summary and 24 plot(e, col="blue", axes=FALSE, xlim=xlim, ylab="", cumulative density function ↪→ xlab="", main="") 25 26 axis(4, ylim=c(0,1.0), col="blue") 27 mtext("Cummulative distribution function", side=4, ↪→ line=2.5) Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 10 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Correlation Coefficients Measures (linear) correlation between two variables Result is between -1 and +1 >0.7: strong positive correlation >0.2: weak positive correlation 0: no correlation, < 0: negative correlation R example 1 library(corrplot) 2 d = iris 3 d$Species = as.numeric(d$Species) # It is normally not Sepal.LengthSepal.WidthPetal.LengthPetal.WidthSpecies ↪→ 1 adviced to convert categorial data to numeric 4 corrplot(cor(d), method = "circle") # the right plot Sepal.Length l 0.8 5 0.6 6 mplot = function(x,y, name){ 7 pdf(name,width=5,height=5) # plot into a PDF Sepal.Width l 0.4 8 p = cor(x,y, method="pearson") # compute correlation 9 k = cor(x,y, method="spearman") 0.2 10 plot(x,y, xlab=sprintf("x\n cor. coeff: %.2f rank coef.: Petal.Length 0 ↪→ %.2f", p, k)) 11 dev.off() −0.2 12 } Petal.Width −0.4 13 14 mplot(iris$Petal.Length, iris$Petal.Width, "iris-corr.pdf") −0.6 15 # cor. coeff: 0.96 rank coef.: 0.94 Species −0.8 16 17 x = 1:10; y = c(1,3,2,5,4,7,6,9,8,10) −1 18 mplot(x,y, "linear.pdf") # cor. coeff: 0.95 rank coef.: 0.95 19 Corrplot 20 mplot(x, x*x*x , "x3.pdf") # cor. coeff: 0.93 rank coef.: 1 Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 11 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Example Correlations for Plots of Two Variables (X, Y) Correlations for x,y plots; Source: [22] Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 12 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary 1 Descriptive Statistics 2 Distribution of Values 3 Inductive Statistics 4 Summary Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 13 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Distribution of Values Data exploration helps to understand distribution of values for a variable To illustrate data distribution, statistics uses: Probability density function (PDF): value and probability; P(x) Cumulative distribution function (CDF): sum from −∞ to value; P(X ≤ x) Many standard distributions exists that can be models for processes Discrete probability functions are drawn from a limited set of values Continuous probability functions are ∈ R Discrete probability functions are derived from useful processes When we explore data, we may identify these common distributions Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 14 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary (Continuous) Uniform Distribution Notation: unif(a, b) Mean: a+b 2 (b−a)2 Variance: 12 (a) Source: IkamusumeFan, PDF of the (b) Source: IkamusumeFan, CDF of the uniform probability distribution (10) uniform probability distribution (10) Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 15 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Normal Distribution Notation: N(μ, σ2) Mean: μ Variance: σ2 Source: Dan Kernler, For the normal distribution, the values less than one standard deviation away from the mean account for 68.27% of the set; while two standard deviations from the mean account for 95.45%; and three standard deviations account for 99.73%. [10] (a) Source: Inductiveload, Probability density (b) Source: Inductiveload, Cumulative function for the normal distribution (10) distribution function for the normal distribution (10) Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 16 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Exponential Distribution Notation: Exponential(λ) Models inter-arrival time of λ (rate) > 0 a Poisson process Mean: λ−1 Memoryless, useful to Variance: λ−2 model failure rates (c) Source: Skbkekas, Probability density (d) Source: Skbkekas, Cumulative distribution function (10) (10) Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 17 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Gamma Distribution Notation: Gamma(k, Θ) k (shape), Θ (scale) > 0; sometimes rate b = 1 Θ Mean: k · Θ Variance: k · Θ2 (e) MarkSweep and Cburnett, Probability (f) MarkSweep and Cburnett, Cumulative density plots of gamma distributions (10) distribution plots of gamma distributions (10) Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 18 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Real Data Observations can be based on several (unknown) processes Fraction of people of a given age that died in 1999 Mark Bailin, Mortality by age [10] Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 19 / 34
Descriptive Statistics Distribution of Values Inductive Statistics Summary Real Data (2) Performance of reading 1 MiB of data from two different storage systems Lustre parallel file systems vs. pooled memory of the XPD Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 20 / 34
Hiển thị 20 trang đầu. Tải tài liệu để xem đầy đủ.
Chưa có bình luận nào. Hãy là người đầu tiên!
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!