第 3 页
Think Stats Exploratory Data Analysis in Python Version 2.0.35 Allen B. Downey Green Tea Press Needham, Massachusetts
资料内容
第 4 页
Copyright ©c 2014 Allen B. Downey. Green Tea Press 9 Washburn Ave Needham MA 02492 Permission is granted to copy, distribute, and/or modify this document under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 Inter- national License, which is available at http://creativecommons.org/licenses/ by-nc-sa/4.0/. The original form of this book is LATEX source code. Compiling this code has the effect of generating a device-independent representation of a textbook, which can be converted to other formats and printed. The LATEX source for this book is available from http://thinkstats2.com.
第 5 页
Preface This book is an introduction to the practical tools of exploratory data anal- ysis. The organization of the book follows the process I use when I start working with a dataset: • Importing and cleaning: Whatever format the data is in, it usually takes some time and effort to read the data, clean and transform it, and check that everything made it through the translation process intact. • Single variable explorations: I usually start by examining one variable at a time, finding out what the variables mean, looking at distributions of the values, and choosing appropriate summary statistics. • Pair-wise explorations: To identify possible relationships between vari- ables, I look at tables and scatter plots, and compute correlations and linear fits. • Multivariate analysis: If there are apparent relationships between vari- ables, I use multiple regression to add control variables and investigate more complex relationships. • Estimation and hypothesis testing: When reporting statistical results, it is important to answer three questions: How big is the effect? How much variability should we expect if we run the same measurement again? Is it possible that the apparent effect is due to chance? • Visualization: During exploration, visualization is an important tool for finding possible relationships and effects. Then if an apparent effect holds up to scrutiny, visualization is an effective way to communicate results.
第 6 页
vi Chapter 0. Preface This book takes a computational approach, which has several advantages over mathematical approaches: • I present most ideas using Python code, rather than mathematical notation. In general, Python code is more readable; also, because it is executable, readers can download it, run it, and modify it. • Each chapter includes exercises readers can do to develop and solidify their learning. When you write programs, you express your under- standing in code; while you are debugging the program, you are also correcting your understanding. • Some exercises involve experiments to test statistical behavior. For example, you can explore the Central Limit Theorem (CLT) by gener- ating random samples and computing their sums. The resulting visu- alizations demonstrate why the CLT works and when it doesn’t. • Some ideas that are hard to grasp mathematically are easy to under- stand by simulation. For example, we approximate p-values by running random simulations, which reinforces the meaning of the p-value. • Because the book is based on a general-purpose programming language (Python), readers can import data from almost any source. They are not limited to datasets that have been cleaned and formatted for a particular statistics tool. The book lends itself to a project-based approach. In my class, students work on a semester-long project that requires them to pose a statistical question, find a dataset that can address it, and apply each of the techniques they learn to their own data. To demonstrate my approach to statistical analysis, the book presents a case study that runs through all of the chapters. It uses data from two sources: • The National Survey of Family Growth (NSFG), conducted by the U.S. Centers for Disease Control and Prevention (CDC) to gather “information on family life, marriage and divorce, pregnancy, infer- tility, use of contraception, and men’s and women’s health.” (See http://cdc.gov/nchs/nsfg.htm.)
第 7 页
0.1. How I wrote this book vii • The Behavioral Risk Factor Surveillance System (BRFSS), conducted by the National Center for Chronic Disease Prevention and Health Promotion to “track health conditions and risk behaviors in the United States.” (See http://cdc.gov/BRFSS/.) Other examples use data from the IRS, the U.S. Census, and the Boston Marathon. This second edition of Think Stats includes the chapters from the first edition, many of them substantially revised, and new chapters on regression, time series analysis, survival analysis, and analytic methods. The previous edition did not use pandas, SciPy, or StatsModels, so all of that material is new. 0.1 How I wrote this book When people write a new textbook, they usually start by reading a stack of old textbooks. As a result, most books contain the same material in pretty much the same order. I did not do that. In fact, I used almost no printed material while I was writing this book, for several reasons: • My goal was to explore a new approach to this material, so I didn’t want much exposure to existing approaches. • Since I am making this book available under a free license, I wanted to make sure that no part of it was encumbered by copyright restrictions. • Many readers of my books don’t have access to libraries of printed ma- terial, so I tried to make references to resources that are freely available on the Internet. • Some proponents of old media think that the exclusive use of electronic resources is lazy and unreliable. They might be right about the first part, but I think they are wrong about the second, so I wanted to test my theory.
第 8 页
viii Chapter 0. Preface The resource I used more than any other is Wikipedia. In general, the arti- cles I read on statistical topics were very good (although I made a few small changes along the way). I include references to Wikipedia pages through- out the book and I encourage you to follow those links; in many cases, the Wikipedia page picks up where my description leaves off. The vocabulary and notation in this book are generally consistent with Wikipedia, unless I had a good reason to deviate. Other resources I found useful were Wol- fram MathWorld and the Reddit statistics forum, http://www.reddit.com/ r/statistics. 0.2 Using the code The code and data used in this book are available from https://github. com/AllenDowney/ThinkStats2. Git is a version control system that allows you to keep track of the files that make up a project. A collection of files under Git’s control is called a repository. GitHub is a hosting service that provides storage for Git repositories and a convenient web interface. The GitHub homepage for my repository provides several ways to work with the code: • You can create a copy of my repository on GitHub by pressing the Fork button. If you don’t already have a GitHub account, you’ll need to create one. After forking, you’ll have your own repository on GitHub that you can use to keep track of code you write while working on this book. Then you can clone the repo, which means that you make a copy of the files on your computer. • Or you could clone my repository. You don’t need a GitHub account to do this, but you won’t be able to write your changes back to GitHub. • If you don’t want to use Git at all, you can download the files in a Zip file using the button in the lower-right corner of the GitHub page. All of the code is written to work in both Python 2 and Python 3 with no translation. I developed this book using Anaconda from Continuum Analytics, which is a free Python distribution that includes all the packages you’ll need to run the
第 9 页
0.2. Using the code ix code (and lots more). I found Anaconda easy to install. By default it does a user-level installation, not system-level, so you don’t need administrative privileges. And it supports both Python 2 and Python 3. You can download Anaconda from http://continuum.io/downloads. If you don’t want to use Anaconda, you will need the following packages: • pandas for representing and analyzing data, http://pandas.pydata. org/; • NumPy for basic numerical computation, http://www.numpy.org/; • SciPy for scientific computation including statistics, http://www. scipy.org/; • StatsModels for regression and other statistical analysis, http:// statsmodels.sourceforge.net/; and • matplotlib for visualization, http://matplotlib.org/. Although these are commonly used packages, they are not included with all Python installations, and they can be hard to install in some environments. If you have trouble installing them, I strongly recommend using Anaconda or one of the other Python distributions that include these packages. After you clone the repository or unzip the zip file, you should have a folder called ThinkStats2/code with a file called nsfg.py. If you run nsfg.py, it should read a data file, run some tests, and print a message like, “All tests passed.” If you get import errors, it probably means there are packages you need to install. Most exercises use Python scripts, but some also use the IPython notebook. If you have not used IPython notebook before, I suggest you start with the documentation at http://ipython.org/ipython-doc/stable/notebook/ notebook.html. I wrote this book assuming that the reader is familiar with core Python, including object-oriented features, but not pandas, NumPy, and SciPy. If you are already familiar with these modules, you can skip a few sections.
第 10 页
x Chapter 0. Preface I assume that the reader knows basic mathematics, including logarithms, for example, and summations. I refer to calculus concepts in a few places, but you don’t have to do any calculus. If you have never studied statistics, I think this book is a good place to start. And if you have taken a traditional statistics class, I hope this book will help repair the damage. — Allen B. Downey is a Professor of Computer Science at the Franklin W. Olin College of Engineering in Needham, MA. Contributor List If you have a suggestion or correction, please send email to downey@allendowney.com. If I make a change based on your feedback, I will add you to the contributor list (unless you ask to be omitted). If you include at least part of the sentence the error appears in, that makes it easy for me to search. Page and section numbers are fine, too, but not quite as easy to work with. Thanks! • Lisa Downey and June Downey read an early draft and made many correc- tions and suggestions. • Steven Zhang found several errors. • Andy Pethan and Molly Farison helped debug some of the solutions, and Molly spotted several typos. • Dr. Nikolas Akerblom knows how big a Hyracotherium is. • Alex Morrow clarified one of the code examples. • Jonathan Street caught an error in the nick of time. • Many thanks to Kevin Smith and Tim Arnold for their work on plasTeX, which I used to convert this book to DocBook. • George Caplan sent several suggestions for improving clarity.
第 11 页
0.2. Using the code xi • Julian Ceipek found an error and a number of typos. • Stijn Debrouwere, Leo Marihart III, Jonathan Hammler, and Kent Johnson found errors in the first print edition. • J¨org Beyer found typos in the book and made many corrections in the doc- strings of the accompanying code. • Tommie Gannert sent a patch file with a number of corrections. • Christoph Lendenmann submitted several errata. • Michael Kearney sent me many excellent suggestions. • Alex Birch made a number of helpful suggestions. • Lindsey Vanderlyn, Griffin Tschurwald, and Ben Small read an early version of this book and found many errors. • John Roth, Carol Willing, and Carol Novitsky performed technical reviews of the book. They found many errors and made many helpful suggestions. • David Palmer sent many helpful suggestions and corrections. • Erik Kulyk found many typos. • Nir Soffer sent several excellent pull requests for both the book and the supporting code. • GitHub user flothesof sent a number of corrections. • Toshiaki Kurokawa, who is working on the Japanese translation of this book, has sent many corrections and helpful suggestions. • Benjamin White suggested more idiomatic Pandas code. • Takashi Sato spotted an code error. Other people who found typos and similar errors are Andrew Heine, G´abor Lipt´ak, Dan Kearney, Alexander Gryzlov, Martin Veillette, Haitao Ma, Jeff Pickhardt, Rohit Deshpande, Joanne Pratt, Lucian Ursu, Paul Glezen, Ting-kuang Lin.
第 13 页
Contents Preface v 0.1 How I wrote this book . . . . . . . . . . . . . . . . . . . . . vii 0.2 Using the code . . . . . . . . . . . . . . . . . . . . . . . . . . viii 1 Exploratory data analysis 1 1.1 A statistical approach . . . . . . . . . . . . . . . . . . . . . . 2 1.2 The National Survey of Family Growth . . . . . . . . . . . . 3 1.3 Importing the data . . . . . . . . . . . . . . . . . . . . . . . 4 1.4 DataFrames . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.5 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.6 Transformation . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.7 Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.8 Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.10 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
第 14 页
xiv Contents 2 Distributions 17 2.1 Histograms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.2 Representing histograms . . . . . . . . . . . . . . . . . . . . 18 2.3 Plotting histograms . . . . . . . . . . . . . . . . . . . . . . . 19 2.4 NSFG variables . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.5 Outliers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.6 First babies . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.7 Summarizing distributions . . . . . . . . . . . . . . . . . . . 25 2.8 Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.9 Effect size . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.10 Reporting results . . . . . . . . . . . . . . . . . . . . . . . . 28 2.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.12 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3 Probability mass functions 31 3.1 Pmfs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.2 Plotting PMFs . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.3 Other visualizations . . . . . . . . . . . . . . . . . . . . . . . 35 3.4 The class size paradox . . . . . . . . . . . . . . . . . . . . . 35 3.5 DataFrame indexing . . . . . . . . . . . . . . . . . . . . . . 39 3.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.7 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
第 15 页
Contents xv 4 Cumulative distribution functions 45 4.1 The limits of PMFs . . . . . . . . . . . . . . . . . . . . . . . 45 4.2 Percentiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.3 CDFs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 4.4 Representing CDFs . . . . . . . . . . . . . . . . . . . . . . . 49 4.5 Comparing CDFs . . . . . . . . . . . . . . . . . . . . . . . . 50 4.6 Percentile-based statistics . . . . . . . . . . . . . . . . . . . . 51 4.7 Random numbers . . . . . . . . . . . . . . . . . . . . . . . . 52 4.8 Comparing percentile ranks . . . . . . . . . . . . . . . . . . 54 4.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 4.10 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 5 Modeling distributions 57 5.1 The exponential distribution . . . . . . . . . . . . . . . . . . 57 5.2 The normal distribution . . . . . . . . . . . . . . . . . . . . 60 5.3 Normal probability plot . . . . . . . . . . . . . . . . . . . . . 62 5.4 The lognormal distribution . . . . . . . . . . . . . . . . . . . 65 5.5 The Pareto distribution . . . . . . . . . . . . . . . . . . . . . 67 5.6 Generating random numbers . . . . . . . . . . . . . . . . . . 69 5.7 Why model? . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 5.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 5.9 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
第 16 页
xvi Contents 6 Probability density functions 75 6.1 PDFs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 6.2 Kernel density estimation . . . . . . . . . . . . . . . . . . . . 77 6.3 The distribution framework . . . . . . . . . . . . . . . . . . 79 6.4 Hist implementation . . . . . . . . . . . . . . . . . . . . . . 80 6.5 Pmf implementation . . . . . . . . . . . . . . . . . . . . . . 81 6.6 Cdf implementation . . . . . . . . . . . . . . . . . . . . . . . 82 6.7 Moments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 6.8 Skewness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 6.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 6.10 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 7 Relationships between variables 91 7.1 Scatter plots . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 7.2 Characterizing relationships . . . . . . . . . . . . . . . . . . 95 7.3 Correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 7.4 Covariance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 7.5 Pearson’s correlation . . . . . . . . . . . . . . . . . . . . . . 98 7.6 Nonlinear relationships . . . . . . . . . . . . . . . . . . . . . 100 7.7 Spearman’s rank correlation . . . . . . . . . . . . . . . . . . 101 7.8 Correlation and causation . . . . . . . . . . . . . . . . . . . 102 7.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 7.10 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
第 17 页
Contents xvii 8 Estimation 105 8.1 The estimation game . . . . . . . . . . . . . . . . . . . . . . 105 8.2 Guess the variance . . . . . . . . . . . . . . . . . . . . . . . 107 8.3 Sampling distributions . . . . . . . . . . . . . . . . . . . . . 109 8.4 Sampling bias . . . . . . . . . . . . . . . . . . . . . . . . . . 112 8.5 Exponential distributions . . . . . . . . . . . . . . . . . . . . 113 8.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 8.7 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 9 Hypothesis testing 117 9.1 Classical hypothesis testing . . . . . . . . . . . . . . . . . . . 117 9.2 HypothesisTest . . . . . . . . . . . . . . . . . . . . . . . . . 119 9.3 Testing a difference in means . . . . . . . . . . . . . . . . . . 121 9.4 Other test statistics . . . . . . . . . . . . . . . . . . . . . . . 123 9.5 Testing a correlation . . . . . . . . . . . . . . . . . . . . . . 124 9.6 Testing proportions . . . . . . . . . . . . . . . . . . . . . . . 125 9.7 Chi-squared tests . . . . . . . . . . . . . . . . . . . . . . . . 127 9.8 First babies again . . . . . . . . . . . . . . . . . . . . . . . . 128 9.9 Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 9.10 Power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 9.11 Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 9.12 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 9.13 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
第 18 页
xviii Contents 10 Linear least squares 137 10.1 Least squares fit . . . . . . . . . . . . . . . . . . . . . . . . . 137 10.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . 139 10.3 Residuals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 10.4 Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 10.5 Goodness of fit . . . . . . . . . . . . . . . . . . . . . . . . . 144 10.6 Testing a linear model . . . . . . . . . . . . . . . . . . . . . 146 10.7 Weighted resampling . . . . . . . . . . . . . . . . . . . . . . 148 10.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 10.9 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 11 Regression 153 11.1 StatsModels . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 11.2 Multiple regression . . . . . . . . . . . . . . . . . . . . . . . 156 11.3 Nonlinear relationships . . . . . . . . . . . . . . . . . . . . . 158 11.4 Data mining . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 11.5 Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 11.6 Logistic regression . . . . . . . . . . . . . . . . . . . . . . . . 163 11.7 Estimating parameters . . . . . . . . . . . . . . . . . . . . . 165 11.8 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . 166 11.9 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 11.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 11.11 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
第 19 页
Contents xix 12 Time series analysis 173 12.1 Importing and cleaning . . . . . . . . . . . . . . . . . . . . . 174 12.2 Plotting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 12.3 Linear regression . . . . . . . . . . . . . . . . . . . . . . . . 178 12.4 Moving averages . . . . . . . . . . . . . . . . . . . . . . . . . 180 12.5 Missing values . . . . . . . . . . . . . . . . . . . . . . . . . . 182 12.6 Serial correlation . . . . . . . . . . . . . . . . . . . . . . . . 183 12.7 Autocorrelation . . . . . . . . . . . . . . . . . . . . . . . . . 185 12.8 Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 12.9 Further reading . . . . . . . . . . . . . . . . . . . . . . . . . 192 12.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 12.11 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 13 Survival analysis 195 13.1 Survival curves . . . . . . . . . . . . . . . . . . . . . . . . . 195 13.2 Hazard function . . . . . . . . . . . . . . . . . . . . . . . . . 198 13.3 Inferring survival curves . . . . . . . . . . . . . . . . . . . . 199 13.4 Kaplan-Meier estimation . . . . . . . . . . . . . . . . . . . . 200 13.5 The marriage curve . . . . . . . . . . . . . . . . . . . . . . . 202 13.6 Estimating the survival curve . . . . . . . . . . . . . . . . . 203 13.7 Confidence intervals . . . . . . . . . . . . . . . . . . . . . . . 204 13.8 Cohort effects . . . . . . . . . . . . . . . . . . . . . . . . . . 206 13.9 Extrapolation . . . . . . . . . . . . . . . . . . . . . . . . . . 209 13.10 Expected remaining lifetime . . . . . . . . . . . . . . . . . . 210 13.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 13.12 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
第 20 页
xx Contents 14 Analytic methods 217 14.1 Normal distributions . . . . . . . . . . . . . . . . . . . . . . 217 14.2 Sampling distributions . . . . . . . . . . . . . . . . . . . . . 219 14.3 Representing normal distributions . . . . . . . . . . . . . . . 220 14.4 Central limit theorem . . . . . . . . . . . . . . . . . . . . . . 221 14.5 Testing the CLT . . . . . . . . . . . . . . . . . . . . . . . . . 222 14.6 Applying the CLT . . . . . . . . . . . . . . . . . . . . . . . . 227 14.7 Correlation test . . . . . . . . . . . . . . . . . . . . . . . . . 228 14.8 Chi-squared test . . . . . . . . . . . . . . . . . . . . . . . . . 230 14.9 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 14.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
显示前 20 页。下载资料查看完整内容。
评论 (0)
暂无评论。快来抢沙发吧！