Page 2
The Art of Data Science A Guide for Anyone Who Works with Data Roger D. Peng and Elizabeth Matsui This book is for sale at http://leanpub.com/artofdatascience This version was published on 2016-07-20 This is a Leanpub book. Leanpub empowers authors and publishers with the Lean Publishing process. Lean Publishing is the act of publishing an in-progress ebook using lightweight tools and many iterations to get reader feedback, pivot until you have the right book and build traction once you do. © 2015 - 2016 Skybrude Consulting, LLC
Page 3
Also By Roger D. Peng R Programming for Data Science Exploratory Data Analysis with R Executive Data Science Report Writing for Data Science in R Conversations On Data Science
Page 5
Contents 1. Data Analysis as Art . . . . . . . . . . . . . . . . . 1 2. Epicycles of Analysis . . . . . . . . . . . . . . . . . 4 2.1 Setting the Scene . . . . . . . . . . . . . . . . 5 2.2 Epicycle of Analysis . . . . . . . . . . . . . . 6 2.3 Setting Expectations . . . . . . . . . . . . . . 8 2.4 Collecting Information . . . . . . . . . . . . 9 2.5 Comparing Expectations to Data . . . . . . 10 2.6 Applying the Epicyle of Analysis Process . . 11 3. Stating and Refining the Question . . . . . . . . 16 3.1 Types of Questions . . . . . . . . . . . . . . . 16 3.2 Applying the Epicycle to Stating and Refin- ing Your Question . . . . . . . . . . . . . . . 20 3.3 Characteristics of a Good Question . . . . . 20 3.4 Translating a Question into a Data Problem 23 3.5 Case Study . . . . . . . . . . . . . . . . . . . . 26 3.6 Concluding Thoughts . . . . . . . . . . . . . 30 4. Exploratory Data Analysis . . . . . . . . . . . . . 31 4.1 Exploratory Data Analysis Checklist: A Case Study . . . . . . . . . . . . . . . . . . . . . . . 33 4.2 Formulate your question . . . . . . . . . . . 33 4.3 Read in your data . . . . . . . . . . . . . . . . 35 4.4 Check the Packaging . . . . . . . . . . . . . . 36 4.5 Look at the Top and the Bottom of your Data 39
Page 6
CONTENTS 4.6 ABC: Always be Checking Your “n”s . . . . . 40 4.7 Validate With at Least One External Data Source . . . . . . . . . . . . . . . . . . . . . . 45 4.8 Make a Plot . . . . . . . . . . . . . . . . . . . 46 4.9 Try the Easy Solution First . . . . . . . . . . 49 4.10 Follow-up Questions . . . . . . . . . . . . . . 53 5. Using Models to Explore Your Data . . . . . . . 55 5.1 Models as Expectations . . . . . . . . . . . . 57 5.2 Comparing Model Expectations to Reality . 60 5.3 Reacting to Data: Refining Our Expectations 64 5.4 Examining Linear Relationships . . . . . . . 67 5.5 When Do We Stop? . . . . . . . . . . . . . . . 73 5.6 Summary . . . . . . . . . . . . . . . . . . . . . 77 6. Inference: A Primer . . . . . . . . . . . . . . . . . 78 6.1 Identify the population . . . . . . . . . . . . 78 6.2 Describe the sampling process . . . . . . . . 79 6.3 Describe a model for the population . . . . . 79 6.4 A Quick Example . . . . . . . . . . . . . . . . 80 6.5 Factors Affecting the Quality of Inference . 84 6.6 Example: Apple Music Usage . . . . . . . . . 86 6.7 Populations Come in Many Forms . . . . . . 89 7. Formal Modeling . . . . . . . . . . . . . . . . . . . 92 7.1 What Are the Goals of Formal Modeling? . 92 7.2 General Framework . . . . . . . . . . . . . . 93 7.3 Associational Analyses . . . . . . . . . . . . . 95 7.4 Prediction Analyses . . . . . . . . . . . . . . . 104 7.5 Summary . . . . . . . . . . . . . . . . . . . . . 111 8. Inference vs. Prediction: Implications for Mod- eling Strategy . . . . . . . . . . . . . . . . . . . . . 112 8.1 Air Pollution and Mortality in New York City113 8.2 Inferring an Association . . . . . . . . . . . . 115 8.3 Predicting the Outcome . . . . . . . . . . . . 121
Page 7
CONTENTS 8.4 Summary . . . . . . . . . . . . . . . . . . . . . 123 9. Interpreting Your Results . . . . . . . . . . . . . . 124 9.1 Principles of Interpretation . . . . . . . . . . 124 9.2 Case Study: Non-diet Soda Consumption and Body Mass Index . . . . . . . . . . . . . 125 10. Communication . . . . . . . . . . . . . . . . . . . 144 10.1 Routine communication . . . . . . . . . . . . 144 10.2 The Audience . . . . . . . . . . . . . . . . . . 146 10.3 Content . . . . . . . . . . . . . . . . . . . . . . 148 10.4 Style . . . . . . . . . . . . . . . . . . . . . . . . 151 10.5 Attitude . . . . . . . . . . . . . . . . . . . . . . 151 11. Concluding Thoughts . . . . . . . . . . . . . . . . 153 12. About the Authors . . . . . . . . . . . . . . . . . . 155
Page 8
1. Data Analysis as Art Data analysis is hard, and part of the problem is that few people can explain how to do it. It’s not that there aren’t any people doing data analysis on a regular basis. It’s that the people who are really good at it have yet to enlighten us about the thought process that goes on in their heads. Imagine you were to ask a songwriter how she writes her songs. There are many tools upon which she can draw. We have a general understanding of how a good song should be structured: how long it should be, how many verses, maybe there’s a verse followed by a chorus, etc. In other words, there’s an abstract framework for songs in general. Similarly, we have music theory that tells us that certain combinations of notes and chords work well together and other combinations don’t sound good. As good as these tools might be, ultimately, knowledge of song structure and music theory alone doesn’t make for a good song. Some- thing else is needed. In Donald Knuth’s legendary 1974 essay Computer Program- ming as an Art1, Knuth talks about the difference between art and science. In that essay, he was trying to get across the idea that although computer programming involved complex machines and very technical knowledge, the act of writing a computer program had an artistic component. In this essay, he says that Science is knowledge which we understand so well that we can teach it to a computer. 1http://www.paulgraham.com/knuth.html 1
Page 9
Data Analysis as Art 2 Everything else is art. At some point, the songwriter must inject a creative spark into the process to bring all the songwriting tools together to make something that people want to listen to. This is a key part of the art of songwriting. That creative spark is difficult to describe, much less write down, but it’s clearly essential to writing good songs. If it weren’t, then we’d have computer programs regularly writing hit songs. For better or for worse, that hasn’t happened yet. Much like songwriting (and computer programming, for that matter), it’s important to realize that data analysis is an art. It is not something yet that we can teach to a computer. Data analysts have many tools at their disposal, from linear regression to classification trees and even deep learning, and these tools have all been carefully taught to computers. But ultimately, a data analyst must find a way to assemble all of the tools and apply them to data to answer a relevant question—a question of interest to people. Unfortunately, the process of data analysis is not one that we have been able to write down effectively. It’s true that there are many statistics textbooks out there, many lining our own shelves. But in our opinion, none of these really addresses the core problems involved in conducting real- world data analyses. In 1991, Daryl Pregibon, a promi- nent statistician previously of AT&T Research and now of Google, said in reference to the process of data analysis2 that “statisticians have a process that they espouse but do not fully understand”. Describing data analysis presents a difficult conundrum. On the one hand, developing a useful framework involves characterizing the elements of a data analysis using abstract 2http://www.nap.edu/catalog/1910/the-future-of-statistical-software- proceedings-of-a-forum
Page 10
Data Analysis as Art 3 language in order to find the commonalities across different kinds of analyses. Sometimes, this language is the language of mathematics. On the other hand, it is often the very details of an analysis that makes each one so difficult and yet interesting. How can one effectively generalize across many different data analyses, each of which has important unique aspects? What we have set out to do in this book is to write down the process of data analysis. What we describe is not a specific “formula” for data analysis—something like “ap- ply this method and then run that test”— but rather is a general process that can be applied in a variety of situ- ations. Through our extensive experience both managing data analysts and conducting our own data analyses, we have carefully observed what produces coherent results and what fails to produce useful insights into data. Our goal is to write down what we have learned in the hopes that others may find it useful.
Page 11
2. Epicycles of Analysis To the uninitiated, a data analysis may appear to follow a linear, one-step-after-the-other process which at the end, arrives at a nicely packaged and coherent result. In reality, data analysis is a highly iterative and non-linear process, better reflected by a series of epicycles (see Figure), in which information is learned at each step, which then informs whether (and how) to refine, and redo, the step that was just performed, or whether (and how) to proceed to the next step. An epicycle is a small circle whose center moves around the circumference of a larger circle. In data analysis, the iterative process that is applied to all steps of the data analysis can be conceived of as an epicycle that is repeated for each step along the circumference of the entire data analysis process. Some data analyses appear to be fixed and linear, such as algorithms embedded into various software platforms, including apps. However, these algorithms are final data analysis products that have emerged from the very non-linear work of developing and refining a data analysis so that it can be “algorithmized.” 4
Page 12
Epicycles of Analysis 5 Epicycles of Analysis 2.1 Setting the Scene Before diving into the “epicycle of analysis,” it’s helpful to pause and consider what we mean by a “data analy- sis.” Although many of the concepts we will discuss in this
Page 13
Epicycles of Analysis 6 book are applicable to conducting a study, the framework and concepts in this, and subsequent, chapters are tailored specifically to conducting a data analysis. While a study in- cludes developing and executing a plan for collecting data, a data analysis presumes the data have already been collected. More specifically, a study includes the development of a hypothesis or question, the designing of the data collection process (or study protocol), the collection of the data, and the analysis and interpretation of the data. Because a data analysis presumes that the data have already been collected, it includes development and refinement of a question and the process of analyzing and interpreting the data. It is important to note that although a data analysis is often performed without conducting a study, it may also be per- formed as a component of a study. 2.2 Epicycle of Analysis There are 5 core activities of data analysis: 1. Stating and refining the question 2. Exploring the data 3. Building formal statistical models 4. Interpreting the results 5. Communicating the results These 5 activities can occur at different time scales: for example, you might go through all 5 in the course of a day, but also deal with each, for a large project, over the course of many months. Before discussing these core activities, which will occur in later chapters, it will be important to first understand the overall framework used to approach each of these activities.
Page 14
Epicycles of Analysis 7 Although there are many different types of activities that you might engage in while doing data analysis, every aspect of the entire process can be approached through an intera- tive process that we call the “epicycle of data analysis”. More specifically, for each of the five core activities, it is critical that you engage in the following steps: 1. Setting Expectations, 2. Collecting information (data), comparing the data to your expectations, and if the expectations don’t match, 3. Revising your expectations or fixing the data so your data and your expectations match. Iterating through this 3-step process is what we call the “epicycle of data analysis.” As you go through every stage of an analysis, you will need to go through the epicycle to continuously refine your question, your exploratory data analysis, your formal models, your interpretation, and your communication. The repeated cycling through each of these five core activi- ties that is done to complete a data analysis forms the larger circle of data analysis (See Figure). In this chapter we go into detail about what this 3-step epicyclic process is and give examples of how you can apply it to your data analysis.
Page 15
Epicycles of Analysis 8 Epicycles of Analysis 2.3 Setting Expectations Developing expectations is the process of deliberately think- ing about what you expect before you do anything, such as inspect your data, perform a procedure, or enter a com- mand. For experienced data analysts, in some circumstances, developing expectations may be an automatic, almost sub- conscious process, but it’s an important activity to cultivate and be deliberate about. For example, you may be going out to dinner with friends at a cash-only establishment and need to stop by the ATM to withdraw money before meeting up. To make a decision about the amount of money you’re going to withdraw, you have to have developed some expectation of the cost of dinner. This may be an automatic expectation because you dine at this establishment regularly so you know what the
Page 16
Epicycles of Analysis 9 typical cost of a meal is there, which would be an example of a priori knowledge. Another example of a priori knowledge would be knowing what a typical meal costs at a restaurant in your city, or knowing what a meal at the most expensive restaurants in your city costs. Using that information, you could perhaps place an upper and lower bound on how much the meal will cost. You may have also sought out external information to de- velop your expectations, which could include asking your friends who will be joining you or who have eaten at the restaurant before and/or Googling the restaurant to find general cost information online or a menu with prices. This same process, in which you use any a priori information you have and/or external sources to determine what you expect when you inspect your data or execute an analysis procedure, applies to each core activity of the data analysis process. 2.4 Collecting Information This step entails collecting information about your question or your data. For your question, you collect information by performing a literature search or asking experts in order to ensure that your question is a good one. In the next chapter, we will discuss characteristics of a good question. For your data, after you have some expectations about what the result will be when you inspect your data or perform the analysis procedure, you then perform the operation. The results of that operation are the data you need to collect, and then you determine if the data you collected matches your expectations. To extend the restaurant metaphor, when you go to the restaurant, getting the check is collecting the data.
Page 17
Epicycles of Analysis 10 2.5 Comparing Expectations to Data Now that you have data in hand (the check at the restaurant), the next step is to compare your expectations to the data. There are two possible outcomes: either your expectations of the cost matches the amount on the check, or they do not. If your expectations and the data match, terrific, you can move onto the next activity. If, on the other hand, your expectations were a cost of 30 dollars, but the check was 40 dollars, your expectations and the data do not match. There are two possible explanations for the discordance: first, your expectations were wrong and need to be revised, or second, the check was wrong and contains an error. You review the check and find that you were charged for two desserts instead of the one that you had, and conclude that there is an error in the data, so ask for the check to be corrected. One key indicator of how well your data analysis is going is how easy or difficult it is to match the data you collected to your original expectations. You want to setup your expec- tations and your data so that matching the two up is easy. In the restaurant example, your expectation was $30 and the data said the meal cost $40, so it’s easy to see that (a) your expectation was off by $10 and that (b) the meal was more expensive than you thought. When you come back to this place, you might bring an extra $10. If our original expectation was that the meal would be between $0 and $1,000, then it’s true that our data fall into that range, but it’s not clear how much more we’ve learned. For example, would you change your behavior the next time you came back? The expectation of a $30 meal is sometimes referred to as a sharp hypothesis because it states something very specific that can be verified with the data.
Page 18
Epicycles of Analysis 11 2.6 Applying the Epicyle of Analysis Process Before we discuss a couple of examples, let’s review the three steps to use for each core data analysis activity. These are : 1. Setting expectations, 2. Collecting information (data), comparing the data to your expectations, and if the expectations don’t match, 3. Revising your expectations or fixing the data so that your expectations and the data match. Example: Asthma prevalence in the U.S. Let’s apply the “data analysis epicycle” to a very basic ex- ample. Let’s say your initial question is to determine the prevalence of asthma among adults, because your company wants to understand how big the market might be for a new asthma drug. You have a general question that has been identified by your boss, but need to: (1) sharpen the question, (2) explore the data, (3) build a statistical model, (4) interpret the results, and (5) communicate the results. We’ll apply the “epicycle” to each of these five core activities. For the first activity, refining the question, you would first develop your expectations of the question, then collect in- formation about the question and determine if the informa- tion you collect matches your expectations, and if not, you would revise the question. Your expectations are that the answer to this question is unknown and that the question is answerable. A literature and internet search, however, re- veal that this question has been answered (and is continually answered by the Centers for Disease Control (CDC)), so you
Page 19
Epicycles of Analysis 12 reconsider the question since you can simply go to the CDC website to get recent asthma prevalence data. You inform your boss and initiate a conversation that re- veals that any new drug that was developed would tar- get those whose asthma was not controlled with currently available medication, so you identify a better question, which is “how many people in the United States have asthma that is not currently controlled, and what are the demographic predictors of uncontrolled asthma?” You repeat the process of collecting information to determine if your question is answerable and is a good one, and continue this process until you are satisfied that you have refined your question so that you have a good question that can be answered with available data. Let’s assume that you have identified a data source that can be downloaded from a website and is a sample that repre- sents the United States adult population, 18 years and older. The next activity is exploratory data analysis, and you start with the expectation that when you inspect your data that there will be 10,123 rows (or records), each representing an individual in the US as this is the information provided in the documentation, or codebook, that comes with the dataset. The codebook also tells you that there will be a variable indicating the age of each individual in the dataset. When you inspect the data, though, you notice that there are only 4,803 rows, so return to the codebook to confirm that your expectations are correct about the number of rows, and when you confirm that your expectations are correct, you return to the website where you downloaded the files and discover that there were two files that contained the data you needed, with one file containing 4,803 records and the second file containing the remaining 5,320 records. You download the second file and read it into your statistical
Page 20
Epicycles of Analysis 13 software package and append the second file to the first. Now you have the correct number of rows, so you move on to determine if your expectations about the age of the pop- ulation matches your expectations, which is that everyone is 18 years or older. You summarize the age variable, so you can view the minimum and maximum values and find that all individuals are 18 years or older, which matches your expectations. Although there is more that you would do to inspect and explore your data, these two tasks are examples of the approach to take. Ultimately, you will use this data set to estimate the prevalence of uncontrolled asthma among adults in the US. The third activity is building a statistical model, which is needed in order to determine the demographic characteris- tics that best predict that someone has uncontrolled asthma. Statistical models serve to produce a precise formulation of your question so that you can see exactly how you want to use your data, whether it is to estimate a specific parameter or to make a prediction. Statistical models also provide a formal framework in which you can challenge your findings and test your assumptions. Now that you have estimated the prevalence of uncon- trolled asthma among US adults and determined that age, gender, race, body mass index, smoking status, and income are the best predictors of uncontrolled asthma available, you move to the fourth core activity, which is interpreting the results. In reality, interpreting results happens along with model building as well as after you’ve finished building your model, but conceptually they are distinct activities. Let’s assume you’ve built your final model and so you are moving on to interpreting the findings of your model. When you examine your final predictive model, initially your ex- pectations are matched as age, African American/black race,
Showing the first 20 pages. Download the document to see it all.
Comments (0)
No comments yet. Be the first!