Seite 2
Big Data Analysis with Python Copyright © 2019 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Authors: Ivan Marin, Ankit Shukla and Sarang VK Technical Reviewer: Namachivayam D Managing Editor: Vishal Kamal Mewada Acquisitions Editor: Kunal Sawant Production Editor: Nitesh Thakur Editorial Board: David Barnes, Ewan Buckingham, Simon Cox, Manasa Kumar, Shivangi Chatterji, Alex Mazonowicz, Douglas Paterson, Dominic Pereira, Shiny Poojary, Saman Siddiqui, Erol Staveley, Ankita Thakur and Mohita Vyas First Published: April 2019 Production Reference: 1080419 ISBN: 978-1-78995-528-6 Published by Packt Publishing Ltd. Livery Place, 35 Livery Street Birmingham B3 2PB, UK Table of Contents Preface
Seite 3
Chapter 1:The Python Data Science Stack Introduction Python Libraries and Packages IPython: A Powerful Interactive Shell Exercise 1: Interacting with the Python Shell Using the IPython Commands The Jupyter Notebook Exercise 2: Getting Started with the Jupyter Notebook IPython or Jupyter? Activity 1: IPython and Jupyter NumPy SciPy Matplotlib Pandas Using Pandas Reading Data Exercise 3: Reading Data with Pandas Data Manipulation Selection and Filtering
Seite 4
Selecting Rows Using Slicing Exercise 4: Data Selection and the .loc Method Applying a Function to a Column Activity 2: Working with Data Problems Data Type Conversion Exercise 5: Exploring Data Types Aggregation and Grouping Exercise 6: Aggregation and Grouping Data NumPy on Pandas Exporting Data from Pandas Exercise 7: Exporting Data in Different Formats Visualization with Pandas Activity 3: Plotting Data with Pandas Summary
Seite 5
Chapter 2: Statistical Visualizations Introduction Types of Graphs and When to Use Them Exercise 8: Plotting an Analytical Function Components of a Graph Exercise 9: Creating a Graph Exercise 10: Creating a Graph for a Mathematical Function Seaborn Which Tool Should Be Used? Types of Graphs Line Graphs Time Series Plots Exercise 11: Creating Line Graphs Using Different Libraries Pandas DataFrames and Grouped Data Activity 4: Line Graphs with the Object-Oriented API and Pandas DataFrames Scatter Plots Activity 5: Understanding Relationships of Variables Using Scatter Plots Histograms
Seite 6
Exercise 12: Creating a Histogram of Horsepower Distribution Boxplots Exercise 13: Analyzing the Behavior of the Number of Cylinders and Horsepower Using a Boxplot Changing Plot Design: Modifying Graph Components Title and Label Configuration for Axis Objects Exercise 14: Configuring a Title and Labels for Axis Objects Line Styles and Color Figure Size Exercise 15: Working with Matplotlib Style Sheets Exporting Graphs Activity 6: Exporting a Graph to a File on Disk Activity 7: Complete Plot Design Summary
Seite 7
Chapter 3:Working with Big Data Frameworks Introduction Hadoop Manipulating Data with the HDFS Exercise 16: Manipulating Files in the HDFS Spark Spark SQL and Pandas DataFrames Exercise 17: Performing DataFrame Operations in Spark Exercise 18: Accessing Data with Spark Exercise 19: Reading Data from the Local Filesystem and the HDFS Exercise 20: Writing Data Back to the HDFS and PostgreSQL Writing Parquet Files Exercise 21: Writing Parquet Files Increasing Analysis Performance with Parquet and Partitions Exercise 22: Creating a Partitioned Dataset Handling Unstructured Data Exercise 23: Parsing Text and Cleaning Activity 8: Removing Stop Words from Text Summary
Seite 8
Chapter 4:Diving Deeper with Spark Introduction Getting Started with Spark DataFrames Exercise 24: Specifying the Schema of a DataFrame Exercise 25: Creating a DataFrame from an Existing RDD Exercise 25: Creating a DataFrame Using a CSV File Writing Output from Spark DataFrames Exercise 27: Converting a Spark DataFrame to a Pandas DataFrame Exploring Spark DataFrames Exercise 28: Displaying Basic DataFrame Statistics Activity 9: Getting Started with Spark DataFrames Data Manipulation with Spark DataFrames Exercise 29: Selecting and Renaming Columns from the DataFrame Exercise 30: Adding and Removing a Column from the DataFrame Exercise 31: Displaying and Counting Distinct Values in a DataFrame Exercise 32: Removing Duplicate Rows and Filtering Rows of a DataFrame Exercise 33: Ordering Rows in a DataFrame
Seite 9
Exercise 34: Aggregating Values in a DataFrame Activity 10: Data Manipulation with Spark DataFrames Graphs in Spark Exercise 35: Creating a Bar Chart Exercise 36: Creating a Linear Model Plot Exercise 37: Creating a KDE Plot and a Boxplot Activity 11: Graphs in Spark Summary
Seite 10
Chapter 5:Handling Missing Values and Correlation Analysis Introduction Setting up the Jupyter Notebook Missing Values Exercise 38: Counting Missing Values in a DataFrame Exercise 39: Counting Missing Values in All DataFrame Columns Fetching Missing Value Records from the DataFrame Handling Missing Values in Spark DataFrames Exercise 40: Removing Records with Missing Values from a DataFrame Exercise 41: Filling Missing Values with a Constant in a DataFrame Column Correlation Exercise 42: Computing Correlation Activity 12: Missing Value Handling and Correlation Analysis with PySpark DataFrames Summary
Seite 11
Chapter 6:Exploratory Data Analysis Introduction Defining a Business Problem Problem Identification Requirement Gathering Data Pipeline and Workflow Identifying Measurable Metrics Documentation and Presentation Translating a Business Problem into Measurable Metrics and Exploratory Data Analysis (EDA) Data Gathering Analysis of Data Generation KPI Visualization Feature Importance Exercise 43: Identify the Target Variable and Related KPIs from the Given Data for the Business Problem Exercise 44: Generate the Feature Importance of the Target Variable and Carry Out EDA Structured Approach to the Data Science Project Life Cycle Data Science Project Life Cycle Phases
Seite 12
Phase 1: Understanding and Defining the Business Problem Phase 2: Data Access and Discovery Phase 3: Data Engineering and Pre-processing Activity 13: Carry Out Mapping to Gaussian Distribution of Numeric Features from the Given Data Phase 4: Model Development Summary
Seite 13
Chapter 7:Reproducibility in Big Data Analysis Introduction Reproducibility with Jupyter Notebooks Introduction to the Business Problem Documenting the Approach and Workflows Explaining the Data Pipeline Explain the Dependencies Using Source Code Version Control Modularizing the Process Gathering Data in a Reproducible Way Functionalities in Markdown and Code Cells Explaining the Business Problem in the Markdown Providing a Detailed Introduction to the Data Source Explain the Data Attributes in the Markdown Exercise 45: Performing Data Reproducibility Code Practices and Standards Environment Documentation Writing Readable Code with Comments Effective Segmentation of Workflows
Seite 14
Workflow Documentation Exercise 46: Missing Value Preprocessing with High Reproducibility Avoiding Repetition Using Functions and Loops for Optimizing Code Developing Libraries/Packages for Code/Algorithm Reuse Activity 14: Carry normalisation of data Summary
Seite 15
Chapter 8:Creating a Full Analysis Report Introduction Reading Data in Spark from Different Data Sources Exercise 47: Reading Data from a CSV File Using the PySpark Object Reading JSON Data Using the PySpark Object SQL Operations on a Spark DataFrame Exercise 48: Reading Data in PySpark and Carrying Out SQL Operations Exercise 49: Creating and Merging Two DataFrames Exercise 50: Subsetting the DataFrame Generating Statistical Measurements Activity 15: Generating Visualization Using Plotly Summary Appendix
Seite 16
Preface About This section briefly introduces the authors, the what this book covers, the technical skills you'll need to get started, and the hardware and software requirements required to complete the activities and exercises. About the Book Processing big data in real time is challenging due to scalability, information inconsistency, and fault tolerance. Big Data Analysis with Python teaches you how to use tools that can control this data avalanche for you. With this book, you'll learn practical techniques to aggregate data into useful dimensions for posterior analysis, extract statistical measurements, and transform datasets into features for other systems. The book begins with an introduction to data manipulation in Python using pandas. You'll then get familiar with statistical analysis and plotting techniques. With multiple hands-on activities in store, you'll be able to analyze data that is distributed on several computers by using Dask. As you progress, you'll study how to aggregate data for plots when the entire data cannot be accommodated in memory. You'll also explore Hadoop (HDFS and YARN), which will help you tackle larger datasets. The book also covers Spark and explains how it interacts with other tools. By the end of this book, you'll be able to bootstrap your own Python environment, process large files, and manipulate data to generate statistics, metrics, and graphs. About the Authors Ivan Marin is a systems architect and data scientist working at Daitan Group, a Campinas-based software company. He designs big data systems for large volumes of data and implements machine learning pipelines end to end using Python and Spark. He is also an active organizer of data science, machine learning, and Python in São Paulo, and has given Python for data science courses at university level. Ankit Shukla is a data scientist working with World Wide Technology, a leading US-based technology solution provider, where he develops and deploys machine learning and artificial intelligence solutions to solve business problems and create actual dollar value for clients. He is also part of the company's R&D initiative, which is responsible for producing intellectual property, building capabilities in new areas, and publishing cutting-edge research in corporate white papers. Besides tinkering with AI/ML models, he likes to read and is a big-time foodie. Sarang VK is a lead data scientist at StraitsBridge Advisors, where his responsibilities include requirement gathering, solutioning, development, and productization of scalable machine learning, artificial intelligence, and analytical solutions using open source technologies. Alongside this, he supports pre-sales and competency. Learning Objectives Use Python to read and transform data into different formats Generate basic statistics and metrics using data on disk Work with computing tasks distributed over a cluster
Seite 17
Convert data from various sources into storage or querying formats Prepare data for statistical analysis, visualization, and machine learning Present data in the form of effective visuals Approach Big Data Analysis with Python takes a hands-on approach to understanding how to use Python and Spark to process data and make something useful out of it. It contains multiple activities that use real-life business scenarios for you to practice and apply your new skills in a highly relevant context. Audience Big Data Analysis with Python is designed for Python developers, data analysts, and data scientists who want to get hands-on with methods to control data and transform it into impactful insights. Basic knowledge of statistical measurements and relational databases will help you to understand various concepts explained in this book. Minimum Hardware Requirements For the optimal student experience, we recommend the following hardware configuration: Processor: Intel or AMD 4-core or better Memory: 8 GB RAM Storage: 20 GB available space Software Requirements You'll need the following software installed in advance. Any of the following operating systems: Windows 7 SP1 32/64-bit Windows 8.1 32/64-bit or Windows 10 32/64-bit Ubuntu 14.04 or later macOS Sierra or later Browser: Google Chrome or Mozilla Firefox Jupyter lab You'll also need the following software installed in advance: Python 3.5+ Anaconda 4.3+ These Python libraries are included with the Anaconda installation:
Seite 18
matplotlib 2.1.0+ iPython 6.1.0+ requests 2.18.4+ NumPy 1.13.1+ pandas 0.20.3+ scikit-learn 0.19.0+ seaborn 0.8.0+ bokeh 0.12.10+ These Python libraries require manual installation: mlxtend version_information ipython-sql pdir2 graphviz Conventions Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "To transform the data into the right data types, we can use conversion functions, such as to_datetime, to_numeric, and astype." A block of code is set as follows: before the sort function:[23, 66, 12, 54, 98, 3] after the sort function: [3, 12, 23, 54, 66, 98] New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: "Pandas (https://pandas.pydata.org) is a data manipulation and analysis library that's widely used in the data science community." Installation and Setup Installing Anaconda: 1. Visit https://www.anaconda.com/download/ in your browser. 2. Click on Windows, Mac, or Linux, depending on the OS you are working on. 3. Next, click on the Download option. Make sure you download the latest version. 4. Open the installer after download.
Seite 19
5. Follow the steps in the installer and that's it! Your Anaconda distribution is ready. PySpark is available on PyPi. To install PySpark run the following command: pip install pyspark --upgrade Updating Jupyter and installing dependencies: 1. Search for Anaconda Prompt and open it. 2. Type the following commands to update Conda and Jupyter: #Update conda conda update conda #Update Jupyter conda update Jupyter #install packages conda install numpy conda install pandas conda install statsmodels conda install matplotlib conda install seaborn 3. To open Jupyter Notebook from Anaconda Prompt, use the following command: jupyter notebook pip install -U scikit-learn Installing the Code Bundle Copy the code bundle for the class to the C:/Code folder. Additional Resources The code bundle for this book is also hosted on GitHub at: https://github.com/TrainingByPackt/Big-Data-Analysis-with-Python. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
Die ersten 19 Seiten werden angezeigt. Laden Sie das Dokument herunter, um alles zu sehen.
Kommentare (0)
Noch keine Kommentare. Seien Sie der Erste!