Page 3
Contents 1 Preface 9 2 Big and Fast Data 11 2.1 How big is “Big”? . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.2 The many Vs of Big data . . . . . . . . . . . . . . . . . . . . . 13 2.3 A brief history of Big Data tech . . . . . . . . . . . . . . . . . 15 2.4 The importance of data . . . . . . . . . . . . . . . . . . . . . 17 2.5 Typical problems solved with Big Data . . . . . . . . . . . . . 17 3 Languages for Big Data processing 19 3.1 Scala and Python . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.2 Hello world . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.3 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.4 Object-Oriented programming . . . . . . . . . . . . . . . . . . 23 4 Programming for Big Data 27 4.1 Basic Data Types . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.2 Functional programming in a nutshell . . . . . . . . . . . . . . 31 4.3 Enumerating datasets . . . . . . . . . . . . . . . . . . . . . . 40 4.4 Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 4.5 Basic transformations . . . . . . . . . . . . . . . . . . . . . . 43 4.6 Key-value pairs . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.7 Immutability . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 5 Introduction to distributed systems 59 5.1 Unreliable networks . . . . . . . . . . . . . . . . . . . . . . . . 61 5.2 Unreliable time . . . . . . . . . . . . . . . . . . . . . . . . . . 62 5.3 Distributed decision making . . . . . . . . . . . . . . . . . . . 66 3
Page 4
4 CONTENTS 5.4 Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 6 Distributed databases 75 6.1 Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 6.2 Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 6.3 Transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 6.4 Distributed transactions . . . . . . . . . . . . . . . . . . . . . 85 7 Processing data with Spark 89 7.1 The world before Spark . . . . . . . . . . . . . . . . . . . . . 89 7.2 Spark and RDDs . . . . . . . . . . . . . . . . . . . . . . . . . 91 7.3 RDDs under the hood . . . . . . . . . . . . . . . . . . . . . . 97 7.4 Spark applications . . . . . . . . . . . . . . . . . . . . . . . . 102 7.5 Effective Spark . . . . . . . . . . . . . . . . . . . . . . . . . . 104 8 Spark Datasets 109 8.1 Spark SQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 8.2 Creating Data Frames and Datasets . . . . . . . . . . . . . . . 111 8.3 SparkSQL under the covers . . . . . . . . . . . . . . . . . . . 114 9 Data processing at the command line 117 9.1 The UNIX operating system . . . . . . . . . . . . . . . . . . . 117 9.2 The UNIX programming environment . . . . . . . . . . . . . . 123 9.3 Task-based tools . . . . . . . . . . . . . . . . . . . . . . . . . 127 9.4 Writing programs . . . . . . . . . . . . . . . . . . . . . . . . . 129
Page 7
List of Figures 2.1 Instagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.2 FaceBook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.3 Data growth rate . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.4 The big data landscape . . . . . . . . . . . . . . . . . . . . . . 16 2.5 Importance of data . . . . . . . . . . . . . . . . . . . . . . . . 17 4.1 A HashTable . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 4.2 Warning: Java Code Ahead! . . . . . . . . . . . . . . . . . . . 38 4.3 Types of joins . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.4 Immutability . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 4.5 A tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 4.6 The tree after addition . . . . . . . . . . . . . . . . . . . . . . 56 5.1 Parallel system . . . . . . . . . . . . . . . . . . . . . . . . . . 59 5.2 Distributed system . . . . . . . . . . . . . . . . . . . . . . . . 60 5.3 Unreliable networks . . . . . . . . . . . . . . . . . . . . . . . . 61 5.4 Soft Watch At The Moment Of First Explosion, by Salvador Dali . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 5.5 The two generals problem setting . . . . . . . . . . . . . . . . 67 5.6 Byzantine Eagle . . . . . . . . . . . . . . . . . . . . . . . . . . 68 5.7 Roles in a Raft cluster . . . . . . . . . . . . . . . . . . . . . . 68 5.8 Raft terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 5.9 Raft leader election . . . . . . . . . . . . . . . . . . . . . . . . 70 5.10 A non-linearisable system . . . . . . . . . . . . . . . . . . . . 71 5.11 Linearisability Compare and Swap . . . . . . . . . . . . . . . 72 5.12 Linearisability Read after Write . . . . . . . . . . . . . . . . . 73 5.13 Linearisability read during conflict . . . . . . . . . . . . . . . 73 7
Page 8
8 LIST OF FIGURES 6.1 Hey I just met you, the network’s laggy . . . . . . . . . . . . . 75 6.2 but here’s my data, so store it maybe . . . . . . . . . . . . . . 76 6.3 MySQL async replication . . . . . . . . . . . . . . . . . . . . 77 6.4 Types of partitioning . . . . . . . . . . . . . . . . . . . . . . . 82 6.5 Sharding and replication in MongoDB . . . . . . . . . . . . . 83 7.1 The Hadoop Workflow . . . . . . . . . . . . . . . . . . . . . . 90 7.2 DryadLINQ architecture . . . . . . . . . . . . . . . . . . . . . 90 7.3 Spark Logo . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 7.4 Partition Dependencies . . . . . . . . . . . . . . . . . . . . . . 99 7.5 Shuffling explained . . . . . . . . . . . . . . . . . . . . . . . . 101 7.6 Spark cluster architecture . . . . . . . . . . . . . . . . . . . . 102 7.7 Word Count job graph . . . . . . . . . . . . . . . . . . . . . . 103 9.1 Ken Thomson and Dennis Ritchie, the original authors of Unix 117 9.2 Brian Kernighan and Rob Pike, authors of the homonymous seminal book . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 9.3 Richard Stallman, founder of the Free Software movement and co-author of many of the Unix tools we use on Linux . . . . . 127 9.4 The Unix way . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
Page 9
Chapter 1 Preface TI2736-B: Big Data Processing is a second year BSc course at TU Delft that, as its title says, aims is to teach students how to Hi there, this is my great book. process big data. It is part of the “Data Processing” variantblok, that also includes courses like Data Mining and Computational Intelligence. In Jan 2017, I took over the course; my collegue who was teaching the course before me had already done a fantastic job with it. I decided to follow a different route; instead of presenting specific systems or technologies, I would focus mostly on how big data systems are programmed. Structure of the book Chapters ?? introduces a new topic, and … Acknowledgments A lot of people helped me when I was writing this book. Wouter Zorgdrager Georgios Gousios 9
Page 10
10 CHAPTER 1. PREFACE Copyright information This work is (c) 2017 - onwards by TU Delft and Georgios Gousios and distributed under the Creative Commons Attribution- NonCommercial-ShareAlike 4.0 International license.
Page 11
Chapter 2 Big and Fast Data What is big data? An overloaded, fuzzy term “Data too large to be efficiently processed on a single computer” “Massive amounts of diverse, unstructured data prodiced by high- performance applications” 2.1 How big is “Big”? Typical numbers associated with Big Data • 2.5 Exabytes (103 TB) produced daily • IoT: 8.4 Billion devices with internet access • Amazon: 600 orders per second up (2016) from 35 in 2012 • Alibaba in 2015: 400 orders per second • Google in 2016: 2+ trillion searches or 65k per second – Each query involves > 1000 machines – Each search touches 200+ services How big is “Big”? – Instagram • 250M daily users, clicking around the app • 50M photos daily 11
Page 12
12 CHAPTER 2. BIG AND FAST DATA Figure 2.1: Instagram • Most followed user: 113M followers How big is “Big”? – FaceBook Figure 2.2: FaceBook Warning: numbers (⇑) from 2014! Today on FB: • 2 Billion users • 1.32 Billion active users per day • 300 million photos per day (136k/min) • Every min: 510k comments, 293k status updates
Page 13
2.2. THE MANY VS OF BIG DATA 13 2.2 The many Vs of Big data Main Vs, by Doug Laney • Volume: large amounts of data • Variety: data comes in many different forms from diverse sources • Velocity: the content is changing quickly More Vs • Value: data alone is not enough; how can value be derived from it? • Veracity: can we trust the data? How accurate is it? • Validity: ensure that the interpreted data is sound • Visibility: data from diverse sources need to be stitched together Volume We call Big Data big because it is really big: • 90% of all the data ever was created in the last 2 years • By 2020, each person will generate 1.7MB per sec • The Big data / data analytics industry will be worth $203 Billion in 2020 Figure 2.3: Data growth rate Variety • Structured data: SQL tables, images, format is known
Page 14
14 CHAPTER 2. BIG AND FAST DATA • Semi-structured data: JSON, XML • Unstructured data: Text, mostly We often need to combine various data sources of different types to come up with a result Velocity Data is not just big; it is generated and needs to be processed fast. Think of: • Datacenters writing to log files • IoT sensors reporting temperatures around the globe • Twitter: 500 million tweets a day (or 6k/sec) • Stock Markets: high-frequency trading (latency costs money) • Online advertising Data needs to be processed with soft or hard real-time guarantees Big Data processing • The ETL cycle – Extract: Convert raw or semi-structured data into structured data – Transform: Convert units, join data sources, cleanup etc – Load: Load the data into another system for further processing • Big data engineering is concerned with building pipelines • Big data analytics is concerned with discovering patters How to process all this data? • Batch processing: All data exists in some data store, a program processes the whole dataset at once • Stream processing: Processing of data as they arrive to the system 2 basic approaches to distribute data processing operations on lots of ma- chines • Divide the data in chunks, apply the same algorithm on all chunks (concurrency) • Divide the problem in chunks, run it on a cluster of machines (paral- lelism)
Page 15
2.3. A BRIEF HISTORY OF BIG DATA TECH 15 Large-scale computing Not a new discipline: • Cray-1 appeared in the late ’70s • Physicists used super computers for simulations in the ’80s • Shared-memory designs still in large scale use (e.g. TOP500 supercom- puters) What is new? Large scale processing on distributed, commodity computers, enabled by advanced software using elastic resource allocation. Software (not HW!) is what drives the Big Data industry 2.3 A brief history of Big Data tech • 2003: Google publishes the Google Filesystem paper, a large-scale dis- tributed file system • 2004: Google publishes the Map/Reduce paper, a distributed data pro- cessing abstraction • 2006: Yahoo creates and open sources Hadoop, inspired by the Google papers • 2006: Amazon lunches its Elastic Compute Cloud, offering cheap, elas- tic resources • 2007: Amazon publishes the DynamoDB paper, sketches the blueprints of a cloud-native database • 2009 – onwards: The NoSQL movement. Schema-less, distributed databases defy the SQL way of storing data • 2010: Matei Zaharia et al. publish the Spark paper, brings FP to in- memory computations • 2012: Both Spark Streaming and Apache Flink appear, able to handle really high volume stream processing • 2012: Alex Krizhevsky et al. publish their deep learning image classifi- cation paper re-igniting interest in neural networks and solidifying the value of big data The Big Data Tech Landscape 2017
Page 17
2.4. THE IMPORTANCE OF DATA 17 Progress is mostly industry-driven D:: Most advancement in Big Data technologies came from the industry. The universities only started contributing late. Why? . . . Data is the new oil 2.4 The importance of data Figure 2.5: Importance of data Figure by Banko and Brill, 2001. They showed that simple algorithms per- form better than complex ones when the data is big enough. 2.5 Typical problems solved with Big Data • Modeling: What factors influence particular outcomes/behaviours? • Information retrieval: Finding needles in haystacks, aka search en- gines
Page 18
18 CHAPTER 2. BIG AND FAST DATA • Collaborative filtering: Recommending items based on items other users with similar tastes have chosen • Outlier detection: Discovering outstanding transactions Group exercise: Big data in practice D: Identify a big data system you interact with every day. Try to answer the following questions: • What types of data does it need to integrate? • How much data does it process per day? • How do the 3 (or 5) Vs apply to it? Use this URL: http://bit.ly/big-data-course-1 Work with the people around you, for 5 mins
Page 19
Chapter 3 Languages for Big Data processing 3.1 Scala and Python The de facto languages of Big Data and data science are • Scala Mostly used for data intensive systems • Python Mostly used for data analytics tasks Other languages include • Java The “assembly” of big data systems; the language that most big data infrastructure is written into. • R The statistician’s tool of choice. Great selection of libraries for seri- ous data analytics, great plotting tools. In our course, we will be using Scala and Python. R will be used in the minor part. Scala and Python from 10k feet • Both support object orientation, functional programming and impera- tive programming – Scala’s strong point is the combination of FP and OO – Python’s strong point is the combination of OO and IP 19
Page 20
20 CHAPTER 3. LANGUAGES FOR BIG DATA PROCESSING • Python is interpreted, Scala is compiled 3.2 Hello world Scala object Hello extends App { println("Hello, world") for (i <- 1 to 10) { System.out.println("Hello") } } • Scala is compiled to JVM bytecode • Can interoperate with JVM libraries • Scala is not sensitive to spaces/tabs. Blocks are denoted by { and } Python #!/usr/bin/env python for i in range(1, 10): print "Hello, world" • Python is interpreted • Python is indentation sensitive: blocks are denoted by a TAB or 2 spaces. 3.3 Declarations Scala val a: Int = 5 val b = 5 b = 6 // re-assignment to val // Type of foo is infered
Affichage des 20 premières pages. Téléchargez le document pour le voir en entier.
Commentaires (0)
Aucun commentaire pour le moment. Soyez le premier !