Fall 2019 Fall 2019 Introduction to Scalable Data Outline Analytics using Apache Spark ⬤ Big Data Problems: Distributing Work, Failures, Slow Machines ⬤ What is Apache Spark? ⬤ Core things of Apache Spark ⬥ RDD ⬤ Core Functionality of Apache Spark ⬤ Simple tutorial Vassilis Christophides christop@csd.uoc.gr http://www.csd.uoc.gr/~hy562 University of Crete, Fall 2019 1 2
Fall 2019 Fall 2019 Hardware for Big Data Big Data Problems: Distributing Work, Failures, Slow Machines Bunch of Hard Drives …. and CPUs ⬤ The Big Data Problem ⬥ Data growing faster than CPU speeds ⬥ Data growing faster than per-machine storage ⬤ Can’t process or store all data on one machine 3 4
Fall 2019 Fall 2019 Hardware for Big Data Hardware for Big Data ⬤ One big box ! (1990s solution) ⬤ Consumer-grade hardware ⬥ All processors share memory ⬥ Not "gold plated" ⬤ Very expensive ⬤ Many desktop-like servers ⬥ Low volume ⬥ Easy to add capacity ⬥ All “premium” HW ⬥ Cheaper per CPU/disk ⬤ Still not big enough! ⬤ But, implies complexity in software Image: Wikimedia Commons / User:Tonusamuel Image: Steve Jurvetson/Flickr 5 6
Fall 2019 Fall 2019 Problems with Cheap HW The Opportunity ⬤ Failures, e.g. (Google numbers) ⬤ Cluster computing is a game-changer! ⬥ 1-5% hard drives/year ⬥ 0.2% DIMMs/year ⬤ Provides access to low-cost computing and storage ⬤ Network speeds vs. shared memory ⬤ Costs decreasing every year ⬥ Much more latency ⬥ Network slower than storage ⬤ The challenge is programming the resources ⬤ Uneven performance ⬤ What’s hard about Cluster computing? ⬥ How do we split work across machines? Google Datacenter 7 8
Fall 2019 Fall 2019 How do you Count the Number of Occurrences of each Word in a Document? Centrilized Approach: Use a Hash Table “I am Sam I: 3 “I am Sam { } I am Sam am: 3 I am Sam Sam I am Sam: 3 Sam I am Do you like do: 1 Do you like Green eggs and ham?” you: 1 Green eggs and ham?” like: 1 … 9 10
Fall 2019 Fall 2019 Centrilized Approach: Use a Hash Table Centrilized Approach: Use a Hash Table “I am Sam { I: 1, } “I am Sam { I: 1, I am Sam I am Sam am: 1, Sam I am Sam I am } Do you like Do you like Green eggs and ham?” Green eggs and ham?” 11 12
Fall 2019 Fall 2019 Centrilized Approach: Use a Hash Table Centrilized Approach: Use a Hash Table “I am Sam { I: 1, “I am Sam { I: 2, I am Sam am: 1, I am Sam am: 1, Sam I am Sam: 1, Sam I am Sam: 1, Do you like } Do you like } Green eggs and ham?” Green eggs and ham?” 13 14
Fall 2019 Fall 2019 What if the Document is Really Big? What if the Document is Really Big? What’s the problem with this approach? Results have to fit on one machine ! 15 16
Fall 2019 Fall 2019 What if the Document is Really Big? What if the Document is Really Big? Can add aggregation layers but results still must fit on one machine Use Divide and Conquer!! 17 18
Fall 2019 Fall 2019 What if the Document is Really Big? Map Reduce for Sorting MAP REDUCE Google Map Reduce 2004 ⬤ “What word is used most?” 19 20
Fall 2019 Fall 2019 What’s Hard About Cluster Computing? How Do We Deal with Machine Failures? ⬤ How to divide work across machines? ⬥ Must consider network, data locality ⬥ Moving data may be very expensive ⬤ How to deal with failures? ⬥ 1 server fails every 3 years => with 10,000 nodes see 10 faults/day ⬤ Even worse: stragglers (not failed, but slow nodes) ⬤ Launch another task! 21 22
How Do We Deal with Slow Tasks? Fall 2019 Fall 2019 MapReduce: Distributed Execution Image: Wikimedia commons (RobH/Tbayer (WMF)) ⬤ Launch another task! ⬤ Each stage passes through the hard drives 23 24
Fall 2019 Fall 2019 Map Reduce: Iterative Jobs Apache Spark Motivation ⬤ Iterative jobs involve a lot of disk I/O for each repetition While MapReduce is simple, it can require asymptotically los lots of disk ⬤ ⬥ Disk I/O is very slow! I/O for complex jobs, interactive queries and online processing Interactive Mining Stream Processing Also, most ML algorithms are iterative! ⬤ MapReduce is great at one-pass computation, but inefficient for multi-pass ⬤ Commonly spend 90% of time doing I/O! algorithms 25 26
Fall 2019 Fall 2019 Tech Trend: Cost of Memory Modern Hardware for Big Data Bunch of Hard Drives …. and CPUs ⬤ Lower cost means can put more memory in each server … and memory! http://www.jcmit.com/mem2014.htm 27 28
Fall 2019 Fall 2019 Opportunity Use Memory Instead of Disk ⬤ Keep more data in-memory ⬤ Create new distributed execution engine : ⬤ One of the most efficient programming frameworks offering abstraction and parallelism for clusters ⬤ It hides complexities of: ⬥ Fault Tolerance ⬥ Slow machines ⬥ Network Failures http://people.csail.mit.edu/matei/papers/2010/hotcloud_spark.pdf 29 30
Fall 2019 Fall 2019 In-Memory Data Sharing Spark and Map Reduce Differences Apache Hadoop Apache Spark Map Reduce Storage Disk only In-memory or on disk Operations Map and Reduce Many transformation and actions, including Map and Reduce Execution Batch Batch, interactive, model streaming Languages Java Scala, Java, R, and Python ⬤ 10-100x faster than network and disk! 31 32
Fall 2019 Fall 2019 Spark: Fault Tolerance Spark: Fault Tolerance ⬤ Hadoop: Once computed, don’t lose it ⬤ Hadoop: Once computed, don’t lose it ⬤ Spark: Remember how to recompute ⬤ Spark: Remember how to recompute 33 34
Fall 2019 Fall 2019 Other Spark and Map Reduce Differences In-Memory Can Make a Big Difference Generalized patterns for computation ⬤ (2013) Two iterative Machine Learning algorithms: ⬤ provide unified engine for many use cases ⬥ K-means Clustering ⬥ ⬥ require 2-5x less code ⬤ Lazy evaluation of the lineage graph ⬥ can optimize, reduce wait states, pipeline better Lower overhead for starting jobs ⬥ Logistic Regression ⬤ ⬤ Less expensive shuffles 35 36
Fall 2019 Fall 2019 A Brief History Recall What’s Hard with Big Data ⬤ Complex combination of processing tasks, storage, systems and modes ⬥ ETL, aggregation, streaming, machine learning ⬤ Hard to get both productivity and performance! 39 40
Hiển thị 20 trang đầu. Tải tài liệu để xem đầy đủ.
Chưa có bình luận nào. Hãy là người đầu tiên!
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!