Fall 2019 Introduction to Scalable Data Analytics using Apache Spark Fall 2019 Outline ⬤ Big Data Problems: Distributing Work, Failures, Slow Machines ⬤ What is Apache Spark? ⬤ Core things of Apache Spark ⬥ RDD ⬤ Core Functionality of Apache Spark ⬤ Simple tutorial Vassilis Christophides christop@csd.uoc.gr http://www.csd.uoc.gr/~hy562 University of Crete, Fall 2019 1 2 Fall 2019 Fall 2019 Hardware for Big Data Big Data Problems: Distributing Work, Failures, Slow Machines Bunch of Hard Drives …. and CPUs The Big Data Problem ⬥ Data growing faster than CPU speeds ⬥ Data growing faster than per-machine storage ⬤ Can’t process or store all data on one machine ⬤ 3 4 Fall 2019 Fall 2019 Hardware for Big Data ⬤ One big box ! (1990s solution) ⬥ All processors share memory ⬤ Very expensive ⬥ Low volume ⬥ All “premium” HW ⬤ Still not big enough! Hardware for Big Data Image: Wikimedia Commons / User:Tonusamuel ⬤ Consumer-grade hardware ⬥ Not "gold plated" ⬤ Many desktop-like servers ⬥ Easy to add capacity ⬥ Cheaper per CPU/disk ⬤ But, implies complexity in software Image: Steve Jurvetson/Flickr 5 6 Fall 2019 Fall 2019 Problems with Cheap HW ⬤ ⬤ ⬤ The Opportunity Failures, e.g. (Google numbers) ⬥ 1-5% hard drives/year ⬥ 0.2% DIMMs/year ⬤ Cluster computing is a game-changer! ⬤ Provides access to low-cost computing and storage Network speeds vs. shared memory ⬥ Much more latency ⬥ Network slower than storage ⬤ Costs decreasing every year ⬤ The challenge is programming the resources Uneven performance ⬤ What’s hard about Cluster computing? ⬥ How do we split work across machines? Google Datacenter 7 8 Fall 2019 How do you Count the Number of Occurrences of each Word in a Document? “I am Sam I am Sam Sam I am Do you like Green eggs and ham?” Fall 2019 Centrilized Approach: Use a Hash Table I: 3 am: 3 Sam: 3 do: 1 you: 1 like: 1 … “I am Sam I am Sam Sam I am Do you like Green eggs and ham?” 9 {} 10 Fall 2019 Centri