Outline I. About me II. Distributed CompuIng at a High Level III. Disk versus Memory based Systems IV. Spark Core I. Brief background II. Benchmarks and Comparisons III. What is an RDD IV. RDD AcIons and TransformaIons V. Caching and SerializaIon VI. Anatomy of a Program VII. The Spark Family 2
Why Distributed CompuIng? Divide and Conquer Problem Single machine cannot complete the computaIon at hand SoluIon Parallelize the job and distribute work among a network of machines 3
Issues Arise in Distributed CompuIng View the world from the eyes of a single worker • How do I distribute an algorithm? • How do I par++on my dataset? • How do I maintain a single consistent view of a shared state? • How do I recover from machine failures? • How do I allocate cluster resources? • ….. 4
Finding majority element in a distributed dataset Think distributed List(1, 18, 1, 18, 1) List(2, 18, 2, 18, 2) List(3, 18, 3, 18, 3) List(4, 18, 4, 18, 4) List(5, 18, 5, 18, 5) 6
Finding majority element in a distributed dataset Think distributed List(1, 18, 1, 18, 1) List(2, 18, 2, 18, 2) List(3, 18, 3, 18, 3) 18 List(4, 18, 4, 18, 4) List(5, 18, 5, 18, 5) 7
Disk Based vs Memory Based Frameworks Acyclic data flow • Disk Based Frameworks ‒Persists intermediate results to disk ‒Data is reloaded from disk with every query ‒Easy failure recovery ‒Best for ETL like work-‐loads ‒Examples: Hadoop, Dryad Map Reduce Input Map Output Reduce Map Image courtesy of MateiZaharia, Introduction to Spark 8
Disk Based vs Memory Based Frameworks Reuse working data set in memory • Memory Based Frameworks ‒Circumvents heavy cost of I/O by keeping intermediate results in memory ‒SensiIve to availability of memory ‒Remembers operaIonsapplied to dataset ‒Best for iteraIve workloads ‒Examples: Spark, Flink iter. 1 iter. 2 . . . Input Image courtesy of MateiZaharia, Introduction to Spark 9
The rest of the talk I. Spark Core I. Brief background II. Benchmarks and Comparisons III. What is an RDD IV. RDD AcIons and TransformaIons V. Spark Cluster VI. Anatomy of a Program VII. The Spark Family 10
Spark Background Arose from an academic selng • Amplab UC Berkley • Project Lead: Dr. Matei Zaharia • First paper published on RDD’s was in 2012 • Open sourced from day one, growing number of contributors • Released its 1.0 version May 2014. Currently in 1.2.1 • Databricks company established to support Spark and all its related technologies. Matei currently sits as its CTO • Amazon, Alibaba, Baidu, eBay, Groupon, Ooyala, OpenTable, Box, Shopify, TechBase, Yahoo! 11
Resilient Distributed Datasets (RDDs) • Main object in Spark’s universe • Think of it as represenIng the data at that stage in the operaIon • Allows for coarse-‐grained transformaIons (e.g. map, group-‐by, join) • Allows for efficient fault recovery using lineage ‒Log one operaIon to apply to many elements ‒Recompute lost parIIons of dataset on failure ‒No cost if nothing fails 14
RDD AcIons and TransformaIons TransformaIons are realized when an acIon is called • TransformaIons ‒ Lazy operaIons applied on an RDD ‒ Creates a new RDD from an exisIng RDD ‒ Allows Spark to perform opImizaIons ‒ e.g. map, filter, flatMap, union, intersecIon, disInct, reduceByKey, groupByKey • AcIons ‒ Returns a value to the driver program a5er computaIon ‒ e.g. reduce, collect, count, first, take, saveAsFile 15
RDD RepresentaIon • Simple common interface: –Set of parIIons –Preferred locaIons for each parIIon –List of parent RDDs –FuncIon to compute a parIIon given parents –OpIonal parIIoning info • Allows capturing wide range of transformaIons Slide courtesy of MateiZaharia, Introduction to Spark 16
Spark Cluster Driver Driver • Entry point of Spark applicaIon • Main Spark applicaIon is ran here • Results of “reduce” operaIons are aggregated here 17
Spark Cluster Driver Master Master • Distributed coordinaIon of Spark workers including: § Health checking workers § Reassignment of failed tasks § Entry point for job and cluster metrics 18
Spark Cluster Worker Worker Driver Master Worker Worker Worker • Spawns executors to perform tasks on parIIons of data 19
Example: Log Mining Load error messages from a log into memory, then interacIvely search for various paserns Base RDD Msgs. 1 lines = spark.textFile(“hdfs://...”) Transformed RDD Worker errors = lines.filter(_.startsWith(“ERROR”)) results messages = errors.map(_.split(‘\t’)(2)) tasks Block 1 Driver messages.persist() AcIon messages.filter(_.contains(“foo”)).count messages.filter(_.contains(“bar”)).count Msgs. 2 Worker . . . Msgs. 3 Worker Block 2 Result:Result:full-‐text search of Wikipedia in <1 sec (scaled to 1 TB data in 5-‐7 sec vs (vs20 sec for on-‐disk data)170 sec for on-‐disk data) Block 3 Slide courtesy of MateiZaharia, Introduction to Spark 20
最初の20ページを表示しています。全文を見るには資料をダウンロードしてください。
まだコメントはありません。最初のコメントを書きましょう！
コメント (0)
まだコメントはありません。最初のコメントを書きましょう！