Analysis in the Big Data Era Massive Data Data Analysis Insight Key to Success = Timely and Cost-Effective Analysis 9/26/2011 Starfish 2
Hadoop MapReduce Ecosystem Popular solution to Big Data Analytics Java / C++ / Elastic R / Python Pig Hive Jaql Oozie MapReduce Hadoop MapReduce Execution Engine HBase Distributed File System 9/26/2011 Starfish 3
Practitioners of Big Data Analytics Who are the users? Data analysts, statisticians, computational scientists… Researchers, developers, testers… You! Who performs setup and tuning? The users! Usually lack expertise to tune the system 9/26/2011 Starfish 4
Tuning Challenges Heavy use of programming languages for MapReduce programs (e.g., Java/python) Data loaded/accessed as opaque files Large space of tuning choices Elasticity is wonderful, but hard to achieve (Hadoop has many useful mechanisms, but policies are lacking) Terabyte-scale data cycles 9/26/2011 Starfish 5
Starfish: Self-tuning System Our goal: Provide good performance automatically Java / C++ / Elastic R / Python Pig Hive Jaql Oozie MapReduce Analytics System Starfish Hadoop HBase MapReduce Execution Engine Distributed File System 9/26/2011 Starfish 6
What are the Tuning Problems? Job-level Cluster sizing MapReduce configuration Data J J layout 1 2 tuning J3 Workflow Workload J4 optimization management 9/26/2011 Starfish 7
Starfish’s Core Approach to Tuning Optimizers Search through space of tuning choices Cluster Job Data layout Profiler What-if Engine Workflow Collects concise Workload Estimates impact of summaries of hypothetical changes execution on execution 1) if Δ(conf. parameters) then what …? 2) if Δ(data properties) then what …? 3) if Δ(cluster properties) then what …? 9/26/2011 Starfish 8
Starfish Architecture Workload Optimizer Elastisizer Profiler What-if Workflow Optimizer Engine Job Optimizer Data Manager Metadata Intermediate Data Layout & Mgr. Data Mgr. Storage Mgr. 9/26/2011 Starfish 9
MapReduce Job Execution job j = < program p, data d, resources r, configuration c > split 0 map split 2 map reduce out 0 split 1 map split 3 map reduce Out 1 Two Map Waves One Reduce Wave 9/26/2011 Starfish 10
What Controls MR Job Execution? job j = < program p, data d, resources r, configuration c > Space of configuration choices: Number of map tasks Number of reduce tasks Partitioning of map outputs to reduce tasks Memory allocation to task-level buffers Multiphase external sorting in the tasks Whether output data from tasks should be compressed Whether combine function should be used 9/26/2011 Starfish 11
Effect of Configuration Settings Rules-of-thumb settings Two-dimensional projection of a multi- dimensional surface (Word Co-occurrence MapReduce Program) Use defaults or set manually (rules-of-thumb) Rules-of-thumb may not suffice 9/26/2011 Starfish 12
MapReduce Job Tuning in a Nutshell Goal: perf = F(p,d,r,c) c = argmin F(p,d,r,c) opt cS Challenges: p is an arbitrary MapReduce program; c is high-dimensional; … Profiler Runs p to collect a job profile (concise execution summary) of <p,d1,r1,c1> What-if Engine Given profile of <p,d1,r1,c1>, estimates virtual profile for <p,d2,r2,c2> Optimizer Enumerates and searches through the optimization space S efficiently 9/26/2011 Starfish 13
Job Profile Concise representation of program execution as a job Records information at the level of “task phases” Generated by Profiler through measurement or by the What-if Engine through estimation Serialize, Memory map Partition Buffer Sort, split [Combine], [Compress] Merge DFS Read Map Collect Spill Merge 9/26/2011 Starfish 14
Job Profile Fields Dataflow: amount of data Costs: execution times at the level of flowing through task phases task phases Map output bytes Read phase time in the map task Number of spills Map phase time in the map task Number of records in buffer per spill Spill phase time in the map task Dataflow Statistics: statistical Cost Statistics: statistical information about dataflow information about resource costs Width of input key-value pairs I/O cost for reading from local disk per byte Map selectivity in terms of records CPU cost for executing the Mapper per record Map output compression ratio CPU cost for uncompressing the input per byte 9/26/2011 Starfish 15
Generating Profiles by Measurement Goals Have zero overhead when profiling is turned off Require no modifications to Hadoop Support unmodified MapReduce programs written in Java or Hadoop Streaming/Pipes (Python/Ruby/C++) Approach: Dynamic (on-demand) instrumentation Event-condition-action rules are specified (in Java) Leads to run-time instrumentation of Hadoop internals Monitors task phases of MapReduce job execution We currently use Btrace (Hadoop internals are in Java) 9/26/2011 Starfish 16
Generating Profiles by Measurement JVM JVM Enable Profiling split 0 map reduce out 0 ECA rules raw data raw data JVM map reduce split 1 map profile profile Use of Sampling raw data • Profile fewer tasks job • profile Execute fewer tasks JVM = Java Virtual Machine, ECA = Event-Condition-Action 9/26/2011 Starfish 17
What-if Engine Possibly Hypothetical Job Input Data Cluster Configuration Profile Properties Resources Settings <p, d1, r1, c1> <d2> <r2> <c2> What-if Engine Job Oracle Virtual Job Profile for <p, d2, r2, c2> Task Scheduler Simulator Properties of Hypothetical job 9/26/2011 Starfish 18
Virtual Profile Estimation Given profile for job j = <p, d1, r1, c1> estimate profile for job j' = <p, d2, r2, c2> Profile for j (Virtual) Profile for j' Dataflow Input Confi- Data d Cardinality guration Statistics 2 Models Dataflow Statistics c2 Cost Resources r White-box Models Statistics 2 Cost Dataflow Relative Statistics Dataflow Black-box Models White-box Models Costs Costs 9/26/2011 Starfish 19
Job Optimizer Job Input Data Cluster Profile Properties Resources <p, d1, r1, c1> <d2> <r2> Just-in-Time Optimizer Subspace Enumeration What-if Recursive Random Search calls Best Configuration Settings <copt> for <p, d2, r2> 9/26/2011 Starfish 20
最初の20ページを表示しています。全文を見るには資料をダウンロードしてください。
まだコメントはありません。最初のコメントを書きましょう！
コメント (0)
まだコメントはありません。最初のコメントを書きましょう！