描述

Analysis in the Big Data Era Massive Data Data Analysis Insight Key to Success = Timely and Cost-Effective Analysis 9/26/2011 Starfish 2 Hadoop MapReduce Ecosystem  Popular solution to Big Data Analytics Java / C++ / R / Python Pig Hive Jaql Oozie Elastic MapReduce Hadoop HBase MapReduce Execution Engine Distributed File System 9/26/2011 Starfish 3 Practitioners of Big Data Analytics  Who are the users?  Data analysts, statisticians, computational scientists…  Researchers, developers, testers…  You!  Who performs setup and tuning?  The users!  Usually lack expertise to tune the system 9/26/2011 Starfish 4 Tuning Challenges  Heavy use of programming languages for MapReduce programs (e.g., Java/python)  Data loaded/accessed as opaque files  Large space of tuning choices  Elasticity is wonderful, but hard to achieve (Hadoop has many useful mechanisms, but policies are lacking)  Terabyte-scale data cycles 9/26/2011 Starfish 5 Starfish: Self-tuning System  Our goal: Provide good performance automatically Java / C++ / R / Python Pig Hive Jaql Oozie Elastic MapReduce Analytics System Starfish Hadoop HBase MapReduce Execution Engine Distributed File System 9/26/2011 Starfish 6 What are the Tuning Problems? Job-level MapReduce configuration J1 Cluster sizing Data layout tuning J2 J3 J4 9/26/2011 Workflow optimization Workload management Starfish 7 Starfish’s Core Approach to Tuning Optimizers Search through space of tuning choices Cluster Job Data layout Profiler Collects concise summaries of execution Workflow Workload What-if Engine Estimates impact of hypothetical changes on execution 1) if Δ(conf. parameters) then what …? 2) if Δ(data properties) then what …? 3) if Δ(cluster properties) then what …? 9/26/2011 Starfish 8 Starfish Architecture Workload Optimizer Profiler Elastisizer Workflow Optimizer What-if Engine Job Optimizer Data Manager Metadata Mgr. 9/26/2011 Intermediate Data Mg

AI 摘要