Performance Aspects Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2017-01-13 Disclaimer: Big Data software is constantly updated, code samples may be outdated. Overview Hardware Assessing Performance Benchmarks Summary Outline 1 Overview 2 Hardware 3 Assessing Performance 4 Benchmarks 5 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 2 / 20 Overview Hardware Assessing Performance Benchmarks Summary Goals Goal (user perspective): minimal time to solution Solution = workflow from data ingestion, programming to analysis results Programmer/User productivity is important Goal (system perspective): cheap total cost of ownership Simple deployment and easy management Cheap hardware Good utilization of (hardware) resources means less hardware ⇒ In this lecture, we focus on the processing of a workflow Julian M. Kunkel Lecture BigData Analytics, 2016 3 / 20 Overview Hardware Assessing Performance Benchmarks Summary Processing Steps 1 Ingesting data into our big data environment 2 Processing the workflow with (multiple) Hive/Pig/... queries Most important factor for the productivity of data scientists Low runtime is crucial for repeated analysis and interactive exploration Multiple steps/different tools can be involved in a complex workflow. We consider only the execution of one job with any tool 3 Post-processing of output with (external) tools to produce insight Strategy: big data workflow – data transfer – local analysis Best: return a final product from the big data workflow Julian M. Kunkel Lecture BigData Analytics, 2016 4 / 20 Overview Hardware Assessing Performance Benchmarks Summary Performance Factors Influencing Processing Time Startup phase Distribution of necessary files/scripts Allocating resources/containers Starting the scripts and loading dependencies Usually fixed costs (in the order of seconds) Job ex