Performance Aspects Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2017-01-13 Disclaimer: Big Data software is constantly updated, code samples may be outdated.
Overview Hardware Assessing Performance Benchmarks Summary Outline 1 Overview 2 Hardware 3 Assessing Performance 4 Benchmarks 5 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 2 / 20
Overview Hardware Assessing Performance Benchmarks Summary Goals Goal (user perspective): minimal time to solution Solution = workflow from data ingestion, programming to analysis results Programmer/User productivity is important Goal (system perspective): cheap total cost of ownership Simple deployment and easy management Cheap hardware Good utilization of (hardware) resources means less hardware ⇒ In this lecture, we focus on the processing of a workflow Julian M. Kunkel Lecture BigData Analytics, 2016 3 / 20
Overview Hardware Assessing Performance Benchmarks Summary Processing Steps 1 Ingesting data into our big data environment 2 Processing the workflow with (multiple) Hive/Pig/... queries Most important factor for the productivity of data scientists Low runtime is crucial for repeated analysis and interactive exploration Multiple steps/different tools can be involved in a complex workflow. We consider only the execution of one job with any tool 3 Post-processing of output with (external) tools to produce insight Strategy: big data workflow – data transfer – local analysis Best: return a final product from the big data workflow Julian M. Kunkel Lecture BigData Analytics, 2016 4 / 20
Overview Hardware Assessing Performance Benchmarks Summary Performance Factors Influencing Processing Time Startup phase Distribution of necessary files/scripts Allocating resources/containers Starting the scripts and loading dependencies Usually fixed costs (in the order of seconds) Job execution: computing the product Costs for computation and necessary communication & I/O depend on Job complexity Software architecture of the big data solution Hardware performance and cluster architecture Cleanup phase Teardown containers, free resources Usually fixed costs (in the order of seconds) Julian M. Kunkel Lecture BigData Analytics, 2016 5 / 20
Overview Hardware Assessing Performance Benchmarks Summary 1 Overview 2 Hardware 3 Assessing Performance 4 Benchmarks 5 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 6 / 20
Overview Hardware Assessing Performance Benchmarks Summary BigData Cluster Characteristics Usually commodity components Cheap (on-board) interconnect, node-local storage Communication (bisection) bandwidth between different racks is low Rack Rack Node Node Node Node … … L:120 μs B: 1 GBit/s B: 48 GBit/s SwitchSwitch SwitchSwitch B: 10 GBit/s SwitchSwitch Architecture of a typical BigData cluster Julian M. Kunkel Lecture BigData Analytics, 2016 7 / 20
Overview Hardware Assessing Performance Benchmarks Summary HPC Cluster Characteristics High-end components Extra fast interconnect, global/shared storage with dedicated servers Switches provide high (full) bisection bandwidth Node … … Node … Node Node … Node L:0,5 μs B: 56 GBit/s B: 24 TBit/s SwitchSwitch SwitchSwitch L:600 ns Cut-Through-SwitchCut-Through-Switch SwitchSwitch L:0,5 μs B: 56 GBit/s I/O-I/O- … I/O-I/O- Server … Server Server Server SAN Architecture of a typical HPC cluster. Julian M. Kunkel Lecture BigData Analytics, 2016 8 / 20
Overview Hardware Assessing Performance Benchmarks Summary Hardware Performance Computation CPU performance (frequency · cores · sockets) e.g., 2.5 GHz · 12 cores · 2 sockets = 60 Gcycles/s The number of cycles per operation depend on the instruction stream Memory (throughput · channels); e.g., 25.6 GB/s per DDR4 DIMM ·3 Communication via the network Throughput, e.g., 125 MiB/s with Gigabit Ethernet Latency, e.g., 0.1 ms with Gigabit Ethernet Input/output devices HDD mechanical parts (head, rotation) lead to expensive seek ⇒ Access data consecutively and not randomly ⇒ Performance depends on the I/O granularity e.g., 150 MiB/s with 10 MiB blocks Julian M. Kunkel Lecture BigData Analytics, 2016 9 / 20
Overview Hardware Assessing Performance Benchmarks Summary Hardware-Aware Strategies for Software Solutions Java is suboptimal: 1.2x - 2x of cycles needed than in C 1 Utilize different hardware components concurrently Pipeline computation, I/O and communication At best hide two of them ⇒ 3x speedup Avoid barriers (waiting for the slowest component) Balance and distribute workload among all available servers Linear scalability is vital (and not the programming language) Add 10x servers, achieve 10x performance Avoid I/O, if possible (keep data in memory) Avoid communication, if possible Allow monitoring of components to see their utilization Examples for Pig/Hive Foreach, filter are node-local operations Sort, group, join need communication 1This does not matter much compared to the other factors. Julian M. Kunkel Lecture BigData Analytics, 2016 10 / 20
Overview Hardware Assessing Performance Benchmarks Summary 1 Overview 2 Hardware 3 Assessing Performance 4 Benchmarks 5 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 11 / 20
Overview Hardware Assessing Performance Benchmarks Summary Basic Approach Question Is the observed performance acceptable? Basic approach Start with a simple model 1 Measure time for the execution of your workload 2 Quantify the workload with some metrics e.g., amount of tuples or data processed, computational operations needed e.g., you may use the statistis output for each Hadoop job 3 Compute wt, the workload you process per time 4 Compare wt with your expectations of the system Refine the model as needed, e.g., include details about intermediate steps Julian M. Kunkel Lecture BigData Analytics, 2016 12 / 20
Overview Hardware Assessing Performance Benchmarks Summary Errors Increase Processing Time [11] Error probability E < 1 increases the processing time A rerun of a job may fail again Processing time with errors: P′ = (E + E2 + ...) · P′ = P/(1 − E) 100 50 20 With 50% chance of errors, twice 10 the processing time 5 With 90% chance, 10x P' (relative batch processing time with errors)2 1 0.0 0.2 0.4 0.6 0.8 1.0 E (Error probability) Processing time & error probability Julian M. Kunkel Lecture BigData Analytics, 2016 13 / 20
Overview Hardware Assessing Performance Benchmarks Summary 1 Overview 2 Hardware 3 Assessing Performance 4 Benchmarks 5 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 14 / 20
Overview Hardware Assessing Performance Benchmarks Summary Daytona GraySort Sort at least 100 TB data in files into an output file Generates 500 TB of disk I/O and 200 TB network I/O [12] Drawback: Benchmark is not very compute intense Data record: 10 byte key, 90 byte data Performance Metric: sort rate (TBs/minute) Source: [12] Julian M. Kunkel Lecture BigData Analytics, 2016 15 / 20
Overview Hardware Assessing Performance Benchmarks Summary Assessing Performance Hadoop 102.5 TB in 4,328 seconds [13] Hardware: 2100 nodes, dual 2.3Ghz 6cores, 64 GB memory, 12 HDDs Sort rate: 23.6 GB/s = 11 MB/s per Node ⇒ 1 MB/s per HDD Clearly this is suboptimal! Apache Spark (on disk) 100 TB in 1,406 seconds [13] Hardware: 207 Amazon EC2, 2.5Ghz 32vCores, 244GB memory, 8 SSDs Sort rate: 71 GB/s = 344 MB/s per node Performance assessment Network: 200 TB ⇒ 687 MiB/s per node Optimal: 1.15 GB/s per Node, but we cannot hide (all) communication I/O: 500 TB ⇒ 1.7 GB/s per node = 212 MB/s per SSD Compute: 17 M records/s per node = 0.5 M/s per core = 4700 cycles/record Julian M. Kunkel Lecture BigData Analytics, 2016 16 / 20
Overview Hardware Assessing Performance Benchmarks Summary Executing the Optimal Algorithm on Given Hardware An Utopic Algorihm Assume 200 nodes and random key distribution 1 Read input file once: 100 TB 2 Pipeline reading and start immediately to scatter data (key): 100 TB 3 Receiving node stores data in likely memory region: 500 GB/node Assume this can be pipelined with the receive 4 Output data to local files: 100 TB Estimating optimal runtime Per node: 500 GByte of data; I/O: keep 1.7 GB/s per node 1 Read: 294s 2 Scatter data: 434s ⇒ reading can be hidden 3 One read/write in memory (2 sockets, 3 channels): 6s 4 Write local file region: 294s Total runtime: 434 + 294 = 728 ⇒ 8.2 T/min Julian M. Kunkel Lecture BigData Analytics, 2016 17 / 20
Overview Hardware Assessing Performance Benchmarks Summary In-Memory Computing Aggregating 10 M integers with 1 thread Spark [14]: 160 MB/s, 500 cycles per operation2 Python (raw): 0.44s = 727 MB/s, 123 cycles per operation Numpy: 0.014s = 22.8 GB/s, 4 cycles per operation Source: [14] Invoking external programming languages is even more expensive! 2But it can use multiple threads easily. Julian M. Kunkel Lecture BigData Analytics, 2016 18 / 20
Overview Hardware Assessing Performance Benchmarks Summary Comparing Pig & Hive Benchmark by IBM [16], similar to Apache Benchmark Tests several operations, data set increases 10x in size Set 1: 772 KB, 2: 6.4 MB, 3: 63 MB, 4: 628 MB, 5: 6.2 GB, 6: 62 GB 5 data/compute nodes, configured to run 8 reduce and 11 map tasks Pig time. Source: B. Jakobus (modified), “Table 1: Averaged performance” [16] Hive time. Source: B. Jakobus (modified), “Table 2: Averaged performance” [16] Julian M. Kunkel Lecture BigData Analytics, 2016 19 / 20
Overview Hardware Assessing Performance Benchmarks Summary Summary Goal (user-perspective): optimize the time-to-solution Runtime of queries/scripts is the main contributor Compute in big data clusters is usually overdimensioned Understanding a few hw throughputs helps assessing performance Linear scalability of the architecture is the crucial performance factor Basic performance analysis 1 Estimate the workload 2 Compute the workload throughput per node 3 Compare with hardware capabilities Error model predicts runtime if jobs must be restarted GreySort with Spark utilizes I/O, communication well Computation even with Spark is much slower than Python Different big data solutions exhibit different performance behaviors Julian M. Kunkel Lecture BigData Analytics, 2016 20 / 20
显示前 20 页。下载资料查看完整内容。
暂无评论。快来抢沙发吧！
评论 (0)
暂无评论。快来抢沙发吧！