Stream Processing (with Storm, Spark, Flink) Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2017-01-27 Disclaimer: Big Data software is constantly updated, code samples may be outdated. Overview Spark Streaming Storm Architecture of Storm Programming and Execution Higher-Level APIs Apache Flink Summary Outline 1 Overview 2 Spark Streaming 3 Storm 4 Architecture of Storm 5 Programming and Execution 6 Higher-Level APIs 7 Apache Flink 8 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 2 / 59 Overview Spark Streaming Storm Architecture of Storm Programming and Execution Higher-Level APIs Apache Flink Summary Stream Processing [12] Stream processing paradigm = dataflow programming Restrictions on the programming model: state and window ⇒ No view of the complete data at any time Uniform streaming: Operation is executed on all elements individually Windowing: sliding (overlapping) windows contain multiple elements Stateless vs. stateful (i.e., keep information for multiple elements) Programming Implement kernel functions (operations) and define data dependencies Advantages Pipelining of operations and massive parallelism is possible Data is in memory and often in CPU cache, i.e., in-memory computation Data dependencies of kernels are known and can be dealt at compile time Element Element Element Element stream Julian M. Kunkel Lecture BigData Analytics, 2016 3 / 59 Overview Spark Streaming Storm Architecture of Storm 1 Overview 2 Spark Streaming 3 Storm 4 Architecture of Storm 5 Programming and Execution 6 Higher-Level APIs 7 Apache Flink 8 Summary Julian M. Kunkel Programming and Execution Lecture BigData Analytics, 2016 Higher-Level APIs Apache Flink Summary 4 / 59 Overview Spark Streaming Storm Architecture of Storm Programming and Execution Higher-Level APIs Apache Flink Summary Spark S