In-Memory Computation with Spark Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2017-01-20 Disclaimer: Big Data software is constantly updated, code samples may be outdated. Concepts Architecture Computation Managing Jobs Examples Higher-Level Abstractions Summary Outline 1 Concepts 2 Architecture 3 Computation 4 Managing Jobs 5 Examples 6 Higher-Level Abstractions 7 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 2 / 53 Concepts Architecture Computation Managing Jobs Examples Higher-Level Abstractions Summary In-Memory Computation/Processing/Analytics [26] In-memory processing: Processing data stored in memory (database) Advantage: No slow I/O necessary ⇒ fast response times Disadvantages Data must fit in the memory of the distributed storage/database Additional persistency (with asynchronous flushing) usually required Fault-tolerance is mandatory BI-Solution: SAP Hana Big data approaches: Apache Spark, Apache Flink Julian M. Kunkel Lecture BigData Analytics, 2016 3 / 53 Concepts Architecture Computation Managing Jobs Examples Higher-Level Abstractions Summary Overview to Spark [10, 12] In-memory processing (and storage) engine Load data from HDFS, Cassandra, HBase Resource management via. YARN, Mesos, Spark, Amazon EC2 ⇒ It can use Hadoop but also works standalone! Task scheduling and monitoring Rich APIs APIs for Java, Scala, Python, R Thrift JDBC/ODBC server for SQL High-level domain-specific tools/languages Advanced APIs simplify typical computation tasks Interactive shells with tight integration spark-shell: Scala (object-oriented functional language running on JVM) pyspark: Python sparkR: R (basic support) Execution in either local (single node) or cluster mode Julian M. Kunkel Lecture BigData Analytics, 2016 4 / 53 Concepts Architecture Computation Managing Jobs Examples Higher-Level Abstractions