Data Flow Languages & Apache Pig Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2017-01-13 Disclaimer: Big Data software is constantly updated, code samples may be outdated. Overview Pig Latin Accessing Data Architecture Summary Outline 1 Overview 2 Pig Latin 3 Accessing Data 4 Architecture 5 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 2 / 32 Overview Pig Latin Accessing Data Architecture Summary General Data Model for Dataflow Languages Data Tuple t = (x1 , ..., xn ) where xi may be of a given type Input/Output = list of tuples (like a table) Typical Operators for Data-Flow Processing Operations process individual tuples Map/Foreach: process or transform data of individual tuples or group transform a tuple: student.Map((matrikel, name) ⇒ (matrikel + 4, name)) count members for each group: groupedStudents.Map((year) ⇒ count()) Filter tuples by comparing a key to a value Operations that require the complete input data Group tuples by a key Sort data according to a key Join multiple relations together Split tuples of a relation into multiple relations (based on a condition) Julian M. Kunkel Lecture BigData Analytics, 2016 3 / 32 Overview Pig Latin Accessing Data Architecture Summary Data Flow Programming Paradigm [68] Focus: data movement and transformation Compare to imperative programming: sequence of commands Models program as directed graph of data flowing between operations Input/output is illustrated as a node Node is an operation, edges are dependencies Operation is run once all inputs become valid An operation might work on a single data element or on the complete data Parallelism is inherently supported by data flow languages States (in the program) Dataflow works best with stateless programs Stateful dataflow graphs support mutable state Data related states, e.g., reductions, may be encoded as data Programming Func