Data Flow Languages & Apache Pig Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2017-01-13 Disclaimer: Big Data software is constantly updated, code samples may be outdated.
Overview Pig Latin Accessing Data Architecture Summary Outline 1 Overview 2 Pig Latin 3 Accessing Data 4 Architecture 5 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 2 / 32
Overview Pig Latin Accessing Data Architecture Summary General Data Model for Dataflow Languages Data Tuple t = (x1, ...,xn) where xi may be of a given type Input/Output = list of tuples (like a table) Typical Operators for Data-Flow Processing Operations process individual tuples Map/Foreach: process or transform data of individual tuples or group transform a tuple: student.Map((matrikel, name) ⇒ (matrikel + 4, name)) count members for each group: groupedStudents.Map((year) ⇒ count()) Filter tuples by comparing a key to a value Operations that require the complete input data Group tuples by a key Sort data according to a key Join multiple relations together Split tuples of a relation into multiple relations (based on a condition) Julian M. Kunkel Lecture BigData Analytics, 2016 3 / 32
Overview Pig Latin Accessing Data Architecture Summary Data Flow Programming Paradigm [68] Focus: data movement and transformation Compare to imperative programming: sequence of commands Models program as directed graph of data flowing between operations Input/output is illustrated as a node Node is an operation, edges are dependencies Operation is run once all inputs become valid An operation might work on a single data element or on the complete data Parallelism is inherently supported by data flow languages States (in the program) Dataflow works best with stateless programs Stateful dataflow graphs support mutable state Data related states, e.g., reductions, may be encoded as data Programming Functional declarative programming model is optimal Example: read(“file.csv”).filter("word” == "big data").reduce(count) Julian M. Kunkel Lecture BigData Analytics, 2016 4 / 32
Overview Pig Latin Accessing Data Architecture Summary Pipe Diagrams1 Goal: Visualize the processing pipeline of data-flows with a schema Optional: Add examples to illustrate processing Elements and diagram concepts Box: Operation e.g., functions, filter, grouping, aggregating, mapping Indicate also changes in schema Arrows show processing order (DAG), joins have two inputs Input (Matrikel, Firstname, Lastname, Female, Birthday) Group by Female Map (Female, count=Count()) Output ⇒(Female, count) 1We will use a variant from [11] Julian M. Kunkel Lecture BigData Analytics, 2016 5 / 32
Overview Pig Latin Accessing Data Architecture Summary Pipe Diagram with Examples Matrikel Firstname Lastname Female Birthdate 22 "Fritz" "Musterman M." false 2000-01-01 23 "Nina" "Musterfrau F." true 2000-01-01 24 "Hans" "Im Glück" false 2001-01-01 Group by Female Matrikel Firstname Lastname Female Birthdate 22 "Fritz" "Musterman M." false 2000-01-01 24 "Hans" "Im Glück" false 2001-01-01 23 "Nina" "Musterfrau F." true 2000-01-01 Map (Female, count=Count()) Female count false 2 true 1 Julian M. Kunkel Lecture BigData Analytics, 2016 6 / 32
Overview Pig Latin Accessing Data Architecture Summary Apache Pig [60, 61, 62] Pig: Infrastructure (language, compiler) for executing big data programs No server (services) required Data is stored on HDFS Uses MapReduce or TEZ execution engine High-level scripting language Pig Latin Describes processing as data flow Compiler parallelizes data flow (into MapReduce / TEZ job) Batch mode and interactive shell (pig) Julian M. Kunkel Lecture BigData Analytics, 2016 7 / 32
Overview Pig Latin Accessing Data Architecture Summary Data Model for Apache Pig [62] Tuple: An ordered set of named fields (data) A field can be a simple type or complex (tuple, bag or map) Fields are referred by name or position ($0 to $n) Bag: Collection of tuples (evtl. with duplicates) Relation: Is a bag (like a table) Data types of fields can be assigned with a schema Not necessarily with a fixed schema Each tuple may have different fields Without defined type, data will be converted if necessary Relations are referred to by name or alias (variable) Example: Loading data with a schema 1 # table with student basic information 2 S = LOAD ’stud.csv’ as (matrikel:int, semester:int, feminine:boolean, name:chararray, ↪→ birthday:datetime); stud.csv 1 4711 5 false "Max Mustermann" 2000-01-01 2 4712 4 true "Nina Musterfrau F." 2000-01-01 Julian M. Kunkel Lecture BigData Analytics, 2016 8 / 32
Overview Pig Latin Accessing Data Architecture Summary 1 Overview 2 Pig Latin 3 Accessing Data 4 Architecture 5 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 9 / 32
Overview Pig Latin Accessing Data Architecture Summary Scripting Language Pig Latin [62] Data-flow oriented imperative programming language Declare execution plan vs. SQL (declare results) Datatypes: basic types, tuples, bags and maps Statement: operator with a named relation as input and output LOAD and STORE operations are exceptions Relations are referred to by name or alias (variable) For computation, additional (arithmetic) operators are provided They are applied to each tuple Preprocessor with parameter substitution and macros (functions) Lazy evaluation for interactive shell Run commands only when output is requested by the user Note: Intermediate relations are stored on temp files on HDFS Julian M. Kunkel Lecture BigData Analytics, 2016 10 / 32
Overview Pig Latin Accessing Data Architecture Summary Relational Operators [62] Input/Output DUMP: Output results on stdout LOAD/STORE: Input/output relations to/from HDFS Subsetting tuples from relations DISTINCT: Removes duplicated tuples FILTER: Select tuples by a a condition SAMPLE: Select random tuples from the relation LIMIT: Limit the number of tuples SPLIT: Partition the relation into relations based on conditions UNION: Merge multiple relations Julian M. Kunkel Lecture BigData Analytics, 2016 11 / 32
Overview Pig Latin Accessing Data Architecture Summary Relational Operators [62] Rearrange tuples GROUP: Group the data based on the values COGROUP: Like group but involves multiple relations ORDER BY: Sort the relation based on fields RANK: To each tuple add the position in the relation (can also apply sort before ranking) Data manipulation FOREACH: Transform tuples of an relation Supports nesting for processing of collections JOIN: Join of multiple relations based on identical field keys CROSS: Cross product of two or more relations CUBE: Aggregates for all combinations of specified groups For n dimensions, this creates 2n aggregates ROLLUP creates n + 1 aggregates based on the hierarchical order Julian M. Kunkel Lecture BigData Analytics, 2016 12 / 32
Overview Pig Latin Accessing Data Architecture Summary Non-relational Operators[62] Arithmetic: +,-,*,/,%, ?:, CASE Boolean: AND, OR, NOT, IN (for collections) Casting: Conversion between data types Comparison (includes regex support) Flatten: Convert tuple elements and bags into tuples Disambiguate: Specifies the relation field, e.g., RELATION_A::f Functions Evaluation functions (reduction): AVG, MIN, MAX, SUM, COUNT, COUNT_STAR (also counts NULL) CONCAT: concatenation TOKENIZE: split string and returns bag String, datetime handling Conversion of strings to types Math functions Julian M. Kunkel Lecture BigData Analytics, 2016 13 / 32
Overview Pig Latin Accessing Data Architecture Summary 1 Overview 2 Pig Latin 3 Accessing Data 4 Architecture 5 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 14 / 32
Overview Pig Latin Accessing Data Architecture Summary Accessing and Manipulating Data with Pig The pig shell is convenient for interactive usage Checks schema and certain language errors Invoke code in other languages via user-defined functions (UDF) Pig Latin can be embedded into, e.g., Python, JavaScript, Java Julian M. Kunkel Lecture BigData Analytics, 2016 15 / 32
Overview Pig Latin Accessing Data Architecture Summary Debugging [62] For testing, run in local mode (pig -x local) For performance analysis, some run statistics are provided Add file names to tuples (e.g., using PigStorage(’,’, ’-tagsource’)) Some operators (with shortcuts) are provided to help debugging Useful operators for debugging ASSERT: Ensure a condition on data (or abort) DUMP (\d): output results on stdout DESCRIBE (\de): show the schema of a relation EXPLAIN (\e): view the execution plans for computation ILLUSTRATE (\i): step-by-step execution of statements Julian M. Kunkel Lecture BigData Analytics, 2016 16 / 32
Overview Pig Latin Accessing Data Architecture Summary Pig Examples: Our Student/Lecture Example stud.csv 1 22,"Fritz","Musterman M.",false,2000-01-01 2 23,"Nina","Musterfrau F.",true,2000-01-01 lecture.csv 1 1;"Big Data";{(22),(23)} 2 2;"Hochleistungsrechnen";{(22)} Pig schema and data loading 1 s = LOAD ’stud.csv’ USING PigStorage(’,’) AS (matrikel:int, name:chararray, ↪→ firstname:chararray, feminine:boolean, birthday:datetime); 2 l = LOAD ’lecture.csv’ USING PigStorage(’;’) AS (id:int, name:chararray, students:bag{T: ↪→ (matrikel:int)}); 3 ASSERT s BY matrikel > 0, ’matrikel must be bigger than 1’; 4 describe s; 5 -- s: {matrikel: int, name: chararray, firstname: chararray, feminine: boolean, ↪→ birthday: datetime} 6 DUMP l; STORE l INTO ’result’ USING PigStorage (’;’); Julian M. Kunkel Lecture BigData Analytics, 2016 17 / 32
Overview Pig Latin Accessing Data Architecture Summary Pig Examples: Our Student/Lecture Example Goal: Identify student names participating in the lecture 1 -- unroll the bag for a join 2 lflat = FOREACH l GENERATE id,name,FLATTEN(students) as matrikel; 3 spart = JOIN lflat by matrikel, s by matrikel; 4 describe spart; 5 -- spart: {lflat::id: int,lflat::name: chararray,lflat::matrikel: int,s::matrikel: ↪→ int,s::name: chararray,s::firstname: chararray,s::feminine: ↪→ boolean,s::birthday: datetime} 6 dump spart; 7 --(2,"Hochleistungsrechnen",22,22,"Fritz","Musterman M.",false, ↪→ 2000-01-01T00:00:00.000+01:00) 8 --(1,"Big Data",22,22,"Fritz","Musterman M.",false,2000-01-01T00:00:00.000+01:00) 9 --(1,"Big Data",23,23,"Nina","Musterfrau F.",true,2000-01-01T00:00:00.000+01:00) Julian M. Kunkel Lecture BigData Analytics, 2016 18 / 32
Overview Pig Latin Accessing Data Architecture Summary Pig Examples: Our Student/Lecture Example Goal: Determine the number of students 1 t = GROUP s ALL; -- we generate only one group containing all tuples 2 c = FOREACH t GENERATE COUNT(s); -- we compute the count for each group 3 -- (2) Goal: Determine the number of participants per lecture 1 c = FOREACH l GENERATE id,COUNT(students) AS participants; 2 -- (1,2) 3 -- (2,1) 4 5 -- alternatively on our flattened table: 6 z = GROUP spart BY id; 7 c = FOREACH z GENERATE group AS id, COUNT(p) AS participants; Julian M. Kunkel Lecture BigData Analytics, 2016 19 / 32
Overview Pig Latin Accessing Data Architecture Summary Pig Examples: Our Student/Lecture Example Goal: Identify female participants in lectures starting with “Big” 1 sf = FILTER s BY (feminine == true); 2 -- Filter the lectures 3 lf = FILTER l BY (name == ’Big.*’); 4 -- Flatten the filtered lectures 5 lfflat = FOREACH lf GENERATE name,FLATTEN(students) as matrikel; 6 7 -- Now join them 8 fp = JOIN lfflat by matrikel, sf by matrikel; 9 -- ("Big Data",23,23,"Nina","Musterfrau F.",true, 2000-01-01T00:00:00.000+01:00) 10 -- only print the name 11 fpn = FOREACH fp GENERATE sf::name; 12 -- ("Nina") Julian M. Kunkel Lecture BigData Analytics, 2016 20 / 32
Hiển thị 20 trang đầu. Tải tài liệu để xem đầy đủ.
Chưa có bình luận nào. Hãy là người đầu tiên!
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!