Page 2
Is MapReduce not Good Enough? ◼ Restricted programming model ◼ Only two phases ◼ Job chain for long data flow ◼ Too many lines of code even for simple logic ◼ How many lines do you have for word count? ◼ Programmers are responsible for this 2
Page 3
Pig to the Rescue ◼ High level dataflow language (Pig Latin) ◼ Much simpler than Java ◼ Simplifies the data processing ◼ Puts the operations at the apropriate phases ◼ Chains multiple MR jobs 3
Page 4
How Pig is used in the Industry ◼ At Yahoo, 70% MapReduce jobs are written in Pig ◼ Used to ◼ Process web logs ◼ Build user behavior models ◼ Process images ◼ Data mining ◼ Also used by Twitter, LinkedIn, eBay, AOL, ... 4
Page 5
Motivation by Example ◼ Suppose we have user data in one file, website data in another file. ◼ We need to find the top 5 most visited pages by users aged 18-25 5
Page 9
Wait a minute ◼ How to map the data to records ❑ By default, one line → one record ❑ User can customize the loading process ◼ How to identify attributes and map them to the schema ❑ Delimiter to separate different attributes ❑ By default, delimiter is tab. Customizable. 9
Page 10
MapReduce Vs. Pig cont. ◼ Join in MapReduce ❑ Various algorithms. None of them are easy to implement in MapReduce ❑ Multi-way join is more complicated ❑ Hard to integrate into SPJA workflow 10
Page 11
MapReduce Vs. Pig cont. ◼ Join in Pig ❑ Various algorithms are already available. ❑ Some of them are generic to support multi-way join ❑ No need to consider integration into SPJA workflow. Pig does that for you! A = LOAD 'input/join/A'; B = LOAD 'input/join/B'; C = JOIN A BY $0, B BY $1; DUMP C; 11
Page 12
Pig Latin ◼ Data flow language ❑ Users specify a sequence of operations to process data ❑ More control on the process, compared with declarative language ◼ Various data types are supported ◼ Schema is supported ◼ User-defined functions are supported 12
Page 13
Statement ◼ A statement represents an operation, or a stage in the data flow ◼ Usually a variable is used to represent the result of the statement ◼ Not limited to data processing operations, but also contains filesystem operations 13
Page 14
Schema ◼ User can optionally define the schema of the input data ◼ Once the schema of the source data is given, the schema of the intermediate relation will be induced by Pig 14
Page 15
Schema cont. ◼ Why schema? ❑ Scripts are more readable (by alias) ❑ Help system validate the input ◼ Similar to Database? ❑ Yes. But schema here is optional ❑ Schema is not fixed for a particular dataset, but changable 15
Page 16
Schema cont. ◼ Schema 1 A = LOAD 'input/A' as (name:chararray, age:int); B = FILTER A BY age != 20; ◼ Schema 2 A = LOAD 'input/A' as (name:chararray, age:chararray); B = FILTER A BY age != '20'; ◼ No Schema A = LOAD 'input/A' ; B = FILTER A BY A.$1 != '20'; 16
Page 17
Data Types ◼ Every attribute can always be interpreted as a bytearray, without further type definition ◼ Simple data types ❑ For each attribute ❑ Defined by user in the schema ❑ Int, double, chararray ... ◼ Complex data types ❑ Usually contructed by relational operations ❑ Tuple, bag, map 17
Page 18
Data Types cont. ◼ Type casting ❑ Pig will try to cast data types when type inconsistency is seen. ❑ Warning will be thrown if casting fails. Process still goes on ◼ Validation ❑ Null will replace the inconvertable data type in type casting ❑ User can tell a corrupted record by detecting whether a particular attribute is null 18
Page 20
Operators ◼ Relational Operators ❑ Represent an operation that will be added to the logical plan ❑ LOAD, STORE, FILTER, JOIN, FOREACH...GENERATE 20
Showing the first 20 pages. Download the document to see it all.
Comments (0)
No comments yet. Be the first!