Đang tạo bản xem trước...
Is MapReduce not Good Enough? ◼ Restricted programming model ◼ Only two phases ◼ Job chain for long data flow ◼ Too many lines of code even for simple logic ◼ How many lines do you have for word count? ◼ Programmers are responsible for this 2
Pig to the Rescue ◼ High level dataflow language (Pig Latin) ◼ Much simpler than Java ◼ Simplifies the data processing ◼ Puts the operations at the apropriate phases ◼ Chains multiple MR jobs 3
How Pig is used in the Industry ◼ At Yahoo, 70% MapReduce jobs are written in Pig ◼ Used to ◼ Process web logs ◼ Build user behavior models ◼ Process images ◼ Data mining ◼ Also used by Twitter, LinkedIn, eBay, AOL, ... 4
Motivation by Example ◼ Suppose we have user data in one file, website data in another file. ◼ We need to find the top 5 most visited pages by users aged 18-25 5
Wait a minute ◼ How to map the data to records ❑ By default, one line → one record ❑ User can customize the loading process ◼ How to identify attributes and map them to the schema ❑ Delimiter to separate different attributes ❑ By default, delimiter is tab. Customizable. 9
MapReduce Vs. Pig cont. ◼ Join in MapReduce ❑ Various algorithms. None of them are easy to implement in MapReduce ❑ Multi-way join is more complicated ❑ Hard to integrate into SPJA workflow 10
MapReduce Vs. Pig cont. ◼ Join in Pig ❑ Various algorithms are already available. ❑ Some of them are generic to support multi-way join ❑ No need to consider integration into SPJA workflow. Pig does that for you! A = LOAD 'input/join/A'; B = LOAD 'input/join/B'; C = JOIN A BY $0, B BY $1; DUMP C; 11
Pig Latin ◼ Data flow language ❑ Users specify a sequence of operations to process data ❑ More control on the process, compared with declarative language ◼ Various data types are supported ◼ Schema is supported ◼ User-defined functions are supported 12
Statement ◼ A statement represents an operation, or a stage in the data flow ◼ Usually a variable is used to represent the result of the statement ◼ Not limited to data processing operations, but also contains filesystem operations 13
Schema ◼ User can optionally define the schema of the input data ◼ Once the schema of the source data is given, the schema of the intermediate relation will be induced by Pig 14
Schema cont. ◼ Why schema? ❑ Scripts are more readable (by alias) ❑ Help system validate the input ◼ Similar to Database? ❑ Yes. But schema here is optional ❑ Schema is not fixed for a particular dataset, but changable 15
Schema cont. ◼ Schema 1 A = LOAD 'input/A' as (name:chararray, age:int); B = FILTER A BY age != 20; ◼ Schema 2 A = LOAD 'input/A' as (name:chararray, age:chararray); B = FILTER A BY age != '20'; ◼ No Schema A = LOAD 'input/A' ; B = FILTER A BY A.$1 != '20'; 16
Data Types ◼ Every attribute can always be interpreted as a bytearray, without further type definition ◼ Simple data types ❑ For each attribute ❑ Defined by user in the schema ❑ Int, double, chararray ... ◼ Complex data types ❑ Usually contructed by relational operations ❑ Tuple, bag, map 17
Data Types cont. ◼ Type casting ❑ Pig will try to cast data types when type inconsistency is seen. ❑ Warning will be thrown if casting fails. Process still goes on ◼ Validation ❑ Null will replace the inconvertable data type in type casting ❑ User can tell a corrupted record by detecting whether a particular attribute is null 18
Operators ◼ Relational Operators ❑ Represent an operation that will be added to the logical plan ❑ LOAD, STORE, FILTER, JOIN, FOREACH...GENERATE 20
Hiển thị 20 trang đầu. Tải tài liệu để xem đầy đủ.
Chưa có bình luận nào. Hãy là người đầu tiên!
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!