Data-intensive Computing Systems Introduction to MapReduce and Hadoop Shivnath Babu Word Count over a Given Set of Web Pages see bob throw see spot run see bob throw see spot run 1 1 1 1 1 1 bob run see spot throw Can we do word count in parallel? 1 1 2 1 1 The MapReduce Framework (pioneered by Google) Automatic Parallel Execution in MapReduce (Google) Handles failures automatically, e.g., restarts tasks if a node fails; runs multiples copies of the same task to avoid a slow task slowing down the whole job MapReduce in Hadoop (1) MapReduce in Hadoop (2) MapReduce in Hadoop (3) Data Flow in a MapReduce Program in Hadoop InputFormat Map function Partitioner Sorting & Merging Combiner Shuffling Merging Reduce function OutputFormat → 1:many Lifecycle of a MapReduce Job Map function Reduce function Run this program as a MapReduce job Lifecycle of a MapReduce Job Map function Reduce function Run this program as a MapReduce job Lifecycle of a MapReduce Job Time Input Splits Map Wave 1 Map Wave 2 Reduce Wave 1 Reduce Wave 2 How are the number of splits, number of map and reduce tasks, memory allocation to tasks, etc., determined? Job Configuration Parameters 190+ parameters in Hadoop Set manually or defaults are used How to sort data using Hadoop? Let us look at a complete example MapReduce program in Hadoop