Pig Optimization and Execution Alan F. Gates @alanfgates © Hortonworks Inc. 2011 Page 1 Who Am I? Pig committer and PMC Member HCatalog committer and mentor Member of ASF and Incubator PMC Co-founder of Hortonworks Author of Programming Pig from O’Reilly Photo credit: Steven Guarnaccia, The Three Little Pigs Who Are You? 3 What Should We Optimize? Minimize scans – Hadoop is still often I/O bound Minimize total number of MR jobs Minimize shuffle size and number of shuffles Avoid spills to disk Reduce or remove skew For small jobs, minimize start-up time 4 Pig Deployment No server, all optimization and planning done on the launching machine Job executes on cluster Pig resides on user machine or gateway Hadoop Cluster User machine Pig Guts (i.e. Pig Architecture), p. 1 Logical Plan Pig Latin Load A = LOAD ‘myfile’ AS (x, y, z); B = GROUP A by x; C = FILTER B by group > 0; D = FOREACH C GENERATE group, COUNT(A); STORE D INTO ‘output’; Group AST Filter Foreach Semantic Checks Store 6 Pig Guts, p. 2 MapReduce Plan Logical Plan Load Load Group Filter Filter Foreach Map Filter Rearrange Group Rule based optimizations Reduce Store 7 Foreach Package Store Foreach Pig Guts, p. 3 MapReduce Plan Map Map Filter Filter Rearrange Rearrange Combine Foreach Physical optimizations Reduce Package Reduce Package Foreach Foreach 8 It would be really cool if… Map Map Reduce Reduce Map Map Reduce Reduce Map Reduce What’s the right join algorithm here? Even with statistics it would be hard to know. Need on the fly execution plan rewrites. 9 Memory Java + memory management = oil + water Java types inefficient memory users (~4x disk size) Very difficult to tell how much memory you are using Originally tried to monitor memory use via MXBeans: FAIL! Now estimate number of records we can hold in memory and spill when we exceed; allow user to tune guess 10 Reducing Spills to Disk Select Map size an