Map Reduce & Hadoop Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2016-11-11 Disclaimer: Big Data software is constantly updated, code samples may be outdated. Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary Outline 1 Hadoop 2 Map Reduce 3 Hadoop 2 4 TEZ Execution Engine 5 Development 6 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 2 / 59 Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary Hadoop Version 1 Hadoop: Framework for scalable processing of data Based on Google’s MapReduce paper Consists of: Hadoop distributed file system (HDFS) MapReduce execution engine: schedules tasks on HDFS Why should we combine storage and execution paradigms? Execution exploits data locality to avoid network data transfer Ship compute to data and not (big) data to compute A complete ecosystem has been layered on top of MapReduce Julian M. Kunkel Lecture BigData Analytics, 2016 3 / 59 Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary Hadoop Distributed File System (HDFS) Goal: Reliable storage on commodity-of-the-shelf hardware Implemented in Java Provides single-writer, multiple-reader concurrency model Has demonstrated scalability to 200 PB of storage and 4500 servers [12] Features Hiearchical namespace (with UNIX/ACL permissions) High availability and automatic recovery Replication of data (pipelined write) Rack-awareness (for performance and high availability) Parallel file access Julian M. Kunkel Lecture BigData Analytics, 2016 4 / 59 Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary Hadoop File System Shell Overview Invoke via: hadoop fs <command> <args> Example: hadoop fs -ls hdfs://abu1.cluster/ HDFS command overview Read files: cat, tail, get, getmerge (useful!) Write files: put, appendToFile, moveFromLocal Permissions: chmod, chgrp, ..., getf