Generating preview...
Map Reduce & Hadoop Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2017-11-10 Disclaimer: Big Data software is constantly updated, code samples may be outdated.
Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary Outline 1 Hadoop 2 Map Reduce 3 Hadoop 2 4 TEZ Execution Engine 5 Development 6 Summary Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 2 / 59
Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary Hadoop Version 1 Hadoop: Framework for scalable processing of data Based on Google’s MapReduce paper Consists of: Hadoop distributed file system (HDFS) MapReduce execution engine: schedules tasks on HDFS Why should we combine storage and execution paradigms? Execution exploits data locality to avoid network data transfer Ship compute to data and not (big) data to compute A complete ecosystem has been layered on top of MapReduce Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 3 / 59
Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary Hadoop Distributed File System (HDFS) Goal: Reliable storage on commodity-of-the-shelf hardware Implemented in Java Provides single-writer, multiple-reader concurrency model Has demonstrated scalability to 200 PB of storage and 4500 servers [12] Features Hiearchical namespace (with UNIX/ACL permissions) High availability and automatic recovery Replication of data (pipelined write) Rack-awareness (for performance and high availability) Parallel file access Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 4 / 59
Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary Hadoop File System Shell Overview Invoke via: hadoop fs <command> <args> Example: hadoop fs -ls hdfs://abu2/ HDFS command overview Read files: cat, tail, get, getmerge (useful!) Write files: put, appendToFile, moveFromLocal Permissions: chmod, chgrp, ..., getfacl Management: ls, rm, rmdir, mkdir, df, du, find, cp, mv, stat, touchz Special commands distcp: map-reduce parallelized copy command between clusters checksum expunge (clear trash) setrep (change replication factor) Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 5 / 59
Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary Architectural Components Namenode: Central manager for the file system namespace Filenames, permissions Information about file block (location) For HA, a secondary NameNode backups data DataNode: Provide storage for objects Directly communicates with other DataNodes for replication TaskTracker: accept and runs map, reduce and shuffle Provides a number of slots for tasks (logical CPUs) A task is tried to be scheduled on a slot of the machine hosting data If all slots are occupied, run the task on the same rack JobTracker: Central manager for running MapReduce jobs For HA, a secondary JobTracker backups data Tools to access and manage the file system (e.g., rebalancing) Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 6 / 59
Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary High-Level Perspective Source: B. Hedlund. [15] Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 7 / 59
Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary System-Level Perspective of Hadoop Clusters Source: B. Hedlund. [15] Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 8 / 59
Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary Parallel Access to Files Files are split into blocks A typical block size is 64 MiB Blocks are distributed across nodes Blocks may be compressed individually MapReduce jobs process input splits Input splits == logical organization of blocks Each input split is processed by one mapper (local processing preferred) Processing for records spanning blocks Skip partial records at the beginning of a split For truncated records, read data from a remote Input splitting (intelligence) depends on the file format File formats that are not splittable must be avoided e.g., XML, JSON Files, compressed text files They enforce sequential read by one mapper Usage of file formats depends on the tools to query data Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 9 / 59
Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary Mapping of Data Blocks to Input Splits [23] Source: [23] Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 10 / 59
Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary Mapping of In-Memory Data Structures to Streams (De)Serialization Data structure ⇒ byte stream ⇒ data structure Serialization is the process creating a byte stream from a data structure De-serialization creates a data structure in memory from the byte stream Byte streams can be transferred via network or stored on block storage Serialization frameworks Provide serialization code for basic types Support writing of datatype-specific serializers Examples: Java: Apache Avro1, Kryo [40] Python: Pickle R: serialize(), unserialize() (functions for objects) Apache Thrift supports multiple languages Requirements: Performance, platform independence 1https://avro.apache.org/ Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 11 / 59
Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary File Formats Text files Delimiters can be choosen This is a simple file.\n Splittable at newlines (only decompressed files) With three lines – \n this is the end. Comma-separated values (CSV) No header supported but JSON records are supported Does not support block compression ’max’, 12.4, 10 \n ’john’, 10.0, 2 \n Sequence files Flat binary file for key/value pairs Supports splitting in HDFS using a synchronization marker Optional block compression for values (and keys) Widely used within Hadoop as internal structure Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 12 / 59
Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary File Formats (2) MapFile [21] Extends the sequence file Provides an index for keys Avro Apache Avro’s serialization system format Self-describing data format2, allows inference of schema Schema can also be changed upon read Enables exchange of data types between tools ⇒ Popular file format for Hadoop ecosystem 2A self-describing format contains information (metadata) needed to understand its contained data, e.g., variable/field names, data types Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 13 / 59
Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary File Formats (3) Record columnar file (RCFile) [10] Partition table into row groups For each group serialize by column-major More efficient to access individual columns Mapping of data structures into files is done by serialization systems Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 14 / 59
Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary The HDFS I/O Path Source: B. Hedlund. [15] Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 15 / 59
Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary The HDFS Write Path Source: B. Hedlund [15] Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 16 / 59
Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary The HDFS Write Path Source: B. Hedlund [15] Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 17 / 59
Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary The HDFS Read Path Source: B. Hedlund [15] Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 18 / 59
Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary Name Node and High Availability Source: B. Hedlund. [15] Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 19 / 59
Hadoop Map Reduce Hadoop 2 TEZ Execution Engine Development Summary Name Node and High Availability Source: B. Hedlund. [15] Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 20 / 59
Showing the first 20 pages. Download the document to see it all.
No comments yet. Be the first!
Comments (0)
No comments yet. Be the first!