Entering the Zettabyte Age Jeffrey Krone 1 Kilobyte 1,000 bits/byte. 1 megabyte 1,000,000 1 gigabyte 1,000,000,000 1 terabyte 1,000,000,000,000 1 petabyte 1,000,000,000,000,000 1 exabyte 1,000,000,000,000,000,000 1 zettabyte 1,000,000,000,000,000,000,000 1
Presentation Outline • The Evolution of Data • Previous Data Solutions (Shard’ing, SANS) • The Hadoop Ecosystem • Hadoop Case Study – Financial Institution • Founded Zettaset to address some of the deficiencies inherent in a Big Data Infrastructure such as Hadoop • Utilizing Flash Drives with Hadoop • Implications of utilizing Hadoop in a Cloud Computing Environment • Open Problems to be addressed 2
Data volume is growing exponentially • Estimated Global Data Volume: – 2011: 1.8 ZB – 2015: 7.9 ZB • The world's information doubles every two years • Over the next 10 years: – The number of servers worldwide will grow by 10x – Amount of information managed by enterprise data centers will grow by 50x – Number of “files” enterprise data center handle will grow by 75x Source: http://www.emc.com/leadership/programs/digital-universe.htm, which was based on the 2011 IDC Digital Universe Study 3 3
The Evolution of Data – In the past, the most difficult problem for businesses was how to store all the data. – The challenge now is no longer to store large amounts of information, but to understand and analyze this data. – By harnessing this data through sophisticated analytics, and by presenting the key metrics in an efficient, easily discernable fashion, we are afforded unprecedented understanding and insight into our data. 4
The Evolution of Data – Unlocking the true value of this massive amount of information will require new systems for centralizing, aggregating, analyzing, and visualizing these enormous data sets. In particular analyzing and understanding petabytes of structured and unstructured data poses the following unique challenges: • Scalability • Robustness • Diversity • Analytics • Visualization of the Data 5
Past Big Data Solutions • Data Shard’ing – Is a “shared nothing” partitioning scheme for large databases across a number of servers increasing scalability of performance of traditional relational database systems. Essentially, you are breaking your database down into smaller chunks called “shards” and spreading them across a number of distributed servers. The advantages of Sharding is as follows: • Easier to manage • Faster • Reduce Costs 7
Past Big Data Solutions • Data Shard’ing Shortcomings: – Reliability – Distributed Queries – Writing Sharding Code is difficult – No automated way to to perform load balancing – Shards are not synchronously replicated 8
Past Big Data Solutions • SANS – SANS are essentially dedicated, high performance storage networks that transfer data between servers and storage devices, separate from the Local Area Network (usually through fiber channels). – ADVANTAGES • Ability to move large blocks of data • High level of performance and availability • Dynamically balances loads across the network. – DISADVANTAGES • Complex to manage a wide scope of devices • Lack of Standardization • SANs are very expensive 9
BIG Data / Apache Hadoop • Apache Hadoop was developed to overcome the deficiencies mentioned previously of prior storage and analytics architectures (e.g. SANS, Sharding, Parallel Databases, BI Tools). • The Apache Hadoop software library framework allows for distributed processing of large datasets across clusters of computers on commodity hardware. This solution is designed for flexibility and scalability, with an architecture that scales to thousands of servers and petabytes of data. The library detects and handles failures at the application layer, delivering a high-availability service on commodity hardware. 10
Hadoop • Hadoop is a Platform which enables you to store and analyze large volumes of data. • Hadoop is batch oriented (high throughput and low latency) and strongly consistent (data is always available). • Hadoop is best utilized for: – Large scale batch analytics – Unstructured or semi-structured data – Flat files • Hadoop is comprised of two major subsystems – HDFS (File System) – Map Reduce 11
Hadoop • HDFS – Is a file system that supports large files – Files are broken into 64MB+ Blocks that are normally triple replicated. – NameNode • Is essentially the master meta data server. • The NameNode only persists metadata. It does not persist the location of each data node that hosts a block. The metadata is stored persistently on a local disk in the form of two files: – Name Space Image File (FS Image) – Edit Log 12
Hadoop • HDFS – Secondary NameNode • Fetches the FS Image and the Edit Log and merges them together into a single file preventing the Edit Log from becoming too large. • Runs on a separate machine then the Primary NameNode • Maintains an out of date image of the merged Name Node image, which could be utilized if the Primary Name Node fails. – DateNode • The purpose of the DataNode is to retrieve blocks of data when it is told to do so by either the Clients and/or NameNode. • Stores all the raw blocks that represent the files being stored. Periodically reports back to the NameNode with lists of blocks it is storing. – NameNode – Determining Replica Placement • How does the NameNode choose which DataNodes to store replica blocks on? 13
Hadoop • HDFS – File Reads (Process) • Data Nodes are sorted by proximity to the Client (Application making the READ request) • Clients contact DataNodes directly to retrieve data and the NameNode simply guides the Client to the best datanode for each block of data being read. • If an error occurs while reading a block from a DataNode, then the NameNode will try the next closest DataNode to the Client in order to retrieve the block of data. DataNodes that fail are reported back to the NameNode. 14
Hadoop / MapReduce • Is a software framework for writing applications which process very large datasets (multi-terabyte data sets) in parallel on large clusters of machines. Essentially enabling the user to run analytics across large blocks of data. • The MapReduce Framework takes care of scheduling tasks, monitoring them, and re-executing failed tasks. 16
Hadoop / MapReduce • The Map Reduce Framework consists of a single master JobTracker and one slave Task Tracker per cluster node. – Job Tracker • Coordinates all the jobs run on the system scheduling tasks to run on Task Trackers. If a Job fails then the Job Tracker can re-schedule it to another Task Tracker. • Stores in-memory information about every running MapReduce Job • Assigns Tasks to machines in the cluster. • When a Job Tracker assigns a task to a machine, It will prioritize the task to machines with Data Locality. – Task Tracker • Runs Tasks and sends progress reports to the Job Tracker • Has a local directory to create a localized cache and localized job • Code is essentially moved to the data (Map Reduce Jars) instead of visa versa. It is more efficient to move around small jar files then moving around data. Map Reduce Jars are sent to Task Trackers to run locally (i.e. machines where the data is local to the task). 17
Hadoop / MapReduce • MapReduce Example: – Input a raw weather data file that is comma delimited and determine the maximum temperature in the dataset. – MAPPER • Assume the ‘Keys’ of the input file are line offsets between each row of data • Our user defined Mapper Function simply extracts the ‘Year’ and ‘Temperature’ from each row of input data. • The Output of our Map Function is sorted before sending it to the Reduce function. Therefore, each key / value in the intermediate output (year, temperature) is grouped by ‘Year’ and sorted by ‘Temperature’ within that year. – REDUCER • The Reducer function takes the sorted Map(s) inputs and simply iterates through the list of temperatures per year and selects a maximum temperature for each year. – Fault Tolerance • What happens if Map or Reducer Tasks fail? 18
HBase • HBase is a distributed Key / Value store built on top of Hadoop and is tightly integrated with the Hadoop MapReduce framework. HBase is an open source, distributed, column oriented database modeled after Google’s BigTable. • HBase shines with large amounts of data, and read/write concurrency. • Automatic Partitioning – as your table grows, they will automatically be split into regions and distributed across all available nodes. • Does not have indexes. Rows are stored sequentially, as are the columns written to each row. • HBase makes Hadoop useable for real time streaming workloads which the Hadoop File System cannot handle itself. 19
OOZIE • Is a workflow scheduler. It manages data processing jobs (e.g. load data, storing data, analyze data, cleaning data, running map reduce jobs, etc.) for Hadoop. • Users create Directed Acyclical Graphs to model their workflow. Oozie at runtime manages the dependencies and execute the actions when the dependencies identified in DAG are satisfied. • Supports all types of Hadoop jobs and is integrated with the Hadoop stack. • Supports data and time triggers, users can specify execution frequency and can wait for data arrival to trigger an action in the workflow. 20
ZooKeeper • Zookeeper is a stripped down filesystem that exposes a few simple operations and abstractions that enable you to build distributed queues, configuration service, distributed locks, and leader election among a group of peers. – Configuration Service – store and allows applications to retrieve or update configuration files – Distributed Lock – is a mechanism for providing mutual exclusion between a collection of processes. At any one time, only a single process may hold the lock. They can be utilized for leader election, where the leader is the process the holds the lock at any point of time. • Zookeeper is highly available running across a collection of machines. 21
Hiển thị 20 trang đầu. Tải tài liệu để xem đầy đủ.
Chưa có bình luận nào. Hãy là người đầu tiên!
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!