Entering the Zettabyte Age Jeffrey Krone . 1 Kilobyte 1,000 bits/byte 1 megabyte 1 gigabyte 1 terabyte 1 petabyte 1 exabyte 1 zettabyte 1,000,000 1,000,000,000 1,000,000,000,000 1,000,000,000,000,000 1,000,000,000,000,000,000 1,000,000,000,000,000,000,000 1 Presentation Outline The Evolution of Data Previous Data Solutions (Shard’ing, SANS) The Hadoop Ecosystem Hadoop Case Study – Financial Institution Founded Zettaset to address some of the deficiencies inherent in a Big Data Infrastructure such as Hadoop Utilizing Flash Drives with Hadoop Implications of utilizing Hadoop in a Cloud Computing Environment Open Problems to be addressed 2 Data volume is growing exponentially Estimated Global Data Volume: 2011: 1.8 ZB 2015: 7.9 ZB The world's information doubles every two years Over the next 10 years: The number of servers worldwide will grow by 10x Amount of information managed by enterprise data centers will grow by 50x Number of “files” enterprise data center handle will grow by 75x Source: http://www.emc.com/leadership/programs/digital-universe.htm, which was based on the 2011 IDC Digital Universe Study 3 3 The Evolution of Data In the past, the most difficult problem for businesses was how to store all the data. The challenge now is no longer to store large amounts of information, but to understand and analyze this data. By harnessing this data through sophisticated analytics, and by presenting the key metrics in an efficient, easily discernable fashion, we are afforded unprecedented understanding and insight into our data. 4 The Evolution of Data Unlocking the true value of this massive amount of information will require new systems for centralizing, aggregating, analyzing, and visualizing these enormous data sets. In particular analyzing and understanding petabytes of structured and unstructured data poses the following unique challenges: Scalability Robustness Diversity Analytics Visual