Big Data Analytics Hadoop and Spark Shelly Garion, Ph.D. IBM Research Haifa 1 What is Big Data? 2 What is Big Data? Big data usually includes data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process data within a tolerable elapsed time. Big data "size" is a constantly moving target, as of 2012 ranging from a few dozen terabytes to many petabytes of data. Big data is a set of techniques and technologies that require new forms of integration to uncover large hidden values from large datasets that are diverse, complex, and of a massive scale. (From Wikipedia) 3 What is Big Data? Volume. Value. Variety. Velocity - the speed of generation of data. Variability - the inconsistency which can be shown by the data at times. Veracity - the quality of the data being captured can vary greatly. Complexity - data management can become a very complex process, especially when large volumes of data come from multiple sources. (From Wikipedia) 4 How to analyze Big Data? 5 Map/Reduce 6 Map/Reduce MapReduce is a framework for processing parallelizable problems across huge datasets using a large number of computers (nodes), collectively referred to as a cluster. Map step: Each worker node applies the "map()" function to the local data, and writes the output to a temporary storage. A master node orchestrates that for redundant copies of input data, only one is processed. Shuffle step: Worker nodes redistribute data based on the output keys (produced by the "map()" function), such that all data belonging to one key is located on the same worker node. Reduce step: Worker nodes now process each group of output data, per key, in parallel. (From Wikipedia) 7 Basic Example: Word Count (Spark & Python) 8 Basic Example: Word Count (Spark & Scala) 9 Map/Reduce – Parallel Computing No dependency among data Data can be split into equal-size chunks Each proces