Big Data Analytics Big Data Analytics C. Distributed Computing Environments / C.2. Resilient Distributed Datasets: Apache Spark Lars Schmidt-Thieme Information Systems and Machine Learning Lab (ISMLL) Institute for Computer Science University of Hildesheim, Germany Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 35 Big Data Analytics Syllabus Tue. 9.4. (1) 0. Introduction Tue. 16.4. Tue. 23.4. Tue. 30.4. (2) (3) (4) A. Parallel Computing A.1 Threads A.2 Message Passing Interface (MPI) A.3 Graphical Processing Units (GPUs) Tue. 7.5. Tue. 14.5. Tue. 21.5. (5) (6) (7) B. Distributed Storage B.1 Distributed File Systems B.2 Partioning of Relational Databases B.3 NoSQL Databases Tue. 28.5. Tue. 4.6. Tue. 11.6. Tue. 18.6. (8) (9) (10) C. Distributed Computing Environments C.1 Map-Reduce C.2 Resilient Distributed Datasets (Spark) Pentecoste Break — C.3 Computational Graphs (TensorFlow) Tue. 25.6. Tue. 2.7. (11) (12) D. Distributed Machine Learning Algorithms D.1 Distributed Stochastic Gradient Descent D.2 Distributed Matrix Factorization Tue. 9.7. (13) Questions and Answers Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 35 Big Data Analytics Outline 1. Introduction 2. Apache Spark 3. Working with Spark 4. MLLib: Machine Learning with Spark Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 35 Big Data Analytics 1. Introduction Outline 1. Introduction 2. Apache Spark 3. Working with Spark 4. MLLib: Machine Learning with Spark Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 35 Big Data Analytics 1. Introduction Core Idea To implement fault-tolerance for primary/original data: I replication: I partition large data into parts I store each part several times on different servers