Big Data Analytics Big Data Analytics C. Distributed Computing Environments / C.2. Resilient Distributed Datasets: Apache Spark Lars Schmidt-Thieme Information Systems and Machine Learning Lab (ISMLL) Institute for Computer Science University of Hildesheim, Germany Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 35
Big Data Analytics Syllabus Tue. 9.4. (1) 0. Introduction A. Parallel Computing Tue. 16.4. (2) A.1 Threads Tue. 23.4. (3) A.2 Message Passing Interface (MPI) Tue. 30.4. (4) A.3 Graphical Processing Units (GPUs) B. Distributed Storage Tue. 7.5. (5) B.1 Distributed File Systems Tue. 14.5. (6) B.2 Partioning of Relational Databases Tue. 21.5. (7) B.3 NoSQL Databases C. Distributed Computing Environments Tue. 28.5. (8) C.1 Map-Reduce Tue. 4.6. (9) C.2 Resilient Distributed Datasets (Spark) Tue. 11.6. — — Pentecoste Break — Tue. 18.6. (10) C.3 Computational Graphs (TensorFlow) D. Distributed Machine Learning Algorithms Tue. 25.6. (11) D.1 Distributed Stochastic Gradient Descent Tue. 2.7. (12) D.2 Distributed Matrix Factorization Tue. 9.7. (13) Questions and Answers Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 35
Big Data Analytics Outline 1. Introduction 2. Apache Spark 3. Working with Spark 4. MLLib: Machine Learning with Spark Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 35
Big Data Analytics 1. Introduction Outline 1. Introduction 2. Apache Spark 3. Working with Spark 4. MLLib: Machine Learning with Spark Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 35
Big Data Analytics 1. Introduction Core Idea To implement fault-tolerance for primary/original data: I replication: I partition large data into parts I store each part several times on different servers I if one server crashes, the data is still available on the others To implement fault-tolerance for secondary/derived data: I replication Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 35
Big Data Analytics 1. Introduction Core Idea To implement fault-tolerance for primary/original data: I replication: I partition large data into parts I store each part several times on different servers I if one server crashes, the data is still available on the others To implement fault-tolerance for secondary/derived data: I replication or I resilience: I partition large data into parts I for each part, store how it was derived (lineage) I from which parts of its input data I by which operations I if a server crashes, recreate its data on the others Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 35
Big Data Analytics 1. Introduction How to store data derivation? journal I sequence of elementary operations I set an element to a value I remove a value/index from a list I insert a value at an index of a list I . . . I generic: supports all types of operations I but too large I often same size as data itself Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 2 / 35
Big Data Analytics 1. Introduction How to store data derivation? journal I sequence of elementary operations I set an element to a value I remove a value/index from a list I insert a value at an index of a list I . . . I generic: supports all types of operations I but too large I often same size as data itself coarse-grained transformations I just store I the executable code of the transformations and I the input I either primary data or itself an RDD Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 2 / 35
Big Data Analytics 1. Introduction Resilient Distributed Datasets (RDD) Represented by 5 components: 1. partition: a list of parts 2. dependencies: a list of parent RDDs 3. transformation: a function to compute the dataset from its parents 4. partitioner: how elements are assigned to parts 5. preferred locations: which hosts store which parts Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 3 / 35
Big Data Analytics 1. Introduction Resilient Distributed Datasets (RDD) Represented by 5 components: 1. partition: a list of parts 2. dependencies: a list of parent RDDs 3. transformation: a function to compute the dataset from its parents 4. partitioner: how elements are assigned to parts 5. preferred locations: which hosts store which parts distinction into two types of dependencies: I narrow depenencies: each parent part is used to derive at most one part of the dataset I wide dependencies: some parent part is used to derive several parts of the dataset Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 3 / 35
Big Data Analytics 1. Introduction How to cope with expensive operations? checkpointing: I traditionally, I a long process is broken into several steps A, B, C etc. I after each step, the state of the process is saved to disk I if the process crashes within step B, I it does not have to be run from the very beginning I but can be restarted at the beginning of step B reading its state at the end of step A. Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 4 / 35
Big Data Analytics 1. Introduction How to cope with expensive operations? checkpointing: I traditionally, I a long process is broken into several steps A, B, C etc. I after each step, the state of the process is saved to disk I if the process crashes within step B, I it does not have to be run from the very beginning I but can be restarted at the beginning of step B reading its state at the end of step A. I in a distributed scenario, I “saving to disk” is not fault-tolerant I replicate the data instead (distributed checkpointing) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 4 / 35
Big Data Analytics 1. Introduction Caching I RDDs are marketed as technology for in memory cluster computing I derived RDDs are not saved to disks, but kept in (distributed) memory I derived RDDs are saved to disks on request (checkpointing) I allows faster operations Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 5 / 35
Big Data Analytics 1. Introduction Limitations I RDDs are read-only (immutable) I as updating would invalidate them as input for possible derived RDDs I transformations have to be deterministic I otherwise lost parts cannot be recreated the very same way I for stochastic transformations: store random seed For more conceptual details see the original paper I Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Franklin, M.J., Shenker, S. and Stoica, I. 2012. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation (2012). Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 6 / 35
Big Data Analytics 2. Apache Spark Outline 1. Introduction 2. Apache Spark 3. Working with Spark 4. MLLib: Machine Learning with Spark Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 7 / 35
Big Data Analytics 2. Apache Spark Apache Spark Overview an open source framework for large scale data processing and analysis. Main Ideas: I Processing occurs where the data resides I Avoid moving data over the network I Works with the data in memory Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 7 / 35
Big Data Analytics 2. Apache Spark Apache Spark Overview an open source framework for large scale data processing and analysis. Main Ideas: I Processing occurs where the data resides I Avoid moving data over the network I Works with the data in memory Technical details: I Written in Scala I Works seamlessly with Java, Python and R I Developed at UC Berkeley Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 7 / 35
Big Data Analytics 2. Apache Spark Apache Spark Stack I Data platform: Distributed file system /data base I Ex: HDFS, HBase, Cassandra I Execution Environment: single machine or a cluster I Standalone, EC2, YARN, Mesos I Spark Core: Spark API I Spark Ecosystem: libraries of common algorithms I MLLib, GraphX, Streaming Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 8 / 35
Big Data Analytics 2. Apache Spark Apache Spark Ecosystem Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 9 / 35
Big Data Analytics 2. Apache Spark How to use Spark Spark can be used through: I The Spark Shell I Available in Python and Scala I Useful for learning the framework I Spark Applications I Available in Python, Java and Scala I For “serious” large scale processing Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 10 / 35
Hiển thị 20 trang đầu. Tải tài liệu để xem đầy đủ.
Chưa có bình luận nào. Hãy là người đầu tiên!
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!