Big Data Analytics Big Data Analytics Lars Schmidt-Thieme Information Systems and Machine Learning Lab (ISMLL) Institute of Computer Science University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 1 / 28
Big Data Analytics Syllabus Tue. 10.4. (1) 0. Introduction A. Parallel Computing Tue. 17.4. (2) A.1 Threads Tue. 24.4. (3) A.2 Message Passing Interface (MPI) Tue. 1.5. (4) A.3 Graphical Processing Units (GPUs) B. Distributed Storage Tue. 8.5. (5) B.1 Distributed File Systems Tue. 15.5. (6) B.2 Partioning of Relational Databases Tue. 22.5. — — Pentecoste Break — Tue. 29.5. (7) B.3 NoSQL Databases C. Distributed Computing Environments Tue. 5.6. (8) C.1 Map-Reduce Tue. 12.6. (9) C.2 Resilient Distributed Datasets (Spark) Tue. 19.6. (10) C.3 Computational Graphs (TensorFlow) D. Distributed Machine Learning Algorithms Tue. 26.6. (11) D.1 Distributed Stochastic Gradient Descent Tue. 3.7. (12) D.2 Distributed Matrix Factorization Tue. 10.7. (13) Questions and Answers Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 1 / 28
Big Data Analytics Outline 1. MPI Basics 2. Point to Point Communication 3. Collective Communication 4. One-sided Communication Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 1 / 28
Big Data Analytics 1. MPI Basics Outline 1. MPI Basics 2. Point to Point Communication 3. Collective Communication 4. One-sided Communication Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 1 / 28
Big Data Analytics 1. MPI Basics The MPI Standard I A standard for parallel and distributed computing I Authored by a consortium of academics and industry. I MPI 1.0 (1994; 236 pages) I MPI 2.0 (1998) I MPI 3.0 (2012) I MPI 3.1 (2015; 868 pages) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 1 / 28
Big Data Analytics 1. MPI Basics The MPI Standard I A standard for parallel and distributed computing I Authored by a consortium of academics and industry. I MPI 1.0 (1994; 236 pages) I MPI 2.0 (1998) I MPI 3.0 (2012) I MPI 3.1 (2015; 868 pages) I Basic concept: I Processes run in parallel I Processes synchronize and exchange data by passing messages from one to another. I http://www.mpi-forum.org/docs/ Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 1 / 28
Big Data Analytics 1. MPI Basics OpenMPI / Compile and Run I OpenMPI: an open source reference implementation of MPI I http://www.openmpi.org I support C++, C and Fortran I see MPI4Py for Python, http://pythonhosted.org/mpi4py/ I compile programs with 1 mpijavac Hello.java I just runs javac with mpi.jar in the classpath. I run programs with 1 mpirun java Hello I option -np N: to start N copies in parallel I option -H h1,h2,h3: to start processes on hosts h1,h2 and h3. I to run on other hosts, one needs: I password-less ssh login to the compute host from the submit host I openmpi installed on both hosts Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 2 / 28
Big Data Analytics 1. MPI Basics Java MPI Skeleton I MPI: service class with static methods and constants: I Init(args): initialize the MPI system I Finalize(): shutdown the MPI system I COMM_WORLD: default communicator (class Intracomm) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 3 / 28
Big Data Analytics 1. MPI Basics Java MPI Skeleton I MPI: service class with static methods and constants: I Init(args): initialize the MPI system I Finalize(): shutdown the MPI system I COMM_WORLD: default communicator (class Intracomm) I The communicator allows interactions with other processes: I getSize(): number of processes in this group. I getRank(): id of this process (between 0 and size-1). I synchronize I exchange data Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 3 / 28
Big Data Analytics 1. MPI Basics Java MPI Skeleton I MPI: service class with static methods and constants: I Init(args): initialize the MPI system I Finalize(): shutdown the MPI system I COMM_WORLD: default communicator (class Intracomm) I The communicator allows interactions with other processes: I getSize(): number of processes in this group. I getRank(): id of this process (between 0 and size-1). I synchronize I exchange data I MPIException: thrown if anything goes wrong. Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 3 / 28
Big Data Analytics 1. MPI Basics Hello World MPI 1 #!/usr/bin/env python3 2 from mpi4py import MPI 3 4 comm = MPI.COMM_WORLD 5 worker = comm.Get_rank() 6 num_workers = comm.Get_size() 7 8 print(’Hello␣world␣from␣worker␣{}␣of␣{}’.format(worker, num_workers)) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 4 / 28
Big Data Analytics 1. MPI Basics Hello World MPI Output Hello world from worker 2 of 4 Hello world from worker 0 of 4 Hello world from worker 3 of 4 Hello world from worker 1 of 4 1 #!/usr/bin/env python3 2 from mpi4py import MPI 3 4 comm = MPI.COMM_WORLD 5 worker = comm.Get_rank() 6 num_workers = comm.Get_size() 7 8 print(’Hello␣world␣from␣worker␣{}␣of␣{}’.format(worker, num_workers)) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 4 / 28
Big Data Analytics 2. Point to Point Communication Outline 1. MPI Basics 2. Point to Point Communication 3. Collective Communication 4. One-sided Communication Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 5 / 28
Big Data Analytics 2. Point to Point Communication Blocking vs. Non-blocking I Send and Receive have to occur paired at a source and a destination process. I Blocking Send and Receive: I process waits/blocks until data was sent or received. I Non-blocking Send and Receive: I returns from the call immediately. I returns a Status object that can be used to I get information if the data has arrived already. I get the data. I wait/block for the data. I Blocking and non-blocking send’s and receive’s can be mixed on both sides. Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 5 / 28
Big Data Analytics 2. Point to Point Communication Blocking Send and Receive Communicator methods: I void send(Object buf, int count, Datatype type, int dest, int tag) I Status recv(Object buf, int count, Datatype type, int source, int tag) I buffers for different types from java.nio I MPI.new<Type>Buffer(int length): create a buffer I Datatype: MPI.INT, MPI.DOUBLE, etc. I dest/source: ID/rank of the destination/source process. I tag: ID to distinguish different messages. I both may throw an MPIException Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 6 / 28
Big Data Analytics 2. Point to Point Communication Computing Pi / Sequential 1 #!/usr/bin/env python3 2 from random import random 3 4 N = 10000000 5 N_circle = 0 6 for i in range(0,N): 7 x = random(); y = random() 8 if x*x + y*y <= 1: 9 N_circle += 1 10 pi = N_circle * 4.0 / N 11 print(’pi␣~␣{}’.format(pi)) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 7 / 28
Big Data Analytics 2. Point to Point Communication Computing Pi / Parallel 1 #!/usr/bin/env python3 2 from mpi4py import MPI 3 from random import random 4 5 comm = MPI.COMM_WORLD 6 worker = comm.Get_rank() 7 num_workers = comm.Get_size() 8 9 N = 10000000 10 N_worker = round(N / num_workers) 11 N_circle = 0 12 for i in range(N_worker): 13 x = random(); y = random() 14 if x*x + y*y <= 1: 15 N_circle += 1 16 17 if worker != 0: 18 data = {’N_circle’: N_circle} 19 comm.send(data, dest=0) 20 else: 21 for w in range(1, num_workers): 22 data = comm.recv() # source=0) 23 N_circle += data[’N_circle’] 24 pi = N_circle * 4.0 / (num_workers * N_worker) 25 print(’pi␣~␣{}’.format(pi)) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 8 / 28
Big Data Analytics 2. Point to Point Communication Computing Pi / Parallel 1 #!/usr/bin/env python3 2 from mpi4py import MPI 3 from random import random 4 5 comm = MPI.COMM_WORLD 6 worker = comm.Get_rank() 7 num_workers = comm.Get_size() 8 9 N = 10000000 10 N_worker = round(N / num_workers) 11 N_circle = 0 12 for i in range(N_worker): 13 x = random(); y = random() implementation runtime [s] 14 if x*x + y*y <= 1: 15 N_circle += 1 sequential 6.363 16 parallel 2.011 17 if worker != 0: 18 data = {’N_circle’: N_circle} (using 4 cores) 19 comm.send(data, dest=0) 20 else: 21 for w in range(1, num_workers): 22 data = comm.recv() # source=0) 23 N_circle += data[’N_circle’] 24 pi = N_circle * 4.0 / (num_workers * N_worker) 25 print(’pi␣~␣{}’.format(pi)) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 8 / 28
Big Data Analytics 2. Point to Point Communication Non-blocking Communication Examples: I Request iSend(Buffer buf, int count, Datatype type, int dest, int tag) I Request iRecv(Buffer buf, int count, Datatype type, int source, int tag) I Request allows to inspect progress on the operation. I testStatus() tests if operation has been completed I returns null, if not, and a Status object, if so. I waitStatus() waits until operation has been completed I returns a Status object Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 9 / 28
Big Data Analytics 2. Point to Point Communication Blocking Communication / Example 1 #!/usr/bin/env python3 2 from mpi4py import MPI 3 from random import randrange 4 from time import sleep 5 6 comm = MPI.COMM_WORLD 7 worker = comm.Get_rank() 8 num_workers = comm.Get_size() 9 10 for i in range(10): 11 dur = randrange(1000) 12 print(’worker␣{}␣,␣round␣{}:␣{}␣ms␣begin’.format(worker, i, dur)) 13 sleep(dur/1000) 14 print(’worker␣{}␣,␣round␣{}:␣{}␣ms␣end’.format(worker, i, dur)) 15 16 if worker != 0: 17 data = { ’dur’: dur } 18 comm.send(data, dest=0) 19 else: 20 totaldur = dur 21 for w in range(1, num_workers): 22 data = comm.recv() 23 # int count = status.getCount(MPI.LONG); 24 totaldur += data[’dur’] 25 print(’--␣total␣round␣{}:␣{}␣ms’.format(i, totaldur)) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany A. Parallel Computing / A.2 Message Passing Interface (MPI) 10 / 28
Hiển thị 20 trang đầu. Tải tài liệu để xem đầy đủ.
Chưa có bình luận nào. Hãy là người đầu tiên!
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!