Big Data Analytics Big Data Analytics B. Distributed Storage / B.1 Distributed File Systems Lars Schmidt-Thieme Information Systems and Machine Learning Lab (ISMLL) Institute for Computer Science University of Hildesheim, Germany Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 29
Big Data Analytics Syllabus Tue. 9.4. (1) 0. Introduction A. Parallel Computing Tue. 16.4. (2) A.1 Threads Tue. 23.4. (3) A.2 Message Passing Interface (MPI) Tue. 30.4. (4) A.3 Graphical Processing Units (GPUs) B. Distributed Storage Tue. 7.5. (5) B.1 Distributed File Systems Tue. 14.5. (6) B.2 Partioning of Relational Databases Tue. 21.5. (7) B.3 NoSQL Databases C. Distributed Computing Environments Tue. 28.5. (8) C.1 Map-Reduce Tue. 4.6. — — Pentecoste Break — Tue. 11.6. (9) C.2 Resilient Distributed Datasets (Spark) Tue. 18.6. (10) C.3 Computational Graphs (TensorFlow) D. Distributed Machine Learning Algorithms Tue. 25.6. (11) D.1 Distributed Stochastic Gradient Descent Tue. 2.7. (12) D.2 Distributed Matrix Factorization Tue. 9.7. (13) Questions and Answers Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 29
Big Data Analytics Outline 1. Why do we need a Distributed File System? 2. What is a Distributed File System? 3. GFS and HDFS 4. Hadoop Distributed File System (HDFS) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 29
Big Data Analytics 1. Why do we need a Distributed File System? Outline 1. Why do we need a Distributed File System? 2. What is a Distributed File System? 3. GFS and HDFS 4. Hadoop Distributed File System (HDFS) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 29
Big Data Analytics 1. Why do we need a Distributed File System? Why do we need a Distributed File System? Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 29
Big Data Analytics 1. Why do we need a Distributed File System? Why do we need a Distributed File System? Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 29
Big Data Analytics 1. Why do we need a Distributed File System? Why do we need a Distributed File System? Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 29
Big Data Analytics 1. Why do we need a Distributed File System? Why do we need a Distributed File System? Read??? - Whole File? - Specific part? Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 29
Big Data Analytics 1. Why do we need a Distributed File System? Why do we need a Distributed File System? Write??? - Append to the end of the file? - Insert content in the middle? Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 29
Big Data Analytics 1. Why do we need a Distributed File System? Why do we need a Distributed File System? We want to: I Read large data fast I scalability: perform multiple parallel reads and writes Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 2 / 29
Big Data Analytics 1. Why do we need a Distributed File System? Why do we need a Distributed File System? We want to: I Read large data fast I scalability: perform multiple parallel reads and writes I Have the files available even if one computer crashes I fault tolerance: replication Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 2 / 29
Big Data Analytics 1. Why do we need a Distributed File System? Why do we need a Distributed File System? We want to: I Read large data fast I scalability: perform multiple parallel reads and writes I Have the files available even if one computer crashes I fault tolerance: replication I Hide parallelization and distribution details I transparency: clients can access it like a local filesystem Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 2 / 29
Big Data Analytics 1. Why do we need a Distributed File System? Data Transfer Rates 10,000 14 m 1000 2.3 h Infiniband 300 RAM DDR4-3200 PCIe 100 23 h 10 Ethernet 10 Infiniband 8 USB3 9.7 d SSD speed [GBit/s] 1 HDD Ethernet 1 97 d Bluray×12 duration to transfer 1 PB 0.1 2.7 yr 4k video 0.01 27 yr disks local area networks buses type Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 3 / 29
Big Data Analytics 2. What is a Distributed File System? Outline 1. Why do we need a Distributed File System? 2. What is a Distributed File System? 3. GFS and HDFS 4. Hadoop Distributed File System (HDFS) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 4 / 29
Big Data Analytics 2. What is a Distributed File System? What is a Distributed File System? File Namespace / /home /home/lucas /home/lucas/big_file Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 4 / 29
Big Data Analytics 2. What is a Distributed File System? What is a Distributed File System? File Namespace / /home /home/john /home/john/big_file Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 5 / 29
Big Data Analytics 2. What is a Distributed File System? Examples I Windows Distributed File System (DFS; Microsoft, 1996) I GFS (Google, 2003) I Lustre (Cluster File Systems, 2003) I BeeGFS (Fraunhofer, 2005) I HDFS (Apache Software Foundation, 2006) I GlusterFS (Red Hat, 2007) I Ceph (Inktank/Red Hat, 2007) I MooseFS (Core Technology/Gemius, 2008) I MapR File System (MapR Technologies, 2010) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 6 / 29
Big Data Analytics 2. What is a Distributed File System? Components A typical distributed filesystem contains the following components I Clients - they interface with the user Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 7 / 29
Big Data Analytics 2. What is a Distributed File System? Components A typical distributed filesystem contains the following components I Clients - they interface with the user I Chunk nodes - stores chunks of files Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 7 / 29
Big Data Analytics 2. What is a Distributed File System? Components A typical distributed filesystem contains the following components I Clients - they interface with the user I Chunk nodes - stores chunks of files I Master node - stores which parts of each file are on which chunk node Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 7 / 29
Showing the first 20 pages. Download the document to see it all.
No comments yet. Be the first!
Comments (0)
No comments yet. Be the first!