Trang 1
E6895 Advanced Big Data and AI Lecture 2: Big Data Foundations Ching-Yung Lin, Ph.D. Adjunct Professor, Dept. of Electrical Engineering and Computer Science January 27th, 2023 E6895 Advanced Big Data Analytics — Lecture 2 © CY Lin, 2023 Columbia University
Trang 2
Milestone 1 Instruction 2 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations
Trang 3
Milestone 1 ◦ Each team will present on either February 3rd or February 10th ◦ Prepare about 10-12 mins of presentation (and expect 3-5 mins of Q&A) ◦ All teams will submit a written Milestone 1 report on February 10th. ◦ Key elements in Milestone 1 presentation: ◦ Task Goal: ◦ What do you want to achieve? ◦ Why is the research and development important? ◦ Is this a new topic that considers challenges of ◦ Volume ◦ Velocity ◦ Variety ◦ Does this topic try to incorporate multi-discipline knowledge? 3 EE6895 Advanced Big Data and AI — Lecture 2 © 2022 CY Lin, Columbia University
Trang 4
Milestone 1 ◦ Literature Survey: ◦ What are the prior arts? What related works were done before? ◦ Which research publications, tools and products may be utilized to build upon them to achieve the goal? ◦ Methodology: ◦ What types of novel algorithms I shall try to invent and implement? ◦ Where will you try to gather the data — Existing dataset? Self- collected dataset? Live dataset? 4 EE6895 Advanced Big Data and AI — Lecture 2 © 2022 CY Lin, Columbia University
Trang 5
Milestone 1 ◦ System: ◦ What will your final system look like — potential backend components and interaction with front end? ◦ How will you create visualization and user interface to help users consume your analysis outcome? ◦ Timeline: ◦ What may you achieve in Milestones 2 and 3, and in the Final Project? Note — it’s good to think about what you want to achieve as complete as possible and study what other people have done. But, like all projects, everything can change when you make progress. Please do not afraid of making bold assumptions and attempt!! 5 EE6895 Advanced Big Data and AI — Lecture 2 © 2022 CY Lin, Columbia University
Trang 6
Big Data Foundations 6 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations
Trang 7
Key Open Source Big Data Foundations 7 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations
Trang 8
Other Important Foundations (visualization and web servers) 8 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations
Trang 9
Reference Foundation (AI Platform) 9 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations
Trang 10
123rf.com 10 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations
Trang 11
123rf.com Where to store data? How to get data in and out? How to manage access of data? 11 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations
Trang 12
Hadoop Key Layers: • Storage • Compute Platform • Shared Services • Compute Frameworks 12 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations
Trang 13
Hadoop Storage is widely used By Tom McCuch, Cloudera, 8/20/2019 13 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations
Trang 14
Hadoop Distributed File System (HDFS) http://hortonworks.com/hadoop/hdfs/ 14 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations
Trang 15
Basic Data Storage Operations on HDFS • Hadoop is designed to work best with a modest number of extremely large files. • Average file sizes ➔ larger than 500MB. • Write Once, Read Often model. • Content of individual files cannot be modified, other than appending new data at the end of the file. • What we can do: – Create a new file – Append content to the end of a file – Delete a file – Rename a file – Modify file attributes like owner 15 © 2022 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Platform
Trang 16
HDFS blocks • File is divided into blocks (default: 64MB) and duplicated in multiple places (default: 3) • Dividing into blocks is normal for a file system. E.g., the default block size in Linux is 4KB. The difference of HDFS is the scale. • Hadoop was designed to operate at the petabyte scale. • Every data block stored in HDFS has its own metadata and needs to be tracked by a central server. 16 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations
Trang 17
Reliable System — High Availability Availability % Downtime Downtime Downtime Downtime per year[note 1] per month per week per day 90% ("one 36.53 days 73.05 16.80 2.40 nine") hours hours hours 99% ("two 3.65 days 7.31 1.68 14.40 nines") hours hours minutes 99.9% ("three 8.77 hours 43.83 10.08 1.44 nines") minutes minutes minutes 99.99% ("four 52.60 4.38 1.01 8.64 nines") minutes minutes minutes seconds There are three principles of systems design in reliability engineering which can help achieve high availability. 1. Elimination of single points of failure. This means adding redundancy to the system so that failure of a component does not mean failure of the entire system. 2. Reliable crossover. In redundant systems, the crossover point itself tends to become a single point of failure. Reliable systems must provide for reliable crossover. 3. Detection of failures as they occur. If the two principles above are observed, then a user may never see a failure – but the maintenance activity must. 17 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations
Trang 18
HDFS blocks • Replication patterns of data blocks in HDFS. • When HDFS stores the replicas of the original blocks across the Hadoop cluster, it tries to ensure that the block replicas are stored in different failure points. 18 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations
Trang 19
Interaction between HDFS components 19 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations
Trang 20
Several useful commands for HDFS • All hadoop commands are invoked by the bin/hadoop script. • % hadoop fsck / -files –blocks: ➔ list the blocks that make up each file in HDFS. • For HDFS, the schema name is hdfs, and for the local file system, the schema name is file. • A file or director in HDFS can be specified in a fully qualified way, such as: hdfs://namenodehost/parent/child or hdfs://namenodehost • The HDFS file system shell command is similar to Linux file commands, with the following general syntax: hadoop hdfs –file_cmd • For instance mkdir runs as: $hadoop hdfs dfs –mkdir /user/directory_name 20 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations
Hiển thị 20 trang đầu. Tải tài liệu để xem đầy đủ.
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!