Advanced Big Data Analytics - Phân tích dữ liệu lớn nâng cao (Lecture 2)

UN
Uni24h
0 lượt tải
99
0

Nội dung tài liệu

Trang 1

E6895 Advanced Big Data and AI Lecture 2: Big Data Foundations Ching-Yung Lin, Ph.D. Adjunct Professor, Dept. of Electrical Engineering and Computer Science January 27th, 2023 E6895 Advanced Big Data Analytics — Lecture 2 © CY Lin, 2023 Columbia University

Trang 2

Milestone 1 Instruction 2 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations

Trang 3

Milestone 1 ◦ Each team will present on either February 3rd or February 10th ◦ Prepare about 10-12 mins of presentation (and expect 3-5 mins of Q&A) ◦ All teams will submit a written Milestone 1 report on February 10th. ◦ Key elements in Milestone 1 presentation: ◦ Task Goal: ◦ What do you want to achieve? ◦ Why is the research and development important? ◦ Is this a new topic that considers challenges of ◦ Volume ◦ Velocity ◦ Variety ◦ Does this topic try to incorporate multi-discipline knowledge? 3 EE6895 Advanced Big Data and AI — Lecture 2 © 2022 CY Lin, Columbia University

Trang 4

Milestone 1 ◦ Literature Survey: ◦ What are the prior arts? What related works were done before? ◦ Which research publications, tools and products may be utilized to build upon them to achieve the goal? ◦ Methodology: ◦ What types of novel algorithms I shall try to invent and implement? ◦ Where will you try to gather the data — Existing dataset? Self- collected dataset? Live dataset? 4 EE6895 Advanced Big Data and AI — Lecture 2 © 2022 CY Lin, Columbia University

Trang 5

Milestone 1 ◦ System: ◦ What will your final system look like — potential backend components and interaction with front end? ◦ How will you create visualization and user interface to help users consume your analysis outcome? ◦ Timeline: ◦ What may you achieve in Milestones 2 and 3, and in the Final Project? Note — it’s good to think about what you want to achieve as complete as possible and study what other people have done. But, like all projects, everything can change when you make progress. Please do not afraid of making bold assumptions and attempt!! 5 EE6895 Advanced Big Data and AI — Lecture 2 © 2022 CY Lin, Columbia University

Trang 6

Big Data Foundations 6 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations

Trang 7

Key Open Source Big Data Foundations 7 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations

Trang 8

Other Important Foundations (visualization and web servers) 8 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations

Trang 9

Reference Foundation (AI Platform) 9 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations

Trang 10

123rf.com 10 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations

Trang 11

123rf.com Where to store data? How to get data in and out? How to manage access of data? 11 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations

Trang 12

Hadoop Key Layers: • Storage • Compute Platform • Shared Services • Compute Frameworks 12 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations

Trang 13

Hadoop Storage is widely used By Tom McCuch, Cloudera, 8/20/2019 13 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations

Trang 14

Hadoop Distributed File System (HDFS) http://hortonworks.com/hadoop/hdfs/ 14 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations

Trang 15

Basic Data Storage Operations on HDFS • Hadoop is designed to work best with a modest number of extremely large files. • Average file sizes ➔ larger than 500MB. • Write Once, Read Often model. • Content of individual files cannot be modified, other than appending new data at the end of the file. • What we can do: – Create a new file – Append content to the end of a file – Delete a file – Rename a file – Modify file attributes like owner 15 © 2022 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Platform

Trang 16

HDFS blocks • File is divided into blocks (default: 64MB) and duplicated in multiple places (default: 3) • Dividing into blocks is normal for a file system. E.g., the default block size in Linux is 4KB. The difference of HDFS is the scale. • Hadoop was designed to operate at the petabyte scale. • Every data block stored in HDFS has its own metadata and needs to be tracked by a central server. 16 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations

Trang 17

Reliable System — High Availability Availability % Downtime Downtime Downtime Downtime per year[note 1] per month per week per day 90% ("one 36.53 days 73.05 16.80 2.40 nine") hours hours hours 99% ("two 3.65 days 7.31 1.68 14.40 nines") hours hours minutes 99.9% ("three 8.77 hours 43.83 10.08 1.44 nines") minutes minutes minutes 99.99% ("four 52.60 4.38 1.01 8.64 nines") minutes minutes minutes seconds There are three principles of systems design in reliability engineering which can help achieve high availability. 1. Elimination of single points of failure. This means adding redundancy to the system so that failure of a component does not mean failure of the entire system. 2. Reliable crossover. In redundant systems, the crossover point itself tends to become a single point of failure. Reliable systems must provide for reliable crossover. 3. Detection of failures as they occur. If the two principles above are observed, then a user may never see a failure – but the maintenance activity must. 17 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations

Trang 18

HDFS blocks • Replication patterns of data blocks in HDFS. • When HDFS stores the replicas of the original blocks across the Hadoop cluster, it tries to ensure that the block replicas are stored in different failure points. 18 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations

Trang 19

Interaction between HDFS components 19 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations

Trang 20

Several useful commands for HDFS • All hadoop commands are invoked by the bin/hadoop script. • % hadoop fsck / -files –blocks: ➔ list the blocks that make up each file in HDFS. • For HDFS, the schema name is hdfs, and for the local file system, the schema name is file. • A file or director in HDFS can be specified in a fully qualified way, such as: hdfs://namenodehost/parent/child or hdfs://namenodehost • The HDFS file system shell command is similar to Linux file commands, with the following general syntax: hadoop hdfs –file_cmd • For instance mkdir runs as: $hadoop hdfs dfs –mkdir /user/directory_name 20 © 2023 CY Lin, Columbia University E6895 Advanced Big Data and AI – Lecture 2: Big Data Foundations

Hiển thị 20 trang đầu. Tải tài liệu để xem đầy đủ.

Bình luận (0)

Đăng nhập để tham gia bình luận

Chưa có bình luận nào. Hãy là người đầu tiên!