E6893 Big Data Analytics Lecture 2: Big Data Analytics Platforms Ching-Yung Lin, Ph.D. Adjunct Professor, Depts. of Electrical Engineering and Computer Science September 15th, 2023 1 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
Remind -- Apache Hadoop The Apache™ Hadoop® project develops open-source software for reliable, scalable, distributed computing. The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures. The project includes these modules: • Hadoop Common: The common utilities that support the other Hadoop modules. • Hadoop Distributed File System (HDFS™): A distributed file system that provides high- throughput access to application data. • Hadoop YARN: A framework for job scheduling and cluster resource management. • Hadoop MapReduce: A YARN-based system for parallel processing of large data sets. http://hadoop.apache.org 2 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
Remind -- Hadoop-related Apache Projects • Ambari™: A web-based tool for provisioning, managing, and monitoring Hadoop clusters.It also provides a dashboard for viewing cluster health and ability to view MapReduce, Pig and Hive applications visually. • Avro™: A data serialization system. • Cassandra™: A scalable multi-master database with no single points of failure. • Chukwa™: A data collection system for managing large distributed systems. • HBase™: A scalable, distributed database that supports structured data storage for large tables. • Hive™: A data warehouse infrastructure that provides data summarization and ad hoc querying. • Mahout™: A Scalable machine learning and data mining library. • Pig™: A high-level data-flow language and execution framework for parallel computation. • Spark™: A fast and general compute engine for Hadoop data. Spark provides a simple and expressive programming model that supports a wide range of applications, including ETL, machine learning, stream processing, and graph computation. • Tez™: A generalized data-flow programming framework, built on Hadoop YARN, which provides a powerful and flexible engine to execute an arbitrary DAG of tasks to process data for both batch and interactive use-cases. • ZooKeeper™: A high-performance coordination service for distributed applications. 3 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
Four distinctive layers of Hadoop 4 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
Common Use Cases for Big Data in Hadoop • Log Data Analysis – most common, fits perfectly for HDFS scenario: Write once & Read often. • Data Warehouse Modernization • Fraud Detection • Risk Modeling • Social Sentiment Analysis • Image Classification • Graph Analysis • Beyond D. deRoos et al, Hadoop for Dummies, John Wiley & Sons, 2014 5 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
Example: Business Value of Log Analysis – “Struggle Detection” D. deRoos et al, Hadoop for Dummies, John Wiley & Sons, 2014 6 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
Remind -- MapReduce example http://www.alex-hanna.com 7 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
MapReduce Process on User Behavior via Log Analysis D. deRoos et al, Hadoop for Dummies, John Wiley & Sons, 2014 8 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
Setting Up the Hadoop Environment • Local (standalone) mode • Pseudo-distributed mode • Fully-distributed mode 9 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
Data Storage Operations on HDFS • Hadoop is designed to work best with a modest number of extremely large files. • Average file sizes ➔ larger than 500MB. • Write Once, Read Often model. • Content of individual files cannot be modified, other than appending new data at the end of the file. • What we can do: – Create a new file – Append content to the end of a file – Delete a file – Rename a file – Modify file attributes like owner 10 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
Remind -- Hadoop Distributed File System (HDFS) http://hortonworks.com/hadoop/hdfs/ 11 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
HDFS blocks • File is divided into blocks (default: 64MB) and duplicated in multiple places (default: 3) • Dividing into blocks is normal for a file system. E.g., the default block size in Linux is 4KB. The difference of HDFS is the scale. • Hadoop was designed to operate at the petabyte scale. • Every data block stored in HDFS has its own metadata and needs to be tracked by a central server. 12 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
HDFS blocks • Replication patterns of data blocks in HDFS. • When HDFS stores the replicas of the original blocks across the Hadoop cluster, it tries to ensure that the block replicas are stored in different failure points. 13 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
HDFS is a User-Space-Level file system 14 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
Interaction between HDFS components 15 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
HDFS Federation • Before Hadoop 2.0, NameNode was a single point of failure and operation limitation. • Before Hadoop 2, Hadoop clusters usually have fewer clusters that were able to scale beyond 3,000 or 4,000 nodes. • Multiple NameNodes can be used in Hadoop 2.x. (HDFS High Availability feature – one is in an Active state, the other one is in a Standby state). http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithNFS.html 16 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
High Availability of the NameNodes • Active NameNode • Standby NameNode – keeping the state of the block locations and block metadata in memory -> HDFS checkpointing responsibilities. • JournalNode – if a failure occurs, the Standby Node reads all completed journal entries to ensure the new Active NameNode is fully consistent with the state of cluster. • Zookeeper – provides coordination and configuration services for distributed systems. 17 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
Several useful commands for HDFS • All hadoop commands are invoked by the bin/hadoop script. • % hadoop fsck / -files –blocks: ➔ list the blocks that make up each file in HDFS. • For HDFS, the schema name is hdfs, and for the local file system, the schema name is file. • A file or director in HDFS can be specified in a fully qualified way, such as: hdfs://namenodehost/parent/child or hdfs://namenodehost • The HDFS file system shell command is similar to Linux file commands, with the following general syntax: hadoop hdfs –file_cmd • For instance mkdir runs as: $hadoop hdfs dfs –mkdir /user/directory_name 18 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
Several useful commands for HDFS -- II 19 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
YARN • YARN – Yet Another Resource Negotiator: – A Tool that enables the other processing frameworks to run on Hadoop. – A general-purpose resource management facility that can schedule and assign CPU cycles and memory (and in the future, other resources, such as network bandwidth) from the Hadoop cluster to waiting applications. ➔YARN has converted Hadoop from simply a batch processing engine into a platform for many different modes of data processing, from traditional batch to interactive queries to streaming analysis. 20 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 2: Big Data Platform
Hiển thị 20 trang đầu. Tải tài liệu để xem đầy đủ.
Chưa có bình luận nào. Hãy là người đầu tiên!
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!