Overview of Tools in the Hadoop Ecosystem Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2017-12-22 Disclaimer: Big Data software is constantly updated, code samples may be outdated.
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Outline 1 Hadoop Ecosystem 2 User/Admin Interfaces 3 Workflows 4 SQL Tools 5 Other BigData Tools 6 Machine Learning 7 Summary Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 2 / 49
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Hortonworks Screenshot from [40] Additionally: Hortonworks offers support, service Build with open-source Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 3 / 49
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Cloudera Enterpise Hadoop Ecosystem [25] Cloudera offers support, services and tools around Hadoop Unified architecture: common infrastructure and data pool for tools Build with open-source tools, some own tools for management, encryption Source: [26] Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 4 / 49
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Supporting Tools1 Ambari: A Tool for Managing Hadoop Clusters Hue: Manage „BigData“ projects in a browser ZooKeeper: coordination/configuration service for services Sqoop: ETL between HDFS and structured data stores Oozie: Workflow scheduler (schedules/triggers workflows) Falcon: Data governance engine for data pipelines Flume: collecting, aggregating and moving large streaming event data Kafka: publish-subscribe distributed messaging system knox: REST API gateway (for all services) Ranger: Integrate ACL permissions into Hadoop (ecosystem) Slider: YARN application supporting monitoring and dynamic scaling of non-YARN apps 1https://hadoop.apache.org/ Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 5 / 49
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Ambari: A Tool for Managing Hadoop Clusters Convenient tool managing 10+ Apache tools Supports installation and management Dealing with data dependencies Service startup Monitoring of health and performance (Re)configuration of services Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 6 / 49
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Management with Ambari: Dashboard Screenshot from the WR-cluster Ambari Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 7 / 49
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Management with Ambari: Configuration Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 8 / 49
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Knox: Security for Hadoop [22] REST API Gateway for Hadoop ecosystem services Supports: HDFS, Hcatalog, HBase, Oozie, Hive, Yarn, Storm Supports multiple clusters Provides authentication, federation/SSO, authorization, auditing Enhances security providing central control and protection SSL encryption Authentication: LDAP, Active Directory, Kerberos Authorization: ACL’s (user, group, IP) on service level2 Source: [22] Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 9 / 49
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Example Accesses via the REST API [22] List a HDFS directory 1 curl -i -k -u guest:guest-password -X GET ↪→ ’https://localhost:8443/gateway/sandbox/webhdfs/v1/?op=LISTSTATUS’ Example response 1 HTTP/1.1 200 OK 2 Content-Type: application/json 3 Content-Length: 450 4 Server: Jetty(6.1.26) 5 6 {"FileStatuses":{"FileStatus":[ 7 {"accessTime":0,"blockSize":0,"group":"hdfs","length":0, ↪→ "modificationTime":1350595859762, "owner":"hdfs", "pathSuffix":"apps", ↪→ "permission":"755", "replication":0,"type":"DIRECTORY"}, 8 {"accessTime":0,"blockSize":0, "group":"mapred","length":0, ↪→ "modificationTime":1350595874024, "owner":"mapred","pathSuffix":"mapred", ↪→ "permission":"755", "replication":0,"type":"DIRECTORY"}, 9 ]}} Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 10 / 49
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Hue [12]: Lightweight Web Server for Hadoop Manage BigData projects in a browser Supports: Hadoop ecosystem HDFS, Pig, Sqoop, Hive, Impala, MapReduce, Spark, ... Features Data upload/download Management of HCatalog tables Query editor (Hive, Pig, Impala) Starting and monitoring of jobs Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 11 / 49
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Hue: Lightweight Web Server for Hadoop Monitoring Oozie Workflows (Live system on gethue.com) Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 12 / 49
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Hue: Lightweight Web Server for Hadoop File browser (Live system on gethue.com) Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 13 / 49
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Hue: Lightweight Web Server for Hadoop Query editor (Live system on gethue.com) Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 14 / 49
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Hue: Lightweight Web Server for Hadoop Visualizing query results in diagrams (Live system on gethue.com) Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 15 / 49
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Zeppelin [39] Web-based notebook for interactive data analytics Add code snippets Arrange them Execute them Visualizes results Supports Spark, Scala, Pig, SQL, Python, R, Hive, Shell, ... Collaborative environment for multiple users Can export paragraph links for embedding into a webpage Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 16 / 49
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Zeppelin Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 17 / 49
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary 1 Hadoop Ecosystem 2 User/Admin Interfaces 3 Workflows 4 SQL Tools 5 Other BigData Tools 6 Machine Learning 7 Summary Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 18 / 49
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Oozie [15, 16] Scalable, reliable and extensible workfow scheduler Jobs are DAGs of actions specified in XML workflows Actions: Map-reduce, Pig, Hive, Sqoop, Spark, Shell actions Workflows can be parameterized Triggers notifications via HTTP GET upon start/end of a node/job Automatic user-retry to repeat actions when fixable errors occur Monitors a few runtime metrics upon execution Interfaces: command line tools, web-service and Java APIs Integrates with HCatalog Coordinator jobs trigger start of jobs By time schedules When data becomes available Requires polling of HDFS (1-10 min invervalls) With HCatalog’s publish-subscribe, jobs can be started immediately Can record events for service level agreement Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 19 / 49
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Workflows [16] A workflow application is a ZIP file to be uploaded Includes workflow definition and coordinator job Bundles scripts, JARs, libraries needed for execution Workflow definition is a DAG with control flow and action nodes Control flow: start, end, decision, fork, join Action nodes: whatever to execute Variables/Parameters3 Default values can be defined in a config-default.xml in the ZIP Expression language functions help in parameterization1 Basic functions: timestamp(), trim(), concat(s1, s2) Workflow functions: wf:errorCode(< action node >) Action specific functions: hadoop:counters("mr-node")["FileSystemCounters"]["FILE_BYTES_READ"] Coordinator job is also an XML file 3They are used with with ${NAME/FUNCTION}, e.g., ${timestamp()} Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 20 / 49
显示前 20 页。下载资料查看完整内容。
暂无评论。快来抢沙发吧！
评论 (0)
暂无评论。快来抢沙发吧！