Overview of Tools in the Hadoop Ecosystem Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2017-02-03 Disclaimer: Big Data software is constantly updated, code samples may be outdated.
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Outline 1 Hadoop Ecosystem 2 User/Admin Interfaces 3 Workflows 4 SQL Tools 5 Other BigData Tools 6 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 2 / 43
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Hortonworks Screenshot from [40] Julian M. Kunkel Lecture BigData Analytics, 2016 3 / 43
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Cloudera Enterpise Hadoop Ecosystem [25] Cloudera offers support, services and tools around Hadoop Unified architecture: common infrastructure and data pool for tools Build with open-source Source: [26] Julian M. Kunkel Lecture BigData Analytics, 2016 4 / 43
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Supporting Tools1 Ambari: A Tool for Managing Hadoop Clusters Hue: Manage „BigData“ projects in a browser ZooKeeper: coordination/configuration service for services Sqoop: ETL between HDFS and structured data stores Oozie: Workflow scheduler (schedules/triggers workflows) Falcon: Data governance engine for data pipelines Flume: collecting, aggregating and moving large streaming event data Kafka: publish-subscribe distributed messaging system knox: REST API gateway (for all services) Ranger: Integrate ACL permissions into Hadoop (ecosystem) Slider: YARN application supporting monitoring and dynamic scaling of non-YARN apps 1https://hadoop.apache.org/ Julian M. Kunkel Lecture BigData Analytics, 2016 5 / 43
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Ambari: A Tool for Managing Hadoop Clusters Convenient tool managing 10+ Apache tools Supports installation and management Dealing with data dependencies Service startup Monitoring of health and performance (Re)configuration of services Julian M. Kunkel Lecture BigData Analytics, 2016 6 / 43
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Management with Ambari: Dashboard Screenshot from the WR-cluster Ambari Julian M. Kunkel Lecture BigData Analytics, 2016 7 / 43
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Management with Ambari: Configuration Julian M. Kunkel Lecture BigData Analytics, 2016 8 / 43
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Knox: Security for Hadoop [22] REST API Gateway for Hadoop ecosystem services Supports: HDFS, Hcatalog, HBase, Oozie, Hive, Yarn, Storm Supports multiple clusters Provides authentication, federation/SSO, authorization, auditing Enhances security providing central control and protection SSL encryption Authentication: LDAP, Active Directory, Kerberos Authorization: ACL’s (user, group, IP) on service level2 Source: [22] Julian M. Kunkel Lecture BigData Analytics, 2016 9 / 43
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Example Accesses via the REST API [22] List a HDFS directory 1 curl -i -k -u guest:guest-password -X GET ↪→ ’https://localhost:8443/gateway/sandbox/webhdfs/v1/?op=LISTSTATUS’ Example response 1 HTTP/1.1 200 OK 2 Content-Type: application/json 3 Content-Length: 450 4 Server: Jetty(6.1.26) 5 6 {"FileStatuses":{"FileStatus":[ 7 {"accessTime":0,"blockSize":0,"group":"hdfs","length":0, ↪→ "modificationTime":1350595859762, "owner":"hdfs", "pathSuffix":"apps", ↪→ "permission":"755", "replication":0,"type":"DIRECTORY"}, 8 {"accessTime":0,"blockSize":0, "group":"mapred","length":0, ↪→ "modificationTime":1350595874024, "owner":"mapred","pathSuffix":"mapred", ↪→ "permission":"755", "replication":0,"type":"DIRECTORY"}, 9 ]}} Julian M. Kunkel Lecture BigData Analytics, 2016 10 / 43
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Hue [12]: Lightweight Web Server for Hadoop Manage BigData projects in a browser Supports: Hadoop ecosystem HDFS, Pig, Sqoop, Hive, Impala, MapReduce, Spark, ... Features Data upload/download Management of HCatalog tables Query editor (Hive, Pig, Impala) Starting and monitoring of jobs Julian M. Kunkel Lecture BigData Analytics, 2016 11 / 43
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Hue: Lightweight Web Server for Hadoop Monitoring Oozie Workflows (Live system on gethue.com) Julian M. Kunkel Lecture BigData Analytics, 2016 12 / 43
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Hue: Lightweight Web Server for Hadoop File browser (Live system on gethue.com) Julian M. Kunkel Lecture BigData Analytics, 2016 13 / 43
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Hue: Lightweight Web Server for Hadoop Query editor (Live system on gethue.com) Julian M. Kunkel Lecture BigData Analytics, 2016 14 / 43
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Hue: Lightweight Web Server for Hadoop Visualizing query results in diagrams (Live system on gethue.com) Julian M. Kunkel Lecture BigData Analytics, 2016 15 / 43
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Zeppelin [39] Web-based notebook for interactive data analytics Add code snippets Arrange them Execute them Visualizes results Supports Spark, Scala, psql, R Collaborative environment Can be embedded into a webpage A bit premature (currently incubating) Julian M. Kunkel Lecture BigData Analytics, 2016 16 / 43
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Zeppelin Julian M. Kunkel Lecture BigData Analytics, 2016 17 / 43
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary 1 Hadoop Ecosystem 2 User/Admin Interfaces 3 Workflows 4 SQL Tools 5 Other BigData Tools 6 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 18 / 43
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Oozie [15, 16] Scalable, reliable and extensible workfow scheduler Jobs are DAGs of actions specified in XML workflows Actions: Map-reduce, Pig, Hive, Sqoop, Spark, Shell actions Workflows can be parameterized Triggers notifications via HTTP GET upon start/end of a node/job Automatic user-retry to repeat actions when fixable errors occur Monitors a few runtime metrics upon execution Interfaces: command line tools, web-service and Java APIs Integrates with HCatalog Coordinator jobs trigger start of jobs By time schedules When data becomes available Requires polling of HDFS (1-10 min invervalls) With HCatalog’s publish-subscribe, jobs can be started immediately Can record events for service level agreement Julian M. Kunkel Lecture BigData Analytics, 2016 19 / 43
Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Workflows [16] A workflow application is a ZIP file to be uploaded Includes workflow definition and coordinator job Bundles scripts, JARs, libraries needed for execution Workflow definition is a DAG with control flow and action nodes Control flow: start, end, decision, fork, join Action nodes: whatever to execute Variables/Parameters 3 Default values can be defined in a config-default.xml in the ZIP Expression language functions help in parameterization1 Basic functions: timestamp(), trim(), concat(s1, s2) Workflow functions: wf:errorCode(< action node >) Action specific functions: hadoop:counters("mr-node")["FileSystemCounters"]["FILE_BYTES_READ"] Coordinator job is also an XML file 3They are used with with ${NAME/FUNCTION}, e.g., ${timestamp()} Julian M. Kunkel Lecture BigData Analytics, 2016 20 / 43
Affichage des 20 premières pages. Téléchargez le document pour le voir en entier.
Aucun commentaire pour le moment. Soyez le premier !
Commentaires (0)
Aucun commentaire pour le moment. Soyez le premier !