Overview of Tools in the Hadoop Ecosystem Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2017-02-03 Disclaimer: Big Data software is constantly updated, code samples may be outdated. Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Outline 1 Hadoop Ecosystem 2 User/Admin Interfaces 3 Workflows 4 SQL Tools 5 Other BigData Tools 6 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 2 / 43 Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Hortonworks Screenshot from [40] Julian M. Kunkel Lecture BigData Analytics, 2016 3 / 43 Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Cloudera Enterpise Hadoop Ecosystem [25] Cloudera offers support, services and tools around Hadoop Unified architecture: common infrastructure and data pool for tools Build with open-source Source: [26] Julian M. Kunkel Lecture BigData Analytics, 2016 4 / 43 Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summary Supporting Tools1 Ambari: A Tool for Managing Hadoop Clusters Hue: Manage „BigData“ projects in a browser ZooKeeper: coordination/configuration service for services Sqoop: ETL between HDFS and structured data stores Oozie: Workflow scheduler (schedules/triggers workflows) Falcon: Data governance engine for data pipelines Flume: collecting, aggregating and moving large streaming event data Kafka: publish-subscribe distributed messaging system knox: REST API gateway (for all services) Ranger: Integrate ACL permissions into Hadoop (ecosystem) Slider: YARN application supporting monitoring and dynamic scaling of non-YARN apps 1 https://hadoop.apache.org/ Julian M. Kunkel Lecture BigData Analytics, 2016 5 / 43 Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Summar