Overview of Tools in the Hadoop Ecosystem Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2017-12-22 Disclaimer: Big Data software is constantly updated, code samples may be outdated. Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Outline 1 Hadoop Ecosystem 2 User/Admin Interfaces 3 Workflows 4 SQL Tools 5 Other BigData Tools 6 Machine Learning 7 Summary Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 2 / 49 Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Hortonworks Screenshot from [40] Additionally: Hortonworks offers support, service Build with open-source Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 3 / 49 Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Cloudera Enterpise Hadoop Ecosystem [25] Cloudera offers support, services and tools around Hadoop Unified architecture: common infrastructure and data pool for tools Build with open-source tools, some own tools for management, encryption Source: [26] Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 4 / 49 Hadoop Ecosystem User/Admin Interfaces Workflows SQL Tools Other BigData Tools Machine Learning Summary Supporting Tools1 Ambari: A Tool for Managing Hadoop Clusters Hue: Manage „BigData“ projects in a browser ZooKeeper: coordination/configuration service for services Sqoop: ETL between HDFS and structured data stores Oozie: Workflow scheduler (schedules/triggers workflows) Falcon: Data governance engine for data pipelines Flume: collecting, aggregating and moving large streaming event data Kafka: publish-subscribe distributed messaging system knox: REST API gateway (for all services) Ranger: Integrate ACL permissions into Hadoop (ecosystem) Slider: YARN application supporti