Introduction Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2016-10-21 Disclaimer: Big Data software is constantly updated, code samples may be outdated. Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary Outline 1 Introduction 2 BigData Challenges 3 Analytical Workflow 4 Use Cases 5 Programming 6 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 2 / 52 Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary About DKRZ German Climate Computing Center (DKRZ) Partner for Climate Research Maximum Compute Performance. Sophisticated Data Management. Competent Service. Julian M. Kunkel Lecture BigData Analytics, 2016 3 / 52 Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary Scientific Computing Research Group of Prof. Ludwig at the University of Hamburg Embedded into DKRZ Research Analysis of parallel I/O Alternative I/O interfaces I/O & energy tracing tools Data reduction techniques Middleware optimization Cost & energy efficiency Julian M. Kunkel Lecture BigData Analytics, 2016 4 / 52 Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary Lecture Concept of the lecture The lecture is focussing on applying technology and some theory Theory Data models and data processing Statistics and machine learning System architectures Algorithms and data structures Applying technology Learning about various state-of-the art technology Hands-on for understanding the key concepts Languages: R and Python, (Java is important but not used) The domain of big data is overwhelming, especially technology It is a crash course for several topics such as statistics and databases ⇒ it is not the goal to learn and understand every aspect in this lecture Julian M. Kunkel Lecture BigData Analytics, 2016 5 / 52 Introduction