Page 1
Introduction Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2016-10-21 Disclaimer: Big Data software is constantly updated, code samples may be outdated.
Page 2
Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary Outline 1 Introduction 2 BigData Challenges 3 Analytical Workflow 4 Use Cases 5 Programming 6 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 2 / 52
Page 3
Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary About DKRZ German Climate Computing Center (DKRZ) Partner for Climate Research Maximum Compute Performance. Sophisticated Data Management. Competent Service. Julian M. Kunkel Lecture BigData Analytics, 2016 3 / 52
Page 4
Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary Scientific Computing Research Group of Prof. Ludwig at the University of Hamburg Embedded into DKRZ Research Analysis of parallel I/O Alternative I/O interfaces I/O & energy tracing tools Data reduction techniques Middleware optimization Cost & energy efficiency Julian M. Kunkel Lecture BigData Analytics, 2016 4 / 52
Page 5
Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary Lecture Concept of the lecture The lecture is focussing on applying technology and some theory Theory Data models and data processing Statistics and machine learning System architectures Algorithms and data structures Applying technology Learning about various state-of-the art technology Hands-on for understanding the key concepts Languages: R and Python, (Java is important but not used) The domain of big data is overwhelming, especially technology It is a crash course for several topics such as statistics and databases ⇒ it is not the goal to learn and understand every aspect in this lecture Julian M. Kunkel Lecture BigData Analytics, 2016 5 / 52
Page 6
Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary Lecture (2) Slides Many openly accessable sources have been used They are cited by a number The reference slide provides the link to the sources For figures, a reference is indicated by Source: [Author]1 [title]1 [ref] In the title, an [ref] means that this reference has been used for the slide, some text may be taken literally Excercise Weekly delivery, processing time about 8 hours / per week estimated Teamwork of two people (groups are important to keep the time limit) Supported by: Hans Ole Hatzel 1If available Julian M. Kunkel Lecture BigData Analytics, 2016 6 / 52
Page 7
Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary Idea of BigData Methods of obtaining knowledge (Erkenntnissprozess) Scientific method: question, hypothesis, prediction, testing, analysis 1 Explorative: start theory with empirical observations of phenomena and experimentation 2 Constructivism: starts with axioms and reason implications (other theoretical approaches) 3 Computational science and simulation The Fourth Paradigm (Big) Data + Analytics ⇒ Insight (prediction of the future) For industry: insight = business advantage and money... Analytics: follow an explorative approach and study the data To infer knowledge, use statistics / machine learning Construct a theory (model) and validate it with the data Julian M. Kunkel Lecture BigData Analytics, 2016 7 / 52
Page 8
Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary Example Models Similarity is a (very) simplistic model and predictor for the world Humans use this approach in their cognitive process Uses the advantage of BigData Weather prediction You may develop and rely on complex models of physics Or use a simple model for a particular day; e.g., expect it to be similar to the weather of the typical day over the last X years Used by humans: rule of thumb for farmers Preferences of Humans Identify a set of people which liked items you like Predict you like also the items those people like (items you haven’t rated so far) Julian M. Kunkel Lecture BigData Analytics, 2016 8 / 52
Page 9
Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary Relevance of Big Data Big Data Analytics is emerging Relevance increases compared to supercomputing Google Search Trends, relative searches Julian M. Kunkel Lecture BigData Analytics, 2016 9 / 52
Page 10
Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary 1 Introduction 2 BigData Challenges 3 Analytical Workflow 4 Use Cases 5 Programming 6 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 10 / 52
Page 11
Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary BigData Challenges & Characteristics Source: MarianVesper [4] Julian M. Kunkel Lecture BigData Analytics, 2016 11 / 52
Page 12
Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary Volume: The size of the Data What is Big Data Terrabytes to 10s of petabytes What is not Big Data A few gigabytes Examples Wikipedia corpus with history ca. 10 TByte Wikimedia commons ca. 23 TByte Google search index ca. 46 Gigawebpages2 YouTube per year 76 PByte (20123) 2http://www.worldwidewebsize.com/ 3https://sumanrs.wordpress.com/2012/04/14/youtube-yearly-costs-for-storagenetworking-estimate/ Julian M. Kunkel Lecture BigData Analytics, 2016 12 / 52
Page 13
Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary Velocity: Data Volume per Time What is Big Data 30 KiB to 30 GiB per second (902 GiB/year to 902 PiB/year) What is not Big Data A never changing data set Examples LHC (Cern) with all experiments about 25 GB/s 4 Square Kilometre Array 700 TB/s (in 2018) 5 50k Google searches per s 6 Facebook 30 Billion content pieces shared per month 7 4http://home.web.cern.ch/about/computing/processing-what-record 5http://venturebeat.com/2014/10/05/how-big-data-is-fueling-a-new-age-in-space-exploration/ 6http://www.internetlivestats.com/google-search-statistics/ 7https://blog.kissmetrics.com/facebook-statistics/ Julian M. Kunkel Lecture BigData Analytics, 2016 13 / 52
Page 14
Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary Data Sources Enterprise data Serves business objectives, well defined Customer information Transactions, e.g., purchases Experimental/Observational data (EOD) Created by machines from sensors/devices Trading systems, satellites Microscopes, video streams, smart meters Social media Created by humans Messages, posts, blogs, Wikis Julian M. Kunkel Lecture BigData Analytics, 2016 14 / 52
Page 15
Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary Variety: Types of Data Structured data Like tables with fixed attributes Traditionally handled by relational databases Unstructured data Usually generated by humans Examples: natural language, voice, Wikipedia, Twitter posts Must be processed into (semi-structured) data to gain value Semi-structured data Has some structure in tags but it changes with documents Examples: HTML, XML, JSON files, server logs What is Big Data Use data from multiple sources and in multiple forms Involve unstructured and semi-structured data Julian M. Kunkel Lecture BigData Analytics, 2016 15 / 52
Page 16
Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary Veracity: Trustworthiness of Data What is Big Data Data involves some uncertainty and ambiguities Mistakes can be introduced by humans and machines Examples People sharing accounts Like sth. today, dislike it tomorrorw Wrong system timestamps Data Quality is vital! Analytics and conclusions rely on good data quality Garbage data + perfect model => garbage results Perfect data + garbage model => garbage results GIGO paradigm: Garbage In – Garbage Out Julian M. Kunkel Lecture BigData Analytics, 2016 16 / 52
Page 17
Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary Value of Data What is Big Data Raw data of Big Data is of low value For example, single observations Analytics and theory about the data increases the value Analytics transform big data into smart (valuable) data! Julian M. Kunkel Lecture BigData Analytics, 2016 17 / 52
Page 18
Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary Types of Data Analytics and Value of Data 1 Descriptive analytics (Beschreiben) “What happened?” 2 Diagnostic analytics “Why did this happen, what went wrong?” 3 Predictive analytics (Vorhersagen) “What will happen?” 4 Prescriptive analytics (Empfehlen) “What should we do and why?” The level of insight and value of data increases from Step 1 to 4 Julian M. Kunkel Lecture BigData Analytics, 2016 18 / 52
Page 19
Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary The Value of Data (alternative view) Source: Dursun Delen, Haluk Demirkan [9] Julian M. Kunkel Lecture BigData Analytics, 2016 19 / 52
Page 20
Introduction BigData Challenges Analytical Workflow Use Cases Programming Summary The Value of Data (alternative view) Source: Forrester report. Understanding The Business Intelligence Growth Opportunity. 20-08-2011 Julian M. Kunkel Lecture BigData Analytics, 2016 20 / 52
Affichage des 20 premières pages. Téléchargez le document pour le voir en entier.
Commentaires (0)
Aucun commentaire pour le moment. Soyez le premier !