Big Data Analytics Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2017-10-20 Disclaimer: Big Data software is constantly updated, code samples may be outdated.
Lecture Big Data Analytics BigData Challenges Gaining Insight with Analytics Use Cases Programming Summary Outline 1 Lecture 2 Big Data Analytics 3 BigData Challenges 4 Gaining Insight with Analytics 5 Use Cases 6 Programming 7 Summary Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 2 / 59
Lecture Big Data Analytics BigData Challenges Gaining Insight with Analytics Use Cases Programming Summary About DKRZ German Climate Computing Center (DKRZ) Partner for Climate Research Maximum Compute Performance. Sophisticated Data Management. Competent Service. Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 3 / 59
Lecture Big Data Analytics BigData Challenges Gaining Insight with Analytics Use Cases Programming Summary Scientific Computing Research Group of Prof. Ludwig at the University of Hamburg Embedded into DKRZ Research Analysis of parallel I/O Alternative I/O interfaces I/O & energy tracing tools Data reduction techniques Middleware optimization Cost & energy efficiency Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 4 / 59
Lecture Big Data Analytics BigData Challenges Gaining Insight with Analytics Use Cases Programming Summary Lecture Concept of the lecture The lecture is focusing on applying technology and some theory Theory Data models and data processing Statistics and machine learning System architectures Algorithms and data structures Applying technology Learning about various state-of-the art technology Hands-on for understanding the key concepts Languages: R and Python, (Java is important but not used) The domain of big data is overwhelming, especially technology It is a crash course for several topics such as statistics and databases ⇒ it is not the goal to learn and understand every aspect in this lecture Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 5 / 59
Lecture Big Data Analytics BigData Challenges Gaining Insight with Analytics Use Cases Programming Summary Lecture (2) Slides Many openly accessible sources have been used They are cited by a number The reference slide provides the link to the sources For figures, a reference is indicated by Source: [Author]1 [title]1 [ref] In the title, an [ref] means that this reference has been used for the slide, some text may be taken literally Exercise Unfortunately no student assistant found so corrections during exercise... Weekly delivery, processing time about 8 hours / per week estimated Teamwork of two people (groups are important to keep the time limit) 1If available Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 6 / 59
Lecture Big Data Analytics BigData Challenges Gaining Insight with Analytics Use Cases Programming Summary 1 Lecture 2 Big Data Analytics 3 BigData Challenges 4 Gaining Insight with Analytics 5 Use Cases 6 Programming 7 Summary Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 7 / 59
Lecture Big Data Analytics BigData Challenges Gaining Insight with Analytics Use Cases Programming Summary Scientific Method Start Based on: The Scientific Method as an Ongoing Process, ArchonMagnus[22] Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 8 / 59
Lecture Big Data Analytics BigData Challenges Gaining Insight with Analytics Use Cases Programming Summary Pillars of the Scientific Method ScienceScience Theory Experimentation Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 9 / 59
Lecture Big Data Analytics BigData Challenges Gaining Insight with Analytics Use Cases Programming Summary Pillars of Science: Modern Perspective ScienceScience Theory Simulation Big Data AnalyticsExperimentation Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 10 / 59
Lecture Big Data Analytics BigData Challenges Gaining Insight with Analytics Use Cases Programming Summary Idea of Big Data Analytics Big Data Vast amounts of data are available Many heterogene data sources Raw data is of low value (fine grained) Analytics Analyzing data ⇒ Insight == value For academia: knowledge For industry: business advantage and money Levels of insight – primary abstraction levels of analytics Exploration: study data and identify properties of (subsets) of data Induction/Inference: infer properties of the full population Big data tools allow to construct a theory/model and validate it with data Statistics and machine learning provide algorithms and models Visual methods support data exploration and analysis Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 11 / 59
Lecture Big Data Analytics BigData Challenges Gaining Insight with Analytics Use Cases Programming Summary Example Models Similarity is a (very) simplistic model and predictor for the world Humans use this approach in their cognitive process Uses the advantage of BigData Weather prediction You may develop and rely on complex models of physics Or use a simple model for a particular day; e.g., expect it to be similar to the weather of the typical day over the last X years Used by humans: rule of thumb for farmers Preferences of Humans Identify a set of people which liked items you like Predict you like also the items those people like but haven’t rated Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 12 / 59
Lecture Big Data Analytics BigData Challenges Gaining Insight with Analytics Use Cases Programming Summary Relevance of Big Data Big Data Analytics is emerging Relevance increases compared to supercomputing Google Search Trends, relative searches Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 13 / 59
Lecture Big Data Analytics BigData Challenges Gaining Insight with Analytics Use Cases Programming Summary Roles in the Big Data Business Data scientist Data science is a systematic method dedicated to knowledge discovery via data analysis [1] In business, optimize organizational processes for efficiency In science, analyze experimental/observational data to derive results Data engineer Data engineering is the domain that develops and provides systems for managing and analyzing big data Build modular and scalable data platforms for data scientists Deploy big data solutions Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 14 / 59
Lecture Big Data Analytics BigData Challenges Gaining Insight with Analytics Use Cases Programming Summary Typical Skills Data scientist Statistics + (mathematics) background Computer science Programming, e.g.: R, (SAS,) Java, Scala, Python Machine learning Some domain knowledge for the problem to solve Data engineer Computer science background Databases Software engineering Massively parallel processing Real-time processing Languages: C++, Java, (Scala,) Python Understand performance factors and limitations of systems Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 15 / 59
Lecture Big Data Analytics BigData Challenges Gaining Insight with Analytics Use Cases Programming Summary 1 Lecture 2 Big Data Analytics 3 BigData Challenges 4 Gaining Insight with Analytics 5 Use Cases 6 Programming 7 Summary Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 16 / 59
Lecture Big Data Analytics BigData Challenges Gaining Insight with Analytics Use Cases Programming Summary BigData Challenges & Characteristics Source: MarianVesper [4] Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 17 / 59
Lecture Big Data Analytics BigData Challenges Gaining Insight with Analytics Use Cases Programming Summary Volume: The size of the Data What is Big Data Terrabytes to 10s of petabytes What is not Big Data A few gigabytes Examples Wikipedia corpus with history ca. 10 TByte Wikimedia commons ca. 23 TByte Google search index ca. 46 Gigawebpages2 YouTube per year 76 PByte (20123) 2http://www.worldwidewebsize.com/ 3https://sumanrs.wordpress.com/2012/04/14/youtube-yearly-costs-for-storagenetworking-estimate/ Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 18 / 59
Lecture Big Data Analytics BigData Challenges Gaining Insight with Analytics Use Cases Programming Summary Velocity: Data Volume per Time What is Big Data 30 KiB to 30 GiB per second (902 GiB/year to 902 PiB/year) What is not Big Data A never changing data set Examples LHC (Cern) with all experiments about 25 GB/s 4 Square Kilometre Array 700 TB/s (in 2018) 5 50k Google searches per s 6 Facebook 30 Billion content pieces shared per month 7 4http://home.web.cern.ch/about/computing/processing-what-record 5http://venturebeat.com/2014/10/05/how-big-data-is-fueling-a-new-age-in-space-exploration/ 6http://www.internetlivestats.com/google-search-statistics/ 7https://blog.kissmetrics.com/facebook-statistics/ Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 19 / 59
Lecture Big Data Analytics BigData Challenges Gaining Insight with Analytics Use Cases Programming Summary Data Sources Enterprise data Serves business objectives, well defined Customer information Transactions, e.g., purchases Experimental/Observational data (EOD) Created by machines from sensors/devices Trading systems, satellites Microscopes, video streams, smart meters Social media Created by humans Messages, posts, blogs, Wikis Julian M. Kunkel Lecture BigData Analytics, WiSe 17/18 20 / 59
Hiển thị 20 trang đầu. Tải tài liệu để xem đầy đủ.
Chưa có bình luận nào. Hãy là người đầu tiên!
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!