Trang 1
Fall 2019 Big Data Processing and Analytics Vassilis Christophides christop@csd.uoc.gr http://www.csd.uoc.gr/~hy562 University of Crete, Fall 2019 1 Fall 2019 The Data Avalanche: From Science to Business 2 1 1
Trang 2
Shifting Paradigm in Sciences Fall 2019 https://aip.scitation.org/doi/10.1063/1.4946894 3 Fall 2019 Data-driven Discovery Data-driven discovery is revolutionizing scientific exploration as well as engineering innovations From hypothesis driven to hypothesis generating R. Leland, R. Murphy, B. Hendrickson, K. Yelick, J. Johnson, J. Berry Large-Scale Data Analytics & its Relationship to Simulation Jan. 20144 2 2
Trang 3
From “Data Door” to “Data Rich” Fall 2019 Scientific Research Astronomy: LSST Physics: LHC Oceanography Biology: Sequencing Sociology: The Web Precision Medicine Neuroscience: EEG, fMRI Sports Data deluge spans biology, climate, cosmology, materials, physics, … M. Franklin Big Data Software: What’s Next? (and what do we have to say about it?) VLDB 2017 5 Fall 2019 New Research Methods Simulation Data: Increasing level of Experimental Data: Light sources, simulation detail and duration, as well genome sequencing, next generation as, model size by orders of magnitude! ARM radars, sky surveys, neuro- sensing and stimulation, … New research methods depend on coupling computation and experiment as well as on integrating data across sources and/or types 6 3 3
Trang 4
Large Synoptic Survey Telescope (LSST)Fall 2019 –100-200 Petabyte image archive 3.2 billion-pixel –20-40 Petabyte database catalog camera 8.4-meter diameter LSST will take more primary mirror = 10 than 800 panoramic square degrees! images each night recording the entire visible sky twice each week Ten-year time series (~2020-2030) imaging of the night sky – mapping the Universe ! www.lsst.org 7 Fall 2019 First Image of a Black Hole Captured by the Event Horizon telescope (EHT), an NSF funded network of eight radio telescopes spanning locations from Antarctica to Spain and Chile, in an effort involving more than 200 scientists achieved resolutions of 22.5 microarcseconds, enabling the array to resolve the event horizon of the black hole at the center of M87 a single-dish telescope would have to be 12000 km in diameter to achieve this same sharpness K. Bauman posing with 5 petabytes of data necessary to image a black hole https://www.for https://www.facebook.com/BusinessInsiderScience/videos/3788973860386158 scientific 4 4
Trang 5
Fall 2019 AI is Changing Drug Discovery! https://medium.com/@ABuvailo/artificial-intelligence-in-drug- discovery-2018-year-in-review-e17b99c99078 12 Fall 2019 Growth of DNA Sequencing blogs.springeropen.com/springeropen/wp-content/uploads/sites/16/ 2018/01/bioData.png 14 5 5
Trang 6
Fall 2019 The Four Industrial Revolutions Everybody & Digital everything is Centralized computing & networked – Mechanical electric power communication networked production infrastructure; technology, information as using the power mass production enhancing a “huge brain” of water and by division of systems’ steam labor intelligence End of 18th Beginning of Mid of 20th century 20th century century Henning Kagermann et.al., Recommendations for implementing the strategic initiative Industrie 4.0 Acatech, 2013 15 Digital Disruption Already Happening !Fall 2019 Largest telco companies owns no telco infrastructure (Skype) World’s largest movie houses owns no cinemas (Netflix) World’s most valuable retailer has no inventory (Alibaba) Most popular media owner creates no content (Facebook) World’s largest taxi company owns no vehicles (Uber) Largest accommodation provider owns no real estate (Airbnb) Faster growing banks have actually no money (BitCoin) http://www.independent.co.uk/news/business/comment/hamish- mcrae/facebook-airbnb-uber-and-the-unstoppable-rise-of-the- content-non-generators-10227207.html 17 6 6
Trang 7
The Data Tsunami: Transactions + Fall 2019 Interactions + Observations Smart farms Your AIs are data hungry! 1021 & food Earth challenge systems areas Creative arts Personal 1018 assistants 'Detoxify' Social Media 1015 Superhuman Doctor Smarter 1012 Cybersecurity Self-Driving 109 Cars Life On Other Planets? https://www.slideshare.net/KeithKraus/gpuaccelerating-udfs- in-pyspark-with-numba-and-pygdf 19 Driving Innovation with Big DataFall 2019 Progress and Innovation is no longer hindered by the ability to collect data but, by the ability to manage, analyze, summarize, visualize, and discover knowledge from the collected data in a timely manner and in a scalable fashion 21 7 7
Trang 8
Fall 2019 What Makes Data, “Big” Data? 23 Definitions Fall 2019 No single standard definition… “Big Data” is data whose scale, diversity, and complexity require new architecture, techniques, algorithms, and analytics to manage it and extract value and hidden knowledge from it… (McKinsey Global Inst.) “Big Data” is high-volume, high- velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making (Gartner) 24 8 8
Trang 9
The Four V’s of Big Data Fall 2019 Terabytes to zetabytes of existing data to process www-05.ibm.com/fr/events/netezzaDM_2012/Solutions_Big_Data.pdf 25 Fall 2019 Characteristics of Big Data: 1-Scale (Volume) 1021 Web data Mobile data 1012 ERP, CRM data Too big: petabyte-scale collections or lots of (not necessarily big) data sets 26 9 9
Trang 10
Fall 2019 Characteristics of Big Data: 2-Speed (Velocity) Financial data IoT data Social data 500 million of Tweets sent per Day 330 million of active Tweeter Users Too fast: needs to be processed quickly and react promptly 27 Fall 2019 Characteristics of Big Data: 3-Complexity (Variety) Measurement data Medical Imaging data Video data Textual data Textual data 500 330 Too diverse: does not fit neatly in an existing tool 28 10 10
Trang 11
Fall 2019 Characteristics of Big Data: 4-Quality (Veracity) Many sources of online information: are all these sources equally ‐ accurate ‐ up‐to‐date ‐ and trustworthy? Too crappie: needs to assess their quality 29 Fall 2019 Big Data Characteristics & Challenges Characteristic Description Challenges Root Cause Volume Over increasing amount of Constantly scale High number of data that must be ingested, hardware and software data sources processed & analyzed. A single infrastructure to High resolution machine can not manage large accommodate very sensors volumes of data efficiently. large storage spaces Velocity Fast data is being ingested Support streaming/ High-rate data and need to be transformed online pprocessing & acquisition, low into insight at a high speed. Real-time Analytics cost of hardware Variety Degree of diversity (in terms Cope with Multi- Social media of content and structuring) of Modality, Complex Scientific data data from sources both inside interactions and Implicit Video and outside an organization Semantics M2M / IoT Veracity Quality and Curate data for missing, Crowd data trustworthiness of data duplicate, erroneous production, values, enchase data Human & traceability Machine Sensing 30 11 11
Trang 12
We’ve Moved into a New Era of DataFall 2019 Analytics Look At All The Data Look Even At Dirty & Noisy Data Leverage Data as it is Captured Let Data Lead the Way @ 2014 IBM Corporation 33 Fall 2019 Data Lakes vs Data Warehouse 34 12 12
Trang 13
Fall 2019 Big Data Mining 35 What to Do with Big Data? Fall 2019 Data contains knowledge and value Nobody knows what’s in data unless it has been processed and analyzed Data value for: Faster, better decision making Cost savings New products and services Grand challenge for data science and engineering: Empower a wide range of users to explore and obtain trustworthy, actionable insights from big data 36 13 13
Trang 14
Data Mining Methods Fall 2019 Predictive: Use some variables Descriptive: Find human- to predict unknown or future interpretable patterns that values of other variables describe the data Supervised Unsupervised Regression Clustering & Dimensionality Reduction Continuous Predict the value of a Finds “natural” grouping of continuous variable instances given unlabeled data Classification Frequent Patterns & Association Rules Categorical Predict the label of an instance Discover interesting co- from pre-label (classified) occurrence relations between instances variables 38 Fall 2019 Data Analysis: ERP & CRM Examples Who are our lowest/highest margin customers ? Who are my customers and what products What is the most are they buying? effective distribution channel? What product prom- Which customers -otions have the biggest are most likely to go impact on revenue? to the competition ? What impact will new products/services have on revenue and margins? D. Agrawal, S. Das, A. El Abbadi Big Data &Cloud Computing VLDB 2010 Tutorial39 14 14
Trang 15
Large-Scale, Real-World Analytics Fall 2019 Question Method How do I segment my customers? K-means Clustering How is product ownership distributed SQL, Cumulative Distribution Functions across customer segments? Does this product appeal to some Log-likelihood segments more than others? What new products should I offer my Cosine similarity, k-Nearest Neighbors, customers? Matrix factorization Which campaign is working better? Mann-Whitney U Test How do I target my marketing efforts Logistic Regression towards customers most likely to churn? What are my customers saying about NLP, sparse vectors the new product launch? How can I identify fraudulent activity? Classification, Logistic Regression Tools and Technologies for Big Data Steven Hillion V.P. Analytics EMC Data Computing Division 2011 40 The WRONG Picture! Fall 2019 Incorrect conclusions can lead to bad decisions 41 15 15
Trang 16
Big Data vs Deep Insights Fall 2019 data knowledge Data exploration is hard regardless of whether data are big or small ! 42 The TRUE Picture! Fall 2019 The time for developing an The time for developing an analysis (with small data) analysis (with big data) Big Data Infrastructures: Exploiting the Power of Big Data T. Sellis School of CS & IT, 2015 Athens 43 16 16
Trang 17
Fall 2019 Big Data Processing & Analytics BIG DATA MINING 44 Fall 2019 Traditional vs. Map/Reduce Approach Don’t move data to workers… Move workers to the data! Store data on the local disks for nodes in the cluster Start up the workers on the node that has the data local! Why? Not enough RAM to hold all the data in memory Common local-area network (LAN) speeds go up to 100 Mbit/s, which is about 12.5MB/s Traditional hard disks provide a lot of storage and transfer speeds around 40-60MB/s https://www.edureka.co/blog/mapreduce-tutorial 46 17 17
Trang 18
What we Need to Make Sense of Big Data?Fall 2019 New Computing Frameworks: Parallel/Distributed architectures: Cloud, HPC, Map/Reduce (Apache Hadoop, SPARK), … Storage solutions: NoSQL, column stores, RDDs Processing Languages: SAPRK SQL, GraphX, Streaming, … But also new Approaches/Algorithms! To explore and process big data integrate, curate, prepare , … To mine data in Big-Data frameworks Several software libraries exist but there are M. Cooper & P. Mell Tackling Big no one-size-fits-all solution! Data NIST Information Technology Laboratory Computer Security often, you have to build your own… Division 48 Big Data Processing & Analytics PlatformsFall 2019 Big Machine Learning Online Machine (Mahout, MillWheel, R/Hadoop) Learning (SAMOA, Rapid Miner, OIIDM) IoT Data Analysis (Parstream, Vitria, Splunk, virdata) (Striim, Storm, Spark, Google RT, Apache S4, MS , Azure, AWS Big Time Kinesis) Series Analytics (InfluxDB, AT&T M2X, IBM Informix TS, OpenTSDB) Source: Vmware 49 18 18
Trang 19
The Big Data Mining Mindset Fall 2019 Data mining overlaps with: Databases (DB): Large-scale data, simple queries Machine Learning (ML): Small data, Complex models Computer Science Theory: (Randomized) Algorithms Big Data urges for a cross-culture curriculum stressing on Scalable Systems Algorithmic Thinking ML/Pattern Computing Architectures Statistics/ AI Recognition Automation for Handling Very Large Datasets Data Mining Databases 50 Fall 2019 Big Data and its Relation to Statistics Statistical methods are the core of what Big Data is today A statistician will typically assume that datasets she/he deals with will fit into the main memory on a single machine Statistics extract most information from a very sparse and expensive to acquire typically small dataset However, now we move from a data poor regime to a data rich regime The goal is not anymore about new fancy mathematical method to squeeze more information from a small dataset The goal is now to about to build new engineering tools to process very large datasets Similarly like statisticians, visualization specialist are less concerned with massive datasets that span across hundreds/thousands of machines on the Internet 51 19 19
Trang 20
Fall 2019 Big Data and its Relation to Business Intelligence (BI) BI aims at descriptive statistics with data with high information density to measure things, detect trends etc. Big Data targets inductive statistics with data with low information density whose huge volume allow to infer laws (regressions…) Software stack designed for BI is very specific and not very adaptable when requirements change Data warehouse and specific dashboards and reports that consume data from the data warehouse in order to answer specific questions Software stack designed for BI is not applicable to Big Data problems where changing requirements is a norm BI engineers do not consumer their own products and make the decisions themselves, while Big Data analysts do 52 Fall 2019 Big Data and its Relation to Data Engineering DB engineers and administrators posses a lot of skills to make them appropriate to Big Data tasks However, they are focused on a particular data model which is usually the relational one (columns and rows) Big data analysts deal with heterogeneous data sources that may include video, audio, text, graphs, images, structures and unstructured data, etc. The relational data model may not be appropriate for some sources To a DB person, data mining is an extreme form of analytic processing – queries that examine large amounts of data Result is the query answer However, to a ML person, data-mining is the inference of models – ML algorithms = “engine” to solve ML models Result is the parameters of the model 53 20 20
Hiển thị 20 trang đầu. Tải tài liệu để xem đầy đủ.
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!