描述

Fall 2019 Relational Data Processing on MapReduce Vassilis Christophides christop@csd.uoc.gr http://www.csd.uoc.gr/~hy562 University of Crete, Fall 2019 1 Peta-scale Data Analysis 30 billion RFID 12+ TBs of tweet data every day tags today (1.3B in 2005) Fall 2019 4.6 billion camera phones world wide ? TBs of data every day 100s of millions of GPS enabled devices sold annually 25+ TBs of log data every day generated by a new user being added every sec. for 3 years 4 billion views/day YouTube is the 2nd most used search engine next to Google 2+ billion 76 million smart meters in 2009… 200M by 2014 people on the Web by end 2011 © 2014 IBM Corporation 2 1 1 Big Data Analysis Fall 2019  A lot of these datasets have some structure Query logs Point-of-sale records User data (e.g., demographics) …  How do we perform data analysis at scale? Relational databases and SQL MapReduce (Hadoop) 3 Fall 2019 Relational Databases vs. MapReduce  Relational databases: Multipurpose: analysis and transactions; batch and interactive Data integrity via ACID transactions Lots of tools in software ecosystem (for ingesting, reporting, etc.) Supports SQL (and SQL integration, e.g., JDBC) Automatic SQL query optimization  MapReduce (Hadoop): Designed for large clusters, fault tolerant Data is accessed in “native format” Supports many query languages Programmers retain control over performance 4 2 2 Fall 2019 Parallel Computation & Data Size Matters! 5 Parallel Relational Databases vs. MapReduce  Parallel relational databases Fall 2019 Shared-nothing architecture for parallel processing Schema on “write” Failures are relatively infrequent “Possessive” of data Mostly proprietary  MapReduce Schema on “read” Failures are relatively common In situ data processing Open source Hadoop NextGen (YARN) architecture 6 3 3 Fall 2019 MapReduce: A Major Step Backwards?  MapReduce is a step backward in database access Separation of the schema

AI 摘要