Trang 2
The World of Big Data eBay had 6.5 PB of user data + 50 TB/day in 2009 From http://www.umiacs.umd.edu/~jimmylin/
Nội dung tài liệu
Hiển thị 2 trang đầu. Tải tài liệu để xem đầy đủ.
Giới thiệu Hệ thống tính toán dữ liệu lớn (Tiếng Anh) - Duke University
0 lượt tải
Nội dung tài liệu
Trang 2
The World of Big Data eBay had 6.5 PB of user data + 50 TB/day in 2009 From http://www.umiacs.umd.edu/~jimmylin/
Trang 4
The World of Big Data eBay had 6.5 PB of user data + 50 TB/day in 2009 How much do they have now? See http://en.wikipedia.org/wiki/Big_data Also see: http://wikibon.org/blog/big-data-statistics/ From http://www.umiacs.umd.edu/~jimmylin/
Trang 5
FOX AUDIENCE NETWORK • Greenplum parallel DB • Also extensive use of R • 42 Sun X4500s (“Thumper”) each and Hadoop with: • 48 500GB drives Yahoo! runs a 4000 • 16GB RAM • 2 dual-core Opterons node Hadoop cluster • Big and growing (probably the largest). • 200 TB data (mirrored) Overall, there are • Fact table of 1.5 trillion rows 38,000 nodes running • Growing 5TB per day 4-7 Billion rows per day Hadoop at Yahoo! • From: http://db.cs.berkeley.edu/jmh/ As reported by FAN, Feb, 2009
Trang 6
How many female WWF How are these people fans under the age of 30 similar to those that visited the Toyota visited Nissan? community over the last 4 days and saw a Class A ad? Open-ended question about statistical densities (distributions) From: http://db.cs.berkeley.edu/jmh/
Trang 7
“No One Size Fits All” Philosophy Non-relational Relational Analytic Hadoop Mapr Hadapt EMC Greenplum SAP Sybase IQ IBM Netezza Dryad MapReduce Online Teradata Aster MapReduce Brisk HP Vertica Infobright Calpont RainStor GridGrain Platfora Spark Shark Dremel Impala Operational In-memory Druid SAP HANA NoSQL Oracle IBM DB2 SQL Server DataStax Enterprise Neo4J PostgreSQL MySQL LevelDB Hypertable DEX -as-a-Service Cassandra HBase OrientDB Amazon RDS ClearDB Sap Sybase Big tables Graph Google Cloud SQL FathomDB ASE Riak AppEngine EnterpriseDB NuvolaBase SQL Azure Database.com Redis SimpleDB Datastore DynamoDB -as-a-Service NewSQL New databases Voldemort Cloudant MemSQL SQLFire VoltDB MongoHQ -as-a-Service NuoDB CouchBase StormDB Drizzle Key value MongoDB Xeround GenieDB Clustrix SchoonerSQL Document RavenDB Storage engines ScaleArc Solr HyperDex ScaleDBClustering/sharding Lucene Lotus Notes Tokutek MySQL Cluster ScaleBase Continuent Xapian Search ElasticSearchSphinx Streaming Esper Gigascope STREAM DejaVu MarkLogic InterSystems Storm Borealis Oracle CEP DataCell S4 SQLStream Versant Starcounter Muppet StreamBaseTruviso Progress An extension of the figure given in http://blogs.the451group.com/information_management/2012/11/02/updated-database-landscape-graphic
Trang 8
What we will cover in class • Scalable data processing – Parallel query plans and operators – Systems based on MapReduce – Scalable key-value stores – Processing rapid, high-speed data streams • Principles of query processing – Indexes – Query execution plans and operators – Query optimization • Data storage – Databases Vs. Filesystems (Google/Hadoop Distributed FileSystem) – Data layouts (row-stores, column-stores, partitioning, compression) • Concurrency control and fault tolerance/recovery – Consistency models for data (ACID, BASE, Serializability) – Write-ahead logging
Trang 9
Course Logistics • Web pages: Course home page will be at Duke, and everything else will be on github • Grading: – Three exams: 10 (Feb) + 15 (March) + 25 (April) = 50% – Project: 10 (Jan 21) + 10 (Feb 1 – Feb 21) + 10 (Feb 22 – March 10) + 20 (March 11 – April 15) = 50% • Books: – No one single book – Hadoop: The Definitive Guide, by Tom White – Database Systems: The Complete Book, by H. Garcia- Molina, J. D. Ullman, and J. Widom
Trang 10
Project Part 0: Due in 2 Weeks • For every single system listed in the “Data Platforms Map”, give as a list of succinct points: – Strengths (with numbered references) – Weaknesses (with numbered references) – References (can be articles, blog posts, research papers, white papers, your own assessment, …) • Your own thoughts only. Don’t plagiarize. List every source of help. We will enforce honor code strictly. • Submit on github (md format) into repository given by Zilong • Outcomes: (a) Score out of 10; (b) Project leader selection.
Trang 11
Project Parts 1, 2, 3 • Shivnath/Zilong will work with project leaders to assign one system per project. Will also try to have one mentor per project • Each student will join one project. Project starts Feb 1 • Part 1: Feb 1 – Feb 21 – Install system – Develop an application workload to exercise the system – Run workload and give demo and report • Part 2: Feb 22 – March 15 – Identify system logs/metrics and other data that will help you understand deeply how the system is running the workload – Collect and send the data to a Kafka/MySQL/ElasticSearch routing and storage system set up by Shivnath/Zilong. Give demo and report • Part 3: March 16 to April 15 – Analyze and visualize the data to bring out some nontrivial aspects of the system related to what we learn in class. Give demo and report
Trang 14
Example: At a Company Query 1: Is there an employee named “Nemo”? Query 2: What is “Nemo’s” salary? Query 3: How many departments are there in the company? Query 4: What is the name of “Nemo’s” department? Query 5: How many employees are there in the “Accounts” department? Employee Department ID Name DeptID Salary … ID Name … 10 Nemo 12 120K … 12 IT … 20 Dory 156 79K … 34 Accounts … 40 Gill 89 76K … 89 HR … 52 Ray 34 85K … 156 Marketing … … … … … … … … …
Trang 15
DataBase Management System (DBMS) High-level Query Q Answer Translates Q into DBMS best execution plan for current conditions, runs plan Data
Trang 16
Example: Store that Sells Cars Owners of Make Model OwnerID ID Name Age Honda Accords Honda Accord 12 12 Nemo 22 who are <= Honda Accord 156 156 Dory 21 23 years old Join (Cars.OwnerID = Owners.ID) Filter (Make = Honda and Filter (Age <= 23) Model = Accord) Cars Owners Make Model OwnerID ID Name Age Honda Accord 12 12 Nemo 22 Toyota Camry 34 34 Ray 42 Mini Cooper 89 89 Gill 36 Honda Accord 156 156 Dory 21 … … … … … …
Trang 17
DataBase Management System (DBMS) High-level Query Q Answer Translates Q into DBMS best execution plan for current conditions, runs plan Keeps data safe and correct despite failures, concurrent Data updates, online processing, etc.
Hiển thị 17 trang đầu. Tải tài liệu để xem đầy đủ.
Bình luận (0)
Đăng nhập để tham gia bình luận
Chưa có bình luận nào. Hãy là người đầu tiên!
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!