CPS 516: Data-intensive Computing Systems Instructor: Shivnath Babu TA: Zilong (Eric) Tan The World of Big Data  eBay had 6.5 PB of user data + 50 TB/day in 2009 From http://www.umiacs.umd.edu/~jimmylin/ From: http://www.cs.duke.edu/smdb10/ The World of Big Data  eBay had 6.5 PB of user data + 50 TB/day in 2009 How much do they have now? See http://en.wikipedia.org/wiki/Big_data Also see: http://wikibon.org/blog/big-data-statistics/ From http://www.umiacs.umd.edu/~jimmylin/ FOX AUDIENCE NETWORK Greenplum parallel DB • 42 Sun X4500s (“Thumper”) each with: 48 500GB drives 16GB RAM 2 dual-core Opterons Big and growing 200 TB data (mirrored) Fact table of 1.5 trillion rows Growing 5TB per day 4-7 Billion rows per day From: http://db.cs.berkeley.edu/jmh/ Also extensive use of R and Hadoop Yahoo! runs a 4000 node Hadoop cluster (probably the largest). Overall, there are 38,000 nodes running Hadoop at Yahoo! As reported by FAN, Feb, 2009 How many female WWF fans under the age of 30 visited the Toyota community over the last 4 days and saw a Class A ad? How are these people similar to those that visited Nissan? Open-ended question about statistical densities (distributions) From: http://db.cs.berkeley.edu/jmh/ “No One Size Fits All” Philosophy Non-relational Relational Analytic Hadoop Mapr EMC Greenplum SAP Sybase IQ IBM Netezza Hadapt Dryad MapReduce Online Teradata Aster Infobright Calpont HP Vertica MapReduce Brisk RainStor Impala Dremel GridGrain Platfora Spark Shark Operational NoSQL In-memory Druid SAP HANA Oracle IBM DB2 SQL Server PostgreSQL MySQL Neo4J DataStax Enterprise Hypertable DEX LevelDB as-a-Service Sap Sybase Cassandra ClearDB Amazon RDS HBase OrientDB ASE Big tables Graph FathomDB Google Cloud SQL Riak AppEngine EnterpriseDB NuvolaBase Database.com SQL Azure Datastore SimpleDB Redis DynamoDB -as-a-Service NewSQL New databases SQLFire VoltDB Voldemort MemSQL as-a-Service MongoHQ Cloudant Driz