EECS E6893 Big Data Analytics Lecture 1: Overview of Big Data Analytics Ching-Yung Lin, Ph.D. Adjunct Professor, Depts. of Electrical Engineering and Computer Science IEEE Fellow September 8th, 2023 E6893 Big Data Analytics — Lecture 1 © CY Lin, 2023 Columbia University Definition and Characteristics of Big Data “Big data is high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.” -- Gartner which was derived from: “While enterprises struggle to consolidate systems and collapse redundant databases to enable greater operational, analytical, and collaborative consistencies, changing economic conditions have made this job more difficult. E-commerce, in particular, has exploded data management challenges along three dimensions: volumes, velocity and variety. In 2001/02, IT organizations much compile a variety of approaches to have at their disposal for dealing each.” – Doug Laney 2 E6893 Big Data Analytics — Lecture 1 © CY Lin 2023, Columbia University What made Big Data needed? “Big Data Analytics”, David Loshin 3 E6893 Big Data Analytics — Lecture 1 © CY Lin 2023, Columbia University Key Computing Resources for Big Data Processing capability: CPU, processor, or node. Memory Storage Network “Big Data Analytics”, David Loshin 4 E6893 Big Data Analytics — Lecture 1 © CY Lin 2023, Columbia University Scalability — Scale Up & Scale Out Scale out Use more resources to distribute workload in parallel Higher data access latency is typically incurred Scale up Efficiently use the resources Architecture-aware algorithm design Example: Resource utilization for a large production cluster at Twitter data center www.stanford.edu/~cdel/2014.asplos.quasar.pdf For independent data ==> scale up may not have obvious advantage than scale out For linked data ==> utilizing scale up as much as possible before scale out 5