E6893 Big Data Analytics Lecture 3: Big Data Analytics Algorithms Ching-Yung Lin, Ph.D. Adjunct Professor, Dept. of Electrical Engineering and Computer Science September 22, 2023 1 © 2023 CY Lin, Columbia University E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms
Key Components of Spark MLlib 2 E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms © 2023 CY Lin, Columbia University
Spark ML Classification and Regression 3 E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms © 2023 CY Lin, Columbia University
Classification — definition 4 E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms © 2023 CY Lin, Columbia University
Classification example: using SVM to recognize a Toyota Camry ML — Support Vector Machine Non-ML Feature Space Positive SVs Rule 1.Symbol has something like bull’s head Rule 2.Big black portion in front of car. Rule 3. …..???? Negative SVs 5 E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms © 2023 CY Lin, Columbia University
Classification example: using SVM to recognize a Toyota Camry ML — Support Vector Machine Positive SVs PCamry > 0.95 Feature Space Negative SVs 6 E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms © 2023 CY Lin, Columbia University
When to use Big Data System for classification? 7 E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms © 2023 CY Lin, Columbia University
The advantage of using Big Data System for classification 8 E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms © 2023 CY Lin, Columbia University
How does a classification system work? 9 E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms © 2023 CY Lin, Columbia University
Key terminology for classification 10 E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms © 2023 CY Lin, Columbia University
Input and Output of a classification model 11 E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms © 2023 CY Lin, Columbia University
Four types of values for predictor variables 12 E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms © 2023 CY Lin, Columbia University
Sample data that illustrates all four value types 13 E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms © 2023 CY Lin, Columbia University
Supervised vs. Unsupervised Learning 14 E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms © 2023 CY Lin, Columbia University
Work flow in a typical classification project 15 E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms © 2023 CY Lin, Columbia University
Classification Example 1 — Color-Fill Position looks promising, especially the x-axis ==> predictor variable. Shape seems to be irrelevant. Target variable is “color-fill” label. 16 E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms © 2023 CY Lin, Columbia University
Classification Example 2 — Color-Fill (another feature) 17 E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms © 2023 CY Lin, Columbia University
Fundamental classification algorithms Example of fundamental classification algorithms: • Naive Bayesian • Complementary Naive Bayesian • Stochastic Gradient Descent (SDG) • Random Forest • Support Vector Machines 18 E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms © 2022 CY Lin, Columbia University
Choose algorithm 19 E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms © 2023 CY Lin, Columbia University
Stochastic Gradient Descent (SGD) 20 E6893 Big Data Analytics – Lecture 3: Big Data Analytics Algorithms © 2023 CY Lin, Columbia University
Showing the first 20 pages. Download the document to see it all.
No comments yet. Be the first!
Comments (0)
No comments yet. Be the first!