E6893 Big Data Analytics Lecture 4: Real-Time Stream Analysis Ching-Yung Lin, Ph.D. Adjunct Professor, Dept. of Electrical Engineering and Computer Science September 29, 2023 1 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis © 2023 CY Lin, Columbia University
Stream Analyses Technical Challenges 2 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis © 2023 CY Lin, Columbia University
Example IP Packet Stream Instantiation rtsp tcp ftp ip http audio udp rtp video ntp Dataflow Inputs Graph By IBM Dense Information Gliding Team 3 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis © 2023 CY Lin, Columbia University
Semantic MM Filtering rtsp Advanced content ftp analysis tcp ip http audio Interest keywords id sess Interest Filtering rtp Routing udp video sess Interested MM streams ntp Dataflow Inputs Packet content analysis Graph per PE 200-500MB/s ~100MB/s 10 MB/s rates 4 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis © 2023 CY Lin, Columbia University
Resource-Accuracy Trade-Offs R X X’ Y’’(X|q,R) Y(X|q) Configurable Parameters of Processing Elements to maximize relevant information: Y’’(X | q, R) > Y’(X | q, R), with resource constraint. Required resource-efficient algorithms for: Classification, routing and filtering of signal-oriented data: (audio, video and, possibly, sensor data) ▪ Input data X – Queries q – Resource R – Y(X | q): Relevant information – Y’(X | q, R) ` Y(X | q): Achievable subset given R 5 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis © 2023 CY Lin, Columbia University
Example: Distributed Video Signal Understanding (Lin et al.) (Server) Concept Detection TV broadcast, (Distributed Smart Sensors) Block diagram of the Processing Elements VCR, smart sensors DVD discs, PE1: 9.2.63.66: 1220 Video Face PE4: 9.2.63.66:1235 File Database, MPEG- Female Webcam CDS Encod- 1/2 GOP Feature Event Features PE2: 9.2.63.67 Meta- PE5: 9.2.63.66: 1240 ing 1.5 Extraction Extraction Extraction Outdoors data 320 22.4 2.8 Male Mbps Kbps Kbps Kbps PE3: 9.2.63.68 PE6: 9.2.63.66 600 Indoors Airplane bps Sensor 1 PE7: 9.2.63.66 Sensor 2 Chair Sensor 3 Smart Cam Sensor N PE100: 9.2.63.66 Clock Display and Information Aggregation User Resource Modules Interests Control Constraints Modules 6 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis © 2023 CY Lin, Columbia University
Semantic Concept Filters E.g.: 7 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis © 2023 CY Lin, Columbia University
Complexity Reduction Introduction • Objective: Real-time classification of instances using Support Vector Machines (SVMs) • Computationally efficient and reasonably accurate solutions • Techniques capable of adjusting tradeoff between accuracy and speed based on available computational resources 8 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis © 2023 CY Lin, Columbia University
SVM formulation SVM 9 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis © 2023 CY Lin, Columbia University
Decision 10 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis © 2023 CY Lin, Columbia University
Problems 11 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis © 2023 CY Lin, Columbia University
Example 12 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis © 2023 CY Lin, Columbia University
Naïve Approach I – Feature Dimension Reduction Feature Dimension 0.8 Slice Color Texture Ratio AP 3 3 3 1 0.7861 3 3 2 0.666666667 0.7861 0.6 3 2 3 0.666666667 0.7757 2 3 3 0.666666667 0.5822 0.4 3 2 2 0.444444444 0.7757 2 3 2 0.444444444 0.5822 0.2 2 2 3 0.444444444 0.5235 Accuracy -- Average Precision 3 3 1 0.333333333 0.4685 0 0 0.25 0.5 0.75 1 3 1 3 0.333333333 0.6581 1 3 3 0.333333333 0.1684 Complexity -- Feature Dimension Ratio 2 2 2 0.296296296 0.5235 3 2 1 0.222222222 0.427 ▪ Experimental Results for Weather_News 3 1 2 0.222222222 0.6581 Detector 2 3 1 0.222222222 0.1241 ▪ Model Selection based on the Model 2 1 3 0.222222222 0.3457 Validation Set 1 3 2 0.222222222 0.1684 ▪ E.g., for Feature Dimension Ratio 0.22, (the 1 2 3 0.222222222 0.1065 best selection of features are: 3 slices, 1 2 2 1 0.148148148 0.0699 color, 2 texture selections), the accuracy is decreased by 17%. 2 1 2 0.148148148 0.3457 1 2 2 0.148148148 0.1065 3 1 1 0.111111111 0.3219 1 3 1 0.111111111 0.0314 1 1 3 0.111111111 0.07 13 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis2 1 © 2023 CY Lin, Columbia University1 0.074074074 0.0318
Naïve Approach II – Reduction on the Number of Support 0.9 0.675 0.45 0.225 Accuracy - Average Precision 0 0 0.25 0.5 0.75 1 Complexity -- Number of Support Vectors ▪ Proposed Novel Reduction Methods: – Ranked Weighting – P/N Cost Reduction – Random Selection – Support Vector Clustering and Centralization ▪ Experimental Results on Weather_News Detectors show that complexity can be at 50% for the cost of 14% decrease on accuracy 14 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis © 2023 CY Lin, Columbia University
Weighted Clustering Approach 15 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis © 2023 CY Lin, Columbia University
Cluster center weight (contd.) 16 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis © 2023 CY Lin, Columbia University
Using the weights 17 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis © 2023 CY Lin, Columbia University
Experiments • Datasets • TREC video datasets (2003 and 2005) • 576 features per instance • > 20000 test instances overall • MNist handwritten digit dataset (RBF kernel) • 576 features • 60000 training instances, 10000 test instances • Performance metrics • Speedup achieved over evaluation with all support vectors • Average precision achieved 18 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis © 2023 CY Lin, Columbia University
Results (Mnist 0-4) 1 0.9875 0 1 2 0.975 3 4 Average Precision 0.9625 0.95 0 75 150 225 300 Speedup Ratio 19 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis © 2023 CY Lin, Columbia University
Results (Mnist 5-9) 1 0.925 5 6 7 0.85 8 9 Average Precision0.775 0.7 1 10 100 1000 Speedup Ratio 20 E6893 Big Data Analytics – Lecture 4: Real-Time Stream Analysis © 2023 CY Lin, Columbia University
最初の20ページを表示しています。全文を見るには資料をダウンロードしてください。
まだコメントはありません。最初のコメントを書きましょう！
コメント (0)
まだコメントはありません。最初のコメントを書きましょう！