MLlib and Distributing the " Singular Value Decomposition Reza Zadeh Outline Example Invocations Benefits of Iterations Singular Value Decomposition All-pairs Similarity Computation MLlib + {Streaming, GraphX, SQL} Future Directions Introduction A General Platform Standard libraries included with Spark Spark Spark SQL Streaming " structured real-time GraphX graph Spark Core MLlib machine learning … MLlib History MLlib is a Spark subproject providing machine learning primitives Initial contribution from AMPLab, UC Berkeley Shipped with Spark since Sept 2013 MLlib: Available algorithms classification: logistic regression, linear SVM," naïve Bayes, least squares, classification tree regression: generalized linear models (GLMs), regression tree collaborative filtering: alternating least squares (ALS), non-negative matrix factorization (NMF) clustering: k-means|| decomposition: SVD, PCA optimization: stochastic gradient descent, L-BFGS Example Invocations Example: K-means Example: PCA Example: ALS Benefits of fast iterations Optimization At least two large classes of optimization problems humans can solve: Convex Programs Singular Value Decomposition Optimization - LR data = spark.textFile(...).map(readPoint).cache() w = numpy.random.rand(D) for i in range(iterations): gradient = data.map(lambda p: (1 / (1 + exp(-­‐p.y * w.dot(p.x)))) * p.y * p.x ).reduce(lambda a, b: a + b) w -­‐= gradient print “Final w: %s” % w MR PageRank Repeatedly multiply sparse matrix and vector Requires repeatedly hashing together page adjacency lists and rank vector Same file grouped over and over Neighbors (id, edges) Ranks (id, rank) … iteration 1 iteration 2 iteration 3 Spark PageRank Using cache(), keep neighbor lists in RAM Using partitioning, avoid repeated hashing partitionBy Neighbors (id, edges) Ranks (id, rank) … join join join Spark PageRank Using cache(), keep neighbor lists in RAM U