Big Data Analytics Big Data Analytics Lars Schmidt-Thieme Information Systems and Machine Learning Lab (ISMLL) Institute of Computer Science University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 1 / 31
Big Data Analytics Syllabus Tue. 9.4. (1) 0. Introduction A. Parallel Computing Tue. 16.4. (2) A.1 Threads Tue. 23.4. (3) A.2 Message Passing Interface (MPI) Tue. 30.4. (4) A.3 Graphical Processing Units (GPUs) B. Distributed Storage Tue. 7.5. (5) B.1 Distributed File Systems Tue. 14.5. (6) B.2 Partioning of Relational Databases Tue. 21.5. (7) B.3 NoSQL Databases C. Distributed Computing Environments Tue. 28.5. (8) C.1 Map-Reduce Tue. 4.6. (9) C.2 Resilient Distributed Datasets (Spark) Tue. 11.6. — — Pentecoste Break — Tue. 18.6. (10) C.3 Computational Graphs (TensorFlow) D. Distributed Machine Learning Algorithms Tue. 25.6. (11) D.1 Distributed Stochastic Gradient Descent Tue. 2.7. (12) D.2 Distributed Matrix Factorization Tue. 9.7. (13) Questions and Answers Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 1 / 31
Big Data Analytics Outline 1. The Computational Graph 2. Variables 3. Example: Linear Regression 4. Automatic Gradients 5. Large Data I: Feeding 6. Large Data II: Reader Nodes 7. Debugging Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 1 / 31
Big Data Analytics 1. The Computational Graph Outline 1. The Computational Graph 2. Variables 3. Example: Linear Regression 4. Automatic Gradients 5. Large Data I: Feeding 6. Large Data II: Reader Nodes 7. Debugging Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 1 / 31
Big Data Analytics 1. The Computational Graph TensorFlow I Computational framework I multi-device, distributed I Core in C/C++, standard interface in Python I several further language bindings, e.g., R, Java I open source I developed by Google I initially released Nov. 2015 I 2nd generation framework I 1st generation framework was called DistBelief I alternative: pytorch I developed by facebook, since 2016, open source Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 1 / 31
Big Data Analytics 1. The Computational Graph Tensors I tensor = multidimensional array I rank = number of dimensions I shape = vector of sizes, one size for each dimension. rank common name shape 0 scalar () 1 vector (size) 2 matrix (numrows, numcols) ≥3 tensor of higher order (numdim1, numdim2, . . ., numdimr) I examples: 1.0 −3.0 2.3 1.7 A = 5.6 0.0 −1.3 3.4 , shape(A) = (3,4), rank(A) = 2 −7.7 −3.3 −2.1 5.2 Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 2 / 31
Big Data Analytics 1. The Computational Graph Computational Graphs I TensorFlow organizes a computation as a directed graph. I Nodes represent a tensor. I or a list of tensors. I Tensors can be: I stored tensors I immutable, value provided at creation time: tf.constant I immutable, value provided when running the graph: tf.placeholder I mutable: tf.Variable I computed tensors (operations): I having one or more input tensors I having one or more output tensors – output index: port I Edges represent dependencies. I Edge x → y if y is computed and x one of its inputs. Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 3 / 31
Big Data Analytics 1. The Computational Graph Sessions I A session represents the state of an ongoing computation on a computational graph. I create with default constructor tf.Session. I compute the value of a tensor node with run. Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 4 / 31
Big Data Analytics 1. The Computational Graph Two Phases 1. Construct the Computational Graph I create tensor nodes I possibly referencing other tensor nodes as inputs 2. Compute values of a node of the Computation Graph (running) I usually specify target tensor(s) I computes all intermediate tensors required for this tensor I yield the value of the target tensor(s) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 5 / 31
Big Data Analytics 1. The Computational Graph Hello TensorFlow: Add two Constants 1 import tensorflow as tf 2 3 a = tf.constant(3.0) 4 b = tf.constant(4.0) Output: 5 x = tf.add(a, b) 6 1 Tensor("Const:0", shape=(), dtype=float32) 7 print(a) 2 Tensor("Const_1:0", shape=(), dtype=float32) 8 print(b) 3 Tensor("Add:0", shape=(), dtype=float32) 9 print(x) 4 7 10 11 sess = tf.Session() 12 x_val = sess.run(x) 13 print(x_val) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 6 / 31
Big Data Analytics 1. The Computational Graph Hello TensorFlow: Add two Constants 1 import tensorflow as tf 2 3 a = tf.constant([3.0, -2.7, 1.2]) 4 b = tf.constant([4.0, 5.1, -1.7]) Output: 5 x = tf.add(a, b) 6 1 Tensor("Const_2:0", shape=(3,), dtype=float32) 7 print(a) 2 Tensor("Const_3:0", shape=(3,), dtype=float32) 8 print(b) 3 Tensor("Add_1:0", shape=(3,), dtype=float32) 9 print(x) 4 [ 7.0 2.4 -0.5 ] 10 11 sess = tf.Session() 12 x_val = sess.run(x) 13 print(x_val) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 7 / 31
Big Data Analytics 1. The Computational Graph Tensor Types I Different element types are represented by tf.DType: dtype description tf.float16 16-bit half-precision floating-point tf.float32 32-bit single-precision floating-point tf.float64 64-bit double-precision floating-point tf.bfloat16 16-bit truncated floating-point tf.complex64 64-bit single-precision complex tf.complex128 128-bit double-precision complex tf.int8 8-bit signed integer tf.uint8 8-bit unsigned integer tf.uint16 16-bit unsigned integer tf.int16 16-bit signed integer tf.int32 32-bit signed integer tf.int64 64-bit signed integer tf.bool Boolean tf.string String tf.qint8 Quantized 8-bit signed integer tf.quint8 Quantized 8-bit unsigned integer tf.qint16 Quantized 16-bit signed integer tf.quint16 Quantized 16-bit unsigned integer tf.qint32 Quantized 32-bit signed integer tf.resource Handle to a mutable resource I if omitted, inferred from values: Output: 1 a = tf.constant(4.0) 2 b = tf.constant(4) 1 Tensor("Const:0", shape=(), dtype=float32) 3 print(a) 2 Tensor("Const_1:0", shape=(), dtype=int32) 4 print(b) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 8 / 31
Big Data Analytics 1. The Computational Graph Operations: Overloaded Operators 1 import tensorflow as tf 2 3 a = tf.constant(3.0) 4 b = tf.constant(4.0) Output: 5 x = a + b 6 1 Tensor("add_1:0", shape=(), dtype=float32) 7 print(x) 2 7 8 9 sess = tf.Session() 10 x_val = sess.run(x) 11 print(x_val) operator operation node + tf.add - tf.subtract * tf.multiply / tf.divide Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 9 / 31
Big Data Analytics 2. Variables Outline 1. The Computational Graph 2. Variables 3. Example: Linear Regression 4. Automatic Gradients 5. Large Data I: Feeding 6. Large Data II: Reader Nodes 7. Debugging Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 10 / 31
Big Data Analytics 2. Variables v = tf.Variable(initial_value = None, . . . ,name = None, . . . , dtype = None) I has immutable element type I has mutable shape (set_shape). I has mutable element values. I set by tf.assign, tf.assign_add (operations) I separate values in each session I has to be initialized before first use: I run v.initializer operation or I run initializers of all variables: 1 init = tf.global_variables_initializer() 2 sess.run(init) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 10 / 31
Big Data Analytics 2. Variables Variables 1 import tensorflow as tf 2 Output: 3 x = tf.Variable(3.0) 4 sess = tf.Session() 1 3.0 5 sess.run( x.initializer ) 2 6 x_val = sess.run(x) 3 0 4.0 7 print(x_val) 4 1 5.0 8 5 2 6.0 9 x_plus_one = tf.assign_add(x, 1.0) 6 3 7.0 10 7 4 8.0 11 for t in range(5): 12 x_val = sess.run(x_plus_one) 13 print(t, x_val) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 11 / 31
Big Data Analytics 2. Variables Initializing from Other Variables 1 import tensorflow as tf 2 3 x = tf.Variable(3.0) 4 y = tf.Variable( tf.multiply(tf.constant(2.0), Output: 5 x.initialized_value()) ) 6 init = tf.global_variables_initializer() 1 6.0 7 sess = tf.Session() 8 sess.run(init) 9 y_val = sess.run(y) 10 print(y_val) I v.initialized_value assures that a variable has been initialized before I do not use 1 y = tf.Variable( tf.multiply(tf.constant(2.0), x) ) as x may be selected to be initialized after y. Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 12 / 31
Big Data Analytics 3. Example: Linear Regression Outline 1. The Computational Graph 2. Variables 3. Example: Linear Regression 4. Automatic Gradients 5. Large Data I: Feeding 6. Large Data II: Reader Nodes 7. Debugging Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 13 / 31
Big Data Analytics 3. Example: Linear Regression Example: Linear Regression yˆ := β0 + Xβ prediction r := y − yˆ residuum 1 ∑N ` := r2 loss/error 2 n n=1 ∂` T − := X r negative gradient w.r.t. β ∂β ∂` ∑N − := rn negative gradient w.r.t. β0 ∂β0 n=1 next ∂` β := β − η update of β ∂β next ∂` β0 := β0 − η update of β0 ∂β0 Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 13 / 31
Big Data Analytics 3. Example: Linear Regression Example: Linear Regression 1 import tensorflow as tf 2 3 Xs_data = [[2,1], [1,2], [4,3], [3,4]] 4 Ys_data = [[+1], [+1], [-1], [-1]] 5 eta_data = 0.01 6 7 Xs = tf.constant(Xs_data, dtype=tf.float32) 8 Ys = tf.constant(Ys_data, dtype=tf.float32) 9 eta = tf.constant(eta_data) 10 11 beta = tf.Variable([[0], [0]], dtype=tf.float32) 12 beta_0 = tf.Variable(0, dtype=tf.float32) 13 14 Yhats = tf.add(beta_0, tf.matmul(Xs, beta)) 15 residua = tf.subtract(Ys, Yhats) 16 error = tf.reduce_sum(tf.square(residua)) 17 18 neg_grad_beta = tf.matmul(Xs, residua, adjoint_a=True) 19 beta_update = tf.assign_add(beta, tf.multiply(eta, neg_grad_beta)) 20 beta_0_update = tf.assign_add(beta_0, tf.multiply(eta, tf.reduce_sum(residua))) 21 22 init = tf.global_variables_initializer() 23 sess = tf.Session() 24 sess.run( init ) 25 26 for t in range(100): 27 error_val, beta_val, beta_0_val = sess.run([error,beta_update,beta_0_update]) 28 print(t, error_val, beta_0_val, beta_val[0,0], beta_val[1,0]) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany C. Distributed Computing Environments / 3. Computational Graphs (TensorFlow) 14 / 31
Hiển thị 20 trang đầu. Tải tài liệu để xem đầy đủ.
Chưa có bình luận nào. Hãy là người đầu tiên!
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!