Trang 2
Contents Website vii Acknowledgments viii Notation xi 1 Introduction 1 1.1 Who Should Read This Book? . . . . . . . . . . . . . . . . . . . . 8 1.2 Historical Trends in Deep Learning . . . . . . . . . . . . . . . . . 11 I Applied Math and Machine Learning Basics 29 2 Linear Algebra 31 2.1 Scalars, Vectors, Matrices and Tensors . . . . . . . . . . . . . . . 31 2.2 Multiplying Matrices and Vectors . . . . . . . . . . . . . . . . . . 34 2.3 Identity and Inverse Matrices . . . . . . . . . . . . . . . . . . . . 36 2.4 Linear Dependence and Span . . . . . . . . . . . . . . . . . . . . 37 2.5 Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.6 Special Kinds of Matrices and Vectors . . . . . . . . . . . . . . . 40 2.7 Eigendecomposition . . . . . . . . . . . . . . . . . . . . . . . . . . 42 2.8 Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . 44 2.9 The Moore-Penrose Pseudoinverse . . . . . . . . . . . . . . . . . . 45 2.10 The Trace Operator . . . . . . . . . . . . . . . . . . . . . . . . . 46 2.11 The Determinant . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 2.12 Example: Principal Components Analysis . . . . . . . . . . . . . 48 3 Probability and Information Theory 53 3.1 Why Probability? . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 i
Trang 3
CONTENTS 3.2 Random Variables . . . . . . . . . . . . . . . . . . . . . . . . . . 56 3.3 Probability Distributions . . . . . . . . . . . . . . . . . . . . . . . 56 3.4 Marginal Probability . . . . . . . . . . . . . . . . . . . . . . . . . 58 3.5 Conditional Probability . . . . . . . . . . . . . . . . . . . . . . . 59 3.6 The Chain Rule of Conditional Probabilities . . . . . . . . . . . . 59 3.7 Independence and Conditional Independence . . . . . . . . . . . . 60 3.8 Expectation, Variance and Covariance . . . . . . . . . . . . . . . 60 3.9 Common Probability Distributions . . . . . . . . . . . . . . . . . 62 3.10 Useful Properties of Common Functions . . . . . . . . . . . . . . 67 3.11 Bayes’ Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 3.12 Technical Details of Continuous Variables . . . . . . . . . . . . . 71 3.13 Information Theory . . . . . . . . . . . . . . . . . . . . . . . . . . 73 3.14 Structured Probabilistic Models . . . . . . . . . . . . . . . . . . . 75 4 Numerical Computation 80 4.1 Overflow and Underflow . . . . . . . . . . . . . . . . . . . . . . . 80 4.2 Poor Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.3 Gradient-Based Optimization . . . . . . . . . . . . . . . . . . . . 82 4.4 Constrained Optimization . . . . . . . . . . . . . . . . . . . . . . 93 4.5 Example: Linear Least Squares . . . . . . . . . . . . . . . . . . . 96 5 Machine Learning Basics 98 5.1 Learning Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . 99 5.2 Capacity, Overfitting and Underfitting . . . . . . . . . . . . . . . 110 5.3 Hyperparameters and Validation Sets . . . . . . . . . . . . . . . . 120 5.4 Estimators, Bias and Variance . . . . . . . . . . . . . . . . . . . . 122 5.5 Maximum Likelihood Estimation . . . . . . . . . . . . . . . . . . 131 5.6 Bayesian Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . 135 5.7 Supervised Learning Algorithms . . . . . . . . . . . . . . . . . . . 140 5.8 Unsupervised Learning Algorithms . . . . . . . . . . . . . . . . . 146 5.9 Stochastic Gradient Descent . . . . . . . . . . . . . . . . . . . . . 151 5.10 Building a Machine Learning Algorithm . . . . . . . . . . . . . . 153 5.11 Challenges Motivating Deep Learning . . . . . . . . . . . . . . . . 155 II Deep Networks: Modern Practices 166 6 Deep Feedforward Networks 168 6.1 Example: Learning XOR . . . . . . . . . . . . . . . . . . . . . . . 171 6.2 Gradient-Based Learning . . . . . . . . . . . . . . . . . . . . . . . 177 ii
Trang 4
CONTENTS 6.3 Hidden Units . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 6.4 Architecture Design . . . . . . . . . . . . . . . . . . . . . . . . . . 197 6.5 Back-Propagation and Other Differentiation Algorithms . . . . . 204 6.6 Historical Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 7 Regularization for Deep Learning 228 7.1 Parameter Norm Penalties . . . . . . . . . . . . . . . . . . . . . . 230 7.2 Norm Penalties as Constrained Optimization . . . . . . . . . . . . 237 7.3 Regularization and Under-Constrained Problems . . . . . . . . . 239 7.4 Dataset Augmentation . . . . . . . . . . . . . . . . . . . . . . . . 240 7.5 Noise Robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 7.6 Semi-Supervised Learning . . . . . . . . . . . . . . . . . . . . . . 243 7.7 Multi-Task Learning . . . . . . . . . . . . . . . . . . . . . . . . . 244 7.8 Early Stopping . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246 7.9 Parameter Tying and Parameter Sharing . . . . . . . . . . . . . . 253 7.10 Sparse Representations . . . . . . . . . . . . . . . . . . . . . . . . 254 7.11 Bagging and Other Ensemble Methods . . . . . . . . . . . . . . . 256 7.12 Dropout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 7.13 Adversarial Training . . . . . . . . . . . . . . . . . . . . . . . . . 268 7.14 Tangent Distance, Tangent Prop, and Manifold Tangent Classifier 270 8 Optimization for Training Deep Models 274 8.1 How Learning Differs from Pure Optimization . . . . . . . . . . . 275 8.2 Challenges in Neural Network Optimization . . . . . . . . . . . . 282 8.3 Basic Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 8.4 Parameter Initialization Strategies . . . . . . . . . . . . . . . . . 301 8.5 Algorithms with Adaptive Learning Rates . . . . . . . . . . . . . 306 8.6 Approximate Second-Order Methods . . . . . . . . . . . . . . . . 310 8.7 Optimization Strategies and Meta-Algorithms . . . . . . . . . . . 317 9 Convolutional Networks 330 9.1 The Convolution Operation . . . . . . . . . . . . . . . . . . . . . 331 9.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 9.3 Pooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339 9.4 Convolution and Pooling as an Infinitely Strong Prior . . . . . . . 345 9.5 Variants of the Basic Convolution Function . . . . . . . . . . . . 347 9.6 Structured Outputs . . . . . . . . . . . . . . . . . . . . . . . . . . 358 9.7 Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360 9.8 Efficient Convolution Algorithms . . . . . . . . . . . . . . . . . . 362 9.9 Random or Unsupervised Features . . . . . . . . . . . . . . . . . 363 iii
Trang 5
CONTENTS 9.10 The Neuroscientific Basis for Convolutional Networks . . . . . . . 364 9.11 Convolutional Networks and the History of Deep Learning . . . . 371 10 Sequence Modeling: Recurrent and Recursive Nets 373 10.1 Unfolding Computational Graphs . . . . . . . . . . . . . . . . . . 375 10.2 Recurrent Neural Networks . . . . . . . . . . . . . . . . . . . . . 378 10.3 Bidirectional RNNs . . . . . . . . . . . . . . . . . . . . . . . . . . 394 10.4 Encoder-Decoder Sequence-to-Sequence Architectures . . . . . . . 396 10.5 Deep Recurrent Networks . . . . . . . . . . . . . . . . . . . . . . 398 10.6 Recursive Neural Networks . . . . . . . . . . . . . . . . . . . . . . 400 10.7 The Challenge of Long-Term Dependencies . . . . . . . . . . . . . 401 10.8 Echo State Networks . . . . . . . . . . . . . . . . . . . . . . . . . 404 10.9 Leaky Units and Other Strategies for Multiple Time Scales . . . . 406 10.10 The Long Short-Term Memory and Other Gated RNNs . . . . . . 408 10.11 Optimization for Long-Term Dependencies . . . . . . . . . . . . . 413 10.12 Explicit Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . 416 11 Practical Methodology 421 11.1 Performance Metrics . . . . . . . . . . . . . . . . . . . . . . . . . 422 11.2 Default Baseline Models . . . . . . . . . . . . . . . . . . . . . . . 425 11.3 Determining Whether to Gather More Data . . . . . . . . . . . . 426 11.4 Selecting Hyperparameters . . . . . . . . . . . . . . . . . . . . . . 427 11.5 Debugging Strategies . . . . . . . . . . . . . . . . . . . . . . . . . 436 11.6 Example: Multi-Digit Number Recognition . . . . . . . . . . . . . 440 12 Applications 443 12.1 Large-Scale Deep Learning . . . . . . . . . . . . . . . . . . . . . . 443 12.2 Computer Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . 452 12.3 Speech Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . 458 12.4 Natural Language Processing . . . . . . . . . . . . . . . . . . . . 461 12.5 Other Applications . . . . . . . . . . . . . . . . . . . . . . . . . . 478 III Deep Learning Research 486 13 Linear Factor Models 489 13.1 Probabilistic PCA and Factor Analysis . . . . . . . . . . . . . . . 490 13.2 Independent Component Analysis (ICA) . . . . . . . . . . . . . . 491 13.3 Slow Feature Analysis . . . . . . . . . . . . . . . . . . . . . . . . 493 13.4 Sparse Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496 iv
Trang 6
CONTENTS 13.5 Manifold Interpretation of PCA . . . . . . . . . . . . . . . . . . . 499 14 Autoencoders 502 14.1 Undercomplete Autoencoders . . . . . . . . . . . . . . . . . . . . 503 14.2 Regularized Autoencoders . . . . . . . . . . . . . . . . . . . . . . 504 14.3 Representational Power, Layer Size and Depth . . . . . . . . . . . 508 14.4 Stochastic Encoders and Decoders . . . . . . . . . . . . . . . . . . 509 14.5 Denoising Autoencoders . . . . . . . . . . . . . . . . . . . . . . . 510 14.6 Learning Manifolds with Autoencoders . . . . . . . . . . . . . . . 515 14.7 Contractive Autoencoders . . . . . . . . . . . . . . . . . . . . . . 521 14.8 Predictive Sparse Decomposition . . . . . . . . . . . . . . . . . . 523 14.9 Applications of Autoencoders . . . . . . . . . . . . . . . . . . . . 524 15 Representation Learning 526 15.1 Greedy Layer-Wise Unsupervised Pretraining . . . . . . . . . . . 528 15.2 Transfer Learning and Domain Adaptation . . . . . . . . . . . . . 536 15.3 Semi-Supervised Disentangling of Causal Factors . . . . . . . . . 541 15.4 Distributed Representation . . . . . . . . . . . . . . . . . . . . . . 546 15.5 Exponential Gains from Depth . . . . . . . . . . . . . . . . . . . 553 15.6 Providing Clues to Discover Underlying Causes . . . . . . . . . . 554 16 Structured Probabilistic Models for Deep Learning 558 16.1 The Challenge of Unstructured Modeling . . . . . . . . . . . . . . 559 16.2 Using Graphs to Describe Model Structure . . . . . . . . . . . . . 563 16.3 Sampling from Graphical Models . . . . . . . . . . . . . . . . . . 580 16.4 Advantages of Structured Modeling . . . . . . . . . . . . . . . . . 582 16.5 Learning about Dependencies . . . . . . . . . . . . . . . . . . . . 582 16.6 Inference and Approximate Inference . . . . . . . . . . . . . . . . 584 16.7 The Deep Learning Approach to Structured Probabilistic Models 585 17 Monte Carlo Methods 590 17.1 Sampling and Monte Carlo Methods . . . . . . . . . . . . . . . . 590 17.2 Importance Sampling . . . . . . . . . . . . . . . . . . . . . . . . . 592 17.3 Markov Chain Monte Carlo Methods . . . . . . . . . . . . . . . . 595 17.4 Gibbs Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 599 17.5 The Challenge of Mixing between Separated Modes . . . . . . . . 599 18 Confronting the Partition Function 605 18.1 The Log-Likelihood Gradient . . . . . . . . . . . . . . . . . . . . 606 18.2 Stochastic Maximum Likelihood and Contrastive Divergence . . . 607 v
Trang 7
CONTENTS 18.3 Pseudolikelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . 615 18.4 Score Matching and Ratio Matching . . . . . . . . . . . . . . . . 617 18.5 Denoising Score Matching . . . . . . . . . . . . . . . . . . . . . . 619 18.6 Noise-Contrastive Estimation . . . . . . . . . . . . . . . . . . . . 620 18.7 Estimating the Partition Function . . . . . . . . . . . . . . . . . . 623 19 Approximate Inference 631 19.1 Inference as Optimization . . . . . . . . . . . . . . . . . . . . . . 633 19.2 Expectation Maximization . . . . . . . . . . . . . . . . . . . . . . 634 19.3 MAP Inference and Sparse Coding . . . . . . . . . . . . . . . . . 635 19.4 Variational Inference and Learning . . . . . . . . . . . . . . . . . 638 19.5 Learned Approximate Inference . . . . . . . . . . . . . . . . . . . 651 20 Deep Generative Models 654 20.1 Boltzmann Machines . . . . . . . . . . . . . . . . . . . . . . . . . 654 20.2 Restricted Boltzmann Machines . . . . . . . . . . . . . . . . . . . 656 20.3 Deep Belief Networks . . . . . . . . . . . . . . . . . . . . . . . . . 660 20.4 Deep Boltzmann Machines . . . . . . . . . . . . . . . . . . . . . . 663 20.5 Boltzmann Machines for Real-Valued Data . . . . . . . . . . . . . 676 20.6 Convolutional Boltzmann Machines . . . . . . . . . . . . . . . . . 683 20.7 Boltzmann Machines for Structured or Sequential Outputs . . . . 685 20.8 Other Boltzmann Machines . . . . . . . . . . . . . . . . . . . . . 686 20.9 Back-Propagation through Random Operations . . . . . . . . . . 687 20.10 Directed Generative Nets . . . . . . . . . . . . . . . . . . . . . . . 692 20.11 Drawing Samples from Autoencoders . . . . . . . . . . . . . . . . 711 20.12 Generative Stochastic Networks . . . . . . . . . . . . . . . . . . . 714 20.13 Other Generation Schemes . . . . . . . . . . . . . . . . . . . . . . 716 20.14 Evaluating Generative Models . . . . . . . . . . . . . . . . . . . . 717 20.15 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 720 Bibliography 721 Index 777 vi
Trang 8
Website www.deeplearningbook.org This book is accompanied by the above website. The website provides a variety of supplementary material, including exercises, lecture slides, corrections of mistakes, and other resources that should be useful to both readers and instructors. vii
Trang 9
Acknowledgments This book would not have been possible without the contributions of many people. We would like to thank those who commented on our proposal for the book and helped plan its contents and organization: Guillaume Alain, Kyunghyun Cho, Çağlar Gülçehre, David Krueger, Hugo Larochelle, Razvan Pascanu and Thomas Rohée. We would like to thank the people who offered feedback on the content of the book itself. Some offered feedback on many chapters: Martín Abadi, Guillaume Alain, Ion Androutsopoulos, Fred Bertsch, Olexa Bilaniuk, Ufuk Can Biçici, Matko Bošnjak, John Boersma, Greg Brockman, Alexandre de Brébisson, Pierre Luc Carrier, Sarath Chandar, Pawel Chilinski, Mark Daoust, Oleg Dashevskii, Laurent Dinh, Stephan Dreseitl, Jim Fan, Miao Fan, Meire Fortunato, Frédéric Francis, Nando de Freitas, Çağlar Gülçehre, Jurgen Van Gael, Javier Alonso García, Jonathan Hunt, Gopi Jeyaram, Chingiz Kabytayev, Lukasz Kaiser, Varun Kanade, Akiel Khan, John King, Diederik P. Kingma, Yann LeCun, Rudolf Mathey, Matías Mattamala, Abhinav Maurya, Kevin Murphy, Oleg Mürk, Roman Novak, Augustus Q. Odena, Simon Pavlik, Karl Pichotta, Kari Pulli, Roussel Rahman, Tapani Raiko, Anurag Ranjan, Johannes Roith, Mihaela Rosca, Halis Sak, César Salgado, Grigory Sapunov, Yoshinori Sasaki, Mike Schuster, Julian Serban, Nir Shabat, Ken Shirriff, Andre Simpelo, Scott Stanley, David Sussillo, Ilya Sutskever, Carles Gelada Sáez, Graham Taylor, Valentin Tolmer, An Tran, Shubhendu Trivedi, Alexey Umnov, Vincent Vanhoucke, Marco Visentini-Scarzanella, David Warde-Farley, Dustin Webb, Kelvin Xu, Wei Xue, Ke Yang, Li Yao, Zygmunt Zając and Ozan Çağlayan. We would also like to thank those who provided us with useful feedback on individual chapters: • Notation: Zhang Yuanhang. • Chapter 1 Introduction, : Yusuf Akgul, Sebastien Bratieres, Samira Ebrahimi, Charlie Gorichanaz, Brendan Loudermilk, Eric Morris, Cosmin Pârvulescu viii
Trang 10
CONTENTS and Alfredo Solano. • Chapter 2, Linear Algebra: Amjad Almahairi, Nikola Banić, Kevin Bennett, Philippe Castonguay, Oscar Chang, Eric Fosler-Lussier, Andrey Khalyavin, Sergey Oreshkov, István Petrás, Dennis Prangle, Thomas Rohée, Colby Toland, Massimiliano Tomassoli, Alessandro Vitale and Bob Welland. • Chapter 3, Probability and Information Theory: John Philip Anderson, Kai Arulkumaran, Vincent Dumoulin, Rui Fa, Stephan Gouws, Artem Oboturov, Antti Rasmus, Alexey Surkov and Volker Tresp. • Chapter 4, Numerical Computation: Tran Lam An, Ian Fischer, and Hu Yuhuang. • Chapter 5, Machine Learning Basics: Dzmitry Bahdanau, Nikhil Garg, Makoto Otsuka, Bob Pepin, Philip Popien, Emmanuel Rayner, Kee-Bong Song, Zheng Sun and Andy Wu. • Chapter 6, Deep Feedforward Networks: Uriel Berdugo, Fabrizio Bottarel, Elizabeth Burl, Ishan Durugkar, Jeff Hlywa, Jong Wook Kim, David Krueger and Aditya Kumar Praharaj. • Chapter 7, Regularization for Deep Learning: Kshitij Lauria, Inkyu Lee, Sunil Mohan and Joshua Salisbury. • Chapter 8, Optimization for Training Deep Models: Marcel Ackermann, Rowel Atienza, Andrew Brock, Tegan Maharaj, James Martens, Klaus Strobl and Martin Vita. • Chapter 9, Convolutional Networks: Martín Arjovsky, Eugene Brevdo, Kon- stantin Divilov, Eric Jensen, Asifullah Khan, Mehdi Mirza, Alex Paino, Eddie Pierce, Marjorie Sayer, Ryan Stout and Wentao Wu. • Chapter 10, Sequence Modeling: Recurrent and Recursive Nets: Gökçen Eraslan, Steven Hickson, Razvan Pascanu, Lorenzo von Ritter, Rui Rodrigues, Dmitriy Serdyuk, Dongyu Shi and Kaiyu Yang. • Chapter 11, Practical Methodology: Daniel Beckstein. • Chapter 12, Applications: George Dahl and Ribana Roscher. • Chapter 15, Representation Learning: Kunal Ghosh. ix
Trang 11
CONTENTS • Chapter 16, Structured Probabilistic Models for Deep Learning: Minh Lê and Anton Varfolom. • Chapter 18, Confronting the Partition Function: Sam Bowman. • Chapter 19, Approximate Inference: Yujia Bao. • Chapter 20, Deep Generative Models: Nicolas Chapados, Daniel Galvez, Wenming Ma, Fady Medhat, Shakir Mohamed and Grégoire Montavon. • Bibliography: Lukas Michelbacher and Leslie N. Smith. We also want to thank those who allowed us to reproduce images, figures or data from their publications. We indicate their contributions in the figure captions throughout the text. We would like to thank Lu Wang for writing pdf2htmlEX, which we used to make the web version of the book, and for offering support to improve the quality of the resulting HTML. We would like to thank Ian’s wife Daniela Flori Goodfellow for patiently supporting Ian during the writing of the book as well as for help with proofreading. We would like to thank the Google Brain team for providing an intellectual environment where Ian could devote a tremendous amount of time to writing this book and receive feedback and guidance from colleagues. We would especially like to thank Ian’s former manager, Greg Corrado, and his current manager, Samy Bengio, for their support of this project. Finally, we would like to thank Geoffrey Hinton for encouragement when writing was difficult. x
Trang 12
Notation This section provides a concise reference describing the notation used throughout this book. If you are unfamiliar with any of the corresponding mathematical concepts, this notation reference may seem intimidating. However, do not despair, we describe most of these ideas in chapters 2-4. Numbers and Arrays a A scalar (integer or real) a A vector A A matrix A A tensor In Identity matrix with n rows and n columns I Identity matrix with dimensionality implied by context e( )i Standard basis vector [0, . . . ,0,1, 0, . . . ,0] with a 1 at position i diag(a) A square, diagonal matrix with diagonal entries given by a a A scalar random variable a A vector-valued random variable A A matrix-valued random variable xi
Trang 13
CONTENTS Sets and Graphs A A set R The set of real numbers {0 1, } The set containing 0 and 1 {0 1, , . . . , n} The set of all integers between 0 and n [a, b] The real interval including a and b (a, b] The real interval excluding a but including b A B\ Set subtraction, i.e., the set containing the ele- ments of A that are not in B G A graph P aG(xi) The parents of xi in G Indexing ai Element i of vector a, with indexing starting at 1 a−i All elements of vector a except for element i Ai,j Element i, j of matrix A Ai,: Row i of matrix A A:,i Column i of matrix A Ai,j,k Element (i, j, k) of a 3-D tensor A A: :, ,i 2-D slice of a 3-D tensor ai Element i of the random vector a Linear Algebra Operations A Transpose of matrix A A+ Moore-Penrose pseudoinverse of A A B Element-wise (Hadamard) product of A and B det(A) Determinant of A xii
Trang 14
CONTENTS Calculus dy Derivative of y with respect to x dx ∂y Partial derivative of y with respect to x ∂x ∇xy Gradient of y with respect to x ∇X y Matrix derivatives of y with respect to X ∇Xy Tensor containing derivatives of y with respect to X ∂f m×n n m Jacobian matrix J ∈ R of f : R → R ∂x ∇2 f(x) or H(f)(x) The Hessian matrix of f at input point x x f(x)dx Definite integral over the entire domain of x f(x)dx Definite integral with respect to x over the set S S Probability and Information Theory a⊥b The random variables a and b are independent a⊥b | c They are are conditionally independent given c P( )a A probability distribution over a discrete variable p( )a A probability distribution over a continuous vari- able, or over a variable whose type has not been specified a ∼ P Random variable a has distribution P Ex∼P[ ( )]f x or Ef x( ) Expectation of f x( ) with respect to P( )x Var( ( ))f x Variance of f x( ) under P( )x Cov( ( )f x , g x( )) Covariance of f x( ) and g x( ) under P( )x H( )x Shannon entropy of the random variable x DKL(PQ) Kullback-Leibler divergence of P and Q N(x μ; ,Σ) Gaussian distribution over x with mean μ and covariance Σ xiii
Trang 15
CONTENTS Functions f : A → B The function f with domain A and range B f ◦ g Composition of the functions f and g f(x θ; ) A function of x parametrized by θ. Sometimes we just write f(x) and ignore the argument θ to lighten notation. logx Natural logarithm of x 1 σ x( ) Logistic sigmoid, 1 + exp(−x) ζ x( ) Softplus, log(1 + exp(x)) ||x|| Lp norm of x p ||x|| L2 norm of x x+ Positive part of x, i.e., max(0, x) 1condition is 1 if the condition is true, 0 otherwise Sometimes we use a function f whose argument is a scalar, but apply it to a vector, matrix, or tensor: f(x), f(X), or f(X). This means to apply f to the array element-wise. For example, if C = σ(X), then Ci,j,k = σ(Xi,j,k) for all valid values of i, j and k. Datasets and distributions pdata The data generating distribution pˆdata The empirical distribution defined by the training set X A set of training examples x( )i The i-th example (input) from a dataset y( )i or y( )i The target associated with x( )i for supervised learn- ing X The m × n matrix with input example x( )i in row Xi,: xiv
Trang 16
Chapter 1 Introduction Inventors have long dreamed of creating machines that think. This desire dates back to at least the time of ancient Greece. The mythical figures Pygmalion, Daedalus, and Hephaestus may all be interpreted as legendary inventors, and Galatea, Talos, and Pandora may all be regarded as artificial life (Ovid and Martin, 2004; Sparkes, 1996; Tandy, 1997). When programmable computers were first conceived, people wondered whether they might become intelligent, over a hundred years before one was built (Lovelace, 1842). Today, artificial intelligence (AI) is a thriving field with many practical applications and active research topics. We look to intelligent software to automate routine labor, understand speech or images, make diagnoses in medicine and support basic scientific research. In the early days of artificial intelligence, the field rapidly tackled and solved problems that are intellectually difficult for human beings but relatively straight- forward for computers—problems that can be described by a list of formal, math- ematical rules. The true challenge to artificial intelligence proved to be solving the tasks that are easy for people to perform but hard for people to describe formally—problems that we solve intuitively, that feel automatic, like recognizing spoken words or faces in images. This book is about a solution to these more intuitive problems. This solution is to allow computers to learn from experience and understand the world in terms of a hierarchy of concepts, with each concept defined in terms of its relation to simpler concepts. By gathering knowledge from experience, this approach avoids the need for human operators to formally specify all of the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones. If we draw a graph showing how these 1
Trang 17
CHAPTER 1. INTRODUCTION concepts are built on top of each other, the graph is deep, with many layers. For this reason, we call this approach to AI deep learning. Many of the early successes of AI took place in relatively sterile and formal environments and did not require computers to have much knowledge about the world. For example, IBM’s Deep Blue chess-playing system defeated world champion Garry Kasparov in 1997 (Hsu, 2002). Chess is of course a very simple world, containing only sixty-four locations and thirty-two pieces that can move in only rigidly circumscribed ways. Devising a successful chess strategy is a tremendous accomplishment, but the challenge is not due to the difficulty of describing the set of chess pieces and allowable moves to the computer. Chess can be completely described by a very brief list of completely formal rules, easily provided ahead of time by the programmer. Ironically, abstract and formal tasks that are among the most difficult mental undertakings for a human being are among the easiest for a computer. Computers have long been able to defeat even the best human chess player, but are only recently matching some of the abilities of average human beings to recognize objects or speech. A person’s everyday life requires an immense amount of knowledge about the world. Much of this knowledge is subjective and intuitive, and therefore difficult to articulate in a formal way. Computers need to capture this same knowledge in order to behave in an intelligent way. One of the key challenges in artificial intelligence is how to get this informal knowledge into a computer. Several artificial intelligence projects have sought to hard-code knowledge about the world in formal languages. A computer can reason about statements in these formal languages automatically using logical inference rules. This is known as the knowledge base approach to artificial intelligence. None of these projects has led to a major success. One of the most famous such projects is Cyc (Lenat and Guha, 1989). Cyc is an inference engine and a database of statements in a language called CycL. These statements are entered by a staff of human supervisors. It is an unwieldy process. People struggle to devise formal rules with enough complexity to accurately describe the world. For example, Cyc failed to understand a story about a person named Fred shaving in the morning (Linde, 1992). Its inference engine detected an inconsistency in the story: it knew that people do not have electrical parts, but because Fred was holding an electric razor, it believed the entity “FredWhileShaving” contained electrical parts. It therefore asked whether Fred was still a person while he was shaving. The difficulties faced by systems relying on hard-coded knowledge suggest that AI systems need the ability to acquire their own knowledge, by extracting patterns from raw data. This capability is known as machine learning. The introduction 2
Trang 18
CHAPTER 1. INTRODUCTION of machine learning allowed computers to tackle problems involving knowledge of the real world and make decisions that appear subjective. A simple machine learning algorithm called logistic regression can determine whether to recommend cesarean delivery (Mor-Yosef et al., 1990). A simple machine learning algorithm called naive Bayes can separate legitimate e-mail from spam e-mail. The performance of these simple machine learning algorithms depends heavily on the representation of the data they are given. For example, when logistic regression is used to recommend cesarean delivery, the AI system does not examine the patient directly. Instead, the doctor tells the system several pieces of relevant information, such as the presence or absence of a uterine scar. Each piece of information included in the representation of the patient is known as a feature. Logistic regression learns how each of these features of the patient correlates with various outcomes. However, it cannot influence the way that the features are defined in any way. If logistic regression was given an MRI scan of the patient, rather than the doctor’s formalized report, it would not be able to make useful predictions. Individual pixels in an MRI scan have negligible correlation with any complications that might occur during delivery. This dependence on representations is a general phenomenon that appears throughout computer science and even daily life. In computer science, opera- tions such as searching a collection of data can proceed exponentially faster if the collection is structured and indexed intelligently. People can easily perform arithmetic on Arabic numerals, but find arithmetic on Roman numerals much more time-consuming. It is not surprising that the choice of representation has an enormous effect on the performance of machine learning algorithms. For a simple visual example, see Fig. 1.1. Many artificial intelligence tasks can be solved by designing the right set of features to extract for that task, then providing these features to a simple machine learning algorithm. For example, a useful feature for speaker identification from sound is an estimate of the size of speaker’s vocal tract. It therefore gives a strong clue as to whether the speaker is a man, woman, or child. However, for many tasks, it is difficult to know what features should be extracted. For example, suppose that we would like to write a program to detect cars in photographs. We know that cars have wheels, so we might like to use the presence of a wheel as a feature. Unfortunately, it is difficult to describe exactly what a wheel looks like in terms of pixel values. A wheel has a simple geometric shape but its image may be complicated by shadows falling on the wheel, the sun glaring off the metal parts of the wheel, the fender of the car or an object in the foreground obscuring part of the wheel, and so on. 3
Trang 19
CHAPTER 1. INTRODUCTION Figure 1.1: Example of different representations: suppose we want to separate two categories of data by drawing a line between them in a scatterplot. In the plot on the left, we represent some data using Cartesian coordinates, and the task is impossible. In the plot on the right, we represent the data with polar coordinates and the task becomes simple to solve with a vertical line. (Figure produced in collaboration with David Warde-Farley) One solution to this problem is to use machine learning to discover not only the mapping from representation to output but also the representation itself. This approach is known as representation learning. Learned representations often result in much better performance than can be obtained with hand-designed representations. They also allow AI systems to rapidly adapt to new tasks, with minimal human intervention. A representation learning algorithm can discover a good set of features for a simple task in minutes, or a complex task in hours to months. Manually designing features for a complex task requires a great deal of human time and effort; it can take decades for an entire community of researchers. The quintessential example of a representation learning algorithm is the au- toencoder. An autoencoder is the combination of an encoder function that converts the input data into a different representation, and a decoder function that converts the new representation back into the original format. Autoencoders are trained to preserve as much information as possible when an input is run through the encoder and then the decoder, but are also trained to make the new representation have various nice properties. Different kinds of autoencoders aim to achieve different kinds of properties. When designing features or algorithms for learning features, our goal is usually to separate the factors of variation that explain the observed data. In this context, we use the word “factors” simply to refer to separate sources of influence; the factors are usually not combined by multiplication. Such factors are often not quantities 4
Trang 20
CHAPTER 1. INTRODUCTION that are directly observed. Instead, they may exist either as unobserved objects or unobserved forces in the physical world that affect observable quantities. They may also exist as constructs in the human mind that provide useful simplifying explanations or inferred causes of the observed data. They can be thought of as concepts or abstractions that help us make sense of the rich variability in the data. When analyzing a speech recording, the factors of variation include the speaker’s age, their sex, their accent and the words that they are speaking. When analyzing an image of a car, the factors of variation include the position of the car, its color, and the angle and brightness of the sun. A major source of difficulty in many real-world artificial intelligence applications is that many of the factors of variation influence every single piece of data we are able to observe. The individual pixels in an image of a red car might be very close to black at night. The shape of the car’s silhouette depends on the viewing angle. Most applications require us to disentangle the factors of variation and discard the ones that we do not care about. Of course, it can be very difficult to extract such high-level, abstract features from raw data. Many of these factors of variation, such as a speaker’s accent, can be identified only using sophisticated, nearly human-level understanding of the data. When it is nearly as difficult to obtain a representation as to solve the original problem, representation learning does not, at first glance, seem to help us. Deep learning solves this central problem in representation learning by introduc- ing representations that are expressed in terms of other, simpler representations. Deep learning allows the computer to build complex concepts out of simpler con- cepts. Fig. 1.2 shows how a deep learning system can represent the concept of an image of a person by combining simpler concepts, such as corners and contours, which are in turn defined in terms of edges. The quintessential example of a deep learning model is the feedforward deep network or multilayer perceptron (MLP). A multilayer perceptron is just a mathe- matical function mapping some set of input values to output values. The function is formed by composing many simpler functions. We can think of each application of a different mathematical function as providing a new representation of the input. The idea of learning the right representation for the data provides one perspec- tive on deep learning. Another perspective on deep learning is that depth allows the computer to learn a multi-step computer program. Each layer of the representation can be thought of as the state of the computer’s memory after executing another set of instructions in parallel. Networks with greater depth can execute more instructions in sequence. Sequential instructions offer great power because later instructions can refer back to the results of earlier instructions. According to this 5
Hiển thị 20 trang đầu. Tải tài liệu để xem đầy đủ.
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!