https://stanford.edu/~shervine CS 229 – Machine Learning VIP Cheatsheet: Deep Learning r Learning rate – The learning rate, often noted η, indicates at which pace the weights get updated. This can be fixed or adaptively changed. The current most popular method is called Adam, which is a method that adapts the learning rate. r Backpropagation – Backpropagation is a method to update the weights in the neural network by taking into account the actual output and the desired output. The derivative with respect to weight w is computed using chain rule and is of the following form: Afshine Amidi and Shervine Amidi August 23, 2018 ∂L(z,y) ∂L(z,y) ∂a ∂z = × × ∂w ∂a ∂z ∂w As a result, the weight is updated as follows: Neural Networks Neural networks are a class of models that are build with layers. Commonly used types of neural networks include convolutional and recurrent neural networks. w ←− w − η r Architecture – The vocabulary around neural networks architectures is described in the figure below: ∂L(z,y) ∂w r Updating weights – In a neural network, weights are updated as follows: Step 1: Take a batch of training data. Step 2: Perform forward propagation to obtain the corresponding loss. Step 3: Backpropagate the loss to get the gradients. By noting i the ith layer of the network and j the j th hidden unit of the layer, we have: [i] [i] T zj = wj Step 4: Use the gradients to update the weights of the network. [i] x + bj r Dropout – Dropout is a technique meant at preventing overfitting the training data by dropping out units in a neural network. In practice, neurons are either dropped with probability p or kept with probability 1 − p. where we note w, b, z the weight, bias and output respectively. r Activation function – Activation functions are used at the end of a hidden unit to introduce non-linear complexities to the model. Here are the most common ones: Sigmoid g(z) = 1 1 + e−z Tanh g(z) = ez − e−z ez + e−z ReLU Leaky ReLU Convolutional