Trang 1
CS 229 – Machine Learning https://stanford.edu/~shervine VIP Cheatsheet: Deep Learning r Learning rate – The learning rate, often noted η, indicates at which pace the weights get updated. This can be fixed or adaptively changed. The current most popular method is called Adam, which is a method that adapts the learning rate. Afshine Amidi and Shervine Amidi r Backpropagation – Backpropagation is a method to update the weights in the neural network by taking into account the actual output and the desired output. The derivative with respect to weight w is computed using chain rule and is of the following form: August 23, 2018 ∂L(z,y) ∂L(z,y) ∂a ∂z = × × ∂w ∂a ∂z ∂w Neural Networks As a result, the weight is updated as follows: Neural networks are a class of models that are build with layers. Commonly used types of neural ∂L(z,y) networks include convolutional and recurrent neural networks. w ←− w − η ∂w r Architecture – The vocabulary around neural networks architectures is described in the figure below: r Updating weights – In a neural network, weights are updated as follows: • Step 1: Take a batch of training data. • Step 2: Perform forward propagation to obtain the corresponding loss. • Step 3: Backpropagate the loss to get the gradients. By noting i the ith layer of the network and j the jth hidden unit of the layer, we have: • Step 4: Use the gradients to update the weights of the network. [i] [i]T [i] z = w x + b j j j where we note w, b, z the weight, bias and output respectively. r Dropout – Dropout is a technique meant at preventing overfitting the training data by dropping out units in a neural network. In practice, neurons are either dropped with probability r Activation function – Activation functions are used at the end of a hidden unit to introduce p or kept with probability 1 − p. non-linear complexities to the model. Here are the most common ones: Sigmoid Tanh ReLU Leaky ReLU Convolutional Neural Networks 1 ez − e−z g(z) = g(z) = g(z) = max(0,z) g(z) = max(z,z) r Convolutional layer requirement – By noting W the input volume size, F the size of the 1 + e−z ez + e−z convolutional layer neurons, P the amount of zero padding, then the number of neurons N that with 1 fit in a given volume is such that: W − F + 2P N = + 1 S r Batch normalization – It is a step of hyperparameter γ, β that normalizes the batch {xi}. By noting μ , σ2 the mean and variance of that we want to correct to the batch, it is done as B B follows: √xi − μB xi ←− γ + β r Cross-entropy loss – In the context of neural networks, the cross-entropy loss L(z,y) is σ2 + commonly used and is defined as follows: B [ ] L(z,y) = − y log(z) + (1 − y) log(1 − z) It is usually done after a fully connected/convolutional layer and before a non-linearity layer and aims at allowing higher learning rates and reducing the strong dependence on initialization. Stanford University 1 Fall 2018
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!