Recurrent neural networks and backpropagation through time

Every network in the course so far took an input of one fixed size. The convolutional networks of Lecture 8 classified images that were all scaled or cropped to the same size, at training time and at test time (32 × 32 on the slide; the video also mentions 227 × 227). The word2vec models of Lecture 10 read a fixed window of \(k\) context words.

Each input was also processed on its own. Show a CNN an apple and it predicts apple, whatever the previous image was: a car, a mango or another apple. The computations, outputs and decisions for two successive inputs have nothing to do with each other.

Many applications break both assumptions. Take auto-completion, familiar from typing messages on a phone. Given the first character d, the model should predict e. Given that e, it predicts the next character, and so on until the word “deep” is complete and it predicts ⟨stop⟩.

Three things are different from the earlier networks.

1. Successive inputs depend on each other. To predict what follows an e, you want to know what came before it too. The lecture's example: if the previous character was z and the current one is e, a b is likely next; ignore the z and many characters could follow the e.
2. The length is not fixed. “deep” has 4 characters, “learn” 5 and “machine” 7, so the number of inputs and the number of predictions both vary.
3. Every step does the same task. Each network takes one character as input and produces one character as output.

Problems like this are called sequence learning problems: the model reads a sequence of dependent inputs and produces an output, or several. Each input corresponds to one time step: input 1 at time step 1, input 2 at time step 2, and so on.

Part-of-speech tagging. For the sentence “man is a social animal”, predict the tag of each word: noun, verb, article, adjective, noun. Once the model has seen the adjective social, it can be almost sure the next word is a noun, and that it is not an article or a verb. So the current output depends on the previous input as well as the current one. Sentences can have any number of words; an average Wikipedia sentence has about 25. Here an output is wanted at every time step, and every step does the same task: a word in, a tag out.

Sentiment of a movie review. Sometimes only the final output matters. To label “The movie was boring and long” as positive or negative, the model has to read the whole review. A prediction made right after “boring” would be premature, since the review could have gone on “but I still loved it”. The answer depends on earlier words as well as the last one. We can still picture the same network at every step, each taking a word and producing a ± output; we just ignore every output except the last.

Activity in a video. Sequences need not be made of words. A video is a sequence of images, and deciding that someone is performing Surya Namaskar needs the whole sequence: stopping partway would show only one posture of it. Speech is another kind of sequence, which the lecture mentions without developing.

How should a model for such tasks look? The lecture first lists what it must do:

1. Account for the dependence between inputs.
2. Account for a variable number of inputs.
3. Make sure the function executed at each time step is the same.

Start with the function at one time step. It has an input, a hidden representation and an output, like any feedforward network:

\(O\) is the output function (a softmax, say), named separately from the hidden nonlinearity \(\sigma\).

To execute the same function at every step, share the network across steps, which means sharing its parameters: the same \(U, V, \mathbf{b}, \mathbf{c}\) at every time step. Sharing also makes the network agnostic to the length of the input. The same function is computed at every step, so the number of steps does not matter: we create as many copies of the network as there are time steps and run each one. That settles items 2 and 3. The copies still don't exchange anything, so item 1 is open.

One way to make the output depend on earlier inputs is to feed them all in: \(\mathbf{x}_1\) at step 1, \(\mathbf{x}_1, \mathbf{x}_2\) at step 2, \(\mathbf{x}_1, \mathbf{x}_2, \mathbf{x}_3\) at step 3, and so on. This handles dependence, but it breaks the other two items on the wishlist.

• The function changes from step to step. \(\mathbf{y}_1 = f_1(\mathbf{x}_1)\), \(\mathbf{y}_2 = f_2(\mathbf{x}_1, \mathbf{x}_2)\), \(\mathbf{y}_3 = f_3(\mathbf{x}_1, \mathbf{x}_2, \mathbf{x}_3)\). A network with two inputs needs weights \(u_1, u_2\) and one with three needs \(u_1, u_2, u_3\), so these are different functions with different parameters.
• The network is sensitive to sequence length. A sequence of length 10 needs \(f_1, \dots, f_{10}\) and one of length 100 needs \(f_1, \dots, f_{100}\). Some weights could be shared between \(f_{99}\) and \(f_{100}\), but the count still grows with the length. And if the longest training sequence had 25 steps while a test sequence has 30, there are no trained parameters for steps 26 to 30 at all.

The solution is to add a recurrent connection: the previous state is fed into the current step through a weight matrix \(W\).

Check it against the wishlist.

• Same function at every step. The equations are written for a general step \(i\) with the same parameters. Only step 1 needs a convention, the assumed \(\mathbf{s}_0\).
• Any length. The same network, with the same parameters, computes \(\mathbf{y}_1, \dots, \mathbf{y}_{10}\) or \(\mathbf{y}_{100}\).
• Dependence on earlier inputs. \(\mathbf{s}_i\) depends on \(\mathbf{x}_i\) and on \(\mathbf{s}_{i-1}\), which depends on \(\mathbf{x}_{i-1}\) and \(\mathbf{s}_{i-2}\), and so on back to \(\mathbf{s}_0\). Recursively, the output at step \(i\) depends on every input so far, and no parameters are added per step.

You will often read about “the state of an RNN” (or of an LSTM or GRU). It means this vector: an encoding of everything the network has read so far. The network is called a recurrent neural network because of the recurrent connection. It can be drawn compactly as one cell whose state feeds back into itself, or unrolled into one copy per time step.

Go back to the earlier problems and add the recurrent connection between time steps. When predicting the character after the second e, the network now has information about the d and the first e. In the review, the final prediction has information from every word before it.

After proposing a network, the next job is to train it. The training algorithm is backpropagation through time (BPTT). First pin down the sizes, because a gradient with respect to a parameter has the same shape as that parameter.

Shape convention (not on the slide). The slide's shapes fit a row-vector convention, in which \(\mathbf{x}_i^\top U\) is \(1 \times d\). With the column-vector equations used on this page, \(U\mathbf{x}_i\) needs \(U \in \mathbb{R}^{d\times n}\) and \(V\mathbf{s}_i\) needs \(V \in \mathbb{R}^{k\times d}\). These are only transposes. \(W\) is \(d \times d\) either way, and the parameter count \(nd + d^2 + dk + d + k\) does not depend on the length of the sequence.

Why does training need a module of its own, if it is just backpropagation? A concrete example shows where plain backpropagation needs care. Return to auto-completion, and for simplicity assume the vocabulary has only four characters: d, e, p and ⟨stop⟩, so \(k = 4\). The inputs are d, e, e, p and the targets are the next characters e, e, p, ⟨stop⟩.

Each step chooses one of 4 classes, so the output function is a softmax and the loss is cross-entropy (Lecture 4). Suppose \(U, V, W\) are initialized randomly and the network predicts these distributions:

The true distribution at each step is one-hot on the true next character. The loss column is worked out here with natural logarithms; the slide shows only the probabilities. Its illustrative distributions add up to 1.1 rather than 1, so read them as rough numbers.

Two questions follow: what is the total loss, and how do we backpropagate it to update the parameters \(\theta = \{U, V, W, \mathbf{b}, \mathbf{c}\}\)? A loss can be incurred at each of the \(T\) steps, and the total is simply their sum.

Backpropagation needs the gradients of \(\mathcal{L}\) with respect to \(W, U, V, \mathbf{b}, \mathbf{c}\). Once we have them, the update is the usual gradient descent step (Lecture 5).

Start with the easy parameter. Because the loss is a sum, so is its gradient:

\(V\) is a matrix, so strictly this is \(\nabla_V \mathcal{L}(\theta)\), a matrix of the same shape as \(V\), and the sum adds \(T\) such matrices. Each term is the gradient of a loss with respect to the weights of an output layer, the shortest chain in backpropagation, which Lecture 4 derived. Nothing new is needed.

For reference, from Lecture 4 (not repeated on this slide): with a softmax output and cross-entropy loss, \(\partial L_t / \partial(V\mathbf{s}_t + \mathbf{c}) = \mathbf{y}_t - \mathbf{e}_c\), where \(\mathbf{e}_c\) is the one-hot vector of the true character. With \(V \in \mathbb{R}^{k\times d}\) this gives \(\partial L_t/\partial V = (\mathbf{y}_t - \mathbf{e}_c)\,\mathbf{s}_t^\top\).

Again the loss is a sum, \(\frac{\partial\mathcal{L}(\theta)}{\partial W} = \sum_{t=1}^{T}\frac{\partial L_t(\theta)}{\partial W}\). By the chain rule, \(\partial L_t/\partial W\) is the sum of the gradients along all paths from \(L_t\) to \(W\). Which paths are there? Take \(L_4\):

• \(L_4\) depends on \(\mathbf{s}_4\);
• \(\mathbf{s}_4\) depends on \(\mathbf{s}_3\) and \(W\);
• \(\mathbf{s}_3\) depends on \(\mathbf{s}_2\) and \(W\);
• \(\mathbf{s}_2\) depends on \(\mathbf{s}_1\) and \(W\);
• \(\mathbf{s}_1\) depends on \(\mathbf{s}_0\) and \(W\), where \(\mathbf{s}_0\) is a constant starting state.

This is an ordered network: its state variables are computed one at a time in a fixed order (first \(\mathbf{s}_1\), then \(\mathbf{s}_2\), and so on), because each needs the one before. The chain rule gives

The first factor is the gradient of the loss with respect to the last hidden layer. It depends on \(V\) only, and backpropagation already computes it. The second factor is the difficulty. Leave out \(U\mathbf{x}_4\), which does not involve \(W\), and recall

We cannot differentiate this treating \(\mathbf{s}_3\) as a constant, because \(\mathbf{s}_3\) itself depends on \(W\), the very parameter we are differentiating with respect to. In such a network the total derivative \(\partial\mathbf{s}_4/\partial W\) has two parts:

• Explicit, written \(\partial^+\mathbf{s}_4/\partial W\): the derivative with all other inputs (here \(\mathbf{s}_3\)) treated as constants.
• Implicit: the sum over all indirect paths from \(\mathbf{s}_4\) to \(W\), which pass through \(\mathbf{s}_3\), \(\mathbf{s}_2\) and \(\mathbf{s}_1\).

Expanding the total derivative M3 · 9:46

Split \(\partial\mathbf{s}_4/\partial W\), then split each total derivative that the split produces:

The last line has no implicit part: \(\mathbf{s}_1\) depends on \(\mathbf{s}_0\), and \(\mathbf{s}_0\) is a constant with no connection to \(W\). Substituting from the bottom up:

The notation is loose. \(\mathbf{s}_4\) is a vector and \(W\) a matrix, so \(\partial\mathbf{s}_4/\partial W\) is a tensor, while \(\partial\mathbf{s}_4/\partial\mathbf{s}_3\) is a matrix. The gory details below work out every shape.

For a compact formula, the lecture short-circuits the chains of Jacobians. Write \(\frac{\partial \mathbf{s}_4}{\partial \mathbf{s}_3}\frac{\partial \mathbf{s}_3}{\partial \mathbf{s}_2}\) as \(\frac{\partial \mathbf{s}_4}{\partial \mathbf{s}_2}\), write the three-factor chain as \(\frac{\partial\mathbf{s}_4}{\partial\mathbf{s}_1}\), and write the first term as \(\frac{\partial\mathbf{s}_4}{\partial\mathbf{s}_4}\frac{\partial^+\mathbf{s}_4}{\partial W}\), which is fair because \(\frac{\partial\mathbf{s}_4}{\partial\mathbf{s}_4}\) is the identity. This is the chain rule read in reverse, so nothing is approximated.

Put back the first factor \(\partial L_4/\partial\mathbf{s}_4\), then replace 4 by a general step \(t\):

That leaves \(U\), which the lecture leaves as an exercise, saying it works out very similarly. It then turns to a problem hidden in this formula. 13:35

Completing the exercise (not in the lecture): \(U\) also enters every state explicitly, through \(U\mathbf{x}_k\), so \(\frac{\partial L_t}{\partial U} = \frac{\partial L_t}{\partial \mathbf{s}_t}\sum_{k=1}^{t}\frac{\partial\mathbf{s}_t}{\partial\mathbf{s}_k}\frac{\partial^+\mathbf{s}_k}{\partial U}\). The explicit part is worked out for \(W\) below; for \(U\), \(\mathbf{x}_k\) takes the place of \(\mathbf{s}_{k-1}\). The bias \(\mathbf{b}\) follows the same pattern, while \(\mathbf{c}\), like \(V\), only touches each step's output.

The BPTT formula sums over \(k = 1, \dots, t\). For the loss at the last step, \(t = T\), and for \(k = 1\), the term \(\partial\mathbf{s}_T/\partial\mathbf{s}_1\) links the last state to the first. Take one such \(\partial\mathbf{s}_t/\partial\mathbf{s}_k\) and undo the short-circuit:

That is \(t - k\) factors, multiplied with later steps on the left. Look at one factor, reindexed as \(\partial\mathbf{s}_j/\partial\mathbf{s}_{j-1}\). Write the pre-activation and the state separately and apply the chain rule:

Both factors are \(d \times d\) matrices, since each is a vector differentiated with respect to a vector. In the first, \(\mathbf{a}_j = [a_{j1}, \dots, a_{jd}]\) and \(\mathbf{s}_j = [\sigma(a_{j1}), \dots, \sigma(a_{jd})]\), and entry \((p, q)\) is \(\partial s_{jp}/\partial a_{jq}\). Since \(s_{jp}\) depends only on \(a_{jp}\), every off-diagonal entry is 0 (for example \(\partial s_{j2}/\partial a_{j1} = 0\)), and the diagonal holds \(\sigma'(a_{jp})\):

In the second factor, \(U\mathbf{x}_j + \mathbf{b}\) does not depend on \(\mathbf{s}_{j-1}\), so \(\partial\mathbf{a}_j/\partial\mathbf{s}_{j-1} = W\).

Take norms. The norm of a product of matrices is at most the product of their norms:

Now bound each factor.

• The activation is a bounded function (logistic or tanh), and so is its derivative; call that bound \(\gamma\). For the logistic, \(\sigma'(a) = \sigma(a)(1 - \sigma(a))\) is largest at \(a = 0\), where it equals \(0.5 \times 0.5 = \tfrac14\). For tanh, \(\sigma'(a) = 1 - \tanh^2(a)\) is at most 1.
• The weights are finite real numbers, so \(\lVert W\rVert\) has some upper bound; call it \(\lambda\).

The product of \(t - k\) such factors is then bounded by the product of their bounds:

The trouble is worst when \(t\) is close to \(T\) and \(k\) is close to 1. The product then has nearly \(T\) factors, and even a number slightly below 1 shrinks fast when raised to that power. With \(\gamma\lambda = 0.9\) the bound is \(0.9^{20} \approx 0.12\) after 20 steps and \(0.9^{50} \approx 0.005\) after 50; with \(\gamma\lambda = 1.1\) it is \(1.1^{50} \approx 117\) after 50 (worked numbers, not from the slide).

Either way training breaks. A vanished gradient means no update, so the weights stay stuck where they are. An exploded gradient makes the step \(W \leftarrow W - \eta\,\nabla_W\mathcal{L}\) enormous, and the weights jump to a point far away from where they were, losing the progress made so far. This is the same saturation problem that deep feedforward networks with sigmoid units have (Lecture 7), now repeated once per time step with the same \(W\).

Beyond the slide. With the spectral norm, \(\lVert W\rVert\) is the largest singular value of \(W\), which is what the figure below lets you set. The \(\gamma\lambda > 1\) case is only an upper bound, which is why the lecture says the gradient could explode: saturated units have \(\sigma'\) far below \(\gamma\). Also, the \(k = t\) term in the BPTT sum has no Jacobian factors at all. What vanishes is the contribution of distant steps, so the network struggles to learn long-range dependencies.

The problem comes from long products. One simple way around it is truncated backpropagation through time: don't backpropagate through all the time steps. At step \(t\), look back only \(\tau\) steps, which restricts each product to at most \(\tau\) factors, with \(\tau\) smaller than the \(t - k\) of the distant steps:

Every product that remains has at most \(\tau\) factors, so its bound is at most \((\gamma\lambda)^\tau\) even when \(\gamma\lambda > 1\).

The lecture adds a second trick aimed at exploding gradients. Gradient descent mostly cares about the direction of the gradient, so when its magnitude is too large, rescale it. Gradient clipping limits the norm of the gradient to a threshold \(\kappa\):

Here \(\mathbf{g}\) is the gradient with all parameters stacked into one vector. The direction is unchanged and the magnitude becomes \(\kappa\). Frameworks such as TensorFlow offer this as an option to clip gradients to a given norm, and other measures of magnitude are also used.

Truncation has a price that is easy to miss: dependencies longer than \(\tau\) steps contribute nothing to the gradient, so they cannot be learned this way. The lecture presents both tricks as partial fixes. Other solutions, Long Short-Term Memory cells and Gated Recurrent Units, are the subject of Lecture 12.

The last module checks that the BPTT formula can really be implemented. First a warning the lecture repeats: the formulas use partial-derivative notation for objects that are gradients, Jacobians and tensors (\(\nabla_W L_t\) would be the honest notation for the left side). Here is each object with its shape:

• \(\partial L_t/\partial\mathbf{s}_t\) is a scalar differentiated with respect to a vector: the gradient with respect to the last hidden layer, which backpropagation already computes.
• \(\partial\mathbf{s}_t/\partial\mathbf{s}_k\) is a vector with respect to a vector. Module 14.4 gave it as \(\prod_{j=k+1}^{t}\operatorname{diag}(\sigma'(\mathbf{a}_j))\,W\).
• \(\partial^+\mathbf{s}_k/\partial W\) is a vector in \(\mathbb{R}^d\) differentiated with respect to a matrix in \(\mathbb{R}^{d\times d}\), so it is a third-order tensor in \(\mathbb{R}^{d\times d\times d}\). It is the only piece we don't yet know how to compute.

The standard recipe for a frightening derivative is to compute one element and then generalize. The \((p, q, r)\)-th element of the tensor is \(\partial^+ s_{kp}/\partial W_{qr}\): how the \(p\)-th entry of \(\mathbf{s}_k\) changes with one entry \(W_{qr}\) of the weight matrix, holding \(\mathbf{s}_{k-1}\) fixed. Recall

(\(U\mathbf{x}_k\) is left out again, since it does not involve \(W\).) Entry \(p\) of \(\mathbf{a}_k\) is row \(p\) of \(W\) times the column \(\mathbf{s}_{k-1}\):

In the sum for \(a_{kp}\), the only term that can contain \(W_{qr}\) is the one with \(i = r\), and it does so only when the row matches, \(p = q\). So \(\partial a_{kp}/\partial W_{qr} = s_{k-1,r}\) if \(p = q\), and 0 otherwise.

So the tensor looks nasty, but split into single elements it is easy, and one element is enough to rebuild all of it. The lecture doesn't ask you to implement it; the point is that you could.

What the sparsity buys (not on the slide). In the formula, the tensor is multiplied by the row vector \(\boldsymbol{\delta}^\top = \frac{\partial L_t}{\partial\mathbf{s}_t}\frac{\partial\mathbf{s}_t}{\partial\mathbf{s}_k}\). Entry \((q, r)\) of the product is \(\sum_p \delta_p\,\partial^+ s_{kp}/\partial W_{qr} = \delta_q\,\sigma'(a_{kq})\,s_{k-1,r}\), so the \(d\times d\) contribution of step \(k\) is an outer product, \(\big(\boldsymbol{\delta}\odot\sigma'(\mathbf{a}_k)\big)\,\mathbf{s}_{k-1}^\top\), the same form as the weight gradients of Lecture 4. No \(d^3\) tensor is ever stored. Implementations also sum over \(t\) first, using \(\frac{\partial\mathcal{L}}{\partial\mathbf{s}_k} = \frac{\partial L_k}{\partial\mathbf{s}_k} + \frac{\partial\mathcal{L}}{\partial\mathbf{s}_{k+1}}\operatorname{diag}(\sigma'(\mathbf{a}_{k+1}))\,W\) in a single backward sweep from \(k = T\) down to 1.

Sequence learning problem

A task whose inputs form a sequence of dependent items of varying length, with the same task at each step and outputs at every step or only some.

Time step

One position in the sequence; input \(\mathbf{x}_i\) arrives at step \(i\).

Parameter sharing

Using the same \(U, V, W, \mathbf{b}, \mathbf{c}\) at every step, so one function runs at each step and any length works.

Recurrent connection

The weight \(W\) that feeds the previous state into the current step: \(\mathbf{s}_i = \sigma(U\mathbf{x}_i + W\mathbf{s}_{i-1} + \mathbf{b})\).

State \(\mathbf{s}_i\)

The hidden vector at step \(i\); it encodes all inputs seen up to step \(i\). \(\mathbf{s}_0\) is a fixed start.

Ordered network

A network whose state variables are computed one at a time in a fixed order, each depending on earlier ones and on the same parameters.

Explicit derivative \(\partial^+\mathbf{s}_k/\partial W\)

The derivative of \(\mathbf{s}_k\) with its other inputs, such as \(\mathbf{s}_{k-1}\), held constant.

Implicit derivative

The sum over indirect paths from \(\mathbf{s}_k\) back to \(W\) through earlier states.

Backpropagation through time

\(\frac{\partial L_t}{\partial W} = \frac{\partial L_t}{\partial \mathbf{s}_t}\sum_{k=1}^{t}\frac{\partial \mathbf{s}_t}{\partial \mathbf{s}_k}\frac{\partial^+ \mathbf{s}_k}{\partial W}\): backpropagation over all previous time steps.

Step Jacobian

\(\partial\mathbf{s}_j/\partial\mathbf{s}_{j-1} = \operatorname{diag}(\sigma'(\mathbf{a}_j))\,W\), with norm at most \(\gamma\lambda\).

Vanishing gradient

\(\lVert\partial\mathbf{s}_t/\partial\mathbf{s}_k\rVert \le (\gamma\lambda)^{t-k} \to 0\) when \(\gamma\lambda < 1\): distant steps stop contributing and weights stop changing.

Exploding gradient

When \(\gamma\lambda > 1\) the bound grows geometrically, so the gradient can become huge and throw the weights far away.

Truncated BPTT

Backpropagating only \(\tau\) steps back from each \(t\), so no product has more than \(\tau\) factors.

Gradient clipping

Rescaling the gradient to norm \(\kappa\) whenever its norm exceeds \(\kappa\), keeping its direction.