LSTMs and GRUs: gated memory for recurrent networks

Lecture 11 ended with the difficulty of training RNNs. Backpropagating from the last time step \(t\) to an early step \(k\) multiplies many Jacobians, and the product can vanish or explode. This lecture builds two architectures that ease the problem, long short-term memory cells (LSTMs) and gated recurrent units (GRUs), from one intuition: selective read, selective write and selective forget.

The state \(\mathbf{s}_t\) of an RNN records information from all previous time steps; that is what the recurrent connection is for. The state has a fixed size, perhaps 100 or 1000 numbers, and at every step the new input morphs what is already stored. After \(t\) steps the information stored at step \(t-k\) (for some \(k < t\)) may be morphed so much that it can no longer be extracted from \(\mathbf{s}_t\). By step 20 or 30 there is no way to say what step 1 or step 2 contributed.

The same difficulty appears when information flows backwards. Training has to decide how much the events at step \(t-k\) are responsible for the error at step \(t\). That responsibility is carried by gradients, and Lecture 11 showed formally that they vanish along long paths. So early steps fade twice: their content in the forward pass, and their share of the blame in the backward pass.

Think of the state as a whiteboard of fixed size. At regular intervals you write something new on it, and every time, part of the earlier content gets changed or wiped. After a long derivation it is hard to see where a quantity from the start went, or how it shaped what is on the board now.

People manage a small board with three habits: they write selectively, read selectively and erase (forget) selectively. The lecture's example: with \(a = 1\), \(b = 3\), \(c = 5\), \(d = 11\), compute \(ac(bd + a) + ad\) on a board that holds only 3 statements at a time. The derivation has six steps: \(ac\), \(bd\), \(bd + a\), \(ac(bd + a)\), \(ad\) and finally \(ac(bd + a) + ad\).

• Selective write. Write only what matters. The board gets \(ac = 5\) and \(bd = 33\); the intermediate lines \(a = 1\), \(c = 5\), \(a \times c = 5\) are skipped. Later, \(ac(bd + a) = 170\) goes up as one line instead of \(5 \times 34\) followed by 170. The reason is the limited space.
• Selective read. To write step 3, \(bd + a = 34\), you read \(bd\) from the board. \(ac\) plays no part in that step.
• Selective forget. Once the board is full, erase the least useful line. \(bd\) can go, because \(bd + a\) already carries what it contributed, and \(ac(bd + a) = 170\) takes its place. Then \(bd + a\) makes way for \(ad = 11\), and \(ac\) makes way for the answer \(181\).

The same three operations show up elsewhere 7:28. The brain holds a finite number of facts, and over time it reads, writes and forgets some of them, mostly without our deciding to. An RNN also has a state of finite size, so the plan is to let it read, write and forget selectively too 8:03. Information will still be morphed, as it was on the whiteboard, but in a principled way that keeps the useful parts and deletes what is not needed. The lecturer adds a warning: the solution that follows moves in this direction without living up to the analogy completely.

Two questions drive this module. Is there a concrete case where an RNN needs to read, write and forget selectively? And how can the intuition be written as equations?

The concrete case is sentiment classification. The RNN reads a review one word per time step and predicts positive or negative at the last step. The slide's review says that the first half of a movie was dry, that the second half picked up pace, and that the lead actor's performance was amazing. By the time the RNN reaches the end of a long document, the information from the first few words is lost. Ideally it would:

• forget the information added by stop words such as a, an and the, which carry no sentiment;
• selectively read the information added by earlier sentiment-bearing words, such as amazing or awesome;
• selectively write new information from the current word into the state.

Long inputs make this pressing. A document of 1000 words is common (many Wikipedia pages are longer), and encoding all of it in one RNN state is hard. LSTMs and GRUs make it easier only to an extent.

Now the equations. The state \(\mathbf{s}_t \in \mathbb{R}^n\) has a finite size and is meant to store all information up to step \(t\). It plays the role of the whiteboard, and sooner or later it gets overloaded. The problem, stated precisely 3:02: from the state \(\mathbf{s}_{t-1}\) and the new input \(\mathbf{x}_t\), compute the new state \(\mathbf{s}_t\), using selective write, read and forget along the way so that only important information is kept in \(\mathbf{s}_t\).

In an RNN the new state uses all of the old one: \(\mathbf{s}_t = \sigma(W\mathbf{s}_{t-1} + U\mathbf{x}_t)\), bias ignored. Instead of passing \(\mathbf{s}_{t-1}\) on as it is, we want to pass on (write) only some portions of it.

The strictest version is a binary decision for each entry 5:21. With \(\mathbf{s}_{t-1} = [-1.4,\ -0.4,\ 1,\ \dots,\ -2]^\top\), keeping the 1st and 3rd entries and deleting the rest gives \([-1.4,\ 0,\ 1,\ \dots,\ 0]^\top\). A more sensible, soft version gives each entry a number between 0 and 1: the fraction of that entry to pass on.

So introduce a vector \(\mathbf{o}_{t-1}\) with entries between 0 and 1, and multiply it element-wise with the state. Each entry of \(\mathbf{o}_{t-1}\) decides what fraction of the matching entry of \(\mathbf{s}_{t-1}\) is passed on. How does the network know which fractions to use? It has to learn them, and the course's recurring recipe applies: to learn a quantity, give it a parametric form and learn the parameters.

The gate at step \(t-1\) depends on that step's input \(\mathbf{x}_{t-1}\) and on \(\mathbf{h}_{t-2}\), the output of the same selective write one step earlier. The sigmoid keeps every entry in \((0, 1)\). This is the place where sigmoids are still used: Lecture 7 argued for tanh or ReLU in their place, but noted that RNNs and LSTMs still use sigmoids 10:25. The new parameters \(W_o, U_o, \mathbf{b}_o\) are learned along with the RNN's \(W, U, V\). \(\mathbf{o}_t\) is called the output gate because it decides how much to pass (write) to the next time step.

In the video the lecturer pauses to check whether the σ on \(\mathbf{s}_{t-1}\) belongs in \(\mathbf{h}_{t-1}\). The slides keep it, and so do these notes.

Nobody tells the gate what to keep M2 · 12:06

The whiteboard analogy hides one difference. On the board we knew exactly which steps mattered. Take a review saying the movie was long but really amazing. The overall sentiment is positive and the word long is misleading, so ideally the gate would drop it. With a label for every word (retain, retain, forget, …) a loss could check whether the gates follow those decisions. No such labels exist. The only signal is the final loss on the positive/negative prediction, backpropagated through the network.

What the gates add is a better modelling choice, in the same sense that a deep network is a different choice from \(y = \mathbf{w}^\top\mathbf{x}\). Their parameters give backpropagation the means to learn selective writing if that lowers the loss 11:34. Suppose \(\mathbf{o}_{t-1}\) were all ones, passing everything to the next state; if that hurts the loss, the gradients will push the fractions toward more useful values.

Next, combine the selectively written \(\mathbf{h}_{t-1}\) with the new input \(\mathbf{x}_t\) into a temporary state:

This is the RNN update with \(\mathbf{h}_{t-1}\) in place of \(\mathbf{s}_{t-1}\), which makes sense because \(\mathbf{h}_{t-1}\) holds only the selectively written values. \(W\), \(U\) and \(\mathbf{b}\) play the same role as the RNN's parameters (the slide's figure leaves \(\mathbf{b}\) out).

\(\tilde{\mathbf{s}}_t\) captures all the information from the previous state and the current input, and we may not want all of it: the current word might be a stop word. So read from it selectively, through another gate called the input gate (the lecture also calls it the read gate):

It has the same form as the output gate with its own parameters. Where things stand:

Selective write and selective read are in place, and \(\mathbf{s}_t\) itself is still not computed.

How should \(\mathbf{s}_{t-1}\) and \(\tilde{\mathbf{s}}_t\) be combined into the new state? One simple and effective way adds the selectively read information to the old state, \(\mathbf{s}_t = \mathbf{s}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{s}}_t\). That keeps all of \(\mathbf{s}_{t-1}\), and we may want to forget some of it. So add a third gate, the forget gate:

This should be confusing, and the lecturer says so: the output gate already wrote \(\mathbf{s}_{t-1}\) selectively, so why also forget selectively? The two gates act at different places. \(\mathbf{o}_{t-1}\) shapes \(\mathbf{h}_{t-1}\), which is only an input to the temporary state and the gates; it is never added into \(\mathbf{s}_t\) directly. \(\mathbf{f}_t\) decides how much of the old state survives when the newly read information is merged back into it. Keeping input, output and forget as separate functions lets each learn on its own.

This exact doubt led to the gated recurrent unit, which merges two of the gates. More generally there are some 15 to 20 variants of the LSTM that tie gates or weights together. For example, the forget gate could be one minus the input gate, or the output gate one minus the input gate. The most parameterized form, used here, gives every gate its own parameters.

Moving the output gate's equation from step \(t-1\) to step \(t\) gives the complete set: three gates of the same form with different parameters, and three states, where \(\tilde{\mathbf{s}}_t\) leads to \(\mathbf{s}_t\) and \(\mathbf{s}_t\) leads to \(\mathbf{h}_t\).

Which part of the slide's diagram is one time step? The lecture asks this as a quiz. The answer is everything indexed by \(t\): the selective read and forget that produce \(\mathbf{s}_t\), plus the selective write after them that produces \(\mathbf{h}_t\). The selective write at the start of the diagram, which produced \(\mathbf{h}_{t-1}\), belongs to step \(t-1\).

An LSTM in TensorFlow or PyTorch returns two vectors at each step, \(\mathbf{h}_t\) and \(\mathbf{s}_t\), since both are states computed at that step. The course writes \(\mathbf{s}\) to match its RNN notation. The literature, and most blogs, call the same vector the cell state and write it \(\mathbf{c}_t\), so read every \(\mathbf{c}\) there as \(\mathbf{s}\).

Not in the lecture: standard implementations squash \(\tilde{\mathbf{s}}_t\) and the cell state inside \(\mathbf{h}_t\) with tanh instead of σ, so \(\tilde{\mathbf{s}}_t \in (-1, 1)\) and an update can also lower an entry of the state. The structure of the equations is the same.

There are many LSTM variants with different numbers and arrangements of gates 26:41. You could tie the input gate to one minus the forget gate, or compute a gate from \(\mathbf{s}_{t-1}\) instead of \(\mathbf{h}_{t-1}\), and so on; all of these are valid LSTMs. The paper LSTM: A Search Space Odyssey compares many of them. The version above is the standard one and the default on most platforms.

Another equally popular variant is the gated recurrent unit (GRU). It has two gates, an output gate and an input gate, and no forget gate:

Two things change compared with the LSTM.

• No explicit forget gate. The forget and input gates are tied. \(\mathbf{i}_t\) says how much to take from the new temporary state, and the rest, \(\mathbf{1} - \mathbf{i}_t\), comes from the previous state.
• The gates read \(\mathbf{s}_{t-1}\) directly, with no intermediate \(\mathbf{h}_{t-1}\). The selective write happens inside the temporary state, as \(\mathbf{o}_t \odot \mathbf{s}_{t-1}\).

Standard names, not used in the lecture: in the GRU paper and in libraries, \(\mathbf{o}_t\) is the reset gate \(\mathbf{r}_t\), \(\mathbf{i}_t\) is the update gate \(\mathbf{z}_t\), and the candidate uses tanh.

At the abstract level 28:23, the original problem was to store information from time step 1 to \(T\) in a state of finite size, which cannot be done. The intuition was that selective read, write and forget would help. LSTMs and GRUs are two ways to realize those operations in equations, and many other choices are possible. When someone says they used an LSTM or a GRU, they most likely mean the equations on this page.

Again, nothing supervises the gates 29:26. They are extra parameters that can learn to read, write and forget selectively if that helps the loss. So when you use RNNs or LSTMs for a task, it is worth visualizing the gates. In a review where the word but overturns everything before it (say, the movie was long, but I really loved it), check whether the gates learn to forget what came before but.

So far the case for gates is an intuition; it does not show that gradients flow back any better. In an RNN the trouble came from the recurrent weight \(W\) between \(\mathbf{s}_{t-1}\) and \(\mathbf{s}_t\). It appears again and again in the gradient, and the bound in Lecture 11 is \((\lambda\gamma)^{t-k}\), where \(\lambda\) bounds \(\lVert W\rVert\) and \(\gamma\) bounds the activation's derivative: above 1 the gradient can explode, below 1 it vanishes. LSTMs and GRUs still have recurrent connections 1:17. Why should things be any easier?

The module concentrates on vanishing gradients, because exploding gradients are easier to deal with 2:20. What we need from a gradient is its direction, so when its magnitude is too large we can normalize it to an acceptable size. This is gradient clipping, and it acts like an extra reduction of the learning rate, which already scales the gradient down. A vanished gradient cannot be rescued that way.

The intuition 2:56. During forward propagation the gates control the flow of information: they decide how much of \(\mathbf{s}_{t-1}\) passes into \(\mathbf{s}_t\) and keep irrelevant information out of the state. During backward propagation the gradients get multiplied by the same gates. Take \(\mathbf{s}_t = \mathbf{f}_t \odot \mathbf{s}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{s}}_t\). Treating the states as single numbers and \(\tilde{s}_t\) as a constant, \(\partial s_t/\partial s_{t-1} = f_t\).

That is not the whole derivative 3:59. This is an ordered network, and \(\tilde{\mathbf{s}}_t\) also depends on \(\mathbf{s}_{t-1}\), through \(\mathbf{h}_{t-1}\). The lecture assumes that this second contribution vanishes. That is the worst case for the argument, an assumption made against what we want to show, and the first term still leaves a gradient proportional to the gate.

Now suppose every forget gate from step 1 to step \(t\) equals 0.5 5:07. In the forward pass the contribution of \(\mathbf{s}_1\) to \(\mathbf{s}_t\) is scaled by \(0.5^{t}\), which vanishes for large \(t\). It works like a game of Chinese whispers, where each player adds noise until the first message is gone. In the backward pass the gradient reaching \(\mathbf{s}_1\) is scaled by the same \(0.5^{t}\) and also vanishes. That is fine: \(\mathbf{s}_1\) did not contribute to \(\mathbf{s}_t\), so there is no reason to blame it for the error at step \(t\).

The second video of Module 15.3 turns this intuition into an illustrative proof. Start from the RNN. Because it is an ordered network, Lecture 11 wrote the gradient with respect to \(W\) using explicit and implicit derivatives:

Each \(\partial\mathbf{s}_{j+1}/\partial\mathbf{s}_j\) is a matrix (a vector differentiated by a vector), the same kind of matrix multiplied again and again, and its norm was bounded by \(\lambda\gamma\) 3:34. Here is why that hurts. Suppose the loss \(\mathcal{L}_4(\theta)\) is high because \(W\) was not good enough to compute \(\mathbf{s}_1\). That news has to reach \(W\) through \(\mathbf{s}_1\). The direct link from \(W\) to \(\mathbf{s}_4\) carries how badly \(W\) did at computing \(\mathbf{s}_4\), which is different information. The path through \(\mathbf{s}_1\) multiplies three Jacobians, and the gradient along it vanishes.

The general principle 5:11: the gradient of \(\mathcal{L}_t(\theta)\) with respect to a parameter \(\theta_i\) adds up contributions from every path between them.

In the RNN only one path reaches \(W\) through \(\mathbf{s}_1\), so when it vanishes that information is lost. If there were another, more direct path, the gradient would survive. For the LSTM this sets the plan of the proof 6:18: to rule out vanishing, show one path along which the gradient flows well; to show that exploding is still possible, show one path along which it can explode.

An LSTM has two kinds of variable, gates and states. Start from the two states at step \(k-1\), \(\mathbf{s}_{k-1}\) and \(\mathbf{h}_{k-1}\), and follow the six computations of step \(k\):

• \(\mathbf{o}_k\), \(\mathbf{i}_k\), \(\mathbf{f}_k\) and \(\tilde{\mathbf{s}}_k\) are each computed from \(\mathbf{h}_{k-1}\) (and \(\mathbf{x}_k\)), for example \(\mathbf{o}_k = \sigma(W_o\mathbf{h}_{k-1} + U_o\mathbf{x}_k + \mathbf{b}_o)\);
• \(\mathbf{s}_k = \mathbf{f}_k \odot \mathbf{s}_{k-1} + \mathbf{i}_k \odot \tilde{\mathbf{s}}_k\) depends on \(\mathbf{s}_{k-1}\), \(\mathbf{f}_k\), \(\mathbf{i}_k\) and \(\tilde{\mathbf{s}}_k\);
• \(\mathbf{h}_k = \mathbf{o}_k \odot \sigma(\mathbf{s}_k)\) depends on \(\mathbf{o}_k\) and \(\mathbf{s}_k\).

From \(\mathbf{s}_k\) and \(\mathbf{h}_k\) the same graph recurs up to the last step, where the loss \(\mathcal{L}_t(\theta)\) depends on \(\mathbf{h}_t\). It has many more paths than the RNN's graph, since each step now has three states and three gates. To keep it readable, the parameters are written on the edges they act on instead of drawn as nodes, and only a few are shown (\(W, W_f, W_i, W_o\)); the \(U\)'s and biases are left out. The argument is the same for every parameter.

The question 10:06: suppose \(\mathcal{L}_t(\theta)\) is high because \(W_f\) failed to compute an appropriate value for \(\mathbf{s}_k\) (through \(\mathbf{f}_k\)). Does that information flow back to \(W_f\) through \(\mathbf{s}_k\)? The same question can be asked about \(W_i\), \(W_o\), \(W\) and the rest. How does the LSTM make sure this gradient does not vanish, even for a step \(k\) far back from \(t\)?

It is enough to show that \(\partial\mathcal{L}_t(\theta)/\partial\mathbf{s}_k\) does not vanish 11:08. Between \(\mathbf{s}_k\) and \(W_f\) there is one short link through \(\mathbf{f}_k\), with nothing recurrent in it, so a gradient that reaches \(\mathbf{s}_k\) also reaches \(W_f\). And there are many paths from \(\mathcal{L}_t(\theta)\) to \(\mathbf{s}_k\); reverse the arrows to see them 11:45. One goes through \(\mathbf{s}_{k+1}\), another through \(\mathbf{h}_k\), and \(\mathbf{h}_k\) is itself reached by many paths, because it has four outgoing edges and each step after it branches again. Their number is combinatorial. What matters is that there are many.

To show that the gradient does not vanish, one good path is enough, even if the gradient vanishes along all the others. Take the highlighted path and call the gradient along it \(t_0\); the full gradient is a sum of many such terms.

The first factor is fine: \(\mathbf{h}_t\) is connected directly to \(\mathcal{L}_t(\theta)\), with no intermediate node that could make the gradient vanish. The remaining factors are worked out one at a time.

\(\partial\mathbf{h}_t/\partial\mathbf{s}_t\) is diagonal M4 · 14:01

Recall \(\mathbf{h}_t = \mathbf{o}_t \odot \sigma(\mathbf{s}_t)\). Entry \(h_{ti}\) depends only on \(o_{ti}\) and \(s_{ti}\), and on no other entry of \(\mathbf{o}_t\) or \(\mathbf{s}_t\). So \(\partial h_{ti}/\partial s_{tj} = 0\) whenever \(i \ne j\), and \(\partial\mathbf{h}_t/\partial\mathbf{s}_t\) is a square \(d \times d\) diagonal matrix whose diagonal is \(\mathbf{o}_t \odot \sigma'(\mathbf{s}_t) \in \mathbb{R}^d\), by the same argument as in Lecture 11. Write \(\mathcal{D}(\mathbf{v})\) for the diagonal matrix with the vector \(\mathbf{v}\) on its diagonal; this factor is \(\mathcal{D}(\mathbf{o}_t \odot \sigma'(\mathbf{s}_t))\).

\(\partial\mathbf{s}_t/\partial\mathbf{s}_{t-1}\): keep only the explicit term M4 · 16:14

In \(\mathbf{s}_t = \mathbf{f}_t \odot \mathbf{s}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{s}}_t\), the temporary state \(\tilde{\mathbf{s}}_t\) also depends on \(\mathbf{s}_{t-1}\), so it cannot be treated as a constant. In an ordered network the derivative is then a sum of an explicit term and an implicit term (Lecture 11). Assume the worst case, that the gradient from the implicit term vanishes 16:47. This works against the claim being proved, because it throws part of the gradient away. The explicit term, with \(\tilde{\mathbf{s}}_t\) treated as a constant, is again diagonal: \(\mathcal{D}(\mathbf{f}_t)\).

Put the path together M4 · 17:57

Here \(\mathcal{L}_t'(\mathbf{h}_t)\) is short for \(\partial\mathcal{L}_t(\theta)/\partial\mathbf{h}_t\). The second line uses the fact that a product of diagonal matrices is diagonal, with the element-wise product of their diagonals on its diagonal.

Every forget gate lies between 0 and 1, so the product can still vanish 18:29. It does so only when the forget gates are small, and the same gates decide how much of \(\mathbf{s}_k\) is carried into \(\mathbf{s}_t\) in the forward pass. If some \(\mathbf{f}_i \to 0\), then \(\mathbf{s}_k\) contributed little to the later states during the forward pass, and the gradient does not reach it during backpropagation either, and there is no reason to hold it responsible. If the gates stayed open, the information from \(\mathbf{s}_k\) reached step \(t\), and the gradient travels all the way back. The forget gates apply the same regulation in both directions, which the lecturer calls fair.

Compare the RNN: there every factor \(\partial\mathbf{s}_{j+1}/\partial\mathbf{s}_j\) contained the weight matrix \(W\), so the size of the product was set by \(W\) through \((\lambda\gamma)^{t-k}\), whatever the states were doing. Along \(t_0\) no weight matrix is repeated, only the forget gates.

So there is one path along which the gradient does not vanish when it should not 20:12. As long as the gradient reaches \(W_f\) through \(\mathbf{s}_k\) along \(t_0\), training gets the signal it needs, and it flows back only when the forget gates allow it. In this sense LSTMs solve the vanishing gradient problem: gradients vanish only when required, while in RNNs they vanish arbitrarily.

To show that the gradient can explode, one path is enough. Take the path from \(\mathcal{L}_t(\theta)\) back to \(\mathbf{h}_{k-1}\) through every output gate: \(\mathcal{L}_t \leftarrow \mathbf{h}_t \leftarrow \mathbf{o}_t \leftarrow \mathbf{h}_{t-1} \leftarrow \cdots \leftarrow \mathbf{h}_k \leftarrow \mathbf{o}_k \leftarrow \mathbf{h}_{k-1}\). A path like this exists whichever gate or state you look at. Call its contribution to \(\partial\mathcal{L}_t(\theta)/\partial\mathbf{h}_{k-1}\) \(t_1\):

The lecture states each bracket without working; it follows from the equations. \(\mathbf{h}_t = \mathbf{o}_t \odot \sigma(\mathbf{s}_t)\) gives \(\partial\mathbf{h}_t/\partial\mathbf{o}_t = \mathcal{D}(\sigma(\mathbf{s}_t))\), and \(\mathbf{o}_t = \sigma(W_o\mathbf{h}_{t-1} + U_o\mathbf{x}_t + \mathbf{b}_o)\) gives \(\partial\mathbf{o}_t/\partial\mathbf{h}_{t-1} = \mathcal{D}(\mathbf{o}'_t)\,W_o\), where \(\mathbf{o}'_t\) is the sigmoid's derivative at the gate's input. Multiplying the two diagonal matrices:

where \(K\) bounds the diagonal matrices.

This is the RNN situation again 21:53: a diagonal matrix times a weight matrix, multiplied \(t-k+1\) times. Both are bounded, but depending on the norm of \(W_o\) the product \(\partial\mathcal{L}_t(\theta)/\partial\mathbf{h}_{k-1}\) may explode, and \(W_i\), \(W_f\) and \(W\) can do the same along their own paths. So LSTMs handle vanishing gradients and leave exploding gradients unsolved, and those still spoil training.

The popular fix is gradient clipping, from Pascanu, Mikolov and Bengio (2013) 22:28. While backpropagating, if the norm of the gradient exceeds a threshold, scale the gradient down so its norm is within the threshold. The direction stays the same; only the norm shrinks.

The formula is the standard way to write the rule; the slides state it in words, with \(\mathbf{g}\) the gradient and \(\tau\) the threshold.

This is acceptable because we care about the gradient's direction, and the learning rate already scales its magnitude (Lecture 5). Clipping keeps an unmanageable magnitude in check while staying faithful to the direction. Vanishing gradients are the more serious problem 23:03: when the whole gradient becomes 0 there is no direction left to keep. That is why it is enough for LSTMs to solve the vanishing problem.

Vanishing: the forget gates let a gradient fade only when the forward contribution faded too. Exploding: still possible through repeated weight matrices, and handled by clipping.

The encoder–decoder models of Lecture 13 are built from these recurrent cells.

Selective write

Passing on only part of the state: \(\mathbf{h}_{t-1} = \mathbf{o}_{t-1} \odot \sigma(\mathbf{s}_{t-1})\).

Selective read

Taking in only part of the new information: \(\mathbf{i}_t \odot \tilde{\mathbf{s}}_t\).

Selective forget

Keeping only part of the old state: \(\mathbf{f}_t \odot \mathbf{s}_{t-1}\).

Gate

A vector \(\sigma(W\mathbf{h}_{t-1} + U\mathbf{x}_t + \mathbf{b})\) with entries in \((0, 1)\), multiplied element-wise; a soft keep/delete decision whose parameters are learned.

Output gate \(\mathbf{o}_t\)

Decides how much of the cell state is written to \(\mathbf{h}_t\) and passed to the next step.

Input gate \(\mathbf{i}_t\)

Decides how much of the temporary state is read into the new state. Also called the read gate.

Forget gate \(\mathbf{f}_t\)

Decides how much of the old state survives into the new one.

Temporary state \(\tilde{\mathbf{s}}_t\)

The RNN-style candidate \(\sigma(W\mathbf{h}_{t-1} + U\mathbf{x}_t + \mathbf{b})\), built from the selectively written \(\mathbf{h}_{t-1}\).

Cell state \(\mathbf{s}_t\)

\(\mathbf{f}_t \odot \mathbf{s}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{s}}_t\); written \(\mathbf{c}_t\) in most of the literature.

GRU

Gated recurrent unit: two gates, forget tied to \(\mathbf{1} - \mathbf{i}_t\), gates computed from \(\mathbf{s}_{t-1}\) with no separate \(\mathbf{h}\).

\(\mathcal{D}(\mathbf{v})\)

The diagonal matrix with \(\mathbf{v}\) on its diagonal; the Jacobian of an element-wise operation.

Path rule

A gradient vanishes only if it vanishes along every path, and explodes if it explodes along any one path.

Gradient clipping

If the gradient's norm exceeds a threshold, scale it back to the threshold, keeping its direction.