Encoder–decoder models and attention

So far the course has built three kinds of networks (feedforward, convolutional and recurrent) and used each on its own. This lecture combines them. It starts from a problem seen before, language modelling: given the first \(t-1\) words (or characters) of a sentence, predict the \(t\)-th one. Autocomplete is the everyday example.

\(\mathcal{V}\) is the vocabulary, say 10,000 to 20,000 English words. At every step the network outputs a probability distribution over all of \(\mathcal{V}\), we pick the most probable word, and we continue until the end of the sentence. An RNN computes this distribution from its state:

Read it from the inside out. \(V\mathbf{s}_t + \mathbf{c}\) is a linear transformation of the state plus a bias, a vector with one entry per vocabulary word. The softmax turns it into a probability distribution, and its \(j\)-th entry is the probability of the \(j\)-th word. The state replaces the whole history \(y_1, \dots, y_{t-1}\) in the condition. That is fair: each step receives only one word as input, but the recurrent connections carry everything seen so far into \(\mathbf{s}_t\).

The five ingredients M1 · 4:19

The output at each step is a composite function of the inputs, so the model is trainable end to end: gradients flow from the loss back to every parameter. Vanishing and exploding gradients are dealt with by replacing the RNN with an LSTM (Lecture 12).

When generating, the input \(\mathbf{x}_t\) is simply the word predicted at the previous step. If word \(j\) received the highest probability at step \(t-1\), then

the one-hot vector of the \(j\)-th vocabulary word. Generation starts from a special \(\langle\text{GO}\rangle\) symbol and ends when the network outputs \(\langle\text{stop}\rangle\); both are added to the vocabulary as extra words. During training the true sentence is known, so the true previous word is used as the input. (Standard name, not used in the lecture: teacher forcing.)

One-hot is only one possible representation of a word 9:13. In practice the input is a pre-trained embedding of the \(j\)-th word, for example its word2vec vector from Lecture 10.

The initial state \(\mathbf{s}_0\) is not computed from anything. Nobody can say what the state "before the first word" should mean, so it is initialized randomly and learned along with the other parameters of the RNN (or LSTM, or GRU). The lecture promises to come back to it; image captioning will give it a meaning.

The models below would be unreadable if every cell were written out with its gates. From here on each cell is a function of its previous state and current input. The gate equations are in Lecture 12.

Any of the three can stand wherever "RNN" appears below. For the LSTM the lecture uses \(\mathbf{s}_t\), or both \(\mathbf{h}_t\) and \(\mathbf{s}_t\), as convenient.

A language model generates a sentence given the previous words. Image captioning generates it given the previous words and an image \(I\): for a photo, "A man throwing a frisbee in a park", one word at a time. The quantity of interest becomes \(P(y_t \mid y_1^{t-1}, I)\), which is again a conditional distribution with one more thing to condition on.

Earlier the history \(y_1^{t-1}\) was replaced by the vector \(\mathbf{s}_t\). The image can be replaced by a vector too. Every layer of a CNN learns an abstract representation of the image, and we trust such a representation to contain the image's information, just as \(\mathbf{s}_t\) contains the history. By convention the representation is taken from fc7, the fully connected layer just before the output layer of a network like VGG-16 (Lecture 8); it is numbered 7 because the five convolutional blocks are counted first. So we model

This is one choice among several. A student suggested a non-neural one in class: detect the objects first (man, frisbee) and condition on their names. Conditioning on \(\text{fc7}(I)\) is the neural way, and there are still two ways to wire it in.

Option 1: the image is the initial state M1 · 16:53

Set \(\mathbf{s}_0 = \text{fc7}(I)\). The mysterious learned \(\mathbf{s}_0\) now has a meaning: the starting point of the description is the image. Then \(\mathbf{s}_1\) depends on \(\mathbf{s}_0\), every later state depends on the one before, and so every \(P(y_t = j)\) depends on \(\text{fc7}(I)\). This computes \(P(y_t = j \mid \mathbf{s}_t, \text{fc7}(I))\). There is one practical detail 17:31. fc7 has 4096 numbers and the state has \(d\), so the image vector is first projected by a learnable \(d \times 4096\) matrix. The first input \(\mathbf{x}_1\) is \(\langle\text{GO}\rangle\).

Option 2: the image at every step M1 · 18:35

Feeding the image only once looks slightly inefficient. The more explicit alternative concatenates it with the input at every step, so each state is computed directly from the image:

You could think of other ways of conditioning \(P(y_t = j)\) on fc7.

The captioning model has two parts. An encoder, here a CNN, reads the input and encodes it as a representation. A decoder, here an RNN, generates (decodes) the output sentence from that encoding. Both are neural networks. This is the encoder–decoder architecture; such models became extremely popular, and the lecture credits them with much of deep learning's popularity. The encoder's output can go in once, as \(\mathbf{s}_0\), or be fed to every step of the decoder.

The diagram, with one box wired to another, is the easy part. What matters is the function behind it. The output \(y_t\) is the output layer applied to the RNN applied to the CNN applied to \(x\), so you should be able to write \(y\) as a function of \(x\), and the gradient can flow back through every piece.

For any encoder–decoder model, write down the chain of equations from the input \(x\) to the output distribution. If you can, you understand the model.

Module 16.2 runs through a series of tasks and asks the same four questions of each:

1. What kind of network can encode the input(s)? (What is an appropriate encoder?)
2. What kind of network can decode the output? (What is an appropriate decoder?)
3. What are the parameters of the model?
4. What is an appropriate loss function?

With the data and the training algorithm, these give the full description of a model.

Image captioning, written out

The slides write the outer sum with index \(i\); it runs over the time steps \(t\) of one caption. The total training loss adds this over the \(N\) training pairs.

The final \(y\) is a composite function of the image: many computations sit in between, but there is a path from \(x\) to \(y\), so the whole model is one network.

Textual entailment: given a premise, generate a hypothesis that follows from it. The premise "It is raining outside" entails "The ground is wet". The data is \(\{x_i = \text{premise}_i,\ y_i = \text{hypothesis}_i\}_{i=1}^N\). Both input and output are word sequences, so the encoder and decoder are both RNNs (replace them with LSTMs if you like). The model becomes one network by handing the encoder's last state to the decoder.

Parameters \(U_{\text{dec}}, V, W_{\text{dec}}, U_{\text{enc}}, W_{\text{enc}}, \mathbf{b}\). The loss is the sum of cross-entropies \(-\sum_t \log P(y_t = \ell_t \mid y_1^{t-1}, x)\), and training is gradient descent with backpropagation, through the decoder, across the hand-off \(\mathbf{s}_0 = \mathbf{h}_T\) and all the way back through the encoder.

Option 2 6:32 feeds the encoding at every step. An RNN takes exactly two inputs, the previous state and the current input, so \(\mathbf{h}_T\) is concatenated with the word embedding:

Parameters, loss and training algorithm stay the same. From here on the lecture states only option 1, since option 2 is the same small change each time.

Machine translation: the English sentence "I am going home" becomes the Hindi "Mein ghar ja raha hoon". The data is \(\{x_i = \text{source}_i,\ y_i = \text{target}_i\}_{i=1}^N\), and the model, parameters, loss and algorithm are exactly the entailment ones, with the same two options.

Transliteration: write a word in another script. It is typically used for names: translating "Thomas" into Hindi means writing "Thomas" in Devanagari. The slides' example takes the characters I-N-D-I-A to इंडिया. Input and output are sequences of characters, so again an RNN encoder and an RNN decoder with the same equations. (The option 2 slide writes the concatenation as \([\mathbf{e}(\hat y_{t-1}), \mathbf{h}_T]\); the order does not matter.)

Given an image and a question about it ("What is the bird's color?"), produce the answer ("White"). To keep it simple the answer is a single word from a fixed vocabulary, so the output is a single probability distribution over the possible answers.

The recipe is to encode each input with the network that suits it, concatenate the encodings, and pass the result to a feed-forward layer that predicts the distribution.

The data is \(\{x_i = \{I, q\}_i,\ y_i = \text{answer}_i\}_{i=1}^N\). The parameters are \(V, \mathbf{b}\) of the output layer, \(U_q, W_q\) of the question RNN and \(W_{\text{conv}}\) of the CNN (the slide lists \(\mathbf{b}\) twice). The loss is a single cross-entropy, \(\mathscr{L}(\theta) = -\log P(y = \ell \mid I, q)\), and gradient descent with backpropagation trains it, back through the question RNN and back into the CNN.

• Document summarization. "India beats Srilanka to win ICC WC 2011. Dhoni and Gambhir's half centuries help beat SL" becomes "India won the world cup". An RNN encoder reads the document word by word, \(\mathbf{s}_0 = \mathbf{h}_T\), and an RNN decoder writes the summary. Parameters, loss and algorithm are those of translation.
• Video captioning. A video is a sequence of frames, and each frame is an image. So encode every frame with a CNN and run an RNN over the frame encodings, \(\mathbf{h}_t = \text{RNN}(\mathbf{h}_{t-1}, \text{CNN}(x_{it}))\). In class this always gets called an RNN of CNNs. Then \(\mathbf{s}_0 = \mathbf{h}_T\) and an RNN decoder writes "A man walking on a rope". Parameters \(U_{\text{dec}}, W_{\text{dec}}, V, \mathbf{b}, W_{\text{conv}}, U_{\text{enc}}, W_{\text{enc}}\); the loss is the sum of cross-entropies, and backpropagation goes through time and through the CNNs.
• Video classification. The same RNN-of-CNNs encoder, but the output is one activity label ("Surya Namaskar"), so the decoder is a feed-forward softmax: \(\mathbf{s} = \mathbf{h}_T\), \(P(y \mid \text{video}) = \operatorname{softmax}(V\mathbf{s} + \mathbf{b})\), with loss \(-\log P(y = \ell \mid \text{video})\). Parameters \(V, \mathbf{b}, W_{\text{conv}}, U_{\text{enc}}, W_{\text{enc}}\).
• Dialog. The utterance "How are you" gets the response "I am fine". RNN encoder, \(\mathbf{s}_0 = \mathbf{h}_T\), RNN decoder, as in translation.

Why has this framework become so popular 14:00? The tasks above span different modalities (images, videos, sentences and combinations of them), and every one is modelled by a single end-to-end network built only from neural components: convolutional, recurrent and feed-forward layers. The lecture stretches it further to video question answering: an RNN for the question, an RNN of CNNs for the video, concatenate the two, and a feed-forward network predicts an answer from a fixed vocabulary. How well each model works is a separate question. Before this framework, many of them could not be written as one end-to-end network at all.

The module ends with a challenge 15:33. Pick a machine learning problem from your own field and model it in the encoder–decoder framework. The example from class: treat a person's DNA as a sequence of genes, encode it with an RNN, and predict a probability distribution over diseases the person may be susceptible to.

The story is not complete. A very important piece is still missing from all these models, attention, and the next module adds it 16:40.

Every model in Module 16.2 is trained with gradient descent and backpropagation (through time, and into the CNN where there is one). "RNN dec." means \(\mathbf{s}_t = \text{RNN}(\mathbf{s}_{t-1}, \mathbf{e}(\hat y_{t-1}))\) with output \(\operatorname{softmax}(V\mathbf{s}_t + \mathbf{b})\); "sum of CE" means \(-\sum_t \log P(y_t = \ell_t \mid y_1^{t-1}, x)\).

The lecture motivates attention with translation, now from Hindi to English: "Main ghar ja raha hoon" becomes "I am going home". In every model so far, the encoder reads the sentence once and encodes it. At each step the decoder then uses that same encoding to produce a new word, whether it was given once as \(\mathbf{s}_0\) or at every step. All the models of Module 16.2 are incomplete in this sense.

People do not translate like this. Reading a sentence once and holding all of it in mind while writing the translation invites forgetting, especially for a typical 25-word Wikipedia sentence. And most input words are irrelevant to any particular output word. A person produces each output word by focusing on only a few input words. At each step there is a distribution over the input words that says how much attention each one gets:

To write "I" it does not matter that you have forgotten the translation of "ghar", as long as you know "Main". "going" draws on both "ja" and "raha"; every other output word comes from one input word. Ideally the decoder would be fed only this relevant information, the encodings of the relevant words, at each step. The vanilla model makes every step look at the entire sentence, and that is the conceptual error to correct.

Suppose an oracle told us which input words matter at decoder step \(t\), and assume the encoder has already run, giving one vector \(\mathbf{h}_j\) per input word. If one word matters, feed its vector to the decoder and nothing else. If two matter, we could concatenate their vectors, but at another step four words might matter, and the input size would keep changing. A weighted average solves this: whatever the weights, it has the same dimension as each \(\mathbf{h}_j\).

At step 3 the oracle's weights are 0.5 on "ja" and 0.5 on "raha", so \(\mathbf{c}_3 = 0.5\,\mathbf{h}_3 + 0.5\,\mathbf{h}_4\). In general we take a weighted combination of all the encoder vectors, with weights from the oracle. Intuitively this should work better than the vanilla model, which overloads the decoder with the whole sentence at every step. Now the decoder gets only the information it needs to produce the current word.

In practice there is no oracle, so the machine has to learn the weights from data, and learning means introducing parameters. First define a score \(e_{jt}\) for how important the \(j\)-th input word is for decoding the \(t\)-th output word. Two things should affect it 6:46:

• The input word itself, represented by its encoder state \(\mathbf{h}_j\). This is a contextual representation: the encoder RNN computed it knowing the neighbouring words.
• What the decoder has done so far, summarized by its state \(\mathbf{s}_{t-1}\). Once the decoder has produced "home" it has no reason to look at "ghar" again.

The scores can be any real numbers, but we want weights that form a distribution over the input words. A softmax over \(j\) does this:

\(\alpha_{jt}\) is the probability of focusing on the \(j\)-th input word to produce the \(t\)-th output word. We now learn the \(\alpha\)s instead of being told them, and that needs a parametric form for \(f_{\text{ATT}}\). One choice among many:

Check the shapes. \(U_{\text{att}}\mathbf{s}_{t-1}\) and \(W_{\text{att}}\mathbf{h}_j\) are added, so both matrices must map to the same size, say \(d_1\): \(U_{\text{att}}, W_{\text{att}} \in \mathbb{R}^{d_1 \times d}\). The tanh then gives a vector in \(\mathbb{R}^{d_1}\). But \(e_{jt}\) is a scalar (it has two indices, yet it is one number: the importance of input \(j\) at step \(t\)), so \(\mathbf{v}_{\text{att}} \in \mathbb{R}^{d_1}\) turns the vector into a number by a dot product. That is the job of the three parameters: let the score depend on \(\mathbf{s}_{t-1}\) and \(\mathbf{h}_j\), and make the output a scalar. The slides take \(d_1 = d\): \(\mathbf{v}_{\text{att}} \in \mathbb{R}^d\), \(U_{\text{att}}, W_{\text{att}} \in \mathbb{R}^{d \times d}\). They are learned with the other parameters of the encoder and decoder.

Notation. One slide writes the encoder states as \(\mathbf{c}_j\); the lecture then fixes \(\mathbf{s}_t\) for decoder states and \(\mathbf{h}_j\) for encoder states, which these notes use throughout, with \(\mathbf{c}_t\) for the context vector. The slides write \(\mathbf{v}_{\text{att}}\) as \(V_{att}\), and call the number of input words \(M\) in one place and \(T\) in another.

The model would make complete sense if the true weights were given at training time. Say that at the step producing "going" the true weights are \(\boldsymbol{\alpha}^{\text{true}} = [0, 0, 0.5, 0.5, 0]\) and the model predicts \(\boldsymbol{\alpha}^{\text{pred}} = [0.1, 0.1, 0.35, 0.35, 0.1]\). We could add a loss \(\mathscr{L}(\boldsymbol{\alpha}^{\text{true}}, \boldsymbol{\alpha}^{\text{pred}})\), squared error for example, to the usual \(\mathscr{L}(\theta)\) and minimize both. Then \(U_{\text{att}}, W_{\text{att}}, \mathbf{v}_{\text{att}}\) would be tuned to bring the predicted weights close to the true ones.

In practice \(\boldsymbol{\alpha}^{\text{true}}\) is very hard to get 14:07. Someone would have to annotate, for every target word of every training sentence, which source words it came from, and nobody can collect that much annotated data. The data stays pairs of source and target sentences. So why should the model work without this supervision?

Because it is a better modelling choice. The lecture explains this with an analogy. Suppose you decide to learn to ride a bicycle without holding the handlebar. It is conceivable that after years of trying you manage it. Now someone suggests you hold the handlebar. That suggestion gives no supervision about how much force to put on the left or right hand; you still find that out yourself by falling to one side and then the other. But the second model, with hands on the handlebar, has extra controls to adjust and is closer to how people actually learn, so you learn better. Attention plays the role of the handlebar. The model is told only that it may focus on different words at different steps; which words, it must discover.

The attention parameters are trained like all the others. If the output at some step is wrong, the cause may be \(V\), the recurrent weights or poor attention weights, and backpropagation adjusts all of them. People learning to translate are not given word-by-word alignments either; they see many translations and work out what to focus on. With enough data the model should learn the attention weights the same way. In the lecture's words, that is the hope, "and hope is a good thing". In practice attention models do work better than vanilla encoder–decoder models.

Now put attention into the translation model and answer the same questions again. The data does not change: source and target sentences, with no information about which words to attend to.

Parameters: \(U_{\text{dec}}, V, W_{\text{dec}}, U_{\text{enc}}, W_{\text{enc}}, \mathbf{b}\), plus the attention parameters \(U_{\text{attn}}, W_{\text{attn}}, \mathbf{v}_{\text{attn}}\). The loss (sum of cross-entropies) and the algorithm (gradient descent with backpropagation) remain the same.

1. One decoder step with attention
2. Score every input word against the decoder's state so far: \(e_{jt}\) from \(\mathbf{h}_j\) and \(\mathbf{s}_{t-1}\), for \(j = 1, \dots, T\).
3. Turn the scores into attention weights: \(\alpha_{jt} = \operatorname{softmax}_j(e_{jt})\).
4. Take the weighted combination of the encoder states: \(\mathbf{c}_t = \sum_j \alpha_{jt}\mathbf{h}_j\).
5. Update the decoder with the previous word and the context: \(\mathbf{s}_t = \text{RNN}(\mathbf{s}_{t-1}, [\mathbf{e}(\hat y_{t-1}), \mathbf{c}_t])\).
6. Output the distribution \(\operatorname{softmax}(V\mathbf{s}_t + \mathbf{b})\); when generating, its most probable word becomes \(\hat y_t\), the next step's input.

The summary slide writes \(W_{\text{attn}}\mathbf{s}_t\) inside the score and swaps the letters (\(U\) now multiplies \(\mathbf{h}_j\)). The letters are just names. The state must be \(\mathbf{s}_{t-1}\), as on the earlier slide and in the video, because \(\mathbf{s}_t\) is computed from \(\mathbf{c}_t\), which needs the scores. The slide's parameter list also leaves out \(W_{\text{attn}}\).

The attention network is a simple feed-forward network, and only these three equations need to be fitted into a model. Like the encoder–decoder framework, this encode–attend–decode framework is generic. Try going back to each application of Module 16.2 and adding attention, answering the same questions (data, encoder, decoder, loss, training algorithm). The data never changes: nobody supplies the \(\alpha\)s. Lecture 14 takes the idea further: transformers drop the recurrence and build the encoder and decoder from attention alone.

Does the model really learn to focus on the most relevant input words at each step? To find out, plot the weights. With \(L\) output steps and \(T\) input words, the weights form an \(L \times T\) matrix. Drawn as a heat map, each cell shows \(\alpha_{tj}\), the importance the model gave the \(j\)-th input word when predicting the \(t\)-th output word; the darker the cell, the more attention. The picture is a soft alignment between input and output. Many papers include such a plot, and it is worth making one whenever you use attention, to check that the weights are meaningful.

• Summarization (Rush et al. 2015). When the summary says "terrorism", the most attended input word is "terrorism"; "Russia" attends mostly to "Russian", "calls" to "called" and "front" to "front". It does not always work this cleanly, but meaningful patterns appear.
• Translation between English and French (Cho et al. 2015). Word order in the two languages largely agrees, so translation is roughly monotonic: the \(k\)-th output word attends to about the \(k\)-th input word, and most of the weight lies on the diagonal.
• Video captioning (Yao et al. 2015). The same kind of plot, with attention over the video's frames as each word is generated.

The module videos stop at the heat maps; this section and the next two follow the slides (Modules 16.4 and 16.5). How should attention work for image captioning? For text, the encoder gives a representation for every location (time step) of the input, and attention chooses among them. For images we have been using the fc7 vector, a single representation that holds no location information. So there is nothing to choose among.

The fix is to take the output of a convolutional layer instead, since convolutional layers keep spatial information. In VGGNet (Lecture 8) the output of the fifth convolutional block is a \(14 \times 14 \times 512\) feature map. Treat it as 196 locations, each with a 512-dimensional representation. The model then learns an attention over these locations, and each location corresponds to an actual region of the image.

Written out in the notation of the text case (the slide shows this as a figure), with \(\mathbf{a}_j\) the vector at location \(j\):

The slides' examples (from Cho et al. 2015) show captions with one word underlined and the attended part of the image in white. The model attends to the correct object when it names it.

Consider a dialog between a user (U) and a bot (B):

The dialog is a sequence of utterances, and each utterance is a sequence of words. The input is a "sequence of sequences". A natural encoder is a two-level hierarchical RNN encoder. The first-level RNN runs over the words of each utterance and gives one representation per utterance. A second RNN runs over this sequence of utterance representations and gives a single representation for the whole context. The decoder generates the response conditioned on it.

Documents have the same shape: a document is a sequence of sentences, and each sentence a sequence of words. Take document classification (or summarization). The slides' example document, which starts "Politics is the process of making decisions…", should be classified as Politics. A hierarchical RNN models it:

Parameters \(W^1_{\text{enc}}, U^1_{\text{enc}}, W^2_{\text{enc}}, U^2_{\text{enc}}, V, \mathbf{b}\); loss cross-entropy; algorithm gradient descent with backpropagation. Here \(\mathbf{w}_{ij}\) is the \(j\)-th word of sentence \(i\), \(T_i\) is the length of sentence \(i\) and \(K\) the number of sentences. Each sentence is summarized by its last word-level state, and the document by the last sentence-level state.

To add attention to a hierarchical model we need it at two levels: first attend to the important (most informative) words in each sentence, then to the important sentences in the document. The slides use the Hierarchical Attention Network of Yang et al.

Parameters \(W_w, W_s, V, \mathbf{b}_w, \mathbf{b}_s, \mathbf{b}, \mathbf{u}_w, \mathbf{u}_s\), plus the weights of the two RNNs. The loss is cross-entropy and the algorithm gradient descent with backpropagation.

Compare with translation. There is no decoder state to score against, because the output is a single class. Instead each state goes through a one-layer network, \(\mathbf{u}_{ij} = \tanh(W_w\mathbf{h}_{ij} + \mathbf{b}_w)\), and is scored by its dot product with a learned vector \(\mathbf{u}_w\) (\(\mathbf{u}_s\) at the sentence level). The attention-weighted sums take the place of "use the last state" (\(\mathbf{s}_i = \mathbf{h}^1_{iT_i}\), \(\mathbf{s} = \mathbf{h}^2_K\)) in the plain hierarchical encoder.

Not in the lecture: Yang et al. call \(\mathbf{u}_w\) and \(\mathbf{u}_s\) the word-level and sentence-level context vectors, and their encoders are bidirectional GRUs, which is why the slide's figure shows forward and backward states.

Language model

A model of \(P(y_t \mid y_1^{t-1})\); an RNN computes it as \(\operatorname{softmax}(V\mathbf{s}_t + \mathbf{c})\).

Encoder

The network that reads the input and encodes it: a CNN for images, an RNN for sequences, an RNN of CNNs for videos.

Decoder

The network that produces the output from the encoding: an RNN for a sequence, a feed-forward softmax for a single label.

fc7(I)

The 4096-dimensional representation of an image from the fully connected layer fc7 of a CNN such as VGG-16.

Options 1 and 2

Feed the encoding once as \(\mathbf{s}_0\), or concatenate it with the decoder's input at every step.

Textual entailment

Generating a hypothesis that follows from a premise ("It is raining outside" → "The ground is wet").

Transliteration

Writing a word, usually a name, in another script, character by character.

RNN of CNNs

A video encoder: a CNN encodes each frame, an RNN runs over the frame encodings.

Attention weights \(\alpha_{jt}\)

Softmax-normalized importance of input position \(j\) when producing output step \(t\); they sum to 1 over \(j\).

Context vector \(\mathbf{c}_t\)

\(\sum_j \alpha_{jt}\mathbf{h}_j\): the weighted combination of encoder states fed to the decoder at step \(t\).

\(f_{\text{ATT}}\)

The scoring function; here \(\mathbf{v}_{\text{att}}^\top\tanh(U_{\text{att}}\mathbf{s}_{t-1} + W_{\text{att}}\mathbf{h}_j)\), a small feed-forward network.

Soft alignment

The \(L \times T\) heat map of attention weights between output and input words.

Hierarchical attention network

A two-level encoder for sequences of sequences, with attention over words in each sentence and over sentences in the document.