Feedforward networks and backpropagation

The course so far went from McCulloch–Pitts neurons and perceptrons (Lecture 2) to sigmoid neurons, gradient descent for a single neuron, and the representation power of networks of sigmoid neurons (Lecture 3). This lecture fixes the notation for a deep network of such neurons and then works out how to learn all of its weights. The same notation is used for the rest of the course, so it is worth learning carefully. The slides credit Hugo Larochelle's video lectures on backpropagation and Andrej Karpathy's CS231n (Winter 2016) lecture as sources.

A feedforward neural network, also called a multilayered network of neurons, has these parts:

• Input. An \(n\)-dimensional vector \(\mathbf{x} \in \mathbb{R}^n\), for example \(n\) measurements of a site (salinity, pressure, temperature, …) used to decide whether there is oil there.
• Hidden layers. \(L-1\) of them, each with \(n\) neurons. The running example has two hidden layers, and the drawings use \(n = 3\).
• Output layer. \(k\) neurons, for example one per class. With two hidden layers the network has \(L = 3\) layers in total.
• Two parts per neuron. Every hidden and output neuron is split into a pre-activation and an activation. For layer \(i\) these form vectors \(\mathbf{a}_i = [a_{i1}, \dots, a_{in}]\) and \(\mathbf{h}_i = [h_{i1}, \dots, h_{in}]\), both in \(\mathbb{R}^n\).
• Layer numbers. The input is layer 0 and the output is layer \(L\). The input is also written \(\mathbf{h}_0 = \mathbf{x}\). The output activation has three names, \(\mathbf{h}_L = \hat{\mathbf{y}} = \hat f(\mathbf{x})\).
• Weights and biases. Every neuron in layer \(i-1\) is connected to every neuron in layer \(i\). Those \(n \times n\) weights form a matrix \(W_i \in \mathbb{R}^{n\times n}\), and each neuron of layer \(i\) has one bias, giving \(\mathbf{b}_i \in \mathbb{R}^n\), for \(0 < i < L\). Into the output layer the weights are \(W_L \in \mathbb{R}^{k\times n}\) and the biases \(\mathbf{b}_L \in \mathbb{R}^k\).

Giving every hidden layer the same \(n\) is only for convenience. Layers can have different sizes, and nothing in the derivations depends on them being equal (see the dimension check further down).

Layer \(i\) takes the previous layer's activation, forms a weighted sum plus a bias, and passes the result through a non-linearity:

\(g\) is the activation function of the hidden layers (logistic, tanh, linear, …; later lectures add ReLU and its relatives, see Lecture 7). \(O\) is the output activation function, for example softmax or linear. All of these quantities depend on the input, \(\mathbf{a}_i(\mathbf{x})\) and \(\mathbf{h}_i(\mathbf{x})\), but the \((\mathbf{x})\) is dropped to keep the notation light.

For layer 1 with \(n = 3\), the pre-activation is one matrix-vector product and one vector addition:

The weight \(W_{1ij}\) sits in row \(i\) and column \(j\), so it carries input \(j\) into neuron \(i\) of layer 1. The same reading holds in every layer: \(W_{kij}\) connects neuron \(j\) of layer \(k-1\) to neuron \(i\) of layer \(k\). The derivations below rely on this. The dimensions match, since an \(n\times n\) matrix times a vector in \(\mathbb{R}^n\), plus a vector in \(\mathbb{R}^n\), is a vector in \(\mathbb{R}^n\).

The activation applies \(g\) to each entry separately; \(g\) is an element-wise function:

Each step on its own is simple. The network becomes a complicated function only because many such steps are stacked.

The output layer gets its own function \(O\) because the right range for the output depends on the problem. A logistic output only produces values between 0 and 1, which is no use for predicting a movie rating out of 100 or a test score. Module 4.3 chooses \(O\) for two common kinds of problem.

Lecture 3 described supervised learning with five components: data, model, parameters, learning algorithm and objective. A network diagram may look like a different kind of object, but following its arrows gives the model as an explicit formula. Start from \(\mathbf{x}\), multiply by \(W_1\) and add \(\mathbf{b}_1\) to get \(\mathbf{a}_1\), apply \(g\) to get \(\mathbf{h}_1\), and repeat for each layer:

So a deep network is one more family of functions in the machine learning toolbox, alongside linear or quadratic functions. It happens to be a very complex composite function. Even a 100-layer network can be written down this way; the formula is just long.

Checking dimensions when layers differ in size

Let the input have \(n\) entries, the first hidden layer \(m\) neurons and the second \(p\). Each neuron of layer 1 receives all \(n\) inputs, so \(W_1 \in \mathbb{R}^{m\times n}\) and \(\mathbf{b}_1 \in \mathbb{R}^m\). Layer 2 receives \(m\) values into each of its \(p\) neurons, so \(W_2 \in \mathbb{R}^{p\times m}\):

In general \(W_i\) has one row per neuron of layer \(i\) and one column per neuron of layer \(i-1\). However many connections the diagram shows, the network is a series of matrix-vector products, each followed by an element-wise non-linearity.

The parameters are all the \(W\)'s and \(\mathbf{b}\)'s. Everything else in the network is computed from them and the input.

Lecture 3 learned the two parameters \(w, b\) of a single sigmoid neuron by gradient descent, repeating \(w_{t+1} \leftarrow w_t - \eta\nabla w_t\) and \(b_{t+1} \leftarrow b_t - \eta\nabla b_t\). Collecting the parameters into one vector \(\theta = [w, b]\) turns the two updates into one:

\(\nabla\theta_t\) is shorthand for the gradient of the loss evaluated at the current values. For a feedforward network, \(\theta = [W_1, \dots, W_L, \mathbf{b}_1, \dots, \mathbf{b}_L]\) instead of \([w, b]\). The algorithm does not care how long \(\theta\) is, because adding and subtracting very long vectors works exactly like adding two-dimensional ones.

1. gradient_descent() for a feedforward network
2. \(t \leftarrow 0\); max_iterations \(\leftarrow 1000\)
3. Initialize \(\theta_0 = [W_1^0, \dots, W_L^0, \mathbf{b}_1^0, \dots, \mathbf{b}_L^0]\)
4. While \(t\)++ \(<\) max_iterations: \(\theta_{t+1} \leftarrow \theta_t - \eta\,\nabla\theta_t\)

What changes is the size of \(\nabla\theta\). It holds one partial derivative per parameter: \(\partial\mathcal{L}/\partial W_{111}\) through \(\partial\mathcal{L}/\partial W_{1nn}\) for the first layer, the same for the second, \(\partial\mathcal{L}/\partial W_{L,11}\) through \(\partial\mathcal{L}/\partial W_{L,kn}\) for the last, and one for each bias. Grouped by layer:

With \(L = 3\) that is \(2n^2 + kn + 2n + k\) numbers. The layers have different sizes, so these numbers form a collection of matrices and vectors rather than one big matrix. For a single neuron, deriving \(\partial\mathcal{L}/\partial w\) and \(\partial\mathcal{L}/\partial b\) already took some work. Deriving a separate formula for each of thousands of weights is out of the question, so the aim is a formula that gives a whole matrix of partial derivatives at once.

Two questions

1. How should the loss function \(\mathcal{L}(\theta)\) be chosen? This comes first, because a derivative cannot be written down before the function is known. (Module 4.3)
2. How can \(\nabla\theta\) be computed, that is, all of \(\nabla W_1, \dots, \nabla W_L\) and \(\nabla\mathbf{b}_1, \dots, \nabla\mathbf{b}_L\)? (Modules 4.4 to 4.8)

The right loss depends on the problem. The lecture takes the two kinds of problem that cover most of practice, regression and classification, and chooses a loss and an output function for each.

Regression example. Predict a movie's ratings from features of the movie (is the actor Damon? is the director Nolan? …). Each movie has three targets, its IMDb rating, its critics' rating and its Rotten Tomatoes rating, so \(\mathbf{y}_i \in \mathbb{R}^3\), for example \(\mathbf{y}_i = [7.5,\ 8.2,\ 7.7]\). The output layer has \(k = 3\) neurons, one per rating.

The loss should measure how far \(\hat{\mathbf{y}}_i\) is from \(\mathbf{y}_i\). For \(\mathbf{y}_i \in \mathbb{R}^k\) the natural choice is the squared error, averaged over the \(N\) training examples:

Which output function? The logistic function would squash every \(\hat y_{ij}\) into \((0, 1)\), while ratings run from 0 to 10, or 0 to 100. So \(O\) should be linear:

Here \(W_O\) is a \(3\times 3\) matrix and \(\mathbf{b}_O \in \mathbb{R}^3\), and \(\hat y_{ij}\) is no longer bounded between 0 and 1.

Since nothing bounds a linear output, what stops the network from predicting a rating of 1000? The loss does. If the true IMDb rating is 9.5 and the network says 1000, the term \((9.5 - 1000)^2\) is enormous, and gradient descent moves the parameters away from settings that produce it. The loss can only become small when the predictions sit in the targets' natural range. With a logistic output, no setting of \(W_1, W_2, W_3, \mathbf{b}_1, \mathbf{b}_2, \mathbf{b}_3\) could ever output 9.5, however strongly the loss pushed.

Before this part the lecture sends students to a separate video, from an earlier course, on information content, entropy and cross-entropy. These notes only need the resulting formula.

Classification example. Classify an image into one of \(k\) classes, say apple, mango, orange and banana. For an apple image the label is the one-hot vector \(\mathbf{y} = [1, 0, 0, 0]\). Squared error could measure the gap between \(\hat{\mathbf{y}}\) and \(\mathbf{y}\) here too, but there is a better choice.

Read \(\mathbf{y}\) as a probability distribution over the classes: the label puts probability 1 on the true class and 0 on the others. Then \(\hat{\mathbf{y}}\) should also be a probability distribution, with every entry in \([0, 1]\) and the entries summing to 1. The output function that ensures this is the softmax:

\(O(\mathbf{a}_L)_j\) is the \(j\)-th entry of \(\hat{\mathbf{y}}\) and \(a_{L,j}\) is the \(j\)-th entry of \(\mathbf{a}_L\).

Example. Suppose the four output pre-activations are \(\mathbf{a}_L = [-10, 10, 20, 30]\). This is clearly not a distribution: it has a negative entry and entries above 1. The softmax turns it into one:

The decimal values are computed for these notes; the lecture leaves the fractions unevaluated.

Two properties make the result a distribution. Every \(e^{a}\) is positive, even for negative \(a\), since \(e^{-10} = 1/e^{10}\). And all entries share one denominator, the sum of all the numerators, so they add up to 1. Positive numbers that add up to 1 each lie between 0 and 1.

Why exponentiate at all? Dividing each raw value by the sum of the raw values also gives entries that add up to 1, but a negative value stays negative: \(-10/(-10 + 10 + 20 + 30) = -0.2\), which cannot be a probability.

With \(\mathbf{y}\) and \(\hat{\mathbf{y}}\) both probability distributions, a loss from probability theory measures how different they are. It is the cross-entropy:

For a one-hot label, \(y_c = 1\) if \(c = \ell\), the true class, and \(y_c = 0\) otherwise. All terms but one vanish:

Minimizing \(-\log\hat y_\ell\) is the same as maximizing \(\log\hat y_\ell\):

Is \(\hat y_\ell\) a function of the parameters? Yes. It is entry \(\ell\) of the network's output,

so changing any weight changes it. \(\hat y_\ell\) is the predicted probability that \(\mathbf{x}\) belongs to class \(\ell\), and training should push it toward 1. \(\log\hat y_\ell\) is called the log-likelihood of the data, so this loss is also called the negative log-likelihood.

For instance, if the network outputs \([0.2, 0.3, 0.4, 0.1]\) for an apple image, the loss is \(-\log 0.2 \approx 1.61\) (our arithmetic). Here and below, \(\log\) is the natural logarithm, which is what gives \(\frac{d}{dx}\log x = \frac1x\) in Module 4.5.

Other problems call for other losses, but these two pairs cover a very large share of what comes up in practice. The rest of the lecture works with a softmax output and the cross-entropy loss. The lecture notes that the results carry over to the regression pair with one small change (see the note at the end of Module 4.5).

The loss is now fixed, which leaves the gradient. Take one weight, say \(W_{112}\) (layer 1, from input 2 into neuron 1). Gradient descent needs a formula for \(\partial\mathcal{L}(\theta)/\partial W_{112}\). The aim is more ambitious than one formula: once the derivative is known for one weight, the same work should give it for every weight in that layer, and then for every layer.

Start with a network that is deep but thin, with one neuron in each layer. Then \(\hat y\) is a plain chain of functions of \(x_1\), and the derivative for the first weight is the chain rule applied link by link, just as for a nested function such as \(\sin(\cos(e^{\log x}))\):

The first five factors together are the derivative of the loss with respect to \(h_{11}\), so the chain compresses to two factors. The same compression works for the weights further up:

This is the idea backpropagation is built on. The chains are long, but most of each chain has already been computed for the weights closer to the loss. Compute from the loss end once, store the partial results, and each weight then needs only its short local factor. In a wide network several weights feed the same neuron, and all of them can share the stored part.

The lecture builds intuition with a story. Feed an input \(\mathbf{x}\) through the network with its current weights, somewhere in the middle of training. It produces \(\hat{\mathbf{y}}\) and a non-zero loss \(-\log\hat y_\ell\). Something is wrong, and we want to find out who is responsible.

1. Ask the output layer why its output is bad. It accepts its share, but points out that it is only as good as the hidden layer and weights below it, since \(\hat{\mathbf{y}} = O(W_L\mathbf{h}_{L-1} + \mathbf{b}_L)\).
2. Ask \(W_L\), \(\mathbf{b}_L\) and the last hidden layer what is wrong with them. The weights and biases take full responsibility: they are the only parts of the network that can be adjusted. The hidden layer's activation says it is only as good as its pre-activation.
3. The pre-activation in turn says it is only as good as the hidden layer and weights below it. The questioning continues down the network.
4. The input is never blamed, since it is given. In the end the responsibility lies with all the weights and biases, the parameters of the model.

The slides name \(h_L\) in step 2; the layer that feeds the output is \(\mathbf{h}_{L-1}\), which is what is written here.

Questioning the first-layer weights directly is hard; questioning them through the output and hidden layers is easy. That is exactly what the chain rule does:

This is written loosely, as on the slides. With several neurons per layer, each factor is a vector or matrix of partial derivatives; Modules 4.5 to 4.7 make each one precise.

Why is a derivative the right measure of responsibility? \(\partial\mathcal{L}/\partial w\) says how much the loss changes when \(w\) changes a tiny bit. If a small change in \(w\) moves the loss a lot, \(w\) has a strong influence on the loss, and adjusting it is an effective way to reduce the loss.

Roadmap

The remaining modules compute three kinds of quantity, each once, in a form that works for every layer:

1. the gradient with respect to the output units (Module 4.5);
2. the gradient with respect to the hidden units (Module 4.6);
3. the gradient with respect to the weights and biases (Module 4.7).

Throughout, the output function is softmax and the loss is cross-entropy.

The output layer has two parts, the activations \(\hat{\mathbf{y}}\) and the pre-activations \(\mathbf{a}_L\). The quantity needed later is \(\nabla_{\mathbf{a}_L}\mathcal{L}(\theta)\). As a warm-up, start with \(\hat{\mathbf{y}}\), and with a single entry of it.

The loss \(\mathcal{L}(\theta) = -\log\hat y_\ell\) contains only one entry of \(\hat{\mathbf{y}}\), the one for the true class. Its derivative with respect to entry \(i\) is

For example, if \(\ell = 2\) the loss is \(-\log\hat y_2\). It does not involve \(\hat y_1\), so that derivative is 0. With respect to \(\hat y_2\) it is \(-1/\hat y_2\), from \(\frac{d}{dx}\log x = \frac1x\). Every other entry gives 0. An indicator variable \(\mathbb{1}_{\text{condition}}\), which is 1 when the condition holds and 0 otherwise, writes the two cases as one expression 5:07:

Now stack the \(k\) entries into the gradient with respect to the vector \(\hat{\mathbf{y}}\). Only the indicator changes from row to row:

\(\mathbf{e}(\ell)\) is the \(k\)-dimensional one-hot vector with 1 in position \(\ell\) and 0 everywhere else (the slides also write it \(e_\ell\)).

The recipe for every gradient in this lecture: first find the partial derivative with respect to one entry of the vector or matrix, then stack the entries and recognize the result as a vector or matrix expression.

Again one entry first. Split the derivative at \(\hat y_\ell\):

Does \(\hat y_\ell\) depend on \(a_{Li}\) when \(i \ne \ell\)? It does, because every entry of \(\mathbf{a}_L\) appears in the softmax denominator:

So unlike the previous step, the derivative is not zero for the other entries. (The summation index is written \(i'\) to keep it apart from the \(i\) we differentiate with respect to.) The derivation uses the quotient rule,

and two small facts. The numerator \(\exp(a_{L\ell})\) depends only on entry \(\ell\), so \(\frac{\partial}{\partial a_{Li}}\exp(a_{L\ell}) = \mathbb{1}_{\ell=i}\exp(a_{L\ell})\). In the denominator only one term of the sum contains \(a_{Li}\), so \(\frac{\partial}{\partial a_{Li}}\sum_{i'}\exp(a_{Li'}) = \exp(a_{Li})\). Step by step:

Line 1 is the chain rule for \(-\log u\) with \(u = \hat y_\ell\). Line 3 applies the quotient rule with \(g = \exp(a_{L\ell})\) and \(h = \sum_{i'}\exp(a_{Li'})\). Line 4 uses the two facts and splits the squared denominator into two copies. Line 5 recognizes the softmax entries, line 6 renames them \(\hat y\), and in line 7 the factor \(\hat y_\ell\) cancels.

Stacking the entries for \(i = 1, \dots, k\) gives the vector:

The result is easy to read. For each wrong class the entry is \(\hat y_i \ge 0\), so a gradient descent step lowers that pre-activation. For the true class it is \(\hat y_\ell - 1 \le 0\), so the step raises it. The entries add up to 0, and the whole vector shrinks to zero as \(\hat{\mathbf{y}}\) approaches \(\mathbf{e}(\ell)\).

Not in the lecture: for the regression pair, with an identity output \(\hat{\mathbf{y}} = \mathbf{a}_L\) and the squared error \(\sum_j(\hat y_j - y_j)^2\) of one example, the same steps give \(\nabla_{\mathbf{a}_L}\mathcal{L} = 2(\hat{\mathbf{y}} - \mathbf{y})\). Only this first gradient changes; everything in Modules 4.6 to 4.8 stays the same.

The hidden layers come next. There may be many of them, so the goal is one formula, derived once, that works for any hidden layer. The output layer was special and got its own formula; the hidden layers should not each need one.

A new feature appears here. In the thin network each quantity fed exactly one later quantity. In a wide network the activation \(h_{22}\) (neuron 2 of hidden layer 2) feeds every neuron of the next layer, \(a_{31}, a_{32}, \dots, a_{3k}\), and the loss depends on all of them. So there are \(k\) paths from \(h_{22}\) to the loss.

The rule for this situation is ordinary calculus. If \(p(z)\) is computed from intermediate results \(q_1(z), \dots, q_m(z)\) that all depend on \(z\), the derivative adds up the chain rule along every path:

The lecture's toy example computes \(q_1 = z^2\), \(q_2 = z^3\) and \(q_3 = e^z\) from \(z\), and then \(p\) from the three \(q\)'s. To check the rule on one concrete choice (ours), take \(p = q_1/q_2 + q_3\), which equals \(1/z + e^z\):

That is the derivative of \(1/z + e^z\), as it should be.

In the network, \(p(z)\) is the loss \(\mathcal{L}(\theta)\), \(z = h_{ij}\), and the intermediate results are the pre-activations of the next layer, \(q_m(z) = a_{i+1,m}\) (on the slide, the output layer's \(a_{Lm}\)).

One entry first, \(h_{ij}\), neuron \(j\) of layer \(i\). Add up over the paths through the next layer's pre-activations:

The sum runs over the neurons of layer \(i+1\): \(k\) of them when \(i+1 = L\), as on the slides, and \(n\) when layer \(i+1\) is hidden. When \(i+1 = L\), the first factor is already known from Module 4.5.

Why \(\partial a_{i+1,m}/\partial h_{ij} = W_{i+1,m,j}\)

Write out \(\mathbf{a}_3 = W_3\mathbf{h}_2 + \mathbf{b}_3\) for two output neurons and three hidden neurons:

Differentiate \(a_{31}\) with respect to \(h_{22}\). Three of the four terms do not contain \(h_{22}\), so \(\partial a_{31}/\partial h_{22} = W_{312}\). Matching indices, \(i+1 = 3\), \(m = 1\) and \(j = 2\): the derivative is entry \((m, j)\) of \(W_{i+1}\).

From a sum to a dot product

Put the two factors of each term into vectors. The first is the gradient with respect to the next layer's pre-activations. The second is column \(j\) of \(W_{i+1}\), written \(W_{i+1,\cdot,j}\), where the dot means "every row":

Their dot product multiplies matching entries and adds the products, which is exactly the sum above:

The whole vector

Stack the entries \(j = 1, \dots, n\). Row \(j\) is column \(j\) of \(W_{i+1}\) dotted with the same gradient vector. The columns of \(W_{i+1}\) are the rows of \(W_{i+1}^\top\), so the stack is one matrix-vector product:

For \(i = L-1\) both factors are available: the current weights \(W_L\), and \(\nabla_{\mathbf{a}_L}\mathcal{L}\) from Module 4.5. For a lower layer the formula needs \(\nabla_{\mathbf{a}_{i+1}}\mathcal{L}\) for a hidden layer. For example, \(\nabla_{\mathbf{h}_1}\mathcal{L} = W_2^\top\nabla_{\mathbf{a}_2}\mathcal{L}\) needs \(\nabla_{\mathbf{a}_2}\mathcal{L}\), which has not been derived yet.

This step is short because there is only one path. Each activation is computed from its own pre-activation, \(h_{ij} = g(a_{ij})\), and \(a_{ij}\) feeds nothing else. So

Stacking over \(j\):

\(\odot\) is the Hadamard product: multiply two vectors entry by entry, which gives a vector. It is different from a dot product, which would give a single number. The second vector collects the derivatives of the activations with respect to their pre-activations. For the usual choices of \(g\) these are cheap to compute (Module 4.9). Write \(g'(\mathbf{a}_i)\) for the vector \([\dots, g'(a_{ij}), \dots]\).

Starting from the output and alternating the two formulas reaches every hidden layer, however deep the network. For \(L = 3\):

Each step needs only the step before it, the current weights and values from the forward pass. Whenever a quantity is needed, everything it depends on has already been computed.

Last come the parameters. We want \(\nabla_{W_k}\mathcal{L}(\theta)\) for a generic layer \(k\). The derivative of a scalar with respect to a matrix is a matrix of the same shape, with one partial derivative per entry. The recipe is the same as before: one entry first.

The weight \(W_{kij}\) connects neuron \(j\) of layer \(k-1\) to neuron \(i\) of layer \(k\), so it enters exactly one pre-activation, \(a_{ki}\). Recall \(\mathbf{a}_k = \mathbf{b}_k + W_k\mathbf{h}_{k-1}\), entry by entry \(a_{ki} = b_{ki} + \sum_j W_{kij}h_{k-1,j}\). There is one path, so

The first factor is already known from Modules 4.5 and 4.6. For the second, the lecture takes \(k = 2\) with three neurons per layer and the weight \(W_{222}\). Row 2 of \(W_2\mathbf{h}_1 + \mathbf{b}_2\) is

All the other terms, the bias included, drop out. What is left is the activation at the source end of the weight, in layer \(k-1 = 1\) at neuron \(j = 2\).

The whole matrix is an outer product

Fill in every entry for a \(3\times 3\) matrix \(W_k\):

Every entry in row \(i\) shares the factor \(\partial\mathcal{L}/\partial a_{ki}\), and every entry in column \(j\) shares \(h_{k-1,j}\). That is the pattern of an outer product. A column vector \(\mathbf{u}\) times a row vector \(\mathbf{v}^\top\) is the matrix with entries \(u_iv_j\):

Here \(\mathbf{u} = \nabla_{\mathbf{a}_k}\mathcal{L}(\theta)\) and \(\mathbf{v} = \mathbf{h}_{k-1}\).

Biases M7 · 11:55

The bias \(b_{ki}\) also enters only \(a_{ki}\), with coefficient 1. From \(a_{ki} = b_{ki} + \sum_j W_{kij}h_{k-1,j}\), \(\partial a_{ki}/\partial b_{ki} = 1\), so

Two ingredients are needed. \(\nabla_{\mathbf{a}_k}\mathcal{L}\) comes from the backward recursion of Modules 4.5 and 4.6. \(\mathbf{h}_{k-1}\) is computed by the forward pass anyway, with no derivative involved; for the first layer it is \(\mathbf{h}_0 = \mathbf{x}\).

All the pieces are now in place:

• the gradient with respect to the output layer, \(\nabla_{\mathbf{a}_L}\mathcal{L}(\theta)\);
• the gradients with respect to the hidden layers, \(\nabla_{\mathbf{h}_k}\mathcal{L}(\theta)\) and \(\nabla_{\mathbf{a}_k}\mathcal{L}(\theta)\) for \(1 \le k < L\);
• the gradients with respect to the weights and biases, \(\nabla_{W_k}\mathcal{L}(\theta)\) and \(\nabla_{\mathbf{b}_k}\mathcal{L}(\theta)\) for \(1 \le k \le L\).

Plugging them into gradient descent gives the full learning algorithm. Each iteration does one forward pass and one backward pass. As written, it works with one training example \((\mathbf{x}, \mathbf{y})\) at a time. Lecture 5 returns to gradient descent and its variants.

1. gradient_descent()
2. \(t \leftarrow 0\); max_iterations \(\leftarrow 1000\)
3. Initialize \(\theta_0 = [W_1^0, \dots, W_L^0, \mathbf{b}_1^0, \dots, \mathbf{b}_L^0]\)
4. While \(t\)++ \(<\) max_iterations:
5. \(\mathbf{a}_1, \mathbf{h}_1,\) \(\mathbf{a}_2, \mathbf{h}_2,\) \(\dots,\) \(\mathbf{a}_{L-1}, \mathbf{h}_{L-1},\) \(\mathbf{a}_L, \hat{\mathbf{y}} = \) forward_propagation\((\theta_t)\)
6. \(\nabla\theta_t = \) back_propagation\((\mathbf{h}_1, \dots, \mathbf{h}_{L-1},\) \(\mathbf{a}_1, \dots, \mathbf{a}_L,\) \(\mathbf{y}, \hat{\mathbf{y}})\)
7. \(\theta_{t+1} \leftarrow \theta_t - \eta\,\nabla\theta_t\)

1. forward_propagation(θ), with \(\mathbf{h}_0 = \mathbf{x}\)
2. For \(k = 1\) to \(L-1\):
3. \(\mathbf{a}_k = \mathbf{b}_k + W_k\mathbf{h}_{k-1}\)
4. \(\mathbf{h}_k = g(\mathbf{a}_k)\)
5. \(\mathbf{a}_L = \mathbf{b}_L + W_L\mathbf{h}_{L-1}\)
6. \(\hat{\mathbf{y}} = O(\mathbf{a}_L)\)

The forward pass involves no derivatives, only matrix-vector products and the element-wise \(g\). The output layer is kept out of the loop because it uses \(O\) instead of \(g\). Once \(\hat{\mathbf{y}}\) is known, the label gives the loss.

1. back_propagation\((\mathbf{h}_1, \dots, \mathbf{h}_{L-1},\) \(\mathbf{a}_1, \dots, \mathbf{a}_L,\) \(\mathbf{y}, \hat{\mathbf{y}})\)
2. Output gradient: \(\nabla_{\mathbf{a}_L}\mathcal{L}(\theta) = -(\mathbf{e}(y) - \hat{\mathbf{y}})\)
3. For \(k = L\) down to \(1\):
4. Parameters: \(\nabla_{W_k}\mathcal{L}(\theta) = \nabla_{\mathbf{a}_k}\mathcal{L}(\theta)\,\mathbf{h}_{k-1}^\top\); \(\nabla_{\mathbf{b}_k}\mathcal{L}(\theta) = \nabla_{\mathbf{a}_k}\mathcal{L}(\theta)\)
5. Layer below: \(\nabla_{\mathbf{h}_{k-1}}\mathcal{L}(\theta) = W_k^\top\,\nabla_{\mathbf{a}_k}\mathcal{L}(\theta)\)
6. Layer below, pre-activation: \(\nabla_{\mathbf{a}_{k-1}}\mathcal{L}(\theta) =\) \(\nabla_{\mathbf{h}_{k-1}}\mathcal{L}(\theta)\odot\big[\dots, g'(a_{k-1,j}), \dots\big]\)

\(\mathbf{e}(y)\) is the one-hot vector of this example's true class, which the training data supplies. Every quantity on a right-hand side is available when it is needed: \(\nabla_{\mathbf{a}_k}\mathcal{L}\) from the previous pass through the loop (or the first line), \(W_k\) from the current parameters, and \(\mathbf{h}_{k-1}\) and \(\mathbf{a}_{k-1}\) from the forward pass. That is why the forward pass hands all of its values to the backward pass.

Not in the lecture: at \(k = 1\) the last two lines would compute gradients with respect to the input \(\mathbf{h}_0 = \mathbf{x}\), which nothing uses, so an implementation can stop after the parameter gradients of layer 1.

The whole algorithm is a short loop of matrix and vector operations. No formula for an individual weight such as \(W_{312}\) or \(W_{222}\) is ever written out; the same three lines serve every layer, and they translate directly into a few lines of Python.

The slides give this its own module, 4.9; the video covers it at the end of M8. The backward pass needs \(g'(a_{k-1,j})\), and for the two activation functions used so far it takes a few lines of calculus.

Logistic

The second line is the chain rule for \(1/u\) with \(u = 1 + e^{-z}\). The fourth writes \(e^{-z}\) as \((1 + e^{-z}) - 1\), so the second fraction is \(1 - \frac{1}{1 + e^{-z}} = 1 - g(z)\).

tanh

The second line is the quotient rule; the third uses \(\frac{d}{dz}(e^{z} - e^{-z}) = e^{z} + e^{-z}\) and \(\frac{d}{dz}(e^{z} + e^{-z}) = e^{z} - e^{-z}\).

Both derivatives are expressions in \(g(z)\) itself, and \(g(\mathbf{a}_k) = \mathbf{h}_k\) is already stored by the forward pass. So \(g'(\mathbf{a}_k) = \mathbf{h}_k\odot(1 - \mathbf{h}_k)\) for the logistic function and \(1 - \mathbf{h}_k\odot\mathbf{h}_k\) for tanh, at almost no extra cost.

A network small enough to check by hand: two inputs, one hidden layer of two logistic neurons and two softmax outputs, so \(L = 2\) with 12 parameters. The pseudocode becomes

The figure starts from \(\mathbf{x} = [1, -0.5]\), true class 1, \(W_1 = \begin{bmatrix} 0.5 & -0.4\\ 0.3 & 0.8 \end{bmatrix}\), \(\mathbf{b}_1 = [0.1, -0.1]\), \(W_2 = \begin{bmatrix} 1.2 & -0.7\\ -0.6 & 0.9 \end{bmatrix}\) and \(\mathbf{b}_2 = [0, 0.1]\), all chosen for these notes. One forward and one backward pass by hand:

The last row is the weight \(W_{112}\) that Module 4.4 started from.

Gradient check (not in the lecture). A standard way to test a backpropagation implementation is to compare each derivative with a finite-difference estimate, \(\big(\mathcal{L}(\theta + \epsilon) - \mathcal{L}(\theta - \epsilon)\big)/2\epsilon\), where one parameter is nudged by \(\epsilon = 10^{-4}\) and everything else is held fixed. The figure does this for whichever parameter you select.

Feedforward neural network

Layers of neurons in which each layer feeds only the next: input layer 0, hidden layers 1 to \(L-1\), output layer \(L\).

Pre-activation \(\mathbf{a}_k\)

A layer's weighted sum plus bias, \(\mathbf{a}_k = \mathbf{b}_k + W_k\mathbf{h}_{k-1}\).

Activation \(\mathbf{h}_k\)

The pre-activation passed through the element-wise activation function, \(\mathbf{h}_k = g(\mathbf{a}_k)\), with \(\mathbf{h}_0 = \mathbf{x}\).

Output function \(O\)

The function at the output layer, \(\hat{\mathbf{y}} = O(\mathbf{a}_L)\): linear for real-valued targets, softmax for class probabilities.

Softmax

\(\hat y_j = e^{a_{L,j}}/\sum_i e^{a_{L,i}}\), which turns any real vector into a probability distribution.

Squared-error loss

\(\frac1N\sum_i\sum_j(\hat y_{ij} - y_{ij})^2\), used with a linear output for regression.

Cross-entropy loss

\(-\sum_c y_c\log\hat y_c\); for a one-hot label it equals \(-\log\hat y_\ell\).

Negative log-likelihood

\(-\log\hat y_\ell\), minus the log of the predicted probability of the true class.

One-hot vector \(\mathbf{e}(\ell)\)

A vector with 1 in position \(\ell\) and 0 everywhere else.

Indicator \(\mathbb{1}_{\text{cond}}\)

1 if the condition holds and 0 otherwise.

Chain rule over several paths

\(\partial p/\partial z = \sum_m(\partial p/\partial q_m)(\partial q_m/\partial z)\), adding one product for every intermediate result that depends on \(z\).

Hadamard product \(\odot\)

The entry-by-entry product of two vectors of the same size; the result is a vector.

Outer product

\(\mathbf{u}\mathbf{v}^\top\), the matrix with entries \(u_iv_j\); it gives \(\nabla_{W_k}\mathcal{L} = \nabla_{\mathbf{a}_k}\mathcal{L}\,\mathbf{h}_{k-1}^\top\).

Backpropagation

Computing every gradient by the chain rule from the output layer down, reusing each layer's result for the layer below, after a forward pass that stores every \(\mathbf{a}_k\) and \(\mathbf{h}_k\).