Optimization: from gradient descent to Adam

The lecture opens with a brisk recap of the gradient descent material from Lecture 3. Lecture 4 carried the same rule over to deep networks with backpropagation. Lecture 5 goes back to the smallest network, a single sigmoid neuron with one weight \(w\) and one bias \(b\), and studies the update rule itself. Each time the rule shows a peculiarity, the lecture proposes a fix, and each fix is a new variant of gradient descent.

Training on the two points \((0.5, 0.2)\) and \((2.5, 0.9)\) means finding \(w^*, b^*\) with \(f(0.5) \to 0.2\) and \(f(2.5) \to 0.9\): a sigmoid curve that passes through both points.

Start with a random guess, \(w = 3,\ b = -1\). The curve is clearly wrong, and the loss says how wrong:

We want the loss as close to zero as possible. Moving \(w\) and \(b\) by hand and keeping any change that lowers the loss gives a sequence like this one from the slides:

With two points and two parameters you could also compute \(\mathcal{L}(w,b)\) over a grid, plot the error surface and pick its lowest point. That does not scale to many data points and parameters, and even here the plot covers only \(w, b \in (-6, 6)\), not \((-\infty, \infty)\). The hand-tuning above is a walk on this surface by someone who cannot see it, and some guesses make the loss worse. We want a principled way to move on the surface.

Collect the parameters in \(\boldsymbol\theta = [w, b]\), initialized randomly. A change is \(\Delta\boldsymbol\theta = [\Delta w, \Delta b]\). To be conservative, move only a small amount \(\eta\) in that direction: \(\boldsymbol\theta_{new} = \boldsymbol\theta + \eta\,\Delta\boldsymbol\theta\). The question is which \(\Delta\boldsymbol\theta\) to use, and Taylor series answers it.

A Taylor series approximates a continuously differentiable function near a point \(w_0\) by a polynomial. Degree 1 gives the linear approximation, degree 2 the quadratic one, and higher degrees do better:

Write \(\mathbf{u} = \Delta\boldsymbol\theta\). For a vector of parameters the expansion is

The approximation drops the higher terms because \(\eta\) is small, so \(\eta^2, \eta^3, \dots \to 0\). The move is worth making only if the new loss is lower, \(\mathcal{L}(\boldsymbol\theta + \eta\mathbf{u}) - \mathcal{L}(\boldsymbol\theta) < 0\), which here means \(\mathbf{u}^\top\nabla_{\boldsymbol\theta}\mathcal{L}(\boldsymbol\theta) < 0\).

How negative can that dot product be? Let \(\beta\) be the angle between \(\mathbf{u}\) and the gradient (only in this derivation; from Module 5.4 on, \(\beta\) is the momentum constant):

The change in loss, \(k\cos\beta\), is most negative when \(\cos\beta = -1\), that is \(\beta = 180^\circ\). So move in the direction opposite to the gradient.

The evaluation point matters for the whole lecture. If \(\mathcal{L} = w^2\), then \(\partial\mathcal{L}/\partial w = 2w\), and at \(w_t = 1\) the gradient \(\nabla w_t\) is 2.

For a single point \((x, y)\) take \(\mathcal{L}(w,b) = \tfrac12(f(x) - y)^2\). The chain rule gives

The bias works the same way without the final \(x\). For the two training points, the gradients are sums over the points:

1. Algorithm · gradient_descent()
2. \(t \leftarrow 0\); max_iterations \(\leftarrow 1000\); initialize \(w, b\) randomly.
3. While \(t <\) max_iterations: \(w_{t+1} \leftarrow w_t - \eta\nabla w_t\) and \(b_{t+1} \leftarrow b_t - \eta\nabla b_t\).
4. \(t \leftarrow t + 1\), and repeat.

The slides' Python version uses X = [0.5, 2.5], Y = [0.2, 0.9], starts at \(w = b = -2\) (a second run starts at \(-4, -4\)), sets \(\eta = 1.0\) and runs 1000 epochs. Its inner loop adds up the gradients of both points before making one update, and its error() returns \(\tfrac12\sum(f(x) - y)^2\).

Run from a start on a flat part of the surface (in the video \(w\) and \(b\) are both near \(-5\) and the loss is about 0.4 to 0.5), the point barely moves. Consecutive \((w, b)\) values are so close that the trail in the \(wb\)-plane looks like one continuous snake. Where the surface drops steeply into the valley, the points suddenly jump far apart. Inside the valley the steps become small again.

The reason is in the update \(w \leftarrow w - \eta\nabla w\). If the derivative is 0, \(w\) does not move; if it is small, \(w\) moves a little; if it is large, \(w\) moves a lot. A derivative measures how sensitive the function is to a change in the input. For \(y = x^2\), moving \(x\) from 1 to 1.5 changes \(y\) by 1.25 (from 1 to 2.25), but moving \(x\) from 2 to 2.5 changes \(y\) by 2.25 (from 4 to 6.25). The input moved by 0.5 both times, but the curve is steeper near 2, so the derivative there is larger. Steep regions have large derivatives and give large updates. Gentle regions give small ones.

So a start on a large flat region can cost thousands of iterations in which the loss barely changes, say from 0.9 to 0.8999. With max_iterations at 1000 you might conclude that nothing is happening, when the algorithm is only stuck on the plateau.

A better start does not remove the problem. The slides suggest \(w_{init} = -2,\ b_{init} = 2\) (and \(-3, 2\), with \(t_{max} = 150\)). This start sits on a slope, so the point drops into the valley quickly with widely spaced iterates. Then it reaches the gentle region near the minimum and crawls again, and 100 iterations run out before it arrives. This always happens near convergence, because the surface flattens around a minimum.

Wherever it starts, once gradient descent reaches a gentle slope its progress slows down. The rest of the lecture looks for ways to move faster on gentle slopes.

3-D plots are awkward: a static picture shows one viewing angle. A contour map shows the same surface in 2-D. Cut the surface with horizontal slices at regular intervals along the vertical axis (in the demo at \(-1, -0.8, -0.6, \dots\)) and look at the slices from the top. Each ring is one slice, so every point on a ring has the same height. For an error surface the horizontal plane is the \(wb\)-plane and the height is the loss, so each ring is a set of \((w, b)\) with equal loss.

The spacing between rings carries the slope. A small distance between two rings means the loss changes quickly along that direction: a steep slope. A large distance means a gentle slope. The same pair of rings can be far apart on one side of the surface and close together on the other.

Two things to remember: the rings are taken at equal steps of the loss, and their spacing tells you the slope in each direction. With these you can picture a 3-D surface from its contour map and the other way round. Run on a contour map, gradient descent moves slowly where rings are far apart and quickly where they are packed together. In the lecture's plots the colour shows the loss level: dark red is high loss, dark blue is low, and pink is in between. From here on most demos are drawn as contour maps.

Gradient descent takes a long time to cross regions with a gentle slope because the gradients there are small. (Every algorithm in this lecture starts its first iteration by computing \(\nabla w_0\).)

The lecture's intuition: you are at the Velachery gate of IIT Madras and ask how to reach Phoenix Market City. Someone points right. You take a few cautious steps and ask again, and the next person also points right. After the third person says the same, you are confident and start walking faster. If you are repeatedly told to go in the same direction, take bigger steps in that direction, the way a ball gains momentum rolling down a slope. Even if the current gradient is small, a long history of pushes in one direction should speed you up.

\(u_t\) is a history vector. Unrolling the recursion shows what it holds:

This is an exponentially weighted sum of the current and all past gradients. With the typical \(\beta = 0.9\), the current gradient has weight 1, the previous one 0.9, the one before 0.81, and gradients from long ago have almost no say. Because the history includes the current gradient plus many earlier pushes in the same direction, steps on a flat region are much larger than plain gradient descent's.

The slides' do_mgd code starts at \(w = b = -2\) with \(\eta = 1\) and \(\beta = 0.9\), initializes both histories to 0, sums the gradients over the data and then updates. Note: the code accumulates \(u \leftarrow \beta u + \eta\nabla w\) and then sets \(w \leftarrow w - u\). With a fixed \(\eta\) this is the same update with \(\eta\) folded into the history.

Is moving fast always good? M4 · 13:46

On the first surface, momentum leaves the plateau far faster than gradient descent, but it shoots past the minimum and has to come back. The analogy continues: so many people have told you to go right that you are now on a scooter, you zoom past the mall, take a U-turn, gather speed again and overshoot again.

A second data set gives a surface with flat regions on every side of a narrow, steep valley, so the error is high on both sides of the minimum. A ball dropped there would roll down, up the other side and back. Momentum does the same: it oscillates in and out of the minimum valley and takes many U-turns before it converges. Even so it converges faster than vanilla gradient descent. From \(w_0 = -2,\ b_0 = -4\) with \(\eta = 1\), after 50 iterations (two seconds of the animation at 25 iterations per second) momentum has reached an error of 0.00001 while gradient descent is still at 0.16. M4 · 17:53

Can we reduce momentum's oscillations? The lecture sums up the intuition as look before you leap. Momentum's step has two parts: the history \(\beta u_{t-1}\), known before the step, and the gradient \(\nabla w_t\) at the current weights. We will move by at least the history, so make that move first, compute the gradient where it lands, and use that gradient for the second part.

With \(\eta\) written outside the history, the history move actually reaches \(w_t - \eta\beta u_{t-1}\). The slide for this module writes \(w_t - \beta u_{t-1}\), which is the same when \(\eta = 1\), as in all the demos. The NAdam slides later write the look-ahead with \(\eta\), and the slides' do_nag code (whose history already includes \(\eta\)) evaluates the gradient at \(w - \beta v\).

A 1-D picture makes the difference clear. You are approaching the minimum from the left with a large accumulated history. At \(w_t\) both the history and the current negative slope push to the right, so momentum takes a big step and can land well past the minimum. NAG first makes only the history move. If that move has already crossed the minimum, the slope at the new point is positive and the gradient pulls back, so the combined step ends close to the minimum.

The demo runs both on the oscillating surface. Momentum (blue in the video) takes a long U-turn, while NAG, having looked ahead and seen that it is overshooting, turns back sooner. Throughout the run NAG's oscillations are shorter. Looking ahead lets NAG correct its course faster than momentum, so its oscillations are smaller and it is less likely to escape the minimum valley.

The algorithm so far (and its momentum and NAG versions) goes over the entire training data, adding up per-point gradients, and only then makes one update. That is because the loss is a sum (or average) over all training points, so its gradient is the sum of the per-point gradients. This is the true gradient with no approximation, and the theoretical guarantee from the Taylor argument holds: each step decreases the loss.

The flip side is cost. With a million training points, one update needs a million gradient computations. Even 100 updates, a small number by modern standards, take 100 million computations, and the weights will hardly have moved.

Stochastic gradient descent moves the update inside the loop over the data: it updates after every single point. It treats one point's gradient as an estimate of the average gradient. With a million points it makes a million updates per epoch, where one epoch is one pass over the entire data and one step is one update of the parameters.

The price is that the gradient is now approximate, or stochastic, because it is estimated from a single point. It is like tossing a biased coin once to estimate the probability of heads. There is no longer a guarantee that each step decreases the loss.

In the demo, gradient descent (green) goes down smoothly while SGD (black) often goes off course and comes back. Each such swing is a point whose gradient was far from the true gradient. SGD makes greedy decisions: each point pushes the parameters in the direction most favourable to itself, without knowing how that affects the other points. An update that helps one point can hurt the next, so the points almost compete with each other.

The estimate can be improved by using more than one point. Mini-batch gradient descent keeps a count of points seen and updates after every mini_batch_size points, accumulating their gradients in between. In the coin analogy you now toss the coin 25 times instead of once. Waiting for a million tosses is too slow and one toss is too few; something in between, like 100, gives a fair estimate.

With \(k = 25\) (the slides use a batch size of 25 and 500 epochs), the mini-batch path (blue) stays close to gradient descent (green), while SGD (black) oscillates much more. The larger \(k\), the more accurate the estimate. Some oscillation remains as long as the gradient is approximate, and with \(k = N\) you recover gradient descent exactly. In practice the batch is much smaller than the data set.

\(N\) is the number of data points and \(B\) the mini-batch size. With 1000 points and a batch of 100, mini-batch gradient descent makes 10 updates per epoch.

The same idea gives stochastic versions of momentum and NAG: the update lines move inside the loop over points. The slides run SGD with momentum at \(\beta = 0.5\) and stochastic NAG at \(\beta = 0.85\). Both oscillate, partly because of noisy estimates and partly from momentum's own overshoot, and early on they may head off in a different direction from the full-batch version because the first few points were not representative. The relative advantage survives: NAG still takes shorter U-turns than momentum. Both are far faster than plain SGD, which after 500 steps is still on the plateau with a high error while momentum and NAG are close to convergence. Mini-batch versions of momentum and NAG work the same way (left as an exercise in the slides).

In practice we use mini-batch versions of all these algorithms. The lecturer adds that modern models can be sensitive to the batch size: in his experiments with Transformers, batch sizes of 1024 and 4096 (usually powers of two) gave quite different results, and the common wisdom is that larger batches are better, while still far below the full data set. Papers and existing implementations are the best guide to a good batch size.

Couldn't a large learning rate fix the flat regions? Multiplying a small gradient by a large \(\eta\) still gives a big step. The demo sets \(\eta = 10\) instead of 1 on the first surface: the point does move fast on the plateau, but in steep regions the already large gradient is blown up further and the point oscillates. What we want is a learning rate that adjusts to the gradient, large on gentle slopes and small on steep ones. The adaptive algorithms of Module 5.9 do exactly that.

The initial learning rate

For standard problems such as machine translation, speech recognition or image recognition, start from the hyperparameters reported in the literature and search a small window around them. For a new problem where nothing is known, try values on a log scale: 0.0001, 0.001, 0.01, 0.1, 1.0. Run a few epochs with each and watch the loss: with rates that are too high it increases (you overshoot into high-loss regions), with good rates it decreases smoothly. Then do a finer search around the best value; if 0.1 was best, try 0.05, 0.2 and 0.3. These are heuristics with no clear winning strategy.

Annealing

Early in training the weights are random and a fairly high rate is fine, but near the minimum a high rate risks overshooting. Annealing means reducing the learning rate as training progresses.

• Step decay. Halve the learning rate every 5 (or 10) epochs, using the epoch count as a proxy for how close you are to the minimum. Or halve it after an epoch whose validation error is higher than the previous epoch's, as in the procedure below.
• Exponential decay. \(\eta = \eta_0 e^{-kt}\), with hyperparameters \(\eta_0\) and \(k\) and step number \(t\). A large \(k\) decays quickly, a small \(k\) smoothly. (The slide prints the formula as \(\eta_0^{-kt}\).)
• \(1/t\) decay. \(\eta = \dfrac{\eta_0}{1 + kt}\), again with \(k\) controlling the speed of decay.

1. Validation-based step decay
2. After every epoch, save the weights and compute the loss on held-out validation data.
3. If the validation loss decreased, continue with the same learning rate.
4. If it increased, throw away the last epoch's updates, restore the weights saved at the end of the previous epoch, halve \(\eta\) and run the epoch again.
5. If the loss keeps increasing even after repeated halving, training has probably converged, so stop.

Both decay formulas add the hyperparameter \(k\), which is itself hard to set. The lecturer's own preference is the validation-based rule: reduce the rate when the validation loss goes up.

A schedule for momentum

Sutskever et al. (2013) suggested increasing \(\beta\) over time:

For \(t < 250\), \(\log_2 1 = 0\) and the first term is \(1 - 2^{-1} = 0.5\). For \(250 \le t < 500\) it is \(1 - 2^{-2} = 0.75\), and for \(750 \le t < 1000\), \(\log_2 4 = 2\) gives \(1 - 2^{-3} = 0.875\). Each value is capped at \(\beta_{max}\), and \(\beta_{max} = 0\) means no momentum. (Equivalently, the first term is \(1 - \frac{1}{2(\lfloor t/250\rfloor + 1)}\).) Early on the history is as unreliable as the current gradient, so it gets little weight. Later the history summarizes many updates that brought you near the minimum, while a single faulty update could throw you away from it, so the history gets more weight.

With one fixed learning rate you are married to it: too high and you oscillate, too low and you move slowly. Line search tries several learning rates at every step and keeps the best result.

1. Gradient descent with line search
2. Compute \(\nabla w_t\) and \(\nabla b_t\) once.
3. For each \(\eta\) in \(\{0.1, 0.5, 1, 2, 10\}\), form the candidate \(w_t - \eta\nabla w_t,\ b_t - \eta\nabla b_t\) and compute its loss.
4. Keep the candidate with the lowest loss as \((w_{t+1}, b_{t+1})\) and discard the others.
5. Repeat from step 1.

The flip side is that each step does many more computations: one extra loss evaluation per candidate rate. In return the choice becomes adaptive. On flat regions the large rates win, and near the valley the large rates would overshoot and raise the loss, so a small one is picked. In the demo (blue: gradient descent with fixed \(\eta = 1\); black: line search), line search moves quickly across the flat region, does not jump out of the valley, and converges much faster. It shows some oscillations, but these are different from the momentum and NAG oscillations.

The discussion of learning rates kept asking for an adaptive rate: small on steep regions, large on gentle ones. The gradients are what they are, but the multiplier in front of them can change. A second argument for adaptation comes from the input features.

Take a sigmoid neuron with four inputs, \(\mathbf{x} = \{x^1, x^2, x^3, x^4\}\) and \(\mathbf{w} = \{w^1, w^2, w^3, w^4\}\), so \(y = f(\mathbf{x}) = \dfrac{1}{1+e^{-(\mathbf{w}^\top\mathbf{x} + b)}}\). Superscripts index the features within one training point; subscripts still index training points. For a single point \((\mathbf{x}, y)\):

The formula is the one-input formula with the last \(x\) replaced by the matching feature, because the derivative of \(\mathbf{w}^\top\mathbf{x} = w^1x^1 + w^2x^2 + \cdots\) with respect to \(w^1\) is \(x^1\). For \(n\) points, sum these over the points.

Now suppose \(x^2\) is sparse: zero for most inputs. Then most terms in the sum for \(\nabla w^2\) are zero, its total gradient is small, and \(w^2\) receives few and small updates. The lecture's example is a feature "the actor is Aamir Khan" over the last 1000 Bollywood movies, which is on for perhaps four or five of them. You might shrug and let such a weight stay put, but sparse features can be important. A feature for the director Christopher Nolan is on for very few films, yet it is a strong signal of a good movie whenever it is present. If its weight starts small and barely moves, its information is lost. Real data has many such features.

So we want a different learning rate for each parameter that accounts for how often its feature appears: higher for rare features, lower for dense ones. With millions of features this cannot be a pile of if-else rules; it has to come out of the update equations.

The intuition: decay a parameter's learning rate in proportion to its update history. A dense feature has received large updates step after step, so be conservative with it. A sparse feature has received only tiny updates, so keep its rate high and let it move.

The effective learning rate \(\eta/\sqrt{v_t + \epsilon}\) is now inversely related to the update history. For a dense feature \(v_t\) grows quickly and the rate shrinks. For a sparse feature \(v_t\) has grown little even by \(t = 100\), so its rate stays high.

Making a feature sparse

A 2-D plot allows only two parameters, \(w\) and \(b\). The input for \(b\) is always 1, so it cannot be sparse; the only option is to make \(x\) sparse. The slides create 500 random \((x, y)\) pairs and set \(x = 0\) for roughly 80% of them. Every algorithm uses \(\eta_0 = 0.1\), and momentum uses \(\beta = 0.9\).

On the contour plot (horizontal axis \(w\), vertical axis \(b\)) the loss is high in two red, fairly flat regions and drops steeply into a blue valley. Gradient descent, momentum and NAG all do something odd: at first they move almost only along \(b\), with very little movement along \(w\). The feature for \(w\) is sparse, so \(w\) gets small gradients and few updates, while \(b\) is dense. Once in the valley there is nothing more to gain along \(b\), and they creep along \(w\) to the minimum. The path is a right angle instead of the direct diagonal; the lecturer jokes that these algorithms were never taught Pythagoras' theorem. Such sparsity is common in large networks with thousands of input features, so this has to be addressed.

AdaGrad moves along \(w\) as well as \(b\). For \(w\) the denominator \(\sqrt{v_t}\) stays small, and when it is below 1 the effective rate is even larger than \(\eta\), so \(w\) moves in proportion to \(b\). The momentum methods still arrive first, because AdaGrad has no momentum term (combining the two comes later, as Adam). And as AdaGrad nears the minimum it slows down a lot, because by then enough history has accumulated to shrink its rate. M9 · 11:53

A closer look at the two histories M9 · 12:24

Since \(\nabla w = (f(x) - y)f(x)(1 - f(x))\,x\) and \(x\) is zero most of the time, the gradient for \(w\) is zero at most steps. So \(v_t^w\) grows gradually and \(\eta/\sqrt{v_t + \epsilon}\) decays slowly; early on it can even exceed \(\eta\). The gradient for \(b\) has no factor \(x\) and is nonzero at almost every step, so \(v_t^b\) grows rapidly and the rate for \(b\) decays rapidly.

Both start from \(\eta_0 = 0.1\). The sparse \(w\) gets a rate above \(\eta_0\), the dense \(b\) one far below it. That is the adaptive behaviour we wanted. M9 · 18:31

Notice that \(v_t^w\) and \(v_t^b\) never become zero, and never decrease, even after the gradients have become zero near the minimum. A running sum keeps all its old terms. Near the minimum the gradients are small and the effective rate is also small, so AdaGrad cannot move fast in these flat regions. M9 · 22:04

Why the square root? Using \(\eta/v_t\) would also make the rate inversely related to the history, but in practice the algorithm works less well without the root. The lecturer's guess is that \(v_t\) then grows so fast that it kills the rate too early, but he knows of no clear theoretical justification.

The flip side: over time the effective rate for \(b\) decays so much that \(b\) stops being updated. AdaGrad found the right \(w\) quickly but takes a long time to find the right \(b\), the mirror image of the other algorithms. M9 · 24:07

AdaGrad lets its denominator grow without limit, so after a while the frequent parameters receive tiny updates. The fix is to decay the denominator: shrink the old history at every step and add only a fraction of the current squared gradient.

Unroll both histories for \(b\) with \(\beta = 0.9\):

This is the same exponentially weighted form as momentum's history (the slide rounds 0.081 to 0.08). RMSProp's \(v_2\) is much smaller than AdaGrad's because every term is a fraction and the old terms keep shrinking. So \(\eta/\sqrt{v_t + \epsilon}\) decays slowly for \(b\) compared with AdaGrad.

In the demo (the slides' code uses \(\eta = 0.1\), \(\beta = 0.5\) and changes only the line that updates \(v\)), RMSProp reaches the minimum much faster than AdaGrad, moving along the diagonal in proportion in \(w\) and \(b\) because it is less aggressive in decaying the rate. But then it starts oscillating around the minimum: the sigmoid already fits the points, yet the iterate keeps moving.

Why RMSProp oscillates

In AdaGrad, \(v_t\) only increases and then saturates. In RMSProp it can also decrease, and the slides' plots show the gradients \(\nabla w\) and \(\nabla b\) themselves oscillating after some iterations. The history works like a moving window: old squared gradients fade, so once recent gradients are small the current weighted sum is less than the earlier one. The learning rate can therefore increase, decrease or stay constant. Near the minimum, after the gradients have been close to zero for a while, both the remaining history and the new terms are tiny and \(v_t\) barely changes from step to step, so the effective learning rate becomes constant.

A constant rate can make the iterate oscillate. On a bowl, an update from one side can land at the mirror point on the other side, where the gradient has the same size and the next update brings it straight back. Even if the derivatives differ slightly, the point keeps jumping from one side to the other instead of settling. M10 · 10:31

Setting the initial learning rate appropriately removes the oscillation in the demo: with \(\eta = 0.05\) and \(\epsilon = 10^{-4}\) (so the very first effective rate, with \(v = 0\), is \(0.05/\sqrt{10^{-4}} = 5\)), RMSProp converges without oscillating. The lecturer stresses that the lesson here is about RMSProp itself: it is sensitive to the initial learning rate, and a value that works on this toy problem may be hard to find on another. M10 · 14:49

Sensitivity to the initial learning rate M10 · 15:23

Keep the surface, the data and the start fixed and change only \(\eta_0\). With \(\eta_0 = 0.6\) the point oscillates widely between high-loss points, and its sigmoid is nowhere near the data. With \(\eta_0 = 0.1\) it moves more slowly and still oscillates, but between low-loss points. With \(\eta_0 = 0.05\) it converges. Converging, oscillating around an acceptable solution, or staying far off: any of the three can happen, depending only on \(\eta_0\).

Plotting the effective rate \(\eta_0/\sqrt{v_t + \epsilon}\) with \(\eta_0 = 0.6\) shows the good part of RMSProp. While the point descends the steep region, gradients grow, \(v_t\) grows and the rate falls. Once it reaches flat ground the new gradients are tiny and the old large ones carry high powers of \(\beta\), so \(v_t\) starts falling and the rate rises again. That matches the wish list: lower the rate on steep curvature, raise it on gentle curvature. With \(\eta_0 = 0.1\) the curve has the same shape but lower values: the rate falls more slowly yet stays lower, and later it cannot grow as much.

Could one \(\eta_0\) be best everywhere? The slides' numbers, taking the denominator to be 1.25 in a steep region and 0.1 in a flat region: M10 · 27:11

The preferred \(\eta_0\) changes along the same surface, yet it is fixed once at the start. So we would like the numerator to change with the gradient as well, adjusting together with the denominator.

Drawbacks so far M10 · 29:22

• Both AdaGrad and RMSProp are sensitive to the initial learning rate, the initial parameter values and the corresponding initial gradients.
• In AdaGrad, large initial gradients make \(v\) large from the start, so the learning rate stays low for the rest of training.
• In AdaGrad, if a gentle curvature is met later there is no way to increase the learning rate, because \(v\) never comes down.

RMSProp's decaying average fixes the last two, but the sensitivity to \(\eta_0\) remains. For well-studied data sets, such as standard machine-translation benchmarks, people know a good range for \(\eta_0\); for a new data set you may not.

AdaDelta avoids setting an initial learning rate at all. Each iteration does five things:

1. AdaDelta · for t in range(1, N), with \(v_{-1} = u_{-1} = 0\)
2. Compute the gradient \(\nabla w_t\).
3. \(v_t = \beta v_{t-1} + (1-\beta)(\nabla w_t)^2\)
4. \(\Delta w_t = -\dfrac{\sqrt{u_{t-1} + \epsilon}}{\sqrt{v_t + \epsilon}}\,\nabla w_t\)
5. \(w_{t+1} = w_t + \Delta w_t\)
6. \(u_t = \beta u_{t-1} + (1-\beta)(\Delta w_t)^2\)

There are now two histories. \(v_t\) tracks squared gradients \(\nabla w_t\), as in RMSProp. \(u_t\) tracks the squared updates \(\Delta w_t\); because the update uses \(\Delta w_t\), the method is called Adaptive Delta. The minus sign sits inside \(\Delta w_t\), so the weight update is \(w_t + \Delta w_t\). The effective learning rate \(\sqrt{u_{t-1} + \epsilon}/\sqrt{v_t + \epsilon}\) is a ratio of two histories, with no \(\eta_0\), and its numerator now depends on past gradients instead of being a constant as in RMSProp and AdaGrad. Two details matter: the \(u_t\) computed at step \(t\) is used only in the next iteration, so the numerator runs one step behind the denominator, and only a fraction \((1-\beta)\) of \((\Delta w_t)^2\) enters \(u_t\).

Starting in a steep region

Take \(\epsilon = 10^{-6}\) and \(\beta = 0.9\), and write everything in terms of the gradients:

(From \(t = 1\) on, \(\epsilon\) is ignored because \(u\) is no longer zero.) Both histories grow at every step, but \(u_t\) stays smaller than \(v_t\): it takes only a fraction of squared steps that were themselves scaled down. At \(t = 2\) the numerator uses \(u_1\) while the denominator \(v_2\) already contains the newest, large gradient. So on a steep slope the ratio, and with it the effective rate, is small. Unlike RMSProp's \(\eta/\sqrt{v_t}\), though, the numerator also grows with the history, so AdaDelta does not cut the rate as aggressively on steep curvature.

Entering a flat region

Suppose the point has moved into low curvature. The newest gradients are small and get the largest weights in \(v_t\) (0.1, 0.09, …), while the earlier large ones now carry high powers of \(\beta\) (weights like 0.034 and 0.038 on the slide). After some iterations \(v_t\) starts decreasing. \(u_{t-1}\) lags one step behind and still remembers the larger earlier steps, so the ratio of numerator to denominator increases: if the gradient stays low, the effective learning rate grows. When the point enters a steep region again, \(v_t\) sees the large current gradient a step before \(u_{t-1}\) does, and the ratio falls. So AdaDelta lets the numerator increase or decrease based on the current and past gradients, which is what RMSProp lacked.

Results M11 · 12:31

The slides' code starts at \(w = b = -4\) with \(\beta = 0.99\) and \(\epsilon = 10^{-4}\). Starting from a moderately high-curvature region, AdaDelta reaches the minimum after 35 iterations, while RMSProp oscillates and AdaGrad is slow. Its plots of \(v_t\) and \(u_t\) have the same shape but different magnitudes, so when \(v_t\) grows, \(u_t\) grows proportionally, and the other way round.

In a second comparison, the slides start RMSProp with \(\eta_0 = 0.013\) (as if chosen by chance) and AdaDelta at \(w_0 = -4,\ b_0 = -4\). AdaDelta converges more quickly because its learning rate adapts sensibly, while RMSProp decays its rate more aggressively.

All the algorithms so far on one surface, from the same start: gradient descent moves slowly; momentum and NAG oscillate (NAG less); all three first move along \(b\) because \(w\) is sparse. AdaGrad, RMSProp and AdaDelta move along \(w\) and \(b\) together, RMSProp shows its oscillation, and AdaDelta converges fastest. M11 · 18:09

Adam stands for adaptive moments. Its idea: do everything RMSProp and AdaDelta do to fix AdaGrad's decay problem, that is, divide by an exponentially weighted history of squared gradients. In addition, use a cumulative history of the gradients themselves, as momentum does, instead of only the current gradient.

\(m_t\) is classical momentum, written with weight \((1-\beta_1)\) on the current gradient. \(v_t\) is RMSProp's denominator. The hats are bias corrections, explained in the next section. Since \(\beta_2 = 0.999\), each new squared gradient enters \(v_t\) with a weight of only 0.001.

The lecture calls \(\sqrt{v_t}\) an \(L^2\) norm. Expanded, \(v_t = (1-\beta_2)\sum_\tau \beta_2^{\,t-\tau}(\nabla w_\tau)^2\). Ignoring the decay weights, this is the sum of squares of all past gradients, and its square root is the \(L^2\) norm of the vector of past gradients. So \(\sqrt{v_t}\) is an exponentially weighted \(L^2\) norm, a view that leads to AdaMax below.

The slides' code starts at \(w = b = -4\) with \(\eta = 0.1\), \(\beta_1 = 0.9\), \(\beta_2 = 0.999\) and \(\epsilon = 10^{-10}\). It uses per-point stochastic gradients and bias-corrects with exponent \(i + 1\), the epoch count. In the demo AdaDelta converges fastest on this surface, Adam is also faster than the others, and RMSProp has its usual oscillation. The lecturer notes this does not mean Adam always wins. Adam sits closer to RMSProp with momentum, and unlike AdaDelta it still has a learning rate \(\eta\).

\(m_t\) is a running average of gradients. We do not want to rely too much on the current gradient, since one sudden large gradient would cause a large jump. What we really want is the expected gradient \(E[\nabla w_t]\), the gradient all the data points together would give, instead of a point estimate from one point or one mini-batch. We compute an exponential moving average instead. Ideally \(E[m_t]\) would equal \(E[\nabla w_t]\). Is that so?

Unroll \(m_t = \beta m_{t-1} + (1-\beta)\nabla w_t\) from \(m_0 = 0\) (writing \(\beta\) for \(\beta_1\)):

This is momentum's sum from Module 5.4 with an extra factor \((1-\beta)\). Take expectations of both sides. Constants come out, and the expectation of a sum is the sum of expectations:

Now assume that every \(\nabla w_\tau\) comes from the same distribution, so \(E[\nabla w_\tau] = E[\nabla w]\) for all \(\tau\). The randomness comes from which data points are picked at each step. As the weights move, the distribution of gradients could change, so this is a simplifying assumption. Under it, \(E[\nabla w]\) leaves the sum, and what remains is a geometric series with ratio \(\beta\):

Without the correction, \(m_t\) and \(v_t\) are biased toward their zero initialization by the factor \(1 - \beta^t\), which is far below 1 for small \(t\). With \(\beta_2 = 0.999\) this lasts for many steps.

What goes wrong without it

Look at the effective rate \(1/\sqrt{v_t}\) (taking \(\eta = 1\) and ignoring \(\epsilon\)) with \(\beta_2 = 0.999\), a first gradient of 0.1 and then a zero gradient:

The slide rounds \(1 - 0.999^2 = 0.001999\) to 0.0019 and gets 0.0052 and 13.8 in the last row.

Without correction the first effective rates are enormous. As the Adam paper puts it, skipping the correction leads to "initial steps that are much larger". Bias correction keeps the initial rates moderate.

The demo compares Adam with and without bias correction (\(\beta_2 = 0.999\), \(\eta_0 = 0.1\)). Adam without correction (blue) makes very large initial updates, rushes ahead and has to come back. It still converges on this toy surface, but in general a huge first step could land in a region from which only a worse minimum is reachable. The plots of the effective rates agree: without correction they reach values around 2 to 2.5, with correction they start smaller. With correction the rate for the sparse \(w\) also keeps increasing as the point enters the valley, where \(w\) still has to move, which is the behaviour we want. M12 · 19:02

Noisy observations of a smooth curve

A second view: gradients computed from mini-batches are noisy versions of the full-batch gradient, which would trace a smooth curve over the iterations. An exponential moving average smooths the noise, much like averaging over a recent window, since older points carry negligible powers of \(\beta\). Without correction the average starts at \(m_0 = 0\) and gives the first observation weight only \(1-\beta\). With \(\beta = 0.999\), a first value near 1 becomes 0.001, the estimate stays stuck near zero, and it takes many steps before the zero start is forgotten and the average reaches the true curve. With correction, \(\hat m_1 = (1-\beta)x_1/(1-\beta) = x_1\) (a first value of 0.5 gives exactly 0.5), so the estimate starts in the right place and stays close. M12 · 26:21

The \(L^p\) norm of a vector is

To visualize it, fix \(L^p = 1\) in two dimensions, so \(|x_1|^p + |x_2|^p = 1\), and vary \(p\). For \(p = 1\) the set is a diamond; the point \((0.5, -0.5)\) is on it because \(0.5 + 0.5 = 1\). For \(p = 2\) it is the unit circle. Any \(p \ge 1\) is allowed, but large \(p\) is numerically unstable: a small \(|x|\) raised to a high power becomes too small to represent, and a value above 1 blows up. That is one reason \(L^2\) is the usual choice.

As \(p \to \infty\) the norm simply becomes \(\max(|x_1|, \dots, |x_n|)\). A proof sketch: without loss of generality let \(|x_1|\) be the largest entry and factor it out,

Every ratio is less than 1, so its \(p\)-th power vanishes, and what remains is the \(p\)-th root of \(|x_1|^p\). (Ties with \(|x_1|\) do not matter either, since \(k^{1/p} \to 1\).) The max is easy to compute and has no numerical instability.

Replace the \(L^2\) norm by the max

Adam's \(\sqrt{v_t}\) is an exponentially weighted \(L^2\) norm of the past gradients. Replace it by the \(L^\infty\) norm of the same weighted history:

The recursion is a pairwise max: the decayed max so far against the current gradient's magnitude. Three things drop out. The current gradient is no longer scaled by \((1-\beta)\). There is no square root, because the \(p\)-th root was already taken when the norm became the max. And there is no bias correction for \(v_t\), because the max is not pulled toward the zero initialization: \(v_1 = \max(0, |\nabla w_1|) = |\nabla w_1|\). On the noisy-observation example, \(\max(\beta v_{t-1}, |\text{noisy } f(t)|)\) starts at the first observation and zigzags along the peaks (tick "max norm" in the widget above).

Why the max suits sparse features M13 · 10:47

Suppose the initial gradient is large and the following gradients are zero because \(x\) is sparse. When the gradient is zero because \(x = 0\), nothing was learned, so ideally the learning rate should not change, and especially not increase. Take 50% of inputs to be zero, so the gradient alternates between 1 and 0, with \(\beta_2 = 0.999\) and \(\eta_0 = 1\):

The slides give steps 1 and 2; step 3 of the exponential average is computed here with the same rule.

With the max the rate stays almost constant, moving only between 1 and 1.001. With exponential averaging (even bias-corrected) the rate jumps to 1.41 on a zero gradient and keeps rising and falling with the zeros. So the max norm is useful when you do not want sparse features to disturb the learning rate.

For a smoother gradient profile, high for the first ten or so steps and then gradually decreasing, the exponential average in RMSProp keeps increasing the learning rate while the max norm is conservative about increasing it. On that smooth convex surface both converge eventually. Low rates on steep parts and high rates on gentle parts are what we want, and on a smooth convex surface increasing the rate does no harm, so there exponential averaging wins. M13 · 17:00

NAG is better than momentum-based gradient descent, and Adam uses momentum, so the obvious next step is to give Adam the Nesterov look-ahead. That is NAdam. To get there, the lecture first rewrites NAG in a form that ports easily.

Two forms of the momentum history are common: \(u_t = \beta u_{t-1} + \nabla w_t\) (Module 5.4) and \(m_t = \beta m_{t-1} + (1-\beta)\nabla w_t\) (Adam). They are interchangeable, and the derivation below drops the factor \((1-\beta)\) for brevity. NAG, written with a gradient variable \(g_t\):

Walk through it with \(\eta = 1\) and \(m_{-1} = 0\):

The momentum vector \(m_{t-1}\) is used twice: once to build the temporary look-ahead point \(w_1 - \beta m_0\), whose gradient is computed and which is then thrown away, and once more in the history. That means two weight updates per step, which is costly for big networks, and the equations mix the time steps \(t-1\), \(t\) and \(t+1\). But \(\beta m_0\) is already known at the previous step, so the look-ahead could have been done there: \(w_1 - \beta m_0 = (w_0 - \nabla w_0) - \beta m_0\).

Walk through again with \(\eta = 1\) and \(m_0 = 0\):

Now the gradient is always taken at the current weights, with no temporary point. The look-ahead is applied in the momentum step: \(w_1\) already includes the extra \(-\beta g_1\), and the gradient of that looked-ahead point is computed at the next step. Only the time steps \(t\) and \(t+1\) appear. The lecturer compares it to moving one iteration of a while loop outside the loop: the look-ahead is done before entering the next iteration, so it is not repeated inside.

The first four equations are Adam's. Ignoring the scale factors, the last one is \(w_{t+1} = w_t - \frac{\eta}{\sqrt{\hat v_{t+1}} + \epsilon}\big(\beta\,\hat m_{t+1} + \nabla w_t\big)\): the history plus the current gradient, the same pattern as \(\beta m_{t+1} + g_{t+1}\) in the rewritten NAG. The \(\beta_1\) and \((1-\beta_1)\) are Adam's weights on history and current gradient, and the denominators \(1-\beta_1^{\,t+1}\) are bias correction. In the demo, NAdam (blue) has smaller oscillations than Adam (black): the Nesterov effect. M14 · 16:32

The lecture's demos use several data sets and surfaces. The widget below runs six of the update rules, exactly as written in the result boxes above, on the two-point problem from Module 5.1, starting from the slides' \(w = b = -2\). Each optimizer starts with the learning rate from the slides' code.

Things to look for, all from the lecture: GD's dots bunch together on the plateau around the start; momentum leaves the plateau sooner but swings past the valley floor; NAG's swings are shorter; AdaGrad's steps keep shrinking as its sum grows; RMSProp gets close fast but can keep hopping around the minimum when its step size stops shrinking; Adam combines momentum's direction with RMSProp's scaling. Then change \(\eta\) for one optimizer, or move the start onto the flat region near \((-4, -4)\), and see which rules still cope.

The slides call this the million-dollar question. Adam seems to be more or less the default choice now, with \(\beta_1 = 0.9\), \(\beta_2 = 0.999\) and \(\epsilon = 10^{-8}\). It is supposed to be robust to the initial learning rate, but the authors have observed that \(\eta = 0.001\) or \(0.0001\) works best for sequence generation problems. Many papers also report that SGD with momentum (Nesterov or classical) and a simple annealing schedule works well in practice, again typically starting with \(\eta = 0.001\) or \(0.0001\) for sequence generation. Adam might just be the best choice overall, but some works point out that Adam fails to converge in some cases, and models trained with Adam often do not generalize as well.

The lecturer's advice: Adam is the default in most papers he has seen, and if you do not know anything else, use Adam or one of its variants such as NAdam. People who really know how to set the initial learning rate, the momentum and the learning-rate schedule for (momentum) gradient descent report results as good as Adam's, but that expertise takes time to build. The slides close the topic with Pedro Domingos's remark that newcomers start with off-the-shelf optimizers and later replace them with custom-designed ones.

Typical constants from the slides: \(\beta = 0.9\) for momentum and RMSProp, \(\beta_1 = 0.9\), \(\beta_2 = 0.999\), \(\epsilon = 10^{-8}\) for Adam.

The last module adds a few more learning-rate schedulers to the map of schemes:

Log search is line search over learning rates on a log scale. The gradient-based schemes are the adaptive optimizers of Module 5.9.

Cyclical learning rate

Consider a loss surface shaped like a saddle. Seen from one side, a point at the end of a valley looks like a minimum, but in another direction you could still go further down. Deep-learning losses are not convex, and multiple papers report that training often stalls at saddle points. Start at the slide's yellow point with an exponentially decaying learning rate: by the time the point reaches the saddle the rate has become tiny, and although escape is possible the algorithm has no way out.

The rationale of the cyclical learning rate (CLR) paper: the difficulty in minimizing the loss often comes from saddle points rather than poor local minima, so a scheme that can increase the rate near saddle points helps. Adaptive methods can do this, but they keep extra histories per parameter, an additional computational cost. A simple alternative is to vary the learning rate cyclically between a low and a high value. If the rate is high when you need it low, or the other way round, you only have to wait out the cycle. The triangular schedule:

\(\mu\) is the step size, the number of iterations to go from the minimum rate to the maximum. With \(\eta_{max} = 0.5\), \(\eta_{min} = 0.01\) and \(\mu = 20\), at \(t = 20\): \(\lfloor 1 + 20/40\rfloor = 1\), so the middle term is \(|20/20 - 2 + 1| = 0\), the max is \(\max(0, 1 - 0) = 1\), and \(\eta_{20} = 0.01 + 0.49 \cdot 1 = 0.5 = \eta_{max}\). At \(t = 30\) the middle term is \(|1.5 - 2 + 1| = 0.5\), giving \(\eta_{30} = 0.01 + 0.49\cdot 0.5 = 0.255\), halfway down again.

The slides' code wraps this as cyclic_lr(iteration, max_lr=0.1, base_lr=0.001, step_size=30) and uses it inside gradient descent for both \(w\) and \(b\), starting from \(w = -2,\ b = 0.0001\). In the demo, gradient descent with an exponentially decaying rate (black) enters the valley and gets stuck at the saddle. The cyclical version (blue) waits until its rate rises, gets pushed out, and continues down.

Changing the rate cyclically can also speed up convergence. In the slides' example, gradient descent with a cosine schedule converges much faster than with the fixed rate of 1 and reaches the minimum at the 46th iteration. Run longer, it crosses the minimum, but a technique such as early stopping (Lecture 6) can roll back to the best point.

\(\eta_{max}\) and \(\eta_{min}\) are the largest and smallest rates, \(t\) is the current epoch, and \(T\) is the restart interval (which can be adaptive). At \(t = 0\) the cosine is 1 and \(\eta_t = \eta_{max}\); at \(t = T\) it is \(\cos\pi = -1\) and \(\eta_t = \eta_{min}\). With \(\eta_{max} = 1\), \(\eta_{min} = 0.1\) and \(T = 50\): at \(t = 50\), \((50 \bmod 51)/50 = 1\) gives \(\eta = 0.1\), and at \(t = 51\), \(51 \bmod 51 = 0\) sends the rate straight back to 1. The abrupt jump after every \(T\) steps is why this is called a warm restart: instead of climbing back gradually, as the triangular schedule does, it restarts from a warm, high rate and anneals down again. The slide notes this is a version of the paper's formula modified for batch gradient descent; the original works with SGD and restarts after a given epoch \(T_i\).

Warm start (warmup) is popular in the Transformer literature (Lecture 14). All the schedules so far, and the adaptive methods, start with some rate and decay it. Warm start lets the rate rise gradually to its maximum first and only then decays it. At the start the parameters are random and the gradients unreliable, so the steps are kept small; as confidence grows the rate increases, and from the peak onward it decays. This is the default schedule for Transformer-based architectures. The number of warmup steps sets how long the rise lasts: the lecture compares 2000 and 4000 warmup steps (the slide uses 4000), and the shorter warmup rises more steeply before decaying.

Standard form (not in the lecture): the original Transformer schedule is \(\eta = d_{model}^{-0.5}\cdot\min\big(\text{step}^{-0.5},\ \text{step}\cdot\text{warmup}^{-1.5}\big)\), a linear rise followed by a decay proportional to \(1/\sqrt{\text{step}}\). It is used for the last panel below.

Any of these schedules can be used together with the optimization algorithms above.

Gradient descent rule

\(w_{t+1} = w_t - \eta\nabla w_t\): the direction that makes the first-order Taylor change most negative.

Contour map

Top view of equal-loss slices; close rings mean a steep slope, distant rings a gentle one.

Momentum

Update with the history \(u_t = \beta u_{t-1} + \nabla w_t = \sum_\tau \beta^{t-\tau}\nabla w_\tau\) instead of the current gradient alone.

Nesterov look-ahead

Compute the gradient after the history move, \(\nabla(w_t - \beta u_{t-1})\), to brake before overshooting.

Epoch and step

An epoch is one pass over the data; a step is one parameter update (1, \(N\) or \(N/B\) per epoch).

Mini-batch

\(B\) points whose summed gradient estimates the true gradient; \(B = 1\) is SGD, \(B = N\) is batch GD.

Annealing

Reducing the learning rate during training: step, exponential, \(1/t\) or validation-based decay.

Line search

Trying several learning rates each step and keeping the update with the lowest loss.

Effective learning rate

The actual multiplier of the gradient, such as \(\eta/\sqrt{v_t + \epsilon}\) in AdaGrad and RMSProp.

Sparse feature

An input that is zero for most data points, so its weight gets small, rare gradients.

Bias correction

Dividing an exponential average started at 0 by \(1-\beta^t\) so its expectation equals the gradient's.

Max (\(L^\infty\)) norm

The limit of the \(L^p\) norm as \(p \to \infty\); in AdaMax, \(v_t = \max(\beta_2 v_{t-1}, |\nabla w_t|)\).

Warm restart

Jumping the learning rate back to its maximum at the end of each cosine-annealing cycle.

Warm start (warmup)

Raising the learning rate from small to its peak over the first steps, then decaying it.