Regularization: from bias and variance to dropout

Until now (Lecture 5) the focus was on minimizing the objective quickly and reliably. The new problem is what a well-minimized objective can hide. Deep learning models often have far more parameters than the training data has samples; the slide's picture is billions of parameters against millions of samples. Such over-parameterized models are prone to overfitting: they can drive the training error to zero on the points they were given and still predict badly on points they have not seen. To understand why, the lecture looks at the bias and variance of a model as its capacity changes.

The curve-fitting experiment

The data points were drawn from a sinusoid, so for once the true relation \(f(x)\) is known. We pretend not to know it and fit two models to the 500 training points:

The simple model has 2 parameters and the complex one 26, so the complex one has much more capacity. Both make an assumption about how \(y\) depends on \(x\), and neither knows the true \(f\). Each is trained by minimizing the squared error \(\sum_i (y_i - \hat f(x_i))^2\) with gradient descent. The experiment is repeated \(k\) times, each time on a different random sample of the training data (in the video, about 400 of the 500 points, 30 times). Each repetition gives one line and one degree-25 curve.

The plots show two things 13:21.

• Simple models trained on different samples barely differ from each other, but all of them are far from the sinusoid. They underfit: even their training error is large.
• Complex models trained on different samples are very different from each other, but on average they follow the sinusoid, and many of them reach close to zero training error.

Definitions

The randomness comes from which sample of the training data a model happened to see. Averaging the 30 fitted curves at each \(x\) estimates \(E[\hat f(x)]\), the expected (average) model. On the slide the green line is this average for the simple model, the blue curve the average for the complex model, and the red curve the true \(f\).

Bias measures how far the average model is from the truth. Variance is the standard statistical variance of the random variable \(\hat f(x)\): roughly, how much models trained on different samples differ from each other. The simple model's average line is far from the sinusoid, so its bias is high; the complex model's average is close, so its bias is low. The simple lines all hug their average (low variance), while some complex curves sit far from theirs (high variance).

There is always a trade-off between the two 24:32. High variance is bad because the model you end up with depends on the training sample you happened to get, so a test point that resembles a different sample can be predicted badly. Low variance is no use if it comes with high bias, because then all the models agree with each other and are all far from the truth. The aim is a moderate amount of each. Both contribute to the mean squared error, as the next module shows.

Take a new point \((x, y)\) that was not seen during training. If \(\hat f\) predicts \(y\), its mean squared error is \(E[(y - \hat f(x))^2]\), the average squared error over many such unseen points (think of every image uploaded after an image classifier was trained). It can be shown to split into three parts.

The slide links to a proof elsewhere. Not in the lecture, but short: write \(y = f + \varepsilon\) with \(E[\varepsilon] = 0\), \(E[\varepsilon^2] = \sigma^2\), and let \(\bar f = E[\hat f(x)]\). For an unseen point the noise \(\varepsilon\) is independent of \(\hat f\), and \(f - \bar f\) is a constant, so in each expansion the cross term has expectation zero:

So the error on unseen data depends on both bias and variance. A high-bias model like the line has a high test error; so does a high-variance model like the degree-25 polynomial. For the test error to be small both must be small, and the experiment showed that they pull against each other. The minimum sits somewhere in the middle, at medium bias and medium variance.

Training error and test error M2 · 7:42

The parameters of \(\hat f\) (all the \(w_i\)) are estimated from a training set of \(n\) points. What we care about is performance on a validation (unseen) set of \(m\) points that was not used for training. That gives two quantities:

Typically, as model complexity grows the training error keeps falling toward zero, while the test error falls at first and then rises. The left end of the curve is the high-bias region: the simple line is bad even on its own training data. The right end is the high-variance region: fits that are excellent on their own sample and poor on new points. Between them lies the sweet spot, the ideal model complexity. The lower panel of the interactive figure above shows this shape on synthetic data.

As model complexity increases, the training error becomes overly optimistic. The validation error gives the real picture of how close \(\hat f\) is to \(f\).

The next two modules make this intuition precise and show how to account for the optimism in the training error.

Let \(D = \{x_i, y_i\}_{i=1}^{m+n}\). Some true function relates \(y\) to \(x\), with noise:

We do not know \(f\). We approximate it by \(\hat f\), estimate its parameters from a training set \(T \subset D\), and predict \(\hat y = \hat f(x)\). The quantity we want is the true error \(E[(\hat f(x) - f(x))^2]\), but it contains the unknown \(f\), so it cannot be computed directly. What we do have are observations \(y_i\) and predictions \(\hat y_i\). Rewrite their squared difference using \(y = f(x) + \varepsilon\), and expand it as \((a - b)^2\) with \(a = \hat f(x) - f(x)\) and \(b = \varepsilon\):

The quantity we want is now written with three terms. The first can be estimated from data, the second is \(\sigma^2\), and the third still contains \(f\) and needs an argument.

Detour: estimating an expectation M3 · 6:13

Suppose a team scored \(z_1 = 2\), \(z_2 = 1\), \(z_3 = 0\), …, \(z_k = 2\) goals in \(k\) matches. The empirical estimate of the expected number of goals is the average, \(E[z] \approx \frac1k\sum_{i=1}^k z_i\). In the same way, from \(m\) observations and predictions,

The right-hand side of the true-error equation can be evaluated with test observations or with training observations, and the two behave very differently.

Case 1: test observations

The last expectation is a covariance because \(\varepsilon\) has mean zero 11:17. For any \(X\) with \(\mu_X = E[X] = 0\):

None of the test observations took part in estimating \(\hat f\); its parameters came from the training data only 14:23. So at a test point, \(\varepsilon = y - f(x)\) is independent of \(\hat f(x) - f(x)\), and the expectation of the product splits into a product of expectations:

Case 2: training observations

Now the argument fails. The training observations were used to estimate the parameters of \(\hat f\), so \(y - f(x)\) is not independent of \(\hat f(x) - f(x)\). The expectation no longer splits into \(E[\varepsilon]\cdot E[\hat f(x) - f(x)]\), and the covariance term is not zero. Reporting the training error alone leaves this term out, so the empirical training error is smaller than the true error and gives a false picture of it. How does this term depend on model complexity?

Estimate the troublesome expectation empirically over the \(n\) training points. Using Stein's lemma (and, as the slide says, some trickery) one can show

The lecture takes the lemma on trust, and so can we. When is \(\partial\hat f(x_i)/\partial y_i\) high? When a small change in one observation causes a large change in the estimate. Say \(x_i = 10\) and \(y_i\) changes from 20 to 20.1: ideally the fitted parameters should barely move. A complex model is more sensitive to such changes than a simple one, which is what the 30 repeated fits already showed. Linking the two:

The slides then check the claim directly: fit a simple and a complex model to some data, change one data point, and refit. The simple model hardly changes compared with the complex one.

Not in the lecture: for a least-squares fit the predictions are linear in the observations, \(\hat{\mathbf y} = S\mathbf y\) with \(S = X(X^\top X)^{-1}X^\top\). Then \(\partial\hat f(x_i)/\partial y_i = S_{ii}\) (the 0.09 and 0.74 above), and \(\sum_i\partial\hat f(x_i)/\partial y_i = \operatorname{tr}(S)\) equals the number of parameters: 2 for the line, 10 for the degree-9 polynomial. In this simple case the complexity term literally counts parameters.

Driving \(\mathscr L_{train}(\theta)\) to zero can raise the complexity term so much that the true error goes up. So while training, instead of the training error alone, minimize

where \(\Omega(\theta)\) is high for complex models and small for simple ones. \(\Omega\) acts as an approximation of \(\frac{\sigma^2}{n}\sum_i\partial\hat f(x_i)/\partial y_i\), which cannot be handled directly. If minimizing \(\Omega\) reduces model complexity, it also reduces the term that made the training error optimistic.

This is the basis for all regularization methods. L1 regularization, L2 regularization, early stopping and injecting noise at the input can all be shown to be instances of it.

Why deep learning needs this

Deep neural networks are highly complex models, with many parameters and many non-linearities. By the representation-power results of Lecture 3, adding layers or neurons lets a network get arbitrarily close to its training data. So it is easy for a deep network to overfit and drive the training error to 0, and some form of regularization is needed. The lecture covers these forms:

1. \(l_2\) regularization
2. Dataset augmentation
3. Parameter sharing and tying
4. Adding noise to the inputs
5. Adding noise to the outputs
6. Early stopping
7. Ensemble methods
8. Dropout

For \(l_2\) regularization the regularizer is the squared norm of the weights:

\(\alpha\) decides how much weight the penalty gets in the total loss. With \(\alpha = 0\) there is no regularization and we are back to the unregularized loss.

Why does a penalty on the size of the weights control model complexity? Without it, any point of weight space is open to the optimizer. With it, moving far from the origin makes \(\lVert\mathbf w\rVert^2\), and so the total loss, large. You may add as many parameters as you like, a million or a billion, but none of them is given the freedom to take very large values. The model keeps its capacity but cannot use all of it.

For SGD and its variants we need the gradient. The derivative of \(\frac\alpha2\mathbf w^\top\mathbf w\) is \(\alpha\mathbf w\), so

This requires a very small modification to gradient descent code: subtract one extra term. (The lecture notes that the change is more involved for optimizers like Adam, and does not go into it.) Grouping the \(\mathbf w_t\) terms gives \(\mathbf w_{t+1} = (1 - \eta\alpha)\mathbf w_t - \eta\nabla\mathscr L(\mathbf w_t)\): every step first shrinks the weights by a constant factor, which is why the L2 penalty is also called weight decay.

Let \(\mathbf w^*\) be the optimal solution of \(\mathscr L(\mathbf w)\), the loss without regularization (not of \(\tilde{\mathscr L}\)). Because it is optimal, \(\nabla\mathscr L(\mathbf w^*) = \mathbf 0\). For a point near it write \(\mathbf u = \mathbf w - \mathbf w^*\) and use the Taylor series up to second order 10:01 (gradient descent in Lecture 3 used only the first-order term). \(H\) is the Hessian, the matrix of second-order partial derivatives:

Adding the gradient of the penalty, \(\nabla\tilde{\mathscr L}(\mathbf w) = \nabla\mathscr L(\mathbf w) + \alpha\mathbf w\), which here is \(H(\mathbf w - \mathbf w^*) + \alpha\mathbf w\). Let \(\tilde{\mathbf w}\) be the optimal solution of the regularized loss. Its gradient is zero:

\(H\) is symmetric positive semi-definite, so for \(\alpha > 0\) the matrix \(H + \alpha I\) is positive definite and the inverse exists. If \(\alpha \to 0\) then \(\tilde{\mathbf w} \to \mathbf w^*\), as it should with no regularization. The case of interest is \(\alpha \ne 0\).

Since \(H\) is symmetric PSD, it has an eigendecomposition \(H = Q\Lambda Q^\top\) with \(Q\) orthogonal (\(QQ^\top = Q^\top Q = I\)) and the eigenvalues \(\lambda_i \ge 0\) on the diagonal of \(\Lambda\). Substitute it, write \(\alpha I = Q(\alpha I)Q^\top\), and use the rule \((ABC)^{-1} = C^{-1}B^{-1}A^{-1}\):

Read the product from right to left 21:57:

1. \(Q^\top\) rotates \(\mathbf w^*\) into the eigenbasis of \(H\), giving \(Q^\top\mathbf w^*\).
2. \(D\) scales element \(i\) of \(Q^\top\mathbf w^*\) by \(\lambda_i/(\lambda_i + \alpha)\).
3. \(Q\) rotates the result back.

If \(\alpha = 0\), \(D = I\) and \(Q\) simply rotates \(Q^\top\mathbf w^*\) back to \(\mathbf w^*\). If \(\alpha \ne 0\) 27:29:

• if \(\lambda_i \gg \alpha\), then \(\frac{\lambda_i}{\lambda_i + \alpha} \approx 1\) and that component is kept;
• if \(\lambda_i \ll \alpha\), then \(\frac{\lambda_i}{\lambda_i + \alpha} \approx 0\) and that component is driven toward zero.

Only the significant directions, those with larger eigenvalues, are retained. Counting each direction by the fraction of it that survives gives the effective number of parameters 30:16:

The network still has \(n\) parameters, but because many are scaled down, fewer of them are fully effective. That is how L2 regularization reduces model complexity.

Draw the contours of the unregularized loss \(\mathscr L(\mathbf w)\), ellipses around \(\mathbf w^*\), and the contours of the penalty \(\lVert\mathbf w\rVert^2\), circles around the origin. The regularized optimum \(\tilde{\mathbf w}\) lies where the two families touch: a compromise between a small loss and small weights.

Two things happen as \(\mathbf w^*\) moves to \(\tilde{\mathbf w}\). First, the weights shrink. Second, they do not shrink uniformly: each shrinks by its own factor, set by the eigenvalues. Along \(w_1\) the loss contours are far apart, so the loss changes slowly in that direction; shrinking \(w_1\) costs little loss, so it is shrunk a lot. Along \(w_2\) the contours are close together, so even a small shrink raises the loss quickly, and \(w_2\) is shrunk only a little. The spacing of the contours is the curvature, which the Hessian's eigenvalues measure. Directions in which the loss is sensitive count as important and keep most of their weight; the others are suppressed. This ensures that only important features are given high weights.

This method changes the data instead of the loss. Take an MNIST training image of the digit 2. Some transformations of the image do not change its label, so each of them yields a new labelled training example at no labelling cost. The slide's examples: the image rotated by 20° or by 65°, shifted vertically, shifted horizontally, blurred, or with some pixels changed. Every one of them still has label 2.

The augmented data is created using knowledge of the task: we know these transformations leave the label alone. Not every transformation is safe for every class; rotating a 6 by 180° gives a 9.

Why it regularizes: the network must now map many variants of the same digit to the same label, so memorizing one particular combination of pixels no longer works. There is more to learn, and overfitting becomes harder, although modern networks can still overfit. Typically, more data means better learning.

• Works well for image classification and object recognition, and has also been shown to work well for speech.
• For some tasks it is not clear how to generate such data. The lecturer's example is text: words are discrete, unlike the continuous signals of images and speech, so meaningful augmentations are harder to design, although libraries for it now exist.

Parameter sharing is used in convolutional neural networks (Lecture 8). A convolutional filter is a small matrix of weights, and the same filter is applied at different positions of the image instead of a separate matrix for each part. Put another way, the same weight matrix acts on different input neurons.

Parameter tying is typically used in autoencoders, which map an input \(\mathbf x \in \mathbb R^n\) to a code \(\mathbf h(\mathbf x) \in \mathbb R^d\) and then to a reconstruction \(\hat{\mathbf x} \in \mathbb R^n\) (autoencoders come up again in Lecture 7). The encoder needs a \(d \times n\) weight matrix and the decoder an \(n \times d\) one. Tying the encoder and decoder weights means the decoder uses the transpose of the encoder's matrix, so one matrix does both jobs. The lecturer adds that some transformer-based models (Lecture 14) similarly reuse one set of weights across layers.

In both cases the regularization effect is plain: fewer separate weights means a smaller model, and a smaller model has lower complexity.

Pass each input through a noise process \(P(\tilde{\mathbf x} \mid \mathbf x)\) and train the network to map the corrupted \(\tilde{\mathbf x}\) to the original target. (The slide draws this on an autoencoder, where the same idea appears.) For binary inputs, a simple noise process keeps each bit with probability 0.8 and flips it with probability 0.2, so about 20% of every input is corrupted. Each epoch sees a different corruption of the same example, and all of them must be mapped to the same \(y\). Memorizing the inputs becomes harder, much as with augmentation, and the slide notes it can be viewed as data augmentation.

For a simple input–output network, with no hidden layer and no non-linearity, adding Gaussian noise to the input can be shown to be equivalent to weight decay. Let the model be \(\hat y = \sum_{i=1}^{n}w_ix_i\) and corrupt each input independently 4:38:

We are interested in the expected squared error of the noisy prediction \(\tilde y\) against the true target \(y\). Group it as \((\hat y - y) + \sum_i w_i\varepsilon_i\) and expand the square:

Two independence facts remove terms. The noise is independent of the clean error \(\hat y - y\) and has mean zero, so the middle term vanishes. Expanding \((\sum_i w_i\varepsilon_i)^2\) gives squares \(w_i^2\varepsilon_i^2\) and cross terms \(w_iw_j\varepsilon_i\varepsilon_j\); since \(\varepsilon_i\) is independent of \(\varepsilon_j\), each cross term has expectation \(w_iw_jE[\varepsilon_i]E[\varepsilon_j] = 0\). Finally \(E[\varepsilon_i^2] = \sigma^2\).

Noise can also go into the targets. In 10-class digit classification, the true distribution for an image of a 2 puts all its mass on class 2: \(\mathbf p = (0, 0, 1, 0, 0, 0, 0, 0, 0, 0)\). These are hard targets. Training minimizes the cross-entropy between \(\mathbf p\) and the estimated distribution \(\mathbf q\) (Lecture 4), \(-\sum_{i=0}^{9}p_i\log q_i\); the slide writes the sum without its minus sign. With many parameters this loss can be driven to zero, mapping each input exactly to its label, which is what overfitting is.

The intuition: do not trust the true labels, as they may be noisy. Take a small probability mass \(\varepsilon\) (a small positive constant) away from the correct class and share it equally among the other nine. These are soft targets:

With hard targets only one term of the sum survives, the one with \(p_i = 1\). With soft targets every class contributes, and the loss becomes a weighted version of the original loss plus an extra term (\(\varepsilon = 0.1\) above is only an example value):

The outputs have been corrupted a little, the majority of the mass is still on the correct label, and the objective again has the form loss plus regularizer, which helps avoid overfitting. Standard name: label smoothing.

Put training steps on the \(x\)-axis instead of model complexity and the familiar picture returns. The longer training runs, the more the parameters adapt to the training data: the training error heads toward zero, while the validation error falls, bottoms out and starts to rise. The goal is the sweet spot where both are low.

1. Early stopping with patience p
2. Train as usual and track the validation error.
3. Fix a patience parameter \(p\), counted in steps or epochs.
4. If at step \(k\) there was no improvement in validation error in the previous \(p\) steps, stop training.
5. Return the model stored at step \(k - p\).

In short, stop training before it drives the training error to 0 and blows up the validation error.

Early stopping is very effective and the most widely used form of regularization. It needs no change to the loss function, it is easy to automate, and it can be used together with other regularizers such as \(l_2\). How does it act as a regularizer? First an intuition, then a mathematical analysis.

Intuition: a limited number of steps limits how far the weights travel M9 · 5:09

Unroll the SGD update \(\mathbf w_{t+1} = \mathbf w_t - \eta\nabla\mathbf w_t\) (writing \(\nabla\mathbf w_t\) for the gradient at step \(t\)) back to the initial weights:

Let \(\tau\) be the largest of these gradients. The sum has \(t\) terms, each no larger than \(\tau\), so

The number of steps \(t\) controls how far \(\mathbf w_t\) can go from the initial \(\mathbf w_0\); in other words, it controls the space of exploration 8:44. Stopping early caps that distance, much as the circles of the L2 penalty cap how far the weights may grow.

Use the same second-order Taylor approximation around the unregularized optimum \(\mathbf w^*\) as in the L2 analysis, which gave \(\nabla\mathscr L(\mathbf w) = H(\mathbf w - \mathbf w^*)\). The SGD update rule then becomes

Substitute the eigendecomposition \(H = Q\Lambda Q^\top\):

If we start with \(\mathbf w_0 = \mathbf 0\), this recursion has a closed form (proof below; the slides write \(\varepsilon\) for the learning rate here):

Compare it with the L2 optimum. Since

the L2 solution can be written in the same shape:

Not in the lecture: matching every eigen-direction exactly is a strong condition. The standard reading (Deep Learning book, §7.8) is an approximate equivalence. Taking logs, \(t\log(1 - \eta\lambda_i) \approx -t\eta\lambda_i\) and \(\log\frac{\alpha}{\lambda_i + \alpha} \approx -\lambda_i/\alpha\) when \(\eta\lambda_i\) and \(\lambda_i/\alpha\) are small, so \(\alpha \approx 1/(\eta t)\). Training for longer is like using weaker L2.

Proof of the closed form (the slides' appendix)

By induction on \(t\), starting from \(\mathbf w_0 = \mathbf 0\). Base case, \(t = 1\): the recursion (first line) and the closed form (second line) agree.

Induction step. Assume the closed form holds at step \(t\), substitute it into the recursion for step \(t + 1\), open the bracket, and use \(Q^\top Q = I\):

Things to remember

Early stopping only allows \(t\) updates to the parameters. If a parameter \(w\) belongs to a direction that is important for the loss \(\mathscr L(\theta)\), then \(\partial\mathscr L(\theta)/\partial w\) is large and the parameter can grow a lot within \(t\) steps. If it is not important, \(\partial\mathscr L(\theta)/\partial w\) is small, its updates are small, and it cannot grow large in \(t\) steps. Early stopping therefore effectively shrinks the parameters of the less important directions, the same thing weight decay does.

An ensemble combines the outputs of different models to reduce generalization error, for example by voting, or by averaging the class distributions the models predict. The models can correspond to:

• different classifiers trained on the same data (the slide's example: logistic regression, an SVM and naive Bayes);
• different instances of the same classifier trained with different hyperparameters (say, networks with different numbers of layers or neurons per layer);
• instances trained with different features (each on a different subset of \(m \lt n\) of the \(n\) input features);
• instances trained on different samples of the training data.

Bagging is an ensemble of different instances of the same classifier. From the given dataset, construct training sets \(T_1, T_2, \dots, T_k\) by sampling with replacement: each sample is drawn from the full dataset, and the points taken for one set are put back before drawing the next. Train the \(i\)-th instance on \(T_i\). Each model overfits to a different sample, so their combination can generalize better.

When would bagging work? M10 · 6:36

Consider \(k\) logistic regression models. On a test example, model \(i\) makes an error \(\varepsilon_i\), so the average prediction of all the models makes the error \(\frac1k\sum_i\varepsilon_i\). Let the \(\varepsilon_i\) be drawn from a zero-mean multivariate normal distribution with

Expand the square of the sum into the \(k\) terms with \(i = j\) and the \(k(k-1)\) terms with \(i \ne j\), then use linearity of expectation:

Real models sit between the extremes 10:42. The smaller the correlation between the models, the larger the benefit; a hundred models that make the same mistakes are no better than one.

Model averaging (a bagging ensemble) typically helps, but for large neural networks it is prohibitively expensive. Option 1, training several networks with different architectures, is obviously expensive. Option 2, training several instances of the same network on different samples of the data, is expensive too. Even if either could be done, combining several models at test time is infeasible in real-time applications, because every test input would have to pass through all of them.

Dropout addresses both issues. It effectively trains several neural networks without significant computational overhead, and it gives an efficient approximate way of combining exponentially many different networks.

• Dropping units. Temporarily remove a node and all its incoming and outgoing connections, which gives a thinned network. Each node is retained with a fixed probability, typically \(p = 0.5\) for hidden nodes and \(p = 0.8\) for visible (input) nodes.
• How many networks? With \(n\) nodes, each either retained or dropped, there are \(2^n\) thinned networks. That is far too many to train separately.

Two tricks make it workable 4:36:

1. Share the weights across all the networks. There is one set of weight matrices, and each thinned network uses the part of it that survives.
2. Sample a different network for each training instance (or mini-batch).

1. Training with dropout
2. Initialize all the parameters (weights) of the network.
3. For the next training instance or mini-batch, apply dropout: keep each node with its probability \(p\). This gives a thinned network.
4. Compute the loss with a forward pass through the thinned network.
5. Backpropagate. Only the parameters that were active get updated; a weight attached to a dropped node took no part in the forward pass and receives no update.
6. Repeat from step 2 with a fresh sample, so each mini-batch trains a different thinned network.

Because the weights are shared, a weight that was active for two mini-batches has received two updates by then, the second starting from the value the first left behind, while a weight active for only one of them has received one 10:15. Each thinned network gets trained rarely, or even never; with a million nodes there are \(2^{1{,}000{,}000}\) of them, far more than the number of training steps. But parameter sharing ensures that no model has untrained or poorly trained parameters. In the lecturer's example, with nodes kept with probability 0.8, a weight between two nodes is active only when both are kept, with probability \(0.8 \times 0.8 = 0.64\), so over 1,000 steps it receives about 640 updates.

In practice nodes are not visited one by one. A layer's output vector is multiplied element-wise by a random 0/1 mask vector (for example with 20% zeros), so the dropped units output zero and take no part in the computation.

At test time it is impossible to aggregate the outputs of \(2^n\) thinned networks. Instead, use the full network with every node present, and scale the output of each node by the fraction of times it was on during training. A node that was present with probability \(p\) during training, with outgoing weights \(w_1, w_2, w_3, w_4\), is always present at test time with weights \(pw_1, pw_2, pw_3, pw_4\). The lecturer's way of putting it: a node that took part in only 80% of the discussions has its say trusted only 80%.

Not in the lecture: most libraries implement inverted dropout, which divides the kept outputs by \(p\) during training so that the test-time network needs no change. It is the same correction applied at the other end.

Why dropout regularizes

Dropout essentially applies a masking noise to the hidden units. This prevents hidden units from co-adapting. A unit cannot rely too much on other units, because they may be dropped at any time, so each hidden unit has to learn to be robust to these random dropouts.

An example of how this builds redundancy 15:53. Suppose \(h_i\) learns to detect a face by firing when it detects a nose. Dropping \(h_i\) then erases the information that a nose exists. The model must either learn another unit that redundantly encodes the presence of a nose, or learn to detect the face from other features, such as the eyes or eyebrows. Either way the network becomes more robust.

There are computationally intensive techniques for visualizing the loss landscape of a deep network in 1-D or 2-D, and the slide points to an interactive explorer at losslandscape.com. Real landscapes are bumpy, with many plateaus, valleys and sharp slopes, unlike the smooth bowls used so far in the course. Researchers have correlated the local flatness (curvature) of the loss surface with the generalization of a model, using different measures of curvature:

• flat surface → better generalization;
• sharp surface → poor generalization.

The lecturer's reasoning: in a flat region a whole neighbourhood of weights gives about the same loss, so small changes do not matter. In a sharp valley surrounded by hills, a small change of the weights changes the loss a lot, and the model is very sensitive. It has been shown that explicit regularization techniques smooth the loss surface; the lecture's example is a paper that studies skip connections (a CNN technique) as a regularizer.

Grouping the techniques

Each technique acts on some part of the training objective \(\mathscr L(\theta) = \frac1N\sum_{i=1}^{N}l\big(f(x_i, \theta), y_i\big)\):

A second grouping splits them into explicit and implicit regularization:

Skip connections, weight sharing and pooling come with CNNs in Lecture 8; at least weight sharing and pooling regularize by reducing the number of parameters. On the implicit side, some papers show that gradient descent itself prefers less complex solutions 3:34, and the initial learning rate also shapes how training proceeds. The lecture mentions these without going into detail.

The slides close with a question the videos do not cover: is regularization, a technique adopted from statistical learning theory, really the fundamental reason deep learning models generalize well? The test is random labelling, from a paper the slides cite (standard reference: Zhang et al., 2017, Understanding deep learning requires rethinking generalization).

1. Take a deep network that performed well on some task, say classification, and fix its complexity (its size, and so its number of parameters), the optimizer used to train it, its initialization, and so on.
2. Replace the true labels of the training data with random labels. This removes the relation (regularity) between the inputs and the corresponding outputs. The random labels are fixed and consistent across epochs.

A toy version first. Points are sampled from a noisy sinusoid, the example from the start of the lecture, and fitted with a model of complexity 19 (19 parameters). After training, the model fits the data points and closely represents the true function. Now randomly shuffle \(y\). With its fixed capacity the model cannot fit the shuffled data, and the training error remains high. This is expected: shuffling removed the regularity the model was exploiting.

Deep networks behave differently. Surprisingly, they drive the training error to zero even with random labels. What the slides conclude:

• Observations on both explicit and implicit regularizers consistently suggest that regularizers, when properly tuned, can improve generalization. It is unlikely, though, that they are the fundamental reason for it, since the networks continue to perform well after all the regularizers are removed.
• Experimental evidence suggests that the value of the objective may imply nothing about generalization (with respect to the Rademacher complexity measure).
• This implies that vastly over-parameterized deep models regularize implicitly while their parameters are optimized with SGD or GD: the discrete steps of gradient descent themselves act as a regularizer.

The lecture's recap lists the explicit methods (\(l_2\) regularization, dataset augmentation, parameter sharing and tying, noise at the inputs, noise at the outputs, early stopping, ensembles, dropout) together with this implicit regularization by the optimizer. Lecture 7 moves on to pre-training, better activation functions, weight initialization and batch normalization.

Over-parameterized model

More parameters than training samples; can drive the training error to zero and overfit.

Bias

\(E[\hat f(x)] - f(x)\): how far the average model over training samples is from the truth. High for simple models.

Variance

\(E[(\hat f(x) - E[\hat f(x)])^2]\): how much models trained on different samples disagree. High for complex models.

Irreducible error \(\sigma^2\)

The noise variance in \(y\); part of every model's expected test error, bias² + variance + σ².

True error

\(E[(\hat f(x) - f(x))^2]\). Equals the test error minus σ², but exceeds what the training error suggests by a complexity term.

Regularizer \(\Omega(\theta)\)

Term added to the training loss that is large for complex models; a stand-in for \(\frac{\sigma^2}{n}\sum_i\partial\hat f(x_i)/\partial y_i\).

Weight decay (L2)

Penalty \(\frac\alpha2\lVert\mathbf w\rVert^2\). Adds \(\alpha\mathbf w\) to the gradient and scales eigen-direction \(i\) of \(\mathbf w^*\) by \(\lambda_i/(\lambda_i + \alpha)\).

Effective number of parameters

\(\sum_i\lambda_i/(\lambda_i + \alpha) \lt n\): how many directions survive L2 regularization.

Dataset augmentation

New training examples from label-preserving transformations such as rotations, shifts and blurring.

Parameter sharing and tying

Reusing one set of weights: a CNN filter at every position; the encoder's \(W\) and decoder's \(W^\top\) in an autoencoder.

Soft targets (label smoothing)

Target \(1 - \varepsilon\) for the true class and \(\varepsilon/9\) for each of the other nine classes.

Early stopping, patience \(p\)

Stop when the validation error has not improved for \(p\) steps and return the model from step \(k - p\).

Bagging

Ensemble of one classifier trained on samples drawn with replacement; \(\text{mse} = \frac1kV + \frac{k-1}{k}C\).

Dropout, thinned network

Keep each unit with probability \(p\) per mini-batch, sharing weights across the \(2^n\) thinned networks; at test time keep all units and scale weights by \(p\).