Pre-training, activation functions, initialization and batch norm

The lecture starts by recalling how networks are trained. For a single sigmoid neuron \(f(x) = \sigma(wx)\) with the squared-error loss of Lecture 3, gradient descent repeats

A wider network, with inputs \(x_1, x_2, x_3\) feeding one sigmoid neuron, changes nothing. Each weight gets its own copy of the rule, \(w_i \leftarrow w_i - \eta\,\nabla w_i\), with \(\nabla w_i = (f(x) - y)\,f(x)(1 - f(x))\,x_i\).

A deeper, thin network chains sigmoid neurons: \(a_i = w_i h_{i-1}\) and \(h_i = \sigma(a_i)\), starting from \(h_0 = x\). The chain rule gives

In general \(\nabla w_i = \frac{\partial \mathcal{L}(w)}{\partial y} \cdots h_{i-1}\). The gradient of a weight is proportional to the input \(h_{i-1}\) it multiplies, and the lecture uses this fact again and again. A network that is both deep and wide applies the same chain rule along every path from the weight to the output and adds the paths up, as in backpropagation (Lecture 4).

Things to remember: training is a game of gradients, played with any gradient-based method from Lecture 5. A gradient measures a parameter's responsibility for the loss, and it is proportional to the input to that parameter.

Backpropagation is old. It existed before 1986, when Rumelhart and colleagues made it popular for neural networks, and it is still how every network in this course is trained. In principle it trains a network of any depth by chaining gradients. In practice, networks as deep as four or five layers were not trained successfully with it: even after a large number of epochs, training did not converge reliably. Until 2006 it was very hard to train deep networks.

Three things have changed since the 1990s. There is far more data, which a network with many parameters needs. Compute is much faster. And several ideas fell into place that make deep networks trainable at all, which is what this lecture is about. The turning point was a 2006 paper by Hinton and Salakhutdinov, "Reducing the dimensionality of data with neural networks" (Science), which revived the field through unsupervised pre-training.

The 2006 paper introduced the idea with restricted Boltzmann machines (RBMs). The lecture explains it with autoencoders instead. Autoencoders are not otherwise part of this offering of the course, so only the part that is needed is introduced here.

Take a deep network from \(\mathbf{x} \in \mathbb{R}^n\) to an output \(y\), and look only at its first two layers, the input \(\mathbf{x}\) and the first hidden layer \(\mathbf{h}_1\). Cut the network after \(\mathbf{h}_1\) and attach a new output layer whose job is to reproduce the input, \(\hat{\mathbf{x}}\), from \(\mathbf{h}_1\). Train these weights to make the reconstruction good on all \(m\) training inputs:

This objective is unsupervised. It uses only the inputs and never the label \(y\).

Why is a good reconstruction useful? Suppose \(\mathbf{x}\) has 1024 dimensions and \(\mathbf{h}_1\) only 256, and the network rebuilds \(\mathbf{x}\) from \(\mathbf{h}_1\) with zero error. Then \(\mathbf{h}_1\) holds everything that matters in \(\mathbf{x}\). That is plausible because real inputs are redundant: weight, height and BMI (BMI follows from the other two), weight in kilograms and in pounds, salary and income tax. Squeezing the input through a smaller layer and rebuilding it is exactly what an autoencoder does, and under certain conditions it is equivalent to principal component analysis. The weights of layer 1 now compute an abstract representation \(\mathbf{h}_1\) of the input, and learning them only required training a network with one hidden layer. Shallow networks were never the problem.

Now repeat one level up. Fix the weights of layer 1 and compute \(\mathbf{h}_1\) for every training input. Treat these as the new data, add \(\mathbf{h}_2\), and train it to reconstruct \(\mathbf{h}_1\) with the same objective:

\(\mathbf{h}_2\) is then an abstract representation of \(\mathbf{h}_1\), and so a deeper abstraction of \(\mathbf{x}\). Continue in the same way up to the last hidden layer, the one just before the output layer.

Finally, return to the original problem. Keep the pre-trained weights, add the output layer with random weights, and train the whole network on the task's supervised objective, \(\min \frac{1}{m}\sum_i \big(y_i - f(\mathbf{x}_i)\big)^2\), with full backpropagation. Every weight is updated, but the hidden layers start from their pre-trained values instead of random ones.

1. Greedy layerwise pre-training, then fine-tuning
2. Set \(\mathbf{h}_0 = \mathbf{x}\) for every training input.
3. For each hidden layer \(k = 1, \dots, L\): keep layers \(1, \dots, k-1\) fixed, and train the weights into \(\mathbf{h}_k\), together with a temporary output layer, to reconstruct \(\mathbf{h}_{k-1}\) from \(\mathbf{h}_k\) with the squared error. Then compute \(\mathbf{h}_k\) for every input.
4. Remove the temporary reconstruction layers and add the real output layer with random weights.
5. Fine-tune all the weights on the supervised loss with backpropagation, starting from the pre-trained values.

In effect, the weights of the network have been initialized with a greedy unsupervised objective and then fine-tuned with the supervised one. It is greedy because each layer is trained to do well on its own, without regard to the layers above it. With this procedure, networks that had been hard to train could be trained effectively (deep for the time, though shallow by today's standards). Unsupervised pre-training still exists, today mostly in the context of Transformers, with different objectives but a similar idea.

Why does pre-training work better? Two explanations were put forward, and studies in 2009 (Erhan et al.; Larochelle et al.) looked at both. Does pre-training lead to better optimization, or to better regularization?

The questions are different. Optimization concerns the training data only: can the training loss be driven down? Regularization concerns generalization, doing better on test data the model was not trained on. So the first question asks whether, without pre-training, the error could not even be brought to zero on the training set.

The error surface of a deep network's supervised objective is highly non-convex, with many hills, plateaus and valleys. Even so, deep networks have large capacity. By the universal approximation theorem (Lecture 3), a large enough network can make the error as small as we like. So it should still be easy to land in one of the many zero-error regions.

Larochelle et al. tested this by increasing the capacity of the last layer: many neurons in the layer before the output means many weights in the last layer. With a large-capacity last layer, \(\mathcal{L}(\theta)\) went to 0 even without pre-training. When capacity was small, pre-training helped, so in that case it was helping the optimization itself.

These were empirical experiments, small in scale compared with deep learning today. Their value was the direction they suggested: if optimization is a bottleneck, it is worth looking for better optimization methods as well.

Regularization constrains the weights to certain regions of the parameter space. L1 regularization pushes most weights to 0 (the diamond-shaped region), L2 regularization keeps most weights from taking large values (the circle), and early stopping has a similar effect (Lecture 6).

Pre-training does the same. It constrains the weights to regions where the characteristics of the data are captured well, as measured by the unsupervised objective

Supervised training of \(\mathcal{L}(\theta)\) then starts inside that region, so where the weights end up depends on where pre-training put them. The unsupervised objective works like an additional constraint on the optimization problem: learning is not greedy with respect to the supervised objective alone, and the solution also satisfies the unsupervised one.

Compare the regularized loss of Lecture 6, \(\mathcal{L}(\theta) + \Omega(\theta)\), which minimizes both terms together. Pre-training minimizes the two terms one after the other: first \(\Omega(\theta)\), then \(\mathcal{L}(\theta)\) starting from where \(\Omega\) left the weights. Seen this way, it acts as a regularizer.

Other experiments (Erhan et al., 2009) showed that pre-training makes training more robust to random initialization. Initialize a network at random, train it, and record its final error. Repeat this many times (around 100) with different random initializations and summarize the errors in a box plot. For shallow networks the spread is small. For deep networks without pre-training it is large: the result depends heavily on the particular initialization. With pre-training inserted between the random initialization and the supervised training, the spread stays small even for deep networks.

One accepted hypothesis is therefore that pre-training leads to better weight initializations, from which the layers capture the internal characteristics of the data.

What has happened since 2006–2009

None of these studies gave a conclusive answer, and that was not what mattered. Each lens (optimization, regularization, initialization) suggested a research direction, and deep learning has advanced along all of them:

Before comparing activation functions, the lecture asks what makes deep networks powerful. Take a thin deep network and replace every sigmoid by the identity, so that \(h_i = a_i\). Then

which is a single linear transformation of \(x\), however many layers there are. (With vectors and matrices the product \(W_4W_3W_2W_1\) is still a single matrix.) Such a network can only learn linear decision boundaries. If one class of points surrounds the other, no line or hyperplane separates them without a large error, and a deep linear network cannot learn the boundary. A deep non-linear network can, as the universal approximation theorem says (Lecture 3).

The power of deep networks comes from the non-linearities, and the non-linearities come from the activation functions. That is why it pays to ask which non-linear functions train well. Much of the material that follows is based on Andrej Karpathy's lecture notes.

The sigmoid compresses every input into the range \([0, 1]\). Training is about gradients, so look at its derivative. In a deep sigmoid network, the chain rule for \(\nabla w_2\) passes through the later layers and meets factors such as

Problem 1: saturated neurons make the gradient vanish

A sigmoid neuron is saturated when its output sits at an extreme, \(\sigma(x) = 0\) or \(\sigma(x) = 1\). There the derivative \(\sigma(1 - \sigma)\) is 0, as the formula or the flat ends of the curve show. Any gradient that has to pass through a saturated neuron vanishes, so the weights before it stop learning. A neuron saturated at 1 is one whose weights contributed strongly to the output, yet that signal never gets back to them.

Why would a neuron saturate? Its output is \(\sigma\big(\sum_{i=1}^{n} w_i x_i\big)\). Inputs are usually standardized, so they are small. If the weights are initialized to large values, the sum is large and the neuron saturates almost at once; large negative weights saturate it at 0. Small weights do not settle the matter either. With a thousand inputs, a sum of a thousand small terms easily reaches 2 or 3, and already \(\sigma(2.5) \approx 0.92\) with \(\sigma'(2.5) \approx 0.07\), against a peak of \(0.25\) at \(x = 0\). The weights must be initialized carefully, which is the topic of Module 7.4.

Problem 2: the outputs are not zero-centred

The sigmoid's outputs lie in \([0, 1]\), centred around 0.5 instead of 0. Take a sigmoid neuron with pre-activation \(a_3 = w_1 h_{21} + w_2 h_{22}\), where \(h_{21}, h_{22}\) come from sigmoid neurons in the previous layer, \(h_3 = \sigma(a_3)\), and output \(y\). Its two weight gradients are

\(h_{21}\) and \(h_{22}\) are sigmoid outputs, so both are positive. The sign of each gradient is therefore the sign of the common factor. If it is positive, both gradients are positive; if it is negative, both are negative. The argument is the same for \(n\) inputs, so all the gradients of the weights feeding a neuron share one sign.

That restricts the directions in which the weights can move. In the plane of \((\nabla w_1, \nabla w_2)\), only the quadrant where both are positive and the quadrant where both are negative can occur. If the optimum requires increasing \(w_1\) while decreasing \(w_2\), no single step can head straight for it, and gradient descent has to zigzag, taking more steps to converge. It is like driving from A to B when certain kinds of turns are forbidden.

Problem 3: cost

Finally, sigmoids are computationally expensive, because every evaluation needs \(\exp(x)\).

tanh compresses its inputs into \([-1, 1]\) and is zero-centred, so the next layer's inputs can be positive or negative and the gradients of a neuron's weights no longer have to share one sign. That removes the restriction on update directions. The other two problems remain. At saturation, \(\tanh(x) = \pm 1\), the derivative \(1 - \tanh^2(x)\) is 0, so gradients still vanish there; the flat ends of the curve show the same thing. It is also still expensive to compute. tanh is not new; it has been in use since the 1990s.

Standard identity, not on the slides: \(\tanh(x) = 2\sigma(2x) - 1\). tanh is a sigmoid stretched to \([-1, 1]\) and recentred at 0, which is why it keeps the sigmoid's flat ends.

The rectified linear unit passes a positive pre-activation through unchanged and clamps a negative one to 0. It looks almost linear, so the lecture first asks whether it is non-linear at all. It is, since a linear function would behave the same way on both sides of 0 and ReLU does not. Combining two ReLUs shows what it can do:

This is a piecewise-linear approximation of a scaled sigmoid, and is called ReLU6, where 6 is its maximum value. Adjusting the pieces brings it even closer to a step.

ReLU has three advantages:

• It does not saturate in the positive region, since nothing clamps that side. It still has zero gradient for negative inputs, so roughly half of the saturation problem is solved.
• It is computationally efficient: one comparison and no exponential.
• In practice it converges much faster than sigmoid or tanh, as Krizhevsky, Sutskever and Hinton found for deep convolutional networks in 2012 (Lecture 8). It has since been used in feedforward, recurrent and Transformer networks too.

The caveat: dead neurons

The derivative of ReLU is 1 for \(x > 0\) and 0 for \(x \lt 0\). Take a ReLU unit \(h_1\) with \(a_1 = w_1x_1 + w_2x_2 + b\), feeding the output through \(w_3\). Suppose that at some point a large gradient updates the bias \(b\) to a large negative value. Then \(w_1x_1 + w_2x_2 + b \lt 0\) for every input, and the unit outputs 0: it is a dead neuron. During backpropagation \(\partial h_1/\partial a_1 = 0\) as well, so

and likewise \(\nabla w_2 = \nabla b = 0\). None of \(w_1\), \(w_2\), \(b\) changes, so on the next input \(b\) is still very negative and the unit is dead again. Once dead, it stays dead for the rest of training.

In practice a large fraction of ReLU units can die if the learning rate is set too high; the lecture mentions more than half. A high learning rate turns one large gradient into a large jump in \(b\). Two remedies are advised:

• Initialize the bias to a positive value, such as 0.01.
• Use a variant of ReLU that lets some gradient through for negative inputs.

Leaky ReLU

Leaky ReLU lets a little through on the negative side. It does not saturate, and its units do not die, because the \(0.1x\) term always passes a small gradient. It is as cheap to compute as ReLU, and since its outputs can now be negative, they are close to zero-centred.

Parametric ReLU

Why fix the slope at 0.1? Parametric ReLU makes \(\alpha\) a parameter of the model, learnt along with everything else. Start with, say, \(\alpha = 0.1\), compute \(\partial\mathcal{L}/\partial\alpha\) during backpropagation, and update \(\alpha \leftarrow \alpha - \eta\,\partial\mathcal{L}/\partial\alpha\) (or with any other update rule). For \(\alpha \lt 1\) the local derivative is \(\partial f/\partial\alpha = x\) when \(x \lt 0\) and 0 otherwise. The slope can then adjust as training goes on.

Exponential linear unit (ELU)

ELU keeps the linear positive side and decays exponentially toward \(-\alpha\) on the negative side. It has all the benefits of ReLU, a small gradient \(\alpha e^{x}\) always flows for negative inputs, and its outputs are close to zero-centred. The price is the exponential, which makes it more expensive.

The slide writes the negative branch as \(ae^x - 1\). The usual ELU is \(\alpha(e^{x} - 1)\), which is 0 at \(x = 0\) and so joins the positive branch without a jump.

Each of these variants was proposed in some setting and found better than ReLU there, some with a theoretical argument. In practice none became very popular. The default ReLU usually works, or one of the newer functions below, GELU and Swish.

Maxout is motivated by comparing dropout with model averaging (Lecture 6). In bagging, \(k\) training sets are built from the original one by sampling with replacement, each of the same size as the original, so each contains duplicates. One model is trained on each set. A model sees its duplicated samples again and again and so overfits to its own bucket of data. At inference the \(k\) predictions are combined, for example by an arithmetic or geometric mean, a max, or a vote.

Dropout imitates model averaging inside one network whose sub-networks share weights. There are exponentially many sub-networks and each is sampled rarely, so none of them gets the chance a bagged model has to specialize on particular samples. Simply raising the learning rate would make bigger updates whenever a sample is seen, but the parameters are shared, so it would disturb the other sub-networks too. Dropout's averaging at inference, scaling each unit by the probability \(p\) that it was on, also treats every unit as equally responsive to every input.

Maxout tries to close this gap. Suppose two ReLU units in a layer are both active on an input, with \(0 \lt a_{21} \lt a_{22}\). The second responds more strongly and is more sensitive to this input, so let it be the one that learns from it. Output the larger of the two:

The gradient now flows only to the stronger unit, which specializes on inputs like this one while the weaker unit is left alone. Different pieces come to specialize on different inputs, much as bagged models specialize on their buckets. Under dropout, every unit that survives the mask gets gradient, whether it responded strongly or weakly. Under maxout, only the strongest piece in each maxout unit stays active, and the weaker pieces are effectively dropped as well.

Maxout generalizes ReLU and its variants. ReLU is a maxout unit with two pieces, \(\max(0\cdot x + 0,\; 1\cdot x + 0)\). Leaky ReLU is \(\max(0.1x,\, x)\) and parametric ReLU is \(\max(\alpha x,\, x)\). More pieces give more non-linearity: \(\max(0.5x, -0.5x) = 0.5|x|\) is a V shape, and the four pieces \(\max(0.5x,\, -0.5x,\, x,\, -x - 0.2)\) give a bowl that the lecture uses to approximate \(x^2\).

• No saturation and no death. The gradient always flows through the winning piece.
• Cost. \(k\) affine pieces per neuron mean \(k\) times as many parameters.
• Power. Two maxout neurons with a sufficient number of affine pieces act as a universal approximator.

Standard detail (Goodfellow et al., 2013): the universal-approximation construction takes the difference of the two maxout units. Each unit is a convex piecewise-linear function, and any continuous function can be approximated by a difference of two such functions.

Despite all this, maxout never became very popular; its special cases, ReLU and the variants, are used far more. It is still worth knowing for the link to dropout, the way it generalizes ReLU, and the fact that two such units can approximate any function.

The Gaussian error linear unit starts from a similarity. The hard threshold and ReLU both decide their output by the sign of the input:

Written this way, ReLU multiplies its input by 1 or by 0. Dropout (Lecture 6) also multiplies a unit's value by 1 or 0, with one difference: it chooses at random, \(\mu(x) = 1\cdot x\) with probability \(p\) and \(0\cdot x\) with probability \(1 - p\) (a slight abuse of notation). GELU combines the two properties in one activation: a random 0/1 multiplier whose probability depends on the input.

Here \(x\) is the neuron's pre-activation. \(\Phi(x)\) is a probability, so it must satisfy \(0 \le \Phi(x) \le 1\). The logistic function would do, but the more natural choice is the cumulative distribution function of the standard normal distribution, \(\Phi(x) = P(X \le x)\) for \(X \sim \mathcal{N}(0, 1)\). It is also S-shaped, close to 0 for very negative \(x\) and close to 1 for very positive \(x\). A large positive pre-activation is then almost always kept, and a large negative one almost always zeroed.

The activation used is the expected value of this random unit:

A quick check, with numbers computed for these notes: \(\Phi(1) \approx 0.841\), so \(\operatorname{GELU}(1) \approx 0.841\) and \(\operatorname{GELU}(-1) = -\Phi(-1) \approx -0.159\). The sigmoid approximation gives 0.846 and \(-0.154\). On its own, \(x\sigma(1.702x)\) looks like an arbitrary formula; the derivation shows it weights the input by the probability that a standard normal value falls below it. GELU is the most common activation in Transformer-based architectures such as BERT and GPT (Lecture 14).

SELU: self-normalizing

It helps if each layer's output activations have zero mean and unit variance. That can be done explicitly with normalization techniques (batch normalization, Module 7.5). The scaled exponential linear unit instead builds it into the activation function, which is called self-normalizing:

ReLU is clearly not zero-centred: it can take large values on the positive side and only 0 on the negative side. ELU has some negative values, so it is somewhat more balanced, and so is GELU. Scaling ELU by a suitable \(\lambda\), with a matching \(\alpha\), balances the two sides and gives near zero-centring.

The slide labels both constants \(\lambda\); the second one, 1.673263, is \(\alpha\). On the negative side SELU flattens at \(-\lambda\alpha \approx -1.76\).

Searching for activation functions

Rather than keep inventing activation functions by hand, one can search for them automatically. Assume a search space of scalar functions, each composed of two kinds of operation:

• Unary operations take one input: \(|x|\), \(-x\), \(\exp(-x)\), \(\sqrt{x}\), and so on.
• Binary operations take two: \(\max(x_1, x_2)\), \(x_1\cdot\sigma(x_2)\), and so on.

The sigmoid, tanh, ReLU, leaky ReLU and \(x\sigma(1.702x)\) all live in this space. Restrict the space, try combinations, and keep those that most improve the accuracy of existing models. The search itself uses reinforcement learning, which is beyond this course. Several good functions came out of it, one of them resembling the ELU and GELU family. The best had the form

SiLU

The family now has three names. \(x\sigma(1.702x)\) is (approximately) GELU, \(x\sigma(\beta x)\) is Swish, and the case \(\beta = 1\),

is the sigmoid-weighted linear unit: a linear unit \(x\), weighted by a sigmoid. It was first proposed in reinforcement learning.

Not in the lecture: the two extremes of Swish make the family easy to picture. With \(\beta = 0\) it is the straight line \(x/2\). As \(\beta\) grows, \(\sigma(\beta x)\) approaches a step and \(x\sigma(\beta x)\) approaches ReLU. Both are easy to see in the explorer below.

The explorer plots each function the lecture covers together with its derivative. Move the probe to the flat ends of the sigmoid and tanh, to the negative side of ReLU and its variants, and compare GELU with its sigmoid approximation.

\(\varphi\) is the standard normal density. The GELU and Swish derivatives come from the product rule and are not on the slides.

Which one should you use? A plot from Papers with Code on the slides shows how the use of these functions has changed over time. ReLU has stayed popular for more than a decade. GELU became popular with Transformers, the setting where it is now the usual choice. Sigmoid and tanh, despite their drawbacks, are still used where their ranges are needed: the sigmoid wherever a value must lie between 0 and 1, as in some attention functions, and tanh in LSTMs and RNNs.

Things to remember: sigmoids are bad as hidden activations. ReLU is more or less the standard unit for convolutional networks, and Leaky ReLU, maxout and ELU are worth exploring. tanh and sigmoid are still used in LSTMs and RNNs (Lecture 12). GELU is the most common choice in Transformer models such as BERT and GPT (Lecture 14).

The module videos end with the activation functions. Modules 7.4 and 7.5 are written from the slides alone.

For learning to happen, gradients must flow from the output layer back to the input layer. Saturated neurons block that flow, because a weight's gradient carries the factor \(\sigma(x)(1 - \sigma(x))\):

The way to keep neurons out of saturation is to initialize the parameters carefully. The right initialization depends on the number of inputs to a layer, \(\text{fan}_\text{in}\), the number of outputs, \(\text{fan}_\text{out}\), the type of activation function and the type of architecture.

Start with the simplest choice, all weights 0. In a network with inputs \(x_1, x_2\), three sigmoid units in layer 1 and one unit in layer 2,

so every unit in layer 1 gets the same activation (here \(\sigma(0) = 0.5\)). Backpropagation then gives

Since \(h_{11} = h_{12}\) and \(a_{11} = a_{12}\), and the two units' outgoing weights are equal as well, every factor matches and \(\nabla w_{11} = \nabla w_{21}\). Both weights get the same update and remain equal. The same is true for \(w_{12}\) and \(w_{22}\), and for all the weights in layer 2 (work through the maths to convince yourself). The symmetry never breaks during training: the units of a layer stay copies of one another and the layer behaves like a single unit. This is the symmetry breaking problem. It happens whenever all the weights in a network are initialized to the same value, 0 or any other.

So the weights must be random. The slides test different scales with a forward pass through a synthetic network:

• Input: 1000 points, each in \(\mathbb{R}^{500}\), drawn from a unit Gaussian.
• The network has 5 layers of 500 neurons each.
• Run only forward propagation and look at each layer's activations under different weight initializations.

num_layers = 5
D = np.random.randn(1000, 500)
for i in range(num_layers):
    x = D if i == 0 else h_cache[i-1]
    W = 0.01*np.random.randn(500, 500)
    a = np.dot(x, W)
    h = sigmoid(a)          # the slides also try tanh

Small random weights: W = 0.01*np.random.randn(500,500)

The slides look at the activations layer by layer, with sigmoid and with tanh. With tanh, they shrink toward 0 as the depth grows. With sigmoid, they bunch ever more tightly around \(\sigma(0) = 0.5\). The explorer at the end of this module reproduces both. Now think about backpropagation. A weight's gradient is proportional to the activation passing through it, so if all the activations of a layer are very close to 0, the gradients of the weights connecting that layer to the next are all close to 0 as well. This is the vanishing gradient problem.

The slides then check real data. Data: MNIST, \(\mathbf{x} \in \mathbb{R}^{784}\). Model: a feedforward network with three hidden layers, \(\mathbf{h}_1, \mathbf{h}_2, \mathbf{h}_3 \in \mathbb{R}^{500}\), and tanh activations. Loss: cross-entropy. Optimizer: mini-batch gradient descent with \(\eta = 0.5\) and batch size 256. With W = 0.01*np.random.randn(500,500), the loss stays almost constant for the first 35 iterations.

Large random weights: W = np.random.randn(500,500)

Now most activations saturate, at 0 or 1 for the sigmoid and at \(\pm 1\) for tanh. At saturation the local gradients are close to 0, so the gradients vanish again. The video makes the same point from the other direction M4 · 11:18: with a thousand inputs, even modest weights give sums large enough to saturate a sigmoid. The slides rerun the MNIST experiment with this initialization as well.

Look at the first pre-activation of layer 1, \(s_{11} = \sum_{i=1}^{n} w_{1i}x_i\), and treat the weights and inputs as independent random variables. For independent \(w\) and \(x\),

so

The same holds for every unit of the layer: \(\operatorname{Var}(s_{1i}) = n\operatorname{Var}(w)\operatorname{Var}(x)\). If \(n\operatorname{Var}(w) \gg 1\), the pre-activations have a large variance and the units saturate. If \(n\operatorname{Var}(w) \to 0\), the variance is small and the activations collapse.

Add one more layer. The same procedure, with the \(s_{1i}\) as the inputs, gives

assuming that the weights in all layers have the same variance. (Passing the \(s_{1i}\) straight to the next layer treats the activation as roughly linear, which is how tanh behaves near 0.)

W = np.random.randn(fan_in, fan_out) / np.sqrt(fan_in)

The slides rerun the synthetic network with this initialization, for tanh and for sigmoid, and the MNIST experiment with W = np.random.randn(500,500)/sqrt(fan_in). In the explorer below, the activations now keep a usable spread through all the layers. With \(n = 500\), the three initializations so far compare as follows (numbers worked out for these notes):

Standard name: scaling by \(1/\sqrt{\text{fan}_\text{in}}\) is often called LeCun or Xavier initialization. The Xavier rule of Glorot and Bengio (2010) balances the forward and backward passes with \(\operatorname{Var}(w) = 2/(\text{fan}_\text{in} + \text{fan}_\text{out})\), which equals \(1/n\) when the two are equal.

The \(1/\sqrt{\text{fan}_\text{in}}\) rule does not work for ReLU neurons. He et al. argue that a factor of 2 is needed. Intuitively, the output of a ReLU is restricted to the positive half of the space: it zeroes roughly half of what passes through it, so the variance reaching the next layer is roughly halved at every layer. The slides show this on a deeper ReLU network, with \(\mathbf{h}_1 \in \mathbb{R}^{100}\) and \(\mathbf{h}_2, \dots, \mathbf{h}_7 \in \mathbb{R}^{50}\), at the layers \(\mathbf{h}_1\), \(\mathbf{h}_4\) and \(\mathbf{h}_7\). Accounting for the factor of 2 gives better performance:

W = np.random.randn(fan_in, fan_out) / np.sqrt(fan_in/2)

Not in the lecture, the short reason for the 2: if \(s\) is symmetric about 0, then \(h = \max(0, s)\) has \(\mathbb{E}[h^2] = \tfrac12\operatorname{Var}(s)\). With zero-mean weights, the next pre-activation has variance \(n\operatorname{Var}(w)\,\mathbb{E}[h^2] = \tfrac12\,n\operatorname{Var}(w)\operatorname{Var}(s)\), which stays constant only if \(n\operatorname{Var}(w) = 2\), that is \(\operatorname{Var}(w) = 2/\text{fan}_\text{in}\).

Batch normalization is a method that allows us to be less careful about initialization.

Consider a deep network and focus on learning the weights between two of its layers, say from \(\mathbf{h}_3\) to \(\mathbf{h}_4\). Training uses mini-batches. If the distribution of \(\mathbf{h}_3\) keeps changing from one mini-batch to the next, these weights keep chasing a moving target, and learning them is hard. It would help if the pre-activations at each layer were unit Gaussians. Batch normalization ensures this explicitly by standardizing each pre-activation, for sample \(i\) and unit \(k\):

The mean and variance are computed from the mini-batch. This explicitly keeps the distribution of the inputs to each layer from changing across batches.

Standard name: Ioffe and Szegedy (2015), who introduced batch normalization, called this changing distribution internal covariate shift.

The network now has a BN step before each non-linearity: linear, BN, tanh, linear, BN, tanh, and so on. Is that legal? Yes. Like the tanh layer, the batch normalization layer is differentiable, so backpropagation passes through it.

Is a unit Gaussian necessarily the best input for the tanh layer? Perhaps not, so let the network learn what is best for it. After the standardization, BN adds one more step:

\(\gamma^{(k)}\) and \(\beta^{(k)}\) are additional parameters of the network, learnt with all the others. If the network learns \(\gamma^{(k)} = \sqrt{\operatorname{Var}(x^{(k)})}\) and \(\beta^{(k)} = \mathbb{E}[x^{(k)}]\), it recovers the original \(s_{ik}\). By adjusting these two parameters, the network can undo the normalization whenever that is more favourable.

How are the statistics computed? Three indices are involved: the layer \(l\), the neuron \(i\) within that layer, and the training sample \(j\) within the mini-batch; picture them as the three axes of a cube. Associate with layer \(l\) an accumulator that stores the activations for the batch inputs, and let \(x_i^j\) be the activation of the \(i\)-th neuron for the \(j\)-th of the \(m\) training samples. Each neuron is normalized with its own statistics, taken over the batch:

1. Batch normalization of neuron i in layer l, for one mini-batch of m samples
2. Mean over the batch: \(\mu_i = \frac{1}{m}\sum_{j=1}^{m} x_i^j\).
3. Variance over the batch: \(\sigma_i^2 = \frac{1}{m}\sum_{j=1}^{m}\big(x_i^j - \mu_i\big)^2\).
4. Standardize each sample's activation: \(\hat{x}_i = \dfrac{x_i - \mu_i}{\sqrt{\sigma_i^2 + \epsilon}}\), where \(\epsilon\) is a small constant that avoids division by zero.
5. Scale and shift: \(\hat{y}_i = \gamma\,\hat{x}_i + \beta\), with \(\gamma\) and \(\beta\) learnt.

The slides then compare a 7-layer network on MNIST with and without batch normalization, using sigmoid activations and w = 0.1*np.random.randn(), and plot the activation distributions at \(\mathbf{h}_1\), \(\mathbf{h}_4\) and \(\mathbf{h}_7\) (blue without BN, red with). To run a comparison of the same kind, tick Batch norm in the explorer of Module 7.4. With BN on, every weight scale gives the same spread at every layer, because standardizing the pre-activations cancels the scale of the weights.

Batch normalization has limitations:

• The mean and variance are estimates, and their accuracy depends on the batch size \(m\). For data such as images and videos, memory limits often force small batches, and a small \(m\) gives a large estimation error.
• A batch size of 1 cannot be used at all. Then \(\mu_i = x_i\) and \(\sigma_i = 0\), so normalization makes no sense: every standardized value is 0.
• Naive use of BN was also found empirically to degrade performance on NLP tasks. A systematic study confirmed this and proposed a modified BN called PowerNorm.

Alternatives such as group normalization and instance normalization were proposed for particular domains. The simplest one is layer normalization: take the average across the outputs of the \(H\) hidden units of the layer, one sample at a time.

The statistics no longer involve the other samples in the batch, so layer normalization works even with a batch size of 1, as needed, for example, in RNNs.

Not in this lecture: layer normalization is the "norm" in the Transformer's add-and-norm steps (Lecture 14).

Greedy layerwise pre-training

Training one hidden layer at a time, each to reconstruct its own input, before fine-tuning the whole network.

Autoencoder objective

Rebuild the input from a hidden layer with small squared error, so that layer keeps the input's important information.

Fine-tuning

Training the whole pre-trained network on the supervised loss, starting from the pre-trained weights.

Saturated neuron

A sigmoid or tanh unit at an end of its range, where its derivative, and every gradient through it, is about 0.

Zero-centred output

Outputs spread around 0. The sigmoid's are all positive, which forces a neuron's weight gradients to share one sign.

Dead ReLU

A ReLU unit whose pre-activation is negative for every input: it outputs 0, gets zero gradient, and never recovers.

Leaky and parametric ReLU

\(\max(\alpha x, x)\), with \(\alpha = 0.1\) fixed or \(\alpha\) learnt, so a gradient always flows.

ELU and SELU

\(x\) for \(x > 0\) and \(\alpha(e^x - 1)\) below; SELU scales this by \(\lambda \approx 1.0507\), with \(\alpha \approx 1.6733\), to self-normalize.

Maxout

The maximum of \(k\) affine functions of the input; only the winning piece receives gradient.

GELU

\(x\Phi(x)\), the expected value of \(m\cdot x\) with \(m \sim \text{Bernoulli}(\Phi(x))\); about \(x\sigma(1.702x)\).

Swish and SiLU

\(x\sigma(\beta x)\), found by automatic search; SiLU is the case \(\beta = 1\).

Symmetry breaking problem

Equal initial weights give equal activations and equal gradients, so the units of a layer stay identical forever.

Fan-in scaling

\(\operatorname{Var}(w) = 1/\text{fan}_\text{in}\) keeps \(n\operatorname{Var}(w) = 1\) and the variance steady across layers; ReLU needs \(2/\text{fan}_\text{in}\).

Batch and layer normalization

Standardize pre-activations, then apply a learnt \(\gamma\) and \(\beta\). BN takes statistics per unit over the batch; LN per sample over the layer's units.