Convolutional neural networks: from the convolution operation to ResNet

The short M1 video sets the plan: the convolution operation, pooling, full convolutional networks and their parameter counts, and the standard architectures from LeNet to ResNet. The lecture itself starts with a tracking problem. A laser sensor reports the position of an aeroplane at discrete time steps, say every few seconds on a flight from Chennai to Delhi. The sensor is noisy, so one reading is not trusted on its own. A better estimate of the position at time \(t\) averages several past readings, and because the plane keeps moving, recent readings should count for more than old ones. That is a weighted average:

\(\mathbf{x}\) is the input, the array of weights \(\mathbf{w}\) is the filter (also called the kernel), and the operation is convolution, written \(\mathbf{x} * \mathbf{w}\). Each weight is indexed by how far back it looks: \(w_0\) multiplies the current reading, \(w_{-1}\) the previous one, and so on.

Readings from long ago tell us little about where the plane is now, so in practice the sum runs over a small window. Using the last 7 readings, \(a = 0, \dots, 6\), the filter has 7 entries. To get the next estimate, slide the filter one step to the right and repeat the multiply-and-add.

The filter, from \(w_{-6}\) to \(w_0\), is 0.01, 0.01, 0.02, 0.02, 0.04, 0.4, 0.5. Its entries add up to 1, so each \(s_t\) is a true weighted average, and 90% of the weight is on the two latest readings. For \(s_6\):

Estimates start at \(t = 6\), the first step with 7 readings behind it. The last row recomputes every \(s_t\) from the slide's \(\mathbf{w}\) and \(\mathbf{x}\): the first three match the slide to within 0.01, and the slide's last three are a little lower than these weights give. The idea is unaffected.

The same story works for a camera. An image is a 2-D input, and if we are unsure of a pixel's value we can re-estimate it from a weighted average of its neighbourhood. The filter is now a 2-D array of size \(m \times n\), one weight per neighbour (in practice square, \(m = n\)). The textbook formula looks at the neighbours before pixel \((i, j)\):

It has \(m \times n\) terms, one per neighbour, each with its own weight from the kernel matrix \(K\). For pixel \((4, 4)\) the term with \(a = b = 1\) uses \(I_{3,3}\). Nothing forces us to look backwards. Different packages use different conventions, and the most common one looks at the neighbours after the pixel instead:

A toy example makes this concrete. Place a \(2 \times 2\) kernel on the top-left corner of a \(3 \times 4\) input, multiply entry by entry and add. Then slide it one column right and repeat, and so on along each row:

The output (\(2 \times 3\)) is smaller than the input (\(3 \times 4\)). Module 11.2 explains exactly by how much. For the rest of the lecture the kernel is centred on the pixel of interest, so it looks at neighbours on both sides M2 · 10:30:

Standard name (not in the lecture): the "succeeding neighbours" version, with no flipping of the kernel, is what deep learning libraries actually compute. Mathematicians call it cross-correlation. Since the kernel is learned, the flip makes no difference to what a network can represent.

Three \(3 \times 3\) kernels show what a convolution can do to an image:

• Blur. Every pixel is replaced by (a multiple of) the average of its \(3 \times 3\) neighbourhood, so sharp transitions are smoothed away.
• Sharpen. Take 5 times the pixel and subtract its four neighbours. Where the neighbours are similar they cancel and the pixel stands out more.
• Edge detector. Add the 8 neighbours and subtract 8 times the pixel. If all neighbours equal the pixel the result is 0; it is nonzero only where something changes, which is along the boundaries of objects.

The blur and sharpen buttons of photo-editing software apply exactly such convolutions across the whole image, and many other visual effects come from some convolution.

Sliding a kernel over the image gives one output value per placement. The resulting array is called a feature map, because each output summarizes some feature of its neighbourhood. In the edge-detector map, a dark (zero) pixel records the feature "this pixel does not lie on a boundary". The weights need not form an average at all. With several different kernels (one blurring, one sharpening, one detecting edges, …) we get several feature maps of the same input.

Three-dimensional inputs

A 1-D filter slides over a 1-D input and a 2-D filter over a 2-D input. A colour image is 3-D: three channels (R, G, B) of size \(m \times n\). The filter then becomes a box with a width, a height and a depth, a volume. Throughout this lecture its depth equals the depth of the input. So the filter can slide along the width and the height but not along the depth, and each placement gives one number. In effect it is a 2-D convolution on a 3-D input, and the output of one filter is 2-D. \(K\) filters give \(K\) such 2-D maps.

If the filter were shallower than the input (depth 2 on a depth-3 input, say), it would also have to slide along the depth and the output would be 3-D. The lecture mentions this only to set it aside; every CNN here uses full-depth filters.

The sizes of inputs, filters and outputs decide the shapes of the matrices in the network, how many parameters it has and how much memory it needs. Module 11.2 makes them precise. The quantities are:

• the input's width \(W_1\), height \(H_1\) and depth \(D_1\) (3 for an RGB image);
• the stride \(S\), defined below;
• the number of filters \(K\);
• the spatial extent \(F\) of each filter, always square, so \(F \times F\), with depth equal to \(D_1\);
• the output volume \(W_2 \times H_2 \times D_2\), which is what we want to compute.

No padding

Take a \(7 \times 7\) input and a \(3 \times 3\) filter centred on each pixel in turn. On a border pixel the filter would stick out of the image, where the values are undefined, so the filter cannot be placed there. The output loses one row and one column on each side and is \(5 \times 5\). A \(5 \times 5\) filter cannot be placed within 2 pixels of the border, so the output is \(3 \times 3\), smaller still. In general a filter of size \(F\) loses \(\lfloor F/2 \rfloor\) columns on each side, \(F - 1\) in total:

Padding

To keep the output as large as the input, add a border of zeros around the input. This is padding: \(P = 1\) adds one row of zeros at the top and bottom and one column at the left and right, which is exactly the strip a \(3 \times 3\) filter could not reach. The filter can now be placed on every original pixel. Each unit of padding adds 2 to the width (one column on each side):

A \(5 \times 5\) filter needs \(P = 2\) to keep the size: \(W_1 - 5 + 4 + 1 = W_1\).

Stride

The stride is the interval at which the filter is applied. So far the filter moved one pixel at a time (\(S = 1\)). With \(S = 2\) it skips every second position, which roughly halves the output's width and height; \(S = 4\) cuts them to about a quarter.

Dividing by the stride gives the final formula for width and height. The depth is simpler: each filter produces one 2-D map, so \(K\) filters produce a volume of depth \(K\).

The lecture notes that when \(W_1 - F + 2P\) is not a multiple of \(S\) the exact count involves a floor, and leaves the detail as an exercise. The usual convention (not stated in the lecture) is \(W_2 = \lfloor (W_1 - F + 2P)/S \rfloor + 1\): a last step that would cross the border is dropped.

Neither exercise is random. The first is the first convolution layer of AlexNet and the second the first layer of LeNet-5; both networks come later in this lecture.

What does convolution have to do with neural networks? The lecture answers through image classification: assign an image to one of a few classes, here car, bus, monument and flower. There are three classical ways to set it up.

1. Raw pixels. Flatten a \(20 \times 20\) image into a 400-dimensional vector and train any multi-class classifier on it.
2. A hand-chosen filter. For these classes the shape of the object matters, while the sky, the lawn and the colour of the car do not. Running an edge detector first keeps the outlines and discards the rest, giving a more refined representation. The kernel was hand-coded from domain knowledge.
3. Engineered features. Before 2012 the computer vision community designed ever better features. The most popular were SIFT and HOG, which describe how pixel gradients change across the image. Much work showed that they worked well on many image tasks.

In all three the feature extraction is static and deterministic. Learning happens only in the classifier on top, which is like training only the output (softmax) layer of a feedforward network. This is the machine learning recipe: engineer features, then learn a classifier.

An edge detector is just a small matrix, and a network is full of matrices that we learn. So the lecture asks a sequence of questions. Can we learn a meaningful kernel instead of hand-coding it? Can we learn several kernels, giving several representations of the input? Can we stack layers of kernels, each building a more abstract representation from the one below? The answer to all three is yes. Treat the kernels as parameters, alongside the classifier's weights, and train everything end to end with backpropagation on the classification loss (cross-entropy for this multi-class problem). Such a network is a convolutional neural network (CNN). In the lecture's framing, this is where deep learning parts from classical machine learning: the representation is learned, not hand-crafted.

How is a CNN different from a regular feedforward network? Take digit recognition (10 classes) on a tiny \(4 \times 4\) image. A feedforward network flattens it into 16 inputs, adds hidden layers and ends in 10 outputs. Every neuron in the first hidden layer, \(h_{11}\) for example, takes input from all 16 pixels: the connections are dense.

Now apply a \(2 \times 2\) convolution with stride 2. \(h_{11}\) is computed from pixels 1, 2, 5 and 6 only, and \(h_{12}\) from pixels 3, 4, 7 and 8. Only a few local inputs feed each output. This sparse connectivity uses what we know about images: the interactions between neighbouring pixels matter more than those between the top-left corner and the bottom-right corner. It also reduces the number of parameters.

Are we losing information by dropping interactions between distant pixels? Not really, because a CNN has several layers. With a window of 3 in the figure below, \(x_1\) and \(x_5\) do not meet in layer 1: no \(h\) depends on both. But \(g_3\) depends on \(h_2\), \(h_3\) and \(h_4\), which together depend on \(x_1, \dots, x_5\). So the two pixels interact one layer up. Early layers capture small neighbourhoods; later layers combine neighbourhoods of neighbourhoods, and eventually all pixels interact.

Standard name (not in the lecture): the set of input pixels that can affect a unit is its receptive field. Here the receptive field of \(g_3\) is \(x_1, \dots, x_5\).

The second property of a CNN concerns the kernel's values. Suppose one \(2 \times 2\) kernel sits on pixels 1, 2, 5, 6 of the \(4 \times 4\) image and another on pixels 11, 12, 15, 16. Should their weights differ? Think about what a filter is for. A kernel that detects edges must have the same configuration wherever it is applied, since an edge looks the same in any part of the image. The blur, sharpen and edge kernels earlier were all applied unchanged at every location.

So a CNN uses the same kernel at every position. This is weight sharing, and it makes learning easier: the network does not have to learn the same weights again and again at different locations. (Lecture 6 met the general idea as parameter sharing; see Lecture 6.)

With a single shared \(2 \times 2\) kernel, layer 1 would have only 4 weights, far too few: the model would underfit. The fix is to use many kernels, each shared across all locations. With \(K\) kernels the layer has \(4K\) weights and produces \(K\) feature maps.

A CNN layer differs from a dense layer in two ways: each output sees only a local window (sparse connectivity), and every window uses the same kernel (weight sharing).

A complete network for handwritten character recognition (LeNet-5, next section) shows the overall pattern. The 2-D input image goes through a convolution layer with 6 filters, giving 6 feature maps. A pooling layer shrinks each map. The 6 shrunken maps form a volume, which is the input to the next convolution layer: 16 full-depth filters give 16 maps, one 2-D map per filter. Another pooling layer follows, and finally a few fully connected layers lead to the output. Each stage's output is the next stage's input, as with hidden layers in a feedforward network.

So a CNN alternates convolution and pooling layers. A pooling layer works on one feature map at a time. Max pooling with a \(2 \times 2\) filter and stride 2 looks at each \(2 \times 2\) block and keeps only its largest value. Each block becomes one number, which halves both width and height (a quarter of the values survive). With stride 1 the windows overlap and the map shrinks less. Average pooling keeps the mean of the window instead of the maximum.

Pooling has no parameters: there is no weight matrix and no \(\mathbf{w}^\top\mathbf{x}\), just a fixed max (or average). It also leaves the depth unchanged, since it never takes a max across feature maps.

LeNet-5 (from around 1997–98, already the fifth version) recognizes handwritten characters from a \(32 \times 32\) grey-scale image. Work out each layer with the size formula; a convolution filter has \(F \times F \times D_1\) weights, where \(D_1\) is the depth of that layer's input.

• Depth counts. In class the first answer for Conv 2 was \(16 \times 25 = 400\). That forgets that each \(5 \times 5\) filter also spans the 6 maps of its input, so it has \(5 \times 5 \times 6 = 150\) weights, and 16 of them make 2,400.
• Slide corrections. The slide lists the pooling layers with \(S = 1\); to go from 28 to 14 the stride must be 2, as the lecturer points out. The slide's output layer has 10 units; for alphabet recognition it would be 26, giving \(84 \times 26 + 26\) parameters. The convolution counts on the slide leave out biases; the fully connected counts include them.
• A fully connected layer flattens the \(5 \times 5 \times 16\) volume into one 400-vector and connects every entry to every one of the 120 units, as in an ordinary feedforward network.

Compare Conv 1 with a dense layer between the same sizes. The input has \(32 \times 32 = 1024\) values and the output \(28 \times 28 = 784\), so a fully connected layer would need \(1024 \times 784 = 802{,}816\) weights against 150 here. Sparse connectivity and weight sharing together make the difference. This is why CNNs with 4 or 5 layers could be trained even before the deep learning revival of 2006 (Lecture 7): they had far fewer parameters than dense networks.

The table also shows a pattern that returns in every network below: the fully connected layers dominate. The convolution layers have hundreds or thousands of parameters; the first fully connected layer alone has 48,120.

For any CNN, you should be able to say for each layer: the input volume, the output volume, the number of filters and the number of parameters.

Backpropagating through a convolution looks daunting, but a CNN can be written as a feedforward network in which only a few weights are active and the rest are fixed at zero. Take a \(3 \times 3\) input and a \(2 \times 2\) kernel:

Seen this way, nothing new is needed. Training a network with some connections switched off is familiar from dropout (Lecture 6), and backpropagation (Lecture 4) applies unchanged, with some careful coding to keep the absent weights out. In practice nobody builds the dense version, which would waste exactly the computation that sparse connections save. Instead the convolution and its derivative are defined directly, and frameworks such as TensorFlow and PyTorch compute the backward pass automatically from the forward convolution. Conceptually it is still the same backpropagation algorithm.

Completing the idea (not worked out in the lecture): because \(w\) multiplies a different input in each of the four outputs, its gradient adds up the four contributions, \(\dfrac{\partial \mathcal{L}}{\partial w} = b\,\dfrac{\partial \mathcal{L}}{\partial l} + c\,\dfrac{\partial \mathcal{L}}{\partial m} + e\,\dfrac{\partial \mathcal{L}}{\partial n} + f\,\dfrac{\partial \mathcal{L}}{\partial o}\). A shared weight always collects gradient from every place it is used.

The lecture raises one more point and leaves it open: is everything in a CNN differentiable? In particular, how does the gradient pass through a max pooling layer?

Not in the lecture: max is differentiable almost everywhere. Within each window, the output equals one input (the largest), so the gradient flows back to that input unchanged and the other inputs in the window get zero. For average pooling each input in the window gets \(1/F^2\) of the gradient.

The competition that made CNNs famous in 2012 is the ImageNet Large Scale Visual Recognition Challenge (ILSVRC). It uses 1000 categories of the ImageNet data set, with roughly a million training images. In 2010 and 2011 the entries were classical: extract SIFT or HOG features, train classifiers on top, and add ensembles and other tricks. An error of 28.2% means about 282 mistakes on 1000 test images.

The rest of the lecture walks through these winners: AlexNet, ZFNet and VGGNet in this module, GoogLeNet and ResNet in the next. ResNet's 3.57% is what prompted claims that CNNs beat humans at this task. A person shown the same 1000 images would also make more than about 3.5% errors, since some images are blurred and some classes, such as dog breeds, are hard to tell apart.

Not in the lecture: these are the challenge's top-5 error rates. A prediction counts as correct if the true label is among the model's five highest-scoring classes.

AlexNet takes a \(227 \times 227 \times 3\) RGB image. The first layer is Exercise 1 from Module 11.2. The slides compute every layer with the size formula; the parameter count of a convolution layer is \((F \times F \times D_1) \times K\), biases left out.

• Shrinking width, growing depth. From \(227 \times 227 \times 3\) the maps shrink to \(27 \times 27 \times 96\) after the first pooling, while the depth grows because more filters capture more patterns. Pooling leaves depth unchanged and has no parameters.
• Eight layers. AlexNet is called an 8-layer network although the table has 12 rows. Only layers with parameters are counted: 5 convolution layers and 3 fully connected layers.
• Where the parameters are. About 24M of the 27.55M (4M + 16M + 4M) are in the fully connected layers.
• Why shrink at all? Pooling and large strides bring \(227 \times 227\) down to \(2 \times 2\) before the fully connected layers. Without them we would be left with maps of order \(200 \times 200\), and a fully connected layer on top would be infeasible, throwing away the savings of sparse connectivity and weight sharing. (Fully convolutional networks with no max pooling also work, the lecture adds; VGGNet below uses long runs of convolutions with few pooling layers.)
• Why this configuration? Nobody really knows. The kernel sizes and filter counts are hyperparameters, presumably chosen by trying many settings on a validation set. Some choices look sensible: large filters on the full image, smaller ones once it has shrunk. Others, such as three convolution layers back to back, have no clear rationale.
• Engineering. Every convolution is followed by a ReLU, applied to each entry of the feature map. The lecture credits AlexNet with bringing ReLU to CNNs in place of sigmoid or tanh (see Lecture 7). Training it in 2012 took many tricks, including splitting the network across GPUs, which is part of why it came two years after the challenge began.

Arithmetic check (not in the lecture): Conv 3 and Conv 5 each have \(3 \times 3 \times 256 \times 384 = 884{,}736 \approx 0.88\)M weights. The slide's 27.55M is the sum of the rounded per-layer figures; the exact products add up to about 28.8M. Also, the published AlexNet pads some of its convolutions, so its later maps are larger than these; the lecture's version uses \(P = 0\) throughout.

ZFNet is AlexNet with a few changes, so the lecture presents it as a layer-by-layer diff. The slide numbers the layers 1 to 10 from Conv 1 to FC 2, pooling layers included. The input is the same \(227 \times 227 \times 3\).

The reasoning behind the first change: an \(11 \times 11\) neighbourhood is more than needed, though \(3 \times 3\) may be too small, so \(7 \times 7\) is a compromise. It also removes a few parameters. The deeper layers get more filters (512, 1024, 512 instead of 384, 384, 256), so ZFNet has more parameters than AlexNet overall. The error fell from 16.4% to 11.7%. The pooling and fully connected layers are unchanged, and like AlexNet it has 8 layers with parameters.

Arithmetic check (not in the lecture): the slide writes the layer 6 and 7 differences as AlexNet minus ZFNet and evaluates them to 0.8M and 0.36M. The products actually come to \(9 \times 376{,}832 \approx 3.39\)M and \(9 \times 425{,}984 \approx 3.83\)M, so the convolution layers add about 7.5M, not 1.45M. Two side effects of the changes are also missing from the slide: with \(F = 7\) and \(S = 4\) the formula gives \((227 - 7)/4 + 1 = 56\), not 55; and with 512 maps of size \(2 \times 2\) at the end, FC 1 would have \(2048 \times 4096\) weights, twice AlexNet's.

VGGNet's distinctive choice is to use \(3 \times 3\) kernels in every convolution layer, from the first to the last, and to stack them in runs between max pooling layers. The slide's figure takes a \(224 \times 224 \times 3\) input. The convolutions keep the width and height (so they must use padding \(P = 1\)), and each max pooling halves them.

That is 13 convolution layers and 3 fully connected layers, 16 layers with parameters: VGG-16. A later variant with 19 such layers is VGG-19. As with AlexNet, there is little intuition for why 16; the architecture is whatever gave the best ImageNet accuracy.

The first fully connected layer is the problem. AlexNet and ZFNet ended with a \(2 \times 2 \times 256\) volume, a 1024-vector. VGGNet ends with \(7 \times 7 \times 512 = 25{,}088\) values, about 25 times more, and connecting each of them to 4096 units costs over 100 million weights. A very deep stack of convolutions holds far fewer parameters than this one layer.

The slide writes the last factor as 1024; with 1000 output classes it is \(4096 \times 1000\). The three terms are about 102.8M, 16.8M and 4.1M, so either way the total is near 123M.

Trained CNNs as feature extractors

The short M7 video adds a point used in the next lecture. Once a network such as AlexNet or VGGNet has been trained on ImageNet (cross-entropy tracked over epochs until the training error is satisfactory), the output of any of its layers, flattened into a vector, is an abstract representation of the input image. So it is common to pass a new image through a network trained on ImageNet, take the vector from a fully connected layer (an "fc representation") or a convolution layer (a "conv representation"), and use it for some other image task. Lecture 9 looks inside these representations.

Consider the output volume at some layer in the middle of a CNN. Next we could apply max pooling, or a \(1 \times 1\), \(3 \times 3\) or \(5 \times 5\) convolution. Every network so far made one uniform choice per layer. But interactions may matter at several scales at once, so GoogLeNet asks: why choose? Apply all of them in parallel and concatenate the resulting feature maps.

Done naively this is expensive. With \(P = 0\) and \(S = 1\), convolving a \(W \times H \times D\) input with an \(F \times F \times D\) filter gives a \((W - F + 1) \times (H - F + 1)\) map, and each entry needs \(O(F \times F \times D)\) multiply-adds:

1×1 convolutions

The fix is the 1×1 convolution. On a single pixel it might seem to do nothing, but remember a filter spans the full depth: a \(1 \times 1\) filter is really \(1 \times 1 \times D\), and it combines the \(D\) values at one location into one number. It aggregates along the depth. Using \(D_1 < D\) such filters (\(S = 1\), \(P = 0\)) turns a \(W \times H \times D\) volume into \(W \times H \times D_1\): the width and height stay, the depth shrinks. A following \(F \times F\) convolution then costs \(O(F \times F \times D_1)\) per entry instead of \(O(F \times F \times D)\).

The 3×3 and 5×5 branches may want different reductions, so each gets its own set of \(1 \times 1\) filters, \(D_1\) before the \(3 \times 3\) and \(D_2\) before the \(5 \times 5\). That adds a little computation, but the trade-off is acceptable. Add a \(3 \times 3\) max pooling branch followed by its own \(1 \times 1\) convolutions, and a plain \(1 \times 1\) branch, and concatenate the four outputs along the depth. This block is the Inception module. Each branch is padded so that all four outputs have the same width and height and can be stacked. Earlier networks were purely serial; an Inception module runs several operations in parallel inside a serial chain.

Worked numbers (not in the lecture): for the 5×5 branch of module 3a, 32 filters of size \(5 \times 5 \times 192\) on the \(28 \times 28\) map cost \(784 \times 32 \times 4800 \approx 120\)M multiplications. With 16 reducing \(1 \times 1\) filters first, the cost is \(784 \times 16 \times 192 \approx 2.4\)M for the reduction plus \(784 \times 32 \times 400 \approx 10\)M for the 5×5, about 12M in all: ten times less.

The lecture stresses that only two ideas here are profound: apply filters of several sizes in parallel, and use \(1 \times 1\) convolutions to keep the computation manageable. The rest is bookkeeping.

GoogLeNet (also called Inception Net) chains many Inception modules. On the slide's figure, an RGB input (229 × 229 on the slide; the lecture says 227 or 229) passes through:

1. GoogLeNet, as on the slide
2. Convolution with 64 filters → 112 × 112 × 64, then max pooling → 56 × 56 × 64.
3. Convolution with 192 filters → 56 × 56 × 192, then max pooling → 28 × 28 × 192.
4. Inception 3a (→ 256) and 3b (→ 480) at 28 × 28, then max pooling → 14 × 14 × 480.
5. Inception 4a, 4b, 4c (→ 512 each), 4d (→ 528) and 4e (→ 832) at 14 × 14, then max pooling → 7 × 7 × 832.
6. Inception 5a (→ 832) and 5b (→ 1024) at 7 × 7.
7. Average pooling over each 7 × 7 map → 1024 values, dropout (40%), a fully connected layer to 1000 units, softmax.

The output depth of each module is the sum of its four branches, for example \(128 + 192 + 96 + 64 = 480\) for 3b. As with the earlier networks, the lecture sees little intuition behind these particular numbers; they are presumably what worked best.

Getting rid of the big fully connected layer

The last Inception module outputs \(7 \times 7 \times 1024\). Flattening it and connecting it to 1000 output units would take \(7 \times 7 \times 1024 \times 1000 = 50{,}176{,}000\) weights (49M on the slide), the same blow-up as in VGGNet. Instead, GoogLeNet averages each of the 1024 feature maps over its \(7 \times 7\) grid, which leaves a 1024-dimensional vector. The output weight matrix becomes \(W \in \mathbb{R}^{1024 \times 1000}\), and the input to the dense layer is 49 times smaller.

Most of the saving comes from dropping the fully connected layer. The Inception modules themselves add parameters; removing the dense layer is what makes them affordable. Twice AlexNet's computation is an acceptable price for a much deeper network: the lecturer counts 14 stages in the figure, each Inception module itself a split layer with parallel branches, and the chart above lists 22 layers. GoogLeNet was also more accurate than AlexNet, ZFNet and VGGNet (6.7% against VGGNet's 7.3%).

Three takeaways from GoogLeNet: filters of several sizes in parallel, 1×1 convolutions to cut computation, and average pooling to avoid a huge fully connected layer.

Suppose a shallow network (shallow is relative: say 8 layers) trains well, reaching near-zero training error and reasonable test accuracy. Now insert a few more layers. Intuitively the deeper network should do at least as well, because one of its possible solutions copies the shallow one: let the new layers compute the identity. The solution space of the shallow network is a subset of the deeper network's.

In practice this does not happen. On CIFAR-10, plain networks of 20, 32, 44 and 56 layers get worse as they get deeper: the 56-layer network has the highest error, on both the training set and the test set. The lecture points to the training error: the deeper networks fit even the training data worse. The argument above relied on the network finding the identity mapping, but that is one solution among a huge number, a needle in a haystack. Obvious to us, it is not easy for gradient-based training to land on.

So make the identity explicit. Two stacked convolution layers take \(\mathbf{x}\) and learn some function \(H(\mathbf{x})\). Instead, let them learn only a residual \(F(\mathbf{x})\), and add the input back through a direct connection:

This follows the earlier argument: a deeper copy of a working shallow network should be fine if its new layers can pass the input through, and the identity connection hands them that copy. With these skip connections the authors trained networks of 152 layers. The last sentence of the figure caption is the standard intuition, not stated in this form in the lecture: pushing a residual towards zero is easier for an optimizer than assembling an identity map from scratch.

ResNet took first place in all five main tracks of the 2015 ImageNet and COCO challenges:

• ImageNet classification: "ultra-deep" 152-layer networks, 3.57% error;
• ImageNet detection: 16% better than the second-best system;
• ImageNet localization (finding where the object is): 27% better;
• COCO detection: 11% better; COCO segmentation: 12% better.

The downside is a very deep network that takes a long time to train, affordable mainly with very large computing resources.

The bag of tricks

The paper's training details, as listed on the slide (most were covered in earlier lectures):

• batch normalization after every convolution layer (Lecture 7);
• "Xavier/2" initialization, which is He initialization (Lecture 7);
• plain SGD with momentum 0.9, no Adam or Adagrad (Lecture 5);
• learning rate 0.1, divided by 10 whenever the validation error plateaus;
• mini-batch size 256;
• weight decay of \(10^{-5}\), that is, L2 regularization with \(\lambda = 10^{-5}\) (Lecture 6);
• no dropout.

These are hyperparameters, separate from the model's parameters. The lecture's advice for reading deep learning papers: study the experimental section where such details are given. Without them you will often fail to reproduce a paper's results even with the architecture coded correctly; with them the chances are much better, though still not guaranteed.

The ImageNet networks side by side

Convolution

Sliding a filter over the input and taking, at each position, the sum of entry-wise products with the window under it.

Filter (kernel)

The small array of weights that is slid over the input; in a CNN its values are learned.

Feature map

The 2-D output of one filter over the whole input; \(K\) filters give \(K\) maps.

Volume filter

A filter of size \(F \times F \times D_1\) spanning the input's full depth, so it slides over width and height only.

Stride \(S\)

The step between successive placements of the filter; larger strides shrink the output.

Padding \(P\)

Rings of zeros around the input so the filter can also be placed on border pixels.

Output size

\(W_2 = (W_1 - F + 2P)/S + 1\), same for \(H_2\), and \(D_2 = K\).

Sparse connectivity

Each output depends only on a small local window of the input, not on every input.

Weight sharing

The same kernel weights are used at every location of the input.

Max / average pooling

A parameter-free layer that replaces each window of a feature map by its maximum or mean, map by map.

Fully connected layer

A dense layer on the flattened volume; in AlexNet and VGGNet it holds most of the parameters.

1×1 convolution

A \(1 \times 1 \times D\) filter that combines values across depth; \(D_1 < D\) of them reduce the depth.

Inception module

Parallel 1×1, 3×3, 5×5 convolutions and max pooling, with 1×1 reductions, concatenated along depth.

Residual connection

A skip connection that adds a block's input to its output: \(H(\mathbf{x}) = F(\mathbf{x}) + \mathbf{x}\).