Visualizing and fooling convolutional networks

This lecture is about understanding what a trained CNN has actually learned: what its filters detect, and what kinds of images make each neuron fire. Accuracy alone does not answer that. You may want to know whether any neuron fires for the dog class at all, or whether most neurons fire for every class, which would mean they have no discriminative power. Much of the material follows Andrej Karpathy's Stanford lectures on visualization, Deep Dream, neural style and adversarial examples.

The first tool needs no new maths. In Lecture 8 we saw that every neuron in every layer depends on a fixed patch of the input, its receptive field (for some layer, say, a 16 × 16 patch). The mapping is deterministic, so for any neuron we can always find which input pixels contributed to it.

1. Find the patches a neuron fires for
2. Pick a neuron in some layer of a trained CNN.
3. Pass as many images as you have (training, test, anything) through the network and record the neuron's output for each.
4. Keep the images where the output is high. These are ReLU neurons, which do not saturate at 1, so "fires" means a large output.
5. For each of them, trace the neuron back to its receptive field and look at that image patch.

Girshick et al. (2014) ran this on 6 neurons of the pool5 layer. Each neuron's top patches had a clear theme: one fired for people's faces, one for dog faces, one for flowers (in bouquets, in pots, on tables, in any orientation), one for numbers and text such as addresses, dates and signboards, one for houses, and one for shiny surfaces. Each neuron fired on the patch that contained its object, without reacting to the surroundings, so the network had learned to localize these characteristics.

Treat this as a debugging tool, like feeding a program different inputs. Does the network need more training? Is there a class no neuron fires for? Are two classes being confused?

The previous section visualized neurons, which are outputs (entries of feature maps). The weights of a CNN are its filters, and those can be visualized too. The lecture borrows an argument from autoencoders (see Lecture 7): which input excites a given hidden neuron the most? The neuron computes \(\mathbf{w}^\top\mathbf{x}\) (before its activation) with its weight vector \(\mathbf{w}\), and the input is restricted to unit norm so the question has an answer:

The lecture quotes this result. One line of proof: by Cauchy–Schwarz \(\mathbf{w}^\top\mathbf{x} \le \lVert\mathbf{w}\rVert\,\lVert\mathbf{x}\rVert = \lVert\mathbf{w}\rVert\), with equality exactly when \(\mathbf{x}\) is a positive multiple of \(\mathbf{w}\).

For MNIST digits the input has 784 dimensions, and so does \(\mathbf{w}\), because 784 weights connect the pixels to the neuron. So \(\mathbf{w}/\lVert\mathbf{w}\rVert\) can be drawn as a 28 × 28 image. In the autoencoder analysis some of these images showed a curve like the one in 2, 8 or 9, others a cusp like the one in 3 or 9. Each image is the pattern that makes its neuron fire.

A CNN filter is much smaller than the image: 3 × 3, 5 × 5 or 7 × 7. Think of the CNN as a feedforward network with sparse connections and weight sharing. In the figure, neuron \(h_{11}\) sees only pixels 1, 2, 5, 6 of a 4 × 4 image and \(h_{12}\) sees pixels 3, 4, 7, 8, both through the same 2 × 2 filter \(W\). Asking what input makes such a neuron fire gives the same answer as before, now for a patch: \(W/\lVert W\rVert\). Because the filter is shared, that patch triggers it wherever it occurs in the image, which is the point of convolution, since a nose should be detected at the top corner, in the centre or at the bottom alike. So filters are pattern detectors.

So we can simply plot each \(K \times K\) filter as an image. For the first convolution layer of a trained network the plots show edges at various orientations (slanted, vertical and others), coloured blobs and textures, even a checkerboard pattern. You want this variety. If the filters all look alike, they are not learning to tell different patterns apart, and something is wrong.

The slides add a claim with a question: this view is only interpretable for the filters of the first convolution layer. Why? The lecture leaves it as a quiz; an answer is in Test yourself.

The next debugging tool asks which portions of an image are responsible for the probability of a class. Insights like these help with design choices, for example whether 5 × 5 or 7 × 7 filters suit your data. Most users will train an off-the-shelf CNN and report accuracy; these tools are for understanding what is going on and how to improve it, and some appear in the assignment.

The idea is the usual way to judge a factor's influence: remove it and see whether the output changes. In a movie-review classifier, dropping words like "amazing" and "great" should lower the probability of the positive label (say from 0.9 to 0.6), while dropping "the", "and" or "for" should change little.

1. Occlusion map
2. Feed the image and note the predicted probability of the true class (for example "Pomeranian").
3. Replace one patch of the image by a gray patch and feed the occluded image again. Record the new probability of the true class.
4. Repeat for patches at every location you like.
5. Plot the probabilities as a heat map over patch positions. In the slide's maps, red regions cause little drop when occluded and blue regions a large drop.

The slide shows three cases, from the figure in Zeiler and Fergus (2014):

• Pomeranian. Covering the dog's face drops the probability the most. That is what we would want: the network is using facial features and not guessing.
• Car wheel. The instructor expected the wheel itself to matter most, and occluding it does not cause a drastic drop. One reading is that the network has learned redundant features, which is good: a face detector should use the nose, ears, hair and eyes, so hiding one part should not destroy it. Here, though, the drop is concentrated on another single region (he reads it as the rear-view mirror), which suggests heavy reliance on one feature and less robustness.
• Afghan hound. Occluding the woman's face lowers the probability. The network has probably learned a wrong association, for example if many training images showed the dog being held by a woman. Given two networks, prefer the one whose map points at the dog.

So far we have visualized neurons, visualized weights and occluded parts of the input. Now we want a principled measure of how much each pixel influences the output, or any neuron in an intermediate layer. The course's usual tool for influence is the gradient.

Flatten the image into \(m \times n\) inputs \(x_0, x_1, \dots, x_{mn}\). The influence of input \(x_i\) on a neuron \(h_j\) is \(\partial h_j / \partial x_i\): if a small change in \(x_i\) causes a large change in \(h_j\), then \(x_i\) has a lot of influence on \(h_j\).

Compute this partial derivative for all \(mn\) inputs, \(\partial h_j/\partial x_0, \dots, \partial h_j/\partial x_{mn}\), and display the resulting gradient matrix as an image. If gray stands for 0, the image should be mostly gray. A specialized neuron should depend on a small patch of pixels; if every pixel influenced it, it would not be discriminating anything. Standard name: such a picture is often called a saliency map.

How do we get gradients with respect to the input? Backpropagation (Lecture 4) always stopped at the weights of the first layer, but nothing in the chain rule forbids one more link. If we called the input \(\mathbf{h}_0\) instead of \(\mathbf{x}\), it would be just another layer. Treat the CNN as a feedforward network with sparse connections and sum over every path from the pixel to the neuron. For the neuron \(h_{32}\) in the slide and pixel \(x_2\), three first-layer neurons \(h_{1i}\) lie on such paths:

The slide writes the first layer as a plain weighted sum. With a ReLU on top, each \(\partial h_{1i}/\partial x_2\) also picks up the ReLU's derivative, 1 if the unit was active and 0 if not. That factor is what the next section modifies.

The resulting gradient image of a cat is murky. Most of it is gray, as expected, but nothing stands out. The whole cat region is non-gray, where we would have liked, say, only the eye pixels to light up for some neuron. Plain gradients do not show sharp influences. Springenberg et al. proposed guided backpropagation, which gives a better picture 4:40.

The lecture calls the idea a bit hacky and heuristic, but it works well. The setting:

1. Influence of the input on one neuron
2. Feed an image to the CNN and do a forward pass.
3. Pick one neuron in some feature map at some layer (the slide uses a 2 × 2 feature map \(\begin{bmatrix}1&0\\3&2\end{bmatrix}\) and the entry 2).
4. Keep that neuron and set all other neurons in the layer to zero, giving \(\begin{bmatrix}0&0\\0&2\end{bmatrix}\). We care only about this neuron's influence.
5. Backpropagate all the way to the image, giving \(\partial h / \partial i_1, \partial h / \partial i_2, \dots\) for every pixel.

On the way back, every ReLU matters. In the forward pass a ReLU lets positive values through and clamps negative values to zero. In the backward pass no gradient flows through such dead ReLU units (Lecture 7), so only the gradients at positions whose forward input was positive pass down. That is ordinary backpropagation.

Guided backpropagation applies the same idea to the gradients themselves: the forward pass does not let negative values go forward, so in the backward pass do not let negative gradients go back. Any gradient arriving from the layer above that is negative is also clamped to zero. Here is the slide's example at one ReLU layer, with \(f^l\) the ReLU's input and \(R^{l+1}\) the gradient arriving from above:

Backprop zeroes the entries where \(f^l \le 0\) (the dead units). Guided backprop also zeroes the entries where \(R^{l+1} \le 0\), so of the slide's nine gradients only 6 and 3 survive.

Not in the video: the figure this slide borrows from Springenberg et al. (2014) also has a "deconvnet" row (the method of Zeiler and Fergus), which keeps only the positive gradients and ignores the forward pass: \(R^{l}_i = \mathbb{I}[R^{l+1}_i > 0]\,R^{l+1}_i\). On the slide's numbers it gives \(\begin{bmatrix}0&3&0\\6&0&1\\2&0&3\end{bmatrix}\). Guided backpropagation is the combination of both masks.

The intuition: there may be pixels that really influence the neuron, but positive and negative gradients flowing back through many paths cancel each other, so most influences come out near zero (the mostly gray image). Neglecting all negative influences and keeping only the positive ones should give a better picture of the true influence of the input 2:34.

The lecture calls this heuristic because it messes with the maths: the correct gradient must flow back whether it is positive or negative, so the guided map is no longer \(\partial h/\partial \mathbf{x}\). Its justification is the analogy with the forward ReLU plus the cancellation argument. In practice the result on the cat is much sharper. It concentrates on the eyes, and the layout of the cat is far clearer than in the plain gradient image. Guided backpropagation is a popular tool for understanding what a CNN does, and one you implement in the assignment.

Take a trained CNN that distinguishes \(k\) classes. What image should we feed it so that it is recognized as a dumbbell (or an ostrich, or a car)? This sounds like an odd goal, but it is the seed of adversarial machine learning. If the \(k\) classes were \(k\) people in a face-recognition system, someone might want to know what image gets them recognized as a famous person. A milder use: an automatic check of whether a product looks like a dumbbell.

The answer reuses everything we know about training, with the roles swapped. Training keeps the image fixed and changes the weights, \(W \leftarrow W - \eta\,\partial\mathcal{L}/\partial W\). Here we keep the weights of the trained network fixed and treat the image itself as the parameter matrix: start from a blank (zero, gray) image and adjust its pixels \(I = (i_0, i_1, \dots, i_{mn})\) so that the score of the class goes up.

1. Class image by gradient steps on the pixels (slide version)
2. Start with a zero image.
3. Set the score vector to \([0, 0, \dots, 1, \dots, 0, 0]\): 1 for the class of interest, 0 elsewhere. This seeds the backward pass so that it computes the gradient of \(S_c\) alone.
4. Compute the gradient \(\partial S_c(I)/\partial i_k\) for every pixel \(i_k\).
5. Update every pixel, \(i_k \leftarrow i_k + \eta\,\partial S_c(I)/\partial i_k\).
6. Do a forward pass through the network again with the modified image.
7. Go to step 2. Stop at convergence, whatever you define it to be, for example once the class gets a score of 0.9 or 0.95 instead of 1.

The slide writes step 4 as \(i_k = i_k - \eta\,\partial S_c(I)/\partial i_k\), the gradient descent form. Since the score is to be maximized, the step must go along the gradient (equivalently, descend on \(-S_c\)); the same applies to the neuron version below.

The results are strange (the images are from Simonyan et al., 2013, a source the slide does not name). For dumbbell, starting from the zero image, many dumbbell-like shapes appear in different orientations. The colours are nothing like real dumbbells, and the network is being redundant: it draws many dumbbells so that some of them make the output fire. Cup gives several trophy-like cups, dalmatian gives black and white spots and dog-like shapes, and bell pepper, lemon and husky are similar. The shapes of the object show up, but nowhere is there a clear image a person would call that object 6:58.

This is nothing to be impressed by. The network assigns these arbitrary images to "dalmatian", so anyone could upload such an image as a dalmatian and an automated system would accept it. These images can later be used to fool the network (Module 13.10).

Any neuron, not just a class M6 · 8:34

The output score is just one neuron. The same procedure works for an arbitrary neuron anywhere in the CNN, with \(A(I)\) the activation of the chosen neuron in some layer:

1. Image that excites one neuron
2. Feed an image through the network.
3. Set the activations in the layer of interest to zero, except for the neuron of interest.
4. Backpropagate to the image.
5. Update each pixel, \(i_k \leftarrow i_k + \eta\,\partial A(I)/\partial i_k\), and repeat.

Starting from different initializations instead of the zero image gives different insights. The slide shows four images, each obtained by focusing on the same neuron in layer 8 and starting from a different initialization, and a similar analysis for layer 7, where some neuron's image looks like a ship with several layers of structure. Different neurons fire for different kinds of patterns, and making such images is a good way to learn more about your network.

Every layer's output can be thought of as an embedding of the image. The input is 227 × 227 × 3; the fc7 layer gives a 4096-dimensional vector for it, and a convolution volume can be flattened into a vector too. Think of autoencoders: a good embedding should capture all the important characteristics of the image, so we should be able to reconstruct the image from it. Can we?

Pose it as optimization over the pixels again. Pass the image of interest (the slide's example is a monkey) through the network and store its embedding \(\phi_0\). Then look for an image that (1) has an embedding close to \(\phi_0\) and (2) looks natural, which is the job of a regularizer. For squared closeness, compare the embeddings dimension by dimension and add up the squared differences.

1. Reconstruct an image from its embedding
2. \(\phi_0\): embedding of the image of interest. \(X\): a random image, say the zero image.
3. Forward pass with \(X\) and compute its embedding \(\phi(X)\).
4. Compute \(\mathcal{L}(I) = \lVert\phi(X) - \phi_0\rVert^2 + \lambda\,\lVert\phi(X)\rVert_6^6\).
5. Update every pixel by gradient descent, \(i_k \leftarrow i_k - \eta\,\partial\mathcal{L}(I)/\partial i_k\), and repeat until convergence.

\(\lVert\mathbf{v}\rVert_6^6 = \sum_k v_k^6\). The slide puts this penalty on \(\phi(X)\). In the original work (Mahendran and Vedaldi, 2015) the 6-norm penalty is applied to the image \(X\) itself, which is what makes it an image prior. It discourages a few extreme pixel values.

The slides then reconstruct the monkey from each layer of the network in turn: Conv-1, ReLU-1, max-pool-1, Norm-1, Conv-2, ReLU-2, max-pool-2, Norm-2, Conv-3, ReLU-3, Conv-4, ReLU-4, Conv-5, ReLU-5, max-pool-5, FC-6, ReLU-6, FC-7, ReLU-7, FC-8.

• From Conv-1 the reconstruction is almost the same image as the original.
• Going deeper, reconstructions become more and more abstract and the monkey stops coming back. Apart from the Conv-1 and Conv-2 layers, they are not accurate reconstructions.
• From FC-6 / FC-7 they look very weird. By then the network has abstracted the image away: perhaps something like a nose or eyes is kept here and there, but the shape and other characteristics are lost 4:45.
• Max-pooling makes it harder still. It is just another embedding, but a much stronger compression: of each pooling window only the maximum survives.

So the deeper representations lose much of the information needed to rebuild the input, which is expected from layers built to abstract.

So far the optimization started from a blank image and pushed it towards a class. Deep Dream starts instead from an actual image, say a picture of the sky. Focus on some layer, check the activations of its neurons, and change the image so that a chosen neuron \(h_{ij}\) fires even more (the slides draw several neurons, but think of a single one).

The parameters are again the pixels. To make "fire more" a neat differentiable objective, maximize the square of the activation, and split the gradient for a pixel \(i_{mn}\) with the chain rule:

The first factor is immediate, and \(\partial h_{ij}/\partial i_{mn}\) is backpropagation with one more link of the chain, as in Module 13.4. Then update the image and feed it back:

The slide writes the step without a learning rate; a step size \(\eta\) can multiply the gradient, as in the other updates.

On the next pass \(h_{ij}\) fires a bit more, because we modified the image for exactly that purpose. Now recall what \(h_{ij}\) captures: certain patterns. If it fires more and more, the image must contain more and more of those patterns. Iterating makes the image look increasingly like whatever makes the neuron fire.

The demos on the slides:

• A sky with clouds fills up with castles: literally castles in the air. In the training data a castle usually has the sky as its background, so the network has learned the correlation. Seeing sky, some castle neurons fire a little, and pushing them harder makes castles appear 6:49.
• A landscape with mountains and green regions sprouts fish eyes and fish-like shapes, and birds and animals on the mountains and green areas, which in the data set appear against such green and brown backgrounds.
• A still from the film Inception (men in brown suits) produces dogs everywhere: in this data set, brown things were mostly dogs. The network cannot dream beyond its data set.

What is happening: the network was trained to detect patterns (dogs, cats, birds and so on) that appear frequently in ImageNet, and it starts seeing them even where they hardly exist. Google's blog post on "inceptionism" describes the loop: a cloud that looks slightly like a bird is made to look more like a bird, so the network recognizes the bird more strongly on the next pass, and so on, until a detailed bird appears seemingly from nowhere.

Take a camera image (the content image, in the slides a picture of Gandalf) and an artwork (the style image), and render the content in that art form. The instructor flags one step as a leap of faith. The method defines two targets and optimizes the pixels of a new image to meet both.

Content

The content targets are the activations of all layers for the content image. The assumption is that the hidden representations capture the essence of the image (the face and its attributes), so a new image in a different style should produce activations close to these. Let \(\mathbf{p}\) and \(\mathbf{x}\) be the activations of the content image and the new image at a layer, indexed by position and channel:

Every entry of the activation volume of the new image should match the content image. The slide shows one layer; the same term can be used for every layer.

Style

How do we capture the style of an image? The lecture takes this on faith from the original paper: if \(V\) holds the activations of a layer with 64 channels of size 256 × 256, then the 64 × 64 matrix \(V^\top V\) captures the style of the image, and deeper layers capture more of it. This matrix is the Gram matrix of the layer.

The slide writes \(V \in \mathbb{R}^{64\times(256\times256)}\) and \(V^\top V \in \mathbb{R}^{64\times 64}\). For the product to be 64 × 64, lay \(V\) out with one column per channel, \(V \in \mathbb{R}^{(256\cdot256)\times 64}\); then \((V^\top V)_{ij} = \sum_k V_{ki}V_{kj}\) is the dot product of feature maps \(i\) and \(j\) over all positions.

To make the new image's style at layer \(\ell\) match the style image, compare the two Gram matrices. With \(G^\ell\) and \(A^\ell\) the Gram matrices at layer \(\ell\) for the style image and the new image respectively, and \(\mathbf{a}\) the style image's activations:

The weights \(w_\ell\) choose how much each layer counts; you can use one layer or all of them.

Optimizing the pixels with this objective (plus a bunch of other tricks) renders Gandalf in the given style. Code for it is available to try. Standard name: this is neural style transfer (Gatys et al., 2015). The broader point is that once you can write an objective over pixels, you can combine two images in whatever way you can imagine.

We have changed images to suit three needs: to make a class fire, to make neurons see patterns that were not really there (Deep Dream), and to produce artistic images (Deep Art). The same idea of optimizing over the input can also fool a CNN.

Feed an image of a bus to a trained network. Instead of setting up the cross-entropy loss for the correct class, set it up for an incorrect class, say ostrich. Backpropagate, leave every weight alone, and change only the image so that the likelihood of "ostrich" increases:

With very minimal changes to the image, the network is soon convinced it is looking at an ostrich (Szegedy et al., 2013). The slide shows triples: the original, the change (magnified to be visible) and their sum. The original and the sum are indistinguishable to a human, yet the network calls every sum an ostrich. The same happens for the other examples: a soap dispenser, a bird, an insect, a temple and a dog.

You do not even need a real starting image. Start from random images and optimize them until the network predicts some class: the results look like noise to us, yet the classifier labels them robin, cheetah, armadillo and so on, and it is 99.6% confident in all these cases (Nguyen, Yosinski and Clune, 2014). That is worse than a random classifier, which over 1000 classes would give each about 1/1000.

An intuitive explanation M10 · 4:51

The explanation is due to Andrej Karpathy.

• Images are extremely high-dimensional objects, points in \(\mathbb{R}^{227\times227}\) on the slide. That space holds an enormous number of points.
• Only a tiny fraction of them are natural images; most are random matrices that look like the noise images above. Even 1 or 10 million training images are a small sample of the natural ones.
• From the training images we fit decision boundaries, for some 1000 classes.
• In doing so we also make decisions for the vast number of points we never saw. Whole regions without a single training point get a class. A boundary drawn around a few cheetah images also encloses many random points that have nothing to do with cheetahs.
• To fool the network, start from a random image and move just far enough to land inside such a region. The classifier was aggressive in fitting the training data and has already labelled it.

One recipe, six objectives

Every method from Module 13.6 onward freezes the network and takes gradient steps on the pixels. Only the objective changes:

This ends the discussion of convolutional networks. Lecture 10 turns to learning vector representations of words.

Receptive field

The input patch a neuron depends on; it can be traced back deterministically through the layers.

Maximally activating input

For weights \(\mathbf{w}\) and a unit-norm input, \(\mathbf{w}^\top\mathbf{x}\) is largest at \(\mathbf{x}^* = \mathbf{w}/\lVert\mathbf{w}\rVert\).

Filter visualization

Plotting each \(K\times K\) first-layer filter as an image to see the pattern it detects.

Occlusion experiment

Gray out a patch, reclassify, and map the probability of the true class over patch positions.

Gradient (saliency) map

\(\partial h_j/\partial x_i\) for every pixel, shown as an image; gray means no influence.

Dead ReLU

A ReLU whose forward input was not positive; no gradient flows back through it.

Guided backpropagation

Backprop that also zeroes negative gradients from the layer above at every ReLU; gives sharper maps.

Deconvnet rule

Passes only positive gradients and ignores the forward sign (in the slide's source figure, not the video).

Optimization over images

Weights frozen, pixels treated as parameters, gradient steps on an objective such as \(S_c(I) - \lambda\Omega(I)\).

Embedding inversion

Finding an image whose embedding at some layer matches a given one; works well only from early layers.

Deep Dream

Gradient ascent on \(h_{ij}^2\) from a natural image, amplifying patterns the neuron already half-sees.

Content loss

\(\sum_{ijk}(p_{ijk} - x_{ijk})^2\) between the activations of the content image and the new image.

Gram matrix

\(V^\top V\) of a layer's flattened channels: channel-by-channel inner products, used as the style.

Fooling (adversarial) image

An image changed imperceptibly, or built from noise, that the network classifies wrongly with high confidence.