Sigmoid neurons, gradient descent and what networks can represent

Lecture 2 was about Boolean functions, \(y = f(\mathbf{x})\) with \(\mathbf{x} \in \{0, 1\}^n\) and \(y \in \{0, 1\}\). Perceptrons accept real inputs, but the discussion stayed with Boolean ones. This lecture moves to arbitrary functions with real inputs and a real output:

Such problems are common.

• Oil. Whether to build a drilling station at a spot in the ocean depends on how much oil it holds. That quantity depends on salinity, density, pressure, temperature, marine diversity and more, all real numbers.
• Interest rate. A bank sets a customer's rate, say between 5% and 20%, from salary, family size, past loans, how quickly they were repaid, their interest rates and any defaults.
• Images. A 100 × 100 greyscale image flattens into a vector of 10,000 real pixel values, a very high-dimensional input.

The goal mirrors Lecture 2. There a network exactly represented a Boolean function if it reproduced every row of the truth table, or every label of a linearly separable training set. The requirement is now relaxed to approximately. Given training pairs \((\mathbf{x}, y)\), such as existing drilling stations with their measurements and the oil they produced, we want a network whose prediction \(\hat{y}\) is close to \(y\) for every training input. Before asking whether such a network exists, we have to graduate from perceptrons to sigmoid neurons.

Recall that a perceptron fires when the weighted sum of its inputs reaches the threshold \(\theta = -w_0\). (Lecture 2 moved the threshold into the sum as a weight \(w_0\) on a constant input \(x_0 = 1\).)

Take the movie example with a single input, the critics' rating \(x_1 \in (0, 1)\), with weight \(w_1 = 1\) and threshold 0.5, so \(w_0 = -0.5\). A movie rated 0.51 is liked; a movie rated 0.49 is disliked. Two nearly identical ratings get opposite decisions, which is not how anyone actually decides.

The same jump appears for any problem, any weights and any threshold, because it comes from the perceptron function itself. Write \(z = \sum_{i=1}^{n} w_i x_i\). The output is 0 while \(z\) is below \(-w_0\) and becomes 1 the moment \(z\) crosses \(-w_0\): a step function. The if/else form of the rule is what makes the decision discrete. For most real applications we would expect a decision function that moves gradually from 0 to 1.

A sigmoid neuron replaces the step with a smooth S-shaped curve. "Sigmoid" names a family of S-shaped functions. The member used here is the logistic function; tanh, which appears later in the course (Lecture 7), is another.

With \(x_0 = 1\), the sum \(w_0 + \sum_{i=1}^{n} w_i x_i = \sum_{i=0}^{n} w_i x_i\) is the dot product \(\mathbf{w}^\top\mathbf{x}\) of \(\mathbf{w} = [w_0, \dots, w_n]\) and \(\mathbf{x} = [x_0, \dots, x_n]\). It is the same quantity the perceptron compares with 0.

To see the S shape, take \(w_0 = 0\) for the moment (a nonzero \(w_0\) only shifts the curve along the axis):

• as \(\mathbf{w}^\top\mathbf{x} \to \infty\), \(e^{-\mathbf{w}^\top\mathbf{x}} \to 0\) and \(y \to 1\);
• as \(\mathbf{w}^\top\mathbf{x} \to -\infty\), the denominator grows without bound and \(y \to 0\);
• at \(\mathbf{w}^\top\mathbf{x} = 0\), \(y = 1/(1 + e^0) = 1/2\).

So the output saturates at both ends and moves smoothly from one to the other, with no sharp transition at the threshold. Two consequences matter.

• The output is a real number between 0 and 1, which can be interpreted as a probability. Instead of a like/dislike decision we get the probability of liking the movie: an output of 0.49 means a 49% chance and 0.51 a 51% chance, a sensibly small difference. Most modelling problems want exactly this, for instance the probability of finding oil, or of a loan being repaid, which a domain expert can then act on.
• It is smooth, continuous and differentiable. The step function is not differentiable at \(-w_0\). Calculus, and derivatives in particular, will drive almost every algorithm in the course, so the functions we work with must be differentiable.

The lecture calls this module the most important one of the course. The pattern from Lecture 2 repeats: after introducing the perceptron we needed an algorithm to learn its weights, and before that algorithm we took a detour through the idea of error, because the algorithm corrected the weights whenever it made an error. The sigmoid neuron needs a learning algorithm too, so error comes back first.

Lecture 2 said a single perceptron cannot handle data that is not linearly separable, such as one class surrounded by the other, or two classes split by a curved boundary. What does that mean in practice? If we ran the perceptron learning algorithm on such data for a fixed number of iterations, it would never converge, yet it would end with some line through the middle of the data. On the slide's examples that line misclassifies a few points of each class, roughly 2 or 3 out of 10, so around 20% error.

For many real applications that is acceptable. Predicting which party a voter will choose with 20% error still gives a rough picture of an election, unless the race is very close or the application is mission-critical. Reaching zero error would need a very complex boundary, with islands carved out around stray points.

From now on we accept that driving the error to 0 is hard in most cases, and aim for the minimum possible error instead.

Every supervised learning problem can be cast in the same mould. We are given \(n\) pairs \(\{(\mathbf{x}_i, y_i)\}_{i=1}^{n}\), where each \(\mathbf{x}_i\) is a vector of \(m\) features (the video uses \(m\) here to keep \(n\) for the number of examples) and \(y_i\) is the value to predict. It is real here for simplicity; it could also be a vector, for example a movie's box-office collection together with its IMDb rating. The oil sites, the bank's past customers and past movies (budget, number of superstars, music director, cinematographer, genre, …) all fit this form.

We have reason to believe that \(y\) depends on \(\mathbf{x}\), that is \(y = f(\mathbf{x})\) for some true function \(f\). If we knew \(f\), there would be nothing left to do: plug in any new location's measurements and read off the oil. We do not know it. So we assume a form \(\hat{f}\) with some parameters and learn those parameters from the data. A school example: assume \(y = mx + c\); two points give two simultaneous equations, and solving them gives \(m\) and \(c\). With many data points and many parameters we need a more principled algorithm, but the idea is the same.

The three example models are familiar: the first is logistic regression, the second linear regression and the third assumes a quadratic relation. None is right or wrong in advance. In deep learning \(\hat{f}\) will be a very complex composite function, but the setup is identical.

Why parameters at all? 14:46 Why not just declare \(\hat{y} = 1/(1 + e^{-x})\)? That function is as arbitrary as declaring \(y = 3x + 2\): nothing in it is learned, so the data could be thrown away. Parameters let the data shape the model. For a sigmoid, \(w\) and \(b\) decide how steep the S is and where it sits.

Why squared error? The loss should be small when every \(\hat{y}_i\) is close to \(y_i\). Simply adding the differences fails: if one prediction is off by \(-0.5\) and another by \(+0.5\), the sum is 0 although both are badly wrong. Squaring removes the sign. The absolute value would also remove it, but it is not differentiable at 0, and the squared error is smooth, which matters because calculus will drive the learning.

\(\mathbf{w}\) has \(m + 1\) real components (counting \(w_0\)), each anywhere in \((-\infty, \infty)\). Of these infinitely many candidates, training should find the one with the smallest loss.

The lecture's analogy 23:57: studying a trigonometry chapter, where you may read the text and aim to get every formula right, is training. The exercises at the end are the validation set: you can still go back and re-read. The exam is the test set: no more studying allowed. Training minimizes the training error, the error on the chapter itself. (Training, validation and test error return in Lecture 6.)

Module 3.3 starts with an algorithm that will not work in practice but builds the intuition for one that does. The model is the sigmoid neuron, written with \(\sigma\) for the logistic function:

To keep the explanation simple:

• there is a single input \(x\), with weight \(w\);
• following the literature, \(w_0\) is renamed \(b\), the bias, a notation kept for the rest of the course;
• instead of like/dislike, the neuron predicts a real value: the critics' rating \(y\) from the IMDb rating \(x\) (the lecture says "for no particular reason"). Nothing in the method depends on the output being real-valued rather than Boolean.

Notation: capital \(N\) counts training pairs; small \(n\) counts inputs, and \(n = 1\) here. The objective reads: find the \(w\) and \(b\) that make the average squared difference between true values and predictions as small as possible.

Now suppose we have only two training points, \((x_1, y_1) = (0.5, 0.2)\) and \((x_2, y_2) = (2.5, 0.9)\). At the end of training we expect \(w^*, b^*\) with \(f(0.5) \to 0.2\) and \(f(2.5) \to 0.9\). Geometrically: we want a sigmoid curve on which both points lie.

Can we find \(w^*, b^*\) by hand? Start with a random guess, \(w = 3\), \(b = -1\). Neither point lies on the resulting curve, so the guess is clearly bad. The loss puts a number on how bad it is. With \(N = 2\):

The slide prints 0.099. Working the last line out gives about 0.092, and about 0.094 without rounding \(f\); either way it is far from 0.

We want \(\mathcal{L}\) as close to 0 as possible, so keep trying values (the slide has sliders for this). The lecture's sequence of guesses and the reasoning behind it:

• \((0.5, 0)\) gives 0.0730, better. A jump to \((-0.1, 0)\) gives 0.14, worse: making \(w\) negative seems to hurt.
• So increase \(w\) and make \(b\) negative: \((0.94, -0.94)\) gives 0.0214, much better.
• Keep going in that direction: \((1.42, -1.73)\), \((1.65, -2.08)\) and finally \((1.78, -2.27)\), where the loss is 0.0000 to four decimals.

The search was not truly random: each step was steered by whether the loss went up or down. Even so, the lecturer admits the values were reverse-engineered from the known answer 12:47. Guessing at random could go on forever without converging.

With only 2 points and 2 parameters, we can do slightly better than guessing: compute \(\mathcal{L}(w, b)\) for a whole grid of \((w, b)\) values and plot it as a surface over the \((w, b)\) plane. The low region is visible at a glance, and we can pick a point there. (The lecture first did this for the perceptron in Lecture 2.)

This does not scale. With many more data points and parameters, computing the loss everywhere becomes intractable. Even here the surface is plotted only for \(w, b \in (-6, 6)\), not \((-\infty, \infty)\), so a lower point outside the range would be missed.

The surface also explains what the guessing was doing. Each guess is a point on the surface. The walk started high at \((3, -1)\), went down to \((0.5, 0)\), climbed back up at \((-0.1, 0)\), then came down steadily from \((0.94, -0.94)\) to \((1.78, -2.27)\) in the low region, where the loss is essentially 0. Without the surface in front of us, the moves were random jumps that sometimes went uphill. The interactive figure in Module 3.4 can draw this walk on the actual surface.

We want a principled way to move on the error surface, one that goes downhill at every step.

The goal of Module 3.4: traverse the error surface so as to reach the minimum quickly, without brute-force search and without random jumps that raise the error. (The slides credit Ryan Harris's YouTube videos on visualizing backpropagation for ideas in this module.)

Collect the parameters into a vector \(\boldsymbol\theta = [w, b]\) and initialize it randomly. A change \(\Delta\boldsymbol\theta = [\Delta w, \Delta b]\) is another vector in \(\mathbb{R}^2\); adding it moves \(\boldsymbol\theta\) in the direction of \(\Delta\boldsymbol\theta\). To be conservative, move only a small amount \(\eta\) in that direction: the direction is kept and the length shrinks.

The question is which \(\Delta\boldsymbol\theta\) to use. The guess-work algorithm made one up, checked the loss and tried again. The principled answer comes from the Taylor series.

The Taylor series approximates any continuously differentiable function by polynomials; the higher the degree, the better the approximation. This applies to us because the loss is differentiable: the sigmoid is differentiable, and so is the squared error on top of it. For one parameter, around a point \(w_0\) where the loss is known:

Why we care: we sit at \(w_0\) at step 0, can compute the loss there but cannot see the whole curve, and want to know whether moving left or right lowers it. The series tells us about \(\mathcal{L}\) at nearby points \(w = w_0 + \Delta\), and we can cut it off after any term.

• Linear approximation (\(n = 1\)) is the tangent line. If a line \(y = mx + c\) passes through \((x_0, y_0)\), then subtracting \(y_0 = mx_0 + c\) from \(y_1 = mx_1 + c\) gives \(y_1 = y_0 + m(x_1 - x_0)\), and here the slope \(m\) is the derivative \(\mathcal{L}'(w_0)\). We pretend the curve is a straight line in a small neighbourhood of \(w_0\) and read the new value off that line.
• Quadratic approximation (\(n = 2\)) is a parabola \(y = ax^2 + bx + c\) whose coefficients come from the first and second derivatives. In the same neighbourhood it hugs the curve more closely than the line.

The lecture's interactive Taylor-series demo makes four points:

1. Small neighbourhoods work. A parameter \(\epsilon\) sets the neighbourhood around \(w_0\). For small \(\epsilon\) the tangent line is close to the curve; move far from \(w_0\) and the value read off the line is way off.
2. A large window can still work where the curve is gentle 16:59: where the slope is small the line stays close over a wide window, and where the slope is high the same window gives a big gap. With a small enough window the approximation is good everywhere.
3. Higher degree, better fit 19:04: for \(\sin w\), degree 2 beats degree 1, degree 3 is good over a much larger neighbourhood, and by degree 7 or 8 the fit is almost perfect. Keeping every term rebuilds the function exactly.
4. It works in several variables. For \(\mathcal{L}(w, b)\) (the demo uses \(0.5\sin x + 0.25\sin y\)) the linear approximation is a plane touching the surface, good in a small neighbourhood. The quadratic approximation stays good over a larger one.

The Taylor approximation is trustworthy only in a small neighbourhood, which is exactly why the step \(\eta\) must be small.

Write \(\mathbf{u} = \Delta\boldsymbol\theta\) for the direction we want to choose. The Taylor series around the current \(\boldsymbol\theta\), with the small step \(\eta\mathbf{u}\) keeping us in the neighbourhood where it is accurate:

Two new objects appear.

• The gradient \(\nabla_{\boldsymbol\theta}\mathcal{L}(\boldsymbol\theta)\). For \(y = x^2\) the derivative is \(dy/dx = 2x\). For a function of several variables, such as \(y = w^2 + b^2\), take a partial derivative with respect to each variable while treating the others as constants: \(\partial y/\partial w = 2w\) and \(\partial y/\partial b = 2b\). The gradient is the vector that collects them, here \([2w, 2b]\). So \(\nabla_{\boldsymbol\theta}\mathcal{L}(\boldsymbol\theta) = [\partial\mathcal{L}/\partial w,\ \partial\mathcal{L}/\partial b]\), and \(\mathbf{u}^\top\nabla_{\boldsymbol\theta}\mathcal{L}\) is the dot product of the step direction with it.
• The Hessian \(\nabla^2_{\boldsymbol\theta}\mathcal{L}(\boldsymbol\theta)\). The gradient of the gradient: differentiate each entry of the gradient with respect to each parameter. The derivative of an \(m\)-dimensional vector with respect to an \(n\)-dimensional vector is an \(m \times n\) matrix, so here it is \(2 \times 2\). The next term would need a third-order derivative, and so on.

Neither the Hessian nor anything after it is needed. \(\eta\) is small, so \(\eta^2, \eta^3, \dots\) are smaller still and those terms can be dropped 9:41. The Taylor demo showed that a linear approximation is good enough when the step is small.

The move \(\eta\mathbf{u}\) is favourable only if the new loss is less than the old one, \(\mathcal{L}(\boldsymbol\theta + \eta\mathbf{u}) - \mathcal{L}(\boldsymbol\theta) < 0\). By the approximation this difference is \(\eta\,\mathbf{u}^\top\nabla_{\boldsymbol\theta}\mathcal{L}(\boldsymbol\theta)\), and \(\eta\) is positive, so the condition is a condition on \(\mathbf{u}\) alone:

Many directions satisfy it. We want the one that satisfies it best, the most negative value, because that is the largest drop in loss. What is the range of this dot product? Let \(\beta\) be the angle between \(\mathbf{u}\) and \(\nabla_{\boldsymbol\theta}\mathcal{L}(\boldsymbol\theta)\):

Multiply through by \(k = \lVert\mathbf{u}\rVert\,\lVert\nabla_{\boldsymbol\theta}\mathcal{L}(\boldsymbol\theta)\rVert\):

So \(\mathcal{L}(\boldsymbol\theta + \eta\mathbf{u}) - \mathcal{L}(\boldsymbol\theta) \approx \eta\,k\cos\beta\), which is most negative when \(\cos\beta = -1\), that is \(\beta = 180°\) 14:49.

Moving opposite the gradient, by a small conservative step \(\eta\) (small because the Taylor series only holds in a small neighbourhood), gives the parameter update rule:

The subscript \(t\) matters. The partial derivative is first a formula (for \(w^2 + b^2\) it is \([2w, 2b]\)), and substituting the current values turns it into a vector of two real numbers that can be added and subtracted. We now have a principled way of moving in the \((w, b)\) plane instead of guessing.

1. gradient_descent()
2. \(t \leftarrow 0\); max_iterations \(\leftarrow 1000\).
3. Initialize \(w, b\) randomly.
4. While \(t <\) max_iterations: \(w_{t+1} \leftarrow w_t - \eta\nabla w_t\); \(b_{t+1} \leftarrow b_t - \eta\nabla b_t\); \(t \leftarrow t + 1\).

The hope is that after the maximum number of iterations we are very close to the minimum.

A 1-D game: land in a minimum M6 · 1:32

The slides set this up as a game. A function \(f(w)\) with several local minima is given; start at \(w = 1.2\) with \(\eta = 1\), read off the gradient, and type in \(w - \eta\,dw\) yourself.

• Step 1: \(\eta\,dw = 0.31\), so \(w = 1.2 - 0.31 = 0.89\). This heads toward a minimum.
• Step 2: at 0.89, \(\eta\,dw = 1.25\), so \(w = 0.89 - 1.25 = -0.36\). This overshoots: the point crosses the minimum and lands on the other side.

The lecture leaves two lessons to discover by playing (they return in Lecture 5). If \(\eta\) is too large the step jumps over the minimum, and the loss can even go up; the Taylor argument only promises a decrease for small steps. And the starting point decides which minimum you reach: start on one slope and you roll into the minimum below it.

Everything in the algorithm is known except \(\nabla w\) and \(\nabla b\), so derive them for the toy network. Assume first there is only one point \((x, y)\) to fit, and use

The factor \(\tfrac12\) is for convenience: scaling every loss value by the same positive number does not move the minimizer, and the \(\tfrac12\) will cancel a 2.

By the chain rule, and since \(y\) is a constant given by the data:

Now the derivative of the sigmoid itself. It is 1 over a function of \(w\), so its derivative is \(-1\) over that function squared, times the derivative of the function (the constant 1 in the denominator contributes nothing):

The first factor is \(f(x)\). The second is \(1 - f(x)\), because \(1 - \frac{1}{1 + e^{-z}} = \frac{1 + e^{-z} - 1}{1 + e^{-z}} = \frac{e^{-z}}{1 + e^{-z}}\). The two minus signs (\(-1\) and \(-x\)) cancel. Substituting back:

With several points the loss is a sum over them, and the derivative of a sum is the sum of the derivatives. For \(\nabla b\) everything is the same except the last factor: \(\partial(wx + b)/\partial b = 1\) instead of \(x\).

The slide's Python code puts the pieces together. It starts at \(w = b = -2\) with \(\eta = 1\) and runs 1000 epochs; each epoch adds up the gradient contributions of both points and then makes one update.

import numpy as np

X = [0.5, 2.5]
Y = [0.2, 0.9]

def f(x, w, b):
    return 1 / (1 + np.exp(-(w*x + b)))

def error(w, b):
    err = 0.0
    for x, y in zip(X, Y):
        fx = f(x, w, b)
        err += (fx - y) ** 2
    return 0.5 * err

def grad_b(x, w, b, y):
    fx = f(x, w, b)
    return (fx - y) * fx * (1 - fx)

def grad_w(x, w, b, y):
    fx = f(x, w, b)
    return (fx - y) * fx * (1 - fx) * x

def do_gradient_descent():
    w, b, eta, max_epochs = -2, -2, 1.0, 1000
    for i in range(max_epochs):
        dw, db = 0, 0
        for x, y in zip(X, Y):
            dw += grad_w(x, w, b, y)
            db += grad_b(x, w, b, y)
        w = w - eta * dw
        b = b - eta * db

In the demo the point starts far from the low region and moves slowly at first. The lecturer asks why it is slow in some regions and fast in others, and leaves the answer for later (Lecture 5). A hint, not given in the lecture: at \((-2, -2)\) both outputs are close to 0, so the factor \(f(1 - f)\) in each gradient is tiny. After 100 iterations the loss is about 0.02, and running longer keeps lowering it. Unlike the guesswork, the loss never goes up: every step moves downhill on the error surface.

Later in the course gradient descent is studied in much more detail, with many variants (Lecture 5). For now it is enough to have an algorithm for learning the parameters of a sigmoid neuron, along with the maths behind it, its geometric picture and code. The cycle from Lecture 2 is complete: a neuron model (perceptron, now sigmoid), a learning algorithm (perceptron learning, now gradient descent), its behaviour on errors. Next in that cycle comes representation power 23:04.

The same template as for perceptrons. Lecture 2 showed by construction that a network of perceptrons with one hidden layer represents any Boolean function exactly, using up to exponentially many hidden perceptrons, each one specializing in one input. The analogous statement for sigmoid neurons changes two words: represent becomes approximate, and Boolean becomes continuous.

Any desired precision is meant in the usual mathematical sense: you choose a tolerance \(\epsilon\) as small as you like, 0.0001 or a hundred zeros followed by a one, and a network can be built to meet it.

Standard fine print, not stated in the lecture: the guarantee is for inputs in a bounded region, such as a box of input values.

This is a very powerful result and much of the appeal of deep learning. Take the oil example: the true \(f\) from salinity, pressure, temperature, marine diversity and so on is unknown and complicated, yet a network exists whose output is close to it. Smaller \(\epsilon\) needs more neurons, and the practical catch is that the hidden layer may need a huge number of them. The lecture mentions that adding a second layer can bring the count per layer down from exponential to something manageable, a point left for an assignment or quiz 6:36.

The actual proof is difficult. The lecture gives an illustrative proof instead, in the same spirit as the proof by construction for perceptrons, based on the illustration in chapter 4 of the online book at neuralnetworksanddeeplearning.com.

Start from an arbitrary 1-D function \(f(x)\), not \(\sin x\), \(x^2\) or anything standard. Can a network of neurons represent it to a desired precision?

Observe first that such a function can be approximated by a row of tower functions: rectangles of some width placed side by side, each as high as the function there. Their sum, the outline of the towers, follows the curve. With a few wide towers the approximation is poor: at some \(x\) the tower top is far from \(f(x)\). With more, narrower towers it gets better and better, and there is always room to make them narrower still. This observation has nothing to do with neural networks yet.

The appealing part is that every tower is the same shape. They differ only in height and in position along the x-axis, so all of them are shifted and scaled copies of one basic tower. (The slide names this the sifting property of the tower function, as for an impulse or delta function.) Learn to build one tower and you can build them all.

Now imagine a black box, a tower maker, that takes the input \(x\) and produces one tower. Use as many as needed, each making a tower of the right height at the right place, and a simple network that adds their outputs approximates \(f\). Thinner towers give a better approximation, which will mean more neurons. This is the cartoon version of the network; what remains is to find out what is inside the black box.

Take two sigmoid neurons on the input \(x\): \(h_{11} = \sigma(w_1 x + b_1)\) and \(h_{12} = \sigma(w_2 x + b_2)\). The lecture's demo varies their parameters:

• With \(w_1 = 0\) and \(b_1 = 0\), \(h_{11} = 1/(1 + e^0) = 0.5\) for every \(x\): a flat line.
• Make \(w_1\) nonzero and the S appears, gently at first. As \(w_1\) grows the slope gets steeper, until the sigmoid is almost a step function: with a very large \(w\) the logistic function recovers the step.
• \(b\) moves the curve left or right: it sets the crossover point where the output goes from 0 to 1, at \(x = -b/w\) (where \(wx + b = 0\)).

Now define \(h_{21} = h_{11} - h_{12}\), with \(h_{11}\) stepping up before \(h_{12}\). Left of both steps it is \(0 - 0 = 0\). Right of both it is \(1 - 1 = 0\). Between them it is \(1 - 0 = 1\). That is a tower. The larger the \(w\)'s, the closer it is to a perfect rectangle, and the gap between the crossover points sets its width. A very narrow tower made with only moderately large \(w\) also loses height, because neither sigmoid finishes climbing.

As a network: \(x\) feeds two sigmoid neurons, and their outputs reach \(h_{21}\) with weights \(+1\) and \(-1\). The slide asks what activation \(h_{21}\) should have. None is needed: it is linear, just the weighted sum \(\sum_i w_i h_{1i}\) with the hand-set weights \(w_1 = 1\), \(w_2 = -1\). This is the tower maker.

What if there is more than one input? Back to oil: decide whether there is oil at a location on the ocean bed (yes/no) from two factors, salinity \(x_1\) and pressure \(x_2\). Each known location is a point in the \((x_1, x_2)\) plane, labelled oil or no oil. The data shows that \(y\) is a complex function of \(x_1\) and \(x_2\): no line separates the classes, but a function that is 1 on the oil regions and 0 elsewhere would. To approximate it we want 2-D towers, which can again be placed side by side. Once 2-D is clear, the same idea extends to 3, 4 or \(n\) dimensions.

A sigmoid neuron with two inputs is

• Vary \(w_1\) with \(w_2 = 0\), \(b = 0\): the surface is flat at 0.5, then an S along the \(x_1\) direction that gets steeper until it is a step.
• Vary \(w_2\) with \(w_1 = 0\), \(b = 0\): the same, along the \(x_2\) direction.
• Vary \(b\) with \(w_1 = k\) (large), \(w_2 = 0\): the step slides along \(x_1\); \(b\) sets where the crossover happens.

Subtract two such steps along \(x_1\) with different \(b\), exactly as in 1-D: \(h_{11} - h_{12}\). Where both are 0 the result is 0, where both are 1 it is 0, and in between it is 1. This is still not a tower. It is a wall-like strip, a tower open on two sides 8:14.

Two more neurons, \(h_{13}\) and \(h_{14}\), with the large weight on \(x_2\) instead (and 0 on \(x_1\)), give a strip in the other direction. Adding the two strips closes all four walls but produces three levels 15:22: 0 where neither strip is, 1 on the arms where exactly one strip is, and 2 in the centre where they cross. The tower we want sits on top, at level 2, surrounded by scaffolding at level 1.

To keep only the top, pass the sum through one more sigmoid whose switch-over sits between levels 1 and 2. Everything at level 0 or 1 maps to about 0, and only the centre, at level 2, maps to about 1. The result is a tower closed on all sides. (The lecture describes the switch-over as being at 1; any value strictly between 1 and 2, such as 1.5, separates the levels.)

The full network: \(x_1, x_2\) feed four sigmoid neurons, two steep in each direction; their outputs are summed with weights \(+1, -1, +1, -1\); the sum goes through a thresholding sigmoid 17:27. Compared with 1-D, the middle layer has grown from two neurons to four. Many such 2-D towers, placed side by side and made thinner, approximate the 2-D function as closely as we like.

One caveat the lecturer points out 18:28: this construction uses two layers of sigmoids (the four neurons, then the threshold), while the theorem promises a single hidden layer. The single-layer argument exists but is more complicated; the illustration shows that a slightly deeper network can approximate any 2-D function to arbitrary precision.

Why this matters: in the real world the true \(f\) can be as arbitrary as you like, and the theorem says a neural network can approximate it to a desired precision if it has enough neurons. Such a network is just a composite function of \(\mathbf{x}\), built from sigmoids and weighted sums, which is exactly the kind of \(\hat{f}\) that machine learning looks for. The count of neurons is still large, and later lectures deal with that. The lecture ends with a question: one input needed 2 neurons per tower and two inputs needed 4, so how many does an \(n\)-dimensional tower need? (See Q8 below.) Lecture 4 continues with deep feedforward networks and backpropagation.

Sigmoid neuron

A neuron whose output is the logistic function \(1/(1 + e^{-\mathbf{w}^\top\mathbf{x}})\): smooth, in \((0, 1)\), differentiable.

Step function

The perceptron's output as a function of \(z\): 0 below \(-w_0\), 1 above it; not differentiable at the jump.

Model

Our assumed approximation \(\hat{f}\) of the unknown relation \(y = f(\mathbf{x})\), with parameters to learn.

Parameters

The quantities learned from data; here \(w\) and \(b\), collected as \(\boldsymbol\theta = [w, b]\).

Bias \(b\)

The new name for \(w_0\); with large \(w\), a sigmoid's step sits at \(x = -b/w\).

Loss function

The objective that measures how far predictions are from the true values; learning minimizes it.

Squared error loss

\(\sum_i (\hat{y}_i - y_i)^2\): signs cannot cancel, and it is differentiable.

Error surface

The loss plotted over the parameter plane \((w, b)\).

Taylor series

A polynomial approximation of a smooth function around a point; the linear one is accurate only nearby.

Gradient

The vector of partial derivatives \(\nabla_{\boldsymbol\theta}\mathcal{L}\); its negative is the direction of steepest descent.

Hessian

The matrix of second partial derivatives, the gradient of the gradient; dropped in the derivation.

Gradient descent

\(\boldsymbol\theta_{t+1} = \boldsymbol\theta_t - \eta\nabla\mathcal{L}(\boldsymbol\theta_t)\), with a small learning rate \(\eta\).

Universal approximation theorem

One hidden layer with enough sigmoid neurons approximates any continuous function within any \(\epsilon\).

Tower function

The difference of two steep sigmoids, \(\sigma(w_1x + b_1) - \sigma(w_2x + b_2)\); sums of towers rebuild a function.