McCulloch–Pitts neurons, perceptrons and multilayer perceptrons

The most basic unit of a deep neural network is the artificial neuron. The name and the idea come from biology. Biological neurons (neural cells, or neural processing units) do the processing in the brain, and the hope is that artificial ones can do useful computation in the same spirit. A biological neuron has four parts that matter here 1:32:

• Dendrites receive signals from other neurons.
• Synapses are the points where one neuron connects to another.
• The soma processes the information, a little like a central processing unit.
• The axon carries this neuron's output on to other neurons.

A cartoon of how this works 2:32: while you watch a comedy, your eyes and ears relay information to neurons. A neuron that is excited enough gets activated and signals others, and eventually a response such as laughter results. In reality no single neuron does all this. The brain is a massively parallel, interconnected network of about \(10^{11}\) (100 billion) neurons, arranged in layers 3:33. The sense organs feed the lowest layer; some of its neurons fire and activate neurons in the next layer, and so on until a physical response is produced.

The network also divides the work: each neuron has a role or responds to a particular stimulus. In the slides' simplified illustration, one neuron fires if the visual is funny, one if the speech style is funny and one if the text is funny. A fourth neuron fires if at least two of these three fired 5:06. That fourth neuron is exactly the kind of unit Module 2.2 formalizes.

The visual cortex shows the layered organization. Information starts at the retina and, following the arrows in the slide's diagram, passes through a series of areas (V1, V2 and on to AIT) that form a hierarchy, ending in an action such as moving a hand. The hierarchy goes from identifying simple visual forms to high-level objects. A simplified version, an idea the slides credit to Hugo Larochelle's lecture slides 6:41: layer 1 detects edges and corners, layer 2 groups them into features such as a nose, eyes and a mouth, and layer 3 detects high-level objects such as a face. Each layer works on a more abstract version of the input than the layer before.

The lecture's disclaimer 8:13: this is an overly simplified account of the brain, but it is enough for the course. The takeaways are a massively parallel network of interconnected neurons, arranged in layers, in which neurons specialize and information flows from one layer to the next. Layered architectures appear throughout the course.

In 1943 McCulloch, a neuroscientist, and Pitts, a logician, proposed a highly simplified computational model of the neuron. Its inputs \(x_1, \dots, x_n\) are boolean, and so is its output \(y \in \{0, 1\}\). For deciding whether to go to a movie, inputs could be "is it raining outside?" or "am I feeling well?". The unit has two parts: \(g\) aggregates the inputs, and \(f\) takes a decision based on that aggregate.

Each input is either excitatory or inhibitory. An inhibitory input is a veto: if it is 1, the output is 0 whatever the other inputs are. The lecture's example is "do I have a high temperature?". If yes, I am not going to the movie, however good the movie or the weather. Excitatory inputs are the ordinary ones that push toward a positive decision. Inhibitory inputs appear in the next few examples and then mostly disappear from the course.

Because each \(x_i\) is 0 or 1, \(g(\mathbf{x})\) simply counts the inputs that are on. The threshold says how many favourable conditions are needed. Someone who goes out only for very specific reasons needs many inputs on and has a high \(\theta\); someone looking for any excuse to go out has a low one. The "at least 2 of 3" neuron of Module 2.1 is this unit with \(n = 3\) and \(\theta = 2\).

An MP unit takes boolean inputs and gives a boolean output, so it computes a boolean function. To implement (or represent) a boolean function means that for every possible input the unit's output equals the function's value 8:12. A unit implements AND of two inputs if it outputs 0 for \((0,0)\), \((0,1)\) and \((1,0)\), and 1 for \((1,1)\), exactly as the truth table says. This idea of a model implementing a function recurs through the course.

The function fixes how many inputs there are. What we choose is \(\theta\), and which inputs are inhibitory.

• AND of three inputs, \(\theta = 3\). The output must be 1 only when all three inputs are 1, that is when \(g(\mathbf{x}) = 3\). For \((1, 1, 0)\), \(g = 2 < 3\) and the output is 0, as AND requires.
• OR of three inputs, \(\theta = 1\) 11:46. The output must be 1 when at least one input is 1. Only \((0, 0, 0)\) has \(g = 0 < 1\).
• \(x_1\) AND !\(x_2\): \(x_2\) inhibitory, \(\theta = 1\). If \(x_2 = 1\) the veto gives 0. Otherwise the output is 1 exactly when \(x_1 = 1\).
• NOR: both inputs inhibitory, \(\theta = 0\). Any input at 1 vetoes the output. For \((0,0)\), \(g = 0 \ge 0\) and the unit fires.
• NOT: one inhibitory input, \(\theta = 0\). Input 1 is vetoed, giving 0. Input 0 gives \(g = 0 \ge \theta = 0\), so the unit fires and outputs 1. The threshold 0 is what makes the unit fire when nothing is on.

Can every boolean function be represented by a single MP unit? So far we have only done arithmetic: sum the inputs and compare with \(\theta\). To answer the question, look at what the unit does geometrically 13:19.

Take the two-input OR unit, which fires when \(x_1 + x_2 \ge 1\). Replace the inequality by an equality for a moment: \(x_1 + x_2 = 1\) is a line, passing through \((0, 1)\) and \((1, 0)\). The plane has many other points, but only four matter, \((0,0)\), \((0,1)\), \((1,0)\) and \((1,1)\), because the inputs are boolean.

The unit outputs 1 for every input with \(x_1 + x_2 \ge 1\), the line's positive half-space, and 0 for every input with \(x_1 + x_2 < 1\), its negative half-space. For OR, three inputs lie in the positive half-space and only \((0, 0)\) in the negative one. (The lecture first says "on or above the line" and then replaces it by half-spaces. Which side counts as "above" depends on how the line is drawn, while the inequality is unambiguous. Module 2.4 has an example where the two disagree.)

• AND 17:30: \(x_1 + x_2 \ge 2\). The line \(x_1 + x_2 = 2\) passes through \((1, 1)\), the only input in the positive half-space.
• Tautology (always on) 18:31: the output must be 1 even when the sum is 0, so \(\theta = 0\). The line \(x_1 + x_2 = 0\) passes through \((0, 0)\) and all four inputs are in the positive half-space.

So a single MP unit divides its inputs into two groups with a linear boundary: output 1 on one side (line included), output 0 on the other. With three inputs the line becomes a plane 20:04. For three-input OR, the plane \(x_1 + x_2 + x_3 = 1\) has \((0, 0, 0)\) on one side and the other seven corners of the cube on it or beyond it. With \(n\) inputs it is a hyperplane, and the picture is the same. (All of this assumes no inhibitory inputs.)

Two questions follow. Are there boolean functions that are not linearly separable, and what can be done about them? Module 2.7 answers the first with XOR, and Module 2.8 the second.

The MP neuron leaves four questions open, and the rest of the lecture works through them:

1. What about non-boolean (say, real) inputs?
2. Do we always need to hand-code the threshold?
3. Are all inputs equal? What if we want to give some inputs more weight (importance) than others? In an MP unit every input counts the same, while real decisions usually hinge on a few important factors. In the brain, too, connections differ in strength.
4. What about functions that are not linearly separable?

Frank Rosenblatt, an American psychologist, proposed the classical perceptron model in 1958. Minsky and Papert refined and carefully analysed it in 1969, and their version is what the course calls the perceptron. It is a more general computational model than the MP neuron. The main differences are numerical weights on the inputs and a mechanism for learning those weights from data, and the inputs are no longer limited to boolean values.

The perceptron fires when the weighted sum of the inputs reaches the threshold 4:37. An input with a small value but a large weight can still contribute a lot.

The second form only moves \(\theta\) to the left side. The more accepted convention goes one step further. Add an imaginary input \(x_0 = 1\) whose weight is \(w_0 = -\theta\). Then \(w_0 x_0 = -\theta\), so a sum that starts at \(i = 0\) already contains the threshold:

The lecture answers three questions with one example: predicting whether you will like a movie. Base the decision on three inputs, binary for simplicity:

• Why boolean functions? Many real problems can be phrased with boolean inputs (is the actor Matt Damon, is the actor Christian Bale, is the genre thriller, is it comedy, is the director Nolan, …) and a boolean output (will I like it). That is why the lecture keeps studying them.
• Why weights? Your past viewing (the data) may show that you liked almost every Nolan film. Then isDirectorNolan deserves a high weight compared to the other inputs. Even when the actor is not Matt Damon and the genre is not thriller, that one input can push the weighted sum across the threshold. Weights encode importance.
• Why is \(w_0 = -\theta\) called the bias? It represents the prior, or prejudice, of the decision maker before looking at the inputs. A very negative \(w_0\) is inertia that the inputs must overcome, so many (or heavily weighted) inputs must be on. A movie buff who watches anything, whatever the genre, actor or director, has \(\theta = 0\) 11:48. A selective viewer who only watches thrillers starring Matt Damon and directed by Nolan has, with all weights equal to 1, \(\theta = 3\): only with all three inputs on does \(-3 + 1 + 1 + 1 = 0\) reach the threshold.

The weights \(w_1, \dots, w_n\) and the bias \(w_0\) depend on the data, here the viewer's history. How to learn them is the subject of Module 2.5 for the perceptron, and of the following lectures in a broader setting.

What kind of functions can a perceptron implement, and is that any different from an MP neuron? Written in the same form, an MP unit without inhibitory inputs fires when \(-\theta + \sum_{i=1}^{n} x_i \ge 0\), and a perceptron when \(\sum_{i=0}^{n} w_i x_i \ge 0\). The only difference is the weights. Both split the input space into two halves with a linear boundary, so a single perceptron, like a single MP unit, can only implement linearly separable functions. What is new is that the weights, the threshold included, can be learned, and the inputs can be real.

Revisit OR with a perceptron. Each row of the truth table gives one inequality on the weights:

One solution is \(w_0 = -1\), \(w_1 = 1.1\), \(w_2 = 1.1\) 16:22. Check: \(-1 < 0\), \(1.1 \ge 1\), \(1.1 \ge 1\) and \(2.2 \ge 1\). The separating line is \(-1 + 1.1x_1 + 1.1x_2 = 0\): \((0, 0)\) falls in the negative half-space and the other three inputs in the positive one. Many other solutions exist. In the video's demo, moving the line shows valid configurations and invalid ones with one or two errors 18:28. Changing \(w_0\) moves the line's intercept on the \(x_2\) axis, while \(w_1\) and \(w_2\) tilt it.

The slides suggest doing the same for an MP neuron, where \(\theta\) is the only unknown. Working it out: \((0, 0) \mapsto 0\) needs \(0 < \theta\); \((0, 1)\) and \((1, 0) \mapsto 1\) need \(1 \ge \theta\); \((1, 1) \mapsto 1\) needs \(2 \ge \theta\). So any \(0 < \theta \le 1\) works, and \(\theta = 1\) is the value used earlier. Finding parameters is not new, then. For the MP unit the answer was obvious; with weights it becomes hard to do by hand, which is why a learning algorithm is needed.

The error of a configuration is the number of inputs it gets wrong. To make it concrete, the video takes the AND function, fixes the threshold at \(w_0 = -1\), and tries different values of \(w_1\) and \(w_2\). The output is 1 when \(-1 + w_1x_1 + w_2x_2 \ge 0\).

• \((w_1, w_2) = (-1, -1)\). The line is \(-1 - x_1 - x_2 = 0\). Every input gives \(-1 - x_1 - x_2 < 0\), so all four lie in the negative half-space, even though they sit "above" the line as drawn 2:04. This is the example promised earlier: the side is decided by the inequality, not by "above" or "below". Only \((1, 1)\), which should give 1, is wrong.
• \((1.5, 0)\). Now \((1, 0)\) scores \(-1 + 1.5 = 0.5 \ge 0\) and is labelled 1, but AND gives 0. A different single error.
• \((10, -10)\) 4:40. \((1, 0)\) scores 9 (wrong) and \((1, 1)\) scores \(-1\) (wrong). Only \((0, 0)\) and \((0, 1)\) are on the correct side.

So with \(w_0\) fixed, the error is a function of \((w_1, w_2)\). The slide plots it over \(w_1, w_2 \in [-4, 4]\), with the error on the vertical axis: the error surface. Since it counts mistakes, it takes only integer values and looks like a staircase of flat steps. Any \((w_1, w_2)\) on the step at height 0 implements the function, so we could pick one from the plot. The slides state the error rule for OR: an error occurs if \((x_1, x_2) = (0, 0)\) but \(w_0 + w_1x_1 + w_2x_2 \ge 0\), or if \((x_1, x_2) \ne (0, 0)\) but \(w_0 + w_1x_1 + w_2x_2 < 0\).

The video says the error can go up to 3 7:42, never 4, because one of the two kinds of point is always on the correct side. With \(w_0 = -1\) fixed, \((0, 0)\) is always correct, and for AND the count actually stops at 2: errors on \((0, 1)\) and \((1, 0)\) need \(w_2 \ge 1\) and \(w_1 \ge 1\), which put \((1, 1)\) on the correct side. A count of 3 occurs for OR, as in the right panel.

Reading the zero-error region off a plot works for two weights. With \(n\) inputs there are \(n + 1\) weights and the surface cannot even be drawn 8:43. We want an algorithm that, whenever the data is linearly separable, finds weights that put every positive point and every negative point on its correct side. That is the perceptron learning algorithm. Error surfaces will keep coming back in the course, starting with gradient descent in Lecture 3.

Implementing boolean functions is not very interesting in itself. The real use of a perceptron is as a binary classifier. Back to the movies: we are given a list of \(m\) movies, each with a label saying whether the user liked it. Each movie is represented by \(n\) features, some boolean (isActorDamon, isGenreThriller, isDirectorNolan, …) and some real-valued (imdbRating, criticsRating, both scaled to 0–1). Assume the data is linearly separable. We want the perceptron to find the separating plane, that is, to learn \(w_0, w_1, \dots, w_n\) from the data.

1. Algorithm: Perceptron Learning Algorithm
2. \(P \leftarrow\) inputs with label 1; \(\;N \leftarrow\) inputs with label 0.
3. Initialize \(\mathbf{w}\) randomly.
4. While not converged: pick a random \(\mathbf{x} \in P \cup N\).
5. If \(\mathbf{x} \in P\) and \(\sum_{i=0}^{n} w_i x_i < 0\), then \(\mathbf{w} \leftarrow \mathbf{w} + \mathbf{x}\).
6. If \(\mathbf{x} \in N\) and \(\sum_{i=0}^{n} w_i x_i \ge 0\), then \(\mathbf{w} \leftarrow \mathbf{w} - \mathbf{x}\).

Here \(\mathbf{x} = [x_0, x_1, \dots, x_n]\) includes the constant \(x_0 = 1\), and \(\mathbf{w} = [w_0, w_1, \dots, w_n]\) includes the bias. The algorithm has converged when all inputs are classified correctly 3:34: \(\sum_i w_i x_i \ge 0\) for every \(\mathbf{x} \in P\) and \(\sum_i w_i x_i < 0\) for every \(\mathbf{x} \in N\). Then neither "if" ever fires again. In practice we cycle through the data until a full pass needs no correction.

The two "if"s fire exactly when the current \(\mathbf{w}\) makes a mistake: a positive point scored below 0, or a negative point scored 0 or more. Why should adding the point, or subtracting it, help 6:42? That takes a little linear algebra and geometry.

With \(\mathbf{w} = [w_0, w_1, \dots, w_n]\) and \(\mathbf{x} = [1, x_1, \dots, x_n]\), the perceptron's sum is a dot product:

We are looking for the line \(\mathbf{w}^\top\mathbf{x} = 0\) that divides the input space into two halves. (In two dimensions it is \(w_0 + w_1x_1 + w_2x_2 = 0\), the familiar \(y = mx + c\) in other notation.) The angle \(\alpha\) between \(\mathbf{w}\) and a point \(\mathbf{x}\) satisfies

• Every point on the line has \(\mathbf{w}^\top\mathbf{x} = 0\), so \(\cos\alpha = 0\) and \(\alpha = 90^\circ\). Since \(\mathbf{w}\) is perpendicular to every point on the line, it is perpendicular to the line itself.
• Points in the positive half-space have \(\mathbf{w}^\top\mathbf{x} > 0\), so \(\cos\alpha > 0\): the angle is less than \(90^\circ\).
• Points in the negative half-space have \(\mathbf{w}^\top\mathbf{x} < 0\): the angle is greater than \(90^\circ\).

So a correct \(\mathbf{w}\) makes an acute angle with every positive point and an obtuse angle with every negative point. Now revisit the updates 12:51.

A positive point with \(\mathbf{w}^\top\mathbf{x} < 0\). Its angle with \(\mathbf{w}\) is more than \(90^\circ\) and should be less. The modest goal is to make the angle smaller, even if it cannot get below \(90^\circ\) in one step. With \(\mathbf{w}_{\text{new}} = \mathbf{w} + \mathbf{x}\):

Since \(\mathbf{x}^\top\mathbf{x} = \lVert\mathbf{x}\rVert^2 > 0\), \(\cos\alpha_{\text{new}} > \cos\alpha\), so \(\alpha_{\text{new}} < \alpha\): \(\mathbf{w}\) turns toward \(\mathbf{x}\), which is what we want.

A negative point with \(\mathbf{w}^\top\mathbf{x} \ge 0\). Its angle is less than \(90^\circ\) and should be more. With \(\mathbf{w}_{\text{new}} = \mathbf{w} - \mathbf{x}\):

so \(\cos\alpha_{\text{new}} < \cos\alpha\) and \(\alpha_{\text{new}} > \alpha\): \(\mathbf{w}\) turns away from \(\mathbf{x}\).

The "\(\propto\)" hides the denominator \(\lVert\mathbf{w}_{\text{new}}\rVert\,\lVert\mathbf{x}\rVert\), which also changes with the update 15:24. The lecture sets it aside on purpose: the argument is meant as intuition, and the exact calculation is trickier. The convergence proof in Module 2.6 does not rely on it.

One worry remains 16:55. A correction for a positive point decreases its angle, and a later correction for a negative point increases that point's angle. What stops the algorithm from toggling like this forever? The intuition says nothing about it; Module 2.6 proves that it cannot happen when the data is linearly separable.

The slides run the algorithm on three positive points \(p_1, p_2, p_3\) and three negative points \(n_1, n_2, n_3\) in the plane. The random initial \(\mathbf{w}\) happens to be the worst possible: \(\mathbf{w}\cdot\mathbf{x} < 0\) for every positive point (angle above \(90^\circ\)) and \(\mathbf{w}\cdot\mathbf{x} \ge 0\) for every negative point, exactly the opposite of what we want. Points are then picked at random:

After step 12 every point has been checked since the last correction and none needed one, so the algorithm has converged. Each greedy step did what it promised for its own point. Whether such steps could, on other data, keep undoing each other is the question Module 2.6 settles.

The figure below runs the same algorithm on the four boolean inputs, with the bias input \(x_0 = 1\) included, so that \(\mathbf{w} = (w_0, w_1, w_2)\) and the line need not pass through the origin. Try OR and AND, then XOR.

With some faith in the algorithm and an intuition for why it works, the lecture turns to a formal proof of convergence.

This answers the toggling worry: the loop cannot run forever. Before the proof, three changes to the setup make the algebra simpler without changing the problem.

1. Negate the negative points. If \(\mathbf{x} \in N\), we need \(\mathbf{w}^\top\mathbf{x} < 0\), which is the same as \(\mathbf{w}^\top(-\mathbf{x}) > 0\). So \(-\mathbf{x}\) behaves like a positive point. Let \(N^-\) contain the negations of all points in \(N\) (the constant input is negated too, so \(-\mathbf{x} = [-1, -x_1, \dots, -x_n]\)) and let \(P' = P \cup N^-\). Now there is a single condition: \(\mathbf{w}^\top\mathbf{p} \ge 0\) for every \(\mathbf{p} \in P'\), and the algorithm needs only one "if".
2. Normalize every point 6:06. Replace each \(\mathbf{p}\) by \(\mathbf{p}/\lVert\mathbf{p}\rVert\), so that \(\lVert\mathbf{p}\rVert = 1\). This does not affect the solution, because \(\mathbf{w}^\top\frac{\mathbf{p}}{\lVert\mathbf{p}\rVert} \ge 0\) exactly when \(\mathbf{w}^\top\mathbf{p} \ge 0\).
3. Name a solution 7:07. The data is linearly separable, so a separating weight vector exists. Let \(\mathbf{w}^*\) be one, normalized so that \(\lVert\mathbf{w}^*\rVert = 1\). We do not know it, but the proof only uses the fact that it exists.

1. Algorithm: Perceptron Learning Algorithm, after the setup
2. \(P \leftarrow\) inputs with label 1; \(\;N \leftarrow\) inputs with label 0.
3. \(N^- \leftarrow\) negations of all points in \(N\); \(\;P' \leftarrow P \cup N^-\).
4. Initialize \(\mathbf{w}\) randomly.
5. While not converged: pick a random \(\mathbf{p} \in P'\) and set \(\mathbf{p} \leftarrow \mathbf{p}/\lVert\mathbf{p}\rVert\), so \(\lVert\mathbf{p}\rVert = 1\).
6. If \(\sum_{i=0}^{n} w_i p_i < 0\), then \(\mathbf{w} \leftarrow \mathbf{w} + \mathbf{p}\).

The second "if" of the original algorithm is no longer needed: by construction every point of \(P'\) has to end up on the positive side.

Notation: \(\mathbf{w}_t\) is the weight vector at time step \(t\), and \(\mathbf{w}_0\) (bold) is the random initial vector, not the bias \(w_0\). The points of \(P'\) are \(\mathbf{p}_1, \mathbf{p}_2, \dots\), all of unit length. \(k\) is the number of corrections made so far.

Step 1: the quantity to watch

Suppose that at time step \(t\) we inspect the point \(\mathbf{p}_i\) and find \(\mathbf{w}_t^\top\mathbf{p}_i \le 0\). (The proof uses \(\le\) here.) We make the correction \(\mathbf{w}_{t+1} = \mathbf{w}_t + \mathbf{p}_i\). Let \(\beta\) be the angle between \(\mathbf{w}^*\) and \(\mathbf{w}_{t+1}\). Since \(\lVert\mathbf{w}^*\rVert = 1\), it does not appear in the denominator:

The plan is to show that the numerator grows at least in proportion to \(k\), while the denominator grows at most like \(\sqrt{k}\).

Step 2: the numerator gains at least δ per correction

Compute the dot product of \(\mathbf{w}^*\) with every point of \(P'\) and let \(\delta\) be the smallest:

Expand the numerator by substituting the update, then bound the new term by \(\delta\):

\(\mathbf{w}_t\) itself came from an earlier correction, \(\mathbf{w}_t = \mathbf{w}_{t-1} + \mathbf{p}_j\) for some point \(\mathbf{p}_j\) of \(P'\) 11:47. The same argument applies to it:

We do not make a correction at every time step, only when the inspected point is misclassified 12:49. Steps without a correction leave \(\mathbf{w}\) unchanged and add nothing, so by time step \(t\) we have made \(k \le t\) corrections, each adding at least \(\delta\). By induction, all the way back to the initial vector:

Step 3: the squared denominator gains at most 1 per correction

Write the squared length as a dot product and expand:

The middle term is at most 0, because \(\mathbf{w}_t\cdot\mathbf{p}_i \le 0\) is what triggered the correction. Dropping it can only make the right side larger. The last term is 1 because the points are normalized:

The same holds at the previous correction, and so on back to the start. Each correction adds at most 1, the same counting argument as for \(\delta\):

Step 4: combine, and use cos β ≤ 1

The numerator is at least \(\mathbf{w}^*\cdot\mathbf{w}_0 + k\delta\) and the denominator at most \(\sqrt{\lVert\mathbf{w}_0\rVert^2 + k}\). A larger numerator over a smaller denominator gives a lower bound on the ratio:

The top of this bound grows like \(k\) and the bottom like \(\sqrt{k}\), so the bound grows in proportion to \(\sqrt{k}\) 18:29. If the algorithm never converged, \(k\) would grow without limit and so would this lower bound on \(\cos\beta\). But a cosine is at most 1 19:00. So \(k\) is bounded by some maximum number: only finitely many corrections can happen, and after the last one every point is classified correctly.

Not in the lecture: two details the slides leave implicit. (1) The argument needs \(\delta > 0\), or the numerator would not grow. It holds when \(\mathbf{w}^*\) puts every point of \(P'\) strictly on the positive side. With the definition above a point of \(P\) could lie exactly on the boundary, but for finitely many points, lowering the threshold slightly moves those points to the positive side while every point of \(N\) stays strictly negative, so such a \(\mathbf{w}^*\) always exists. The lower bound in step 4 also assumes its numerator is nonnegative, which is true once \(k\) is large, the only case that matters. (2) Starting from \(\mathbf{w}_0 = \mathbf{0}\), the bound reads \(\delta\sqrt{k} \le 1\), so \(k \le 1/\delta^2\). Since all points have unit length, \(\delta\) is the margin by which \(\mathbf{w}^*\) separates them. This is the standard form of the result (Novikoff, 1962), usually written \(k \le R^2/\gamma^2\) for points of length at most \(R\) and margin \(\gamma\).

Back to the four questions M6 · 20:03

The perceptron, and its learning guarantee, only work for linearly separable data. So which boolean functions are not linearly separable? The simplest is XOR, which is 1 when exactly one input is 1. Its truth table gives four inequalities:

The fourth condition contradicts the second and third. Adding those two gives \(w_1 + w_2 \ge -2w_0\), and since \(w_0 < 0\) by the first condition, \(-2w_0 > -w_0\). So \(w_1 + w_2 > -w_0\), the opposite of the fourth inequality. No weights satisfy all four, so no line puts \((0, 1)\) and \((1, 0)\) on one side and \((0, 0)\) and \((1, 1)\) on the other. The picture says the same: adjust the line as much as you like, the two pairs of diagonal corners cannot be split. Choose XOR in the perceptron figure to watch the learning algorithm cycle.

This matters because most real-world data is not linearly separable. It usually contains outliers: some people have every characteristic of those who like machine learning and still do not like it; in a locality where most people speak one language, a few speak another. And data can be non-separable without any outliers 4:06, for example one class in an inner circle and the other in a ring around it: no line separates them. A model that only handles linearly separable data is not acceptable in the long run. A single perceptron cannot deal with such data, but a network of perceptrons can, as Module 2.8 shows.

With two inputs there are four input combinations, and each can be mapped to 0 or 1, so there are \(2^4 = 2^{2^2} = 16\) boolean functions. They include the always-off function \(f_1\), the always-on function \(f_{16}\), AND (\(f_2\)) and OR (\(f_8\)). All of them are linearly separable except XOR (\(f_7\)) and its negation !XOR (\(f_{10}\)) 6:40.

With \(n\) inputs there are \(2^n\) input combinations and \(2^{2^n}\) boolean functions. How many of them are not linearly separable? The lecture notes that this is an unsolved problem in general, with no formula known; the point here is only that some of them are not, and that real data is messier still. (Not in the lecture: the counts are known for small \(n\) by enumeration. 104 of the 256 functions of three inputs are linearly separable, and 1,882 of the 65,536 functions of four inputs.)

The rest of the lecture stays with boolean inputs and outputs and asks how to handle functions that are not linearly separable. Real inputs and outputs come later in the course.

The aim of the last module is to implement any boolean function, linearly separable or not, with a network of perceptrons. The argument is a proof by construction: build the network and check that it works. The setup for two inputs:

• For this discussion, True \(= +1\) and False \(= -1\), instead of 1 and 0.
• There are 4 perceptrons. Each input is connected to all 4 with a fixed weight of \(-1\) (red edges on the slides) or \(+1\) (blue edges).
• The bias of each of the 4 is \(w_0 = -2\), so it fires only if the weighted sum of its inputs is at least 2.
• The 4 perceptrons are connected to an output perceptron by weights \(w_1, w_2, w_3, w_4\), which need to be learned. Its output \(y\) is the output of the network.

Terminology that stays with us for the rest of the course: the network has three layers. The input layer holds \(x_1, x_2\). The hidden layer holds the 4 perceptrons, whose outputs are \(h_1, h_2, h_3, h_4\); it is called hidden because it sits between input and output and the outside world sees only \(y\). The output layer holds the one output neuron. The red and blue edges are the layer 1 weights, and \(w_1, \dots, w_4\) are the layer 2 weights.

The claim 4:36: this one network can implement any boolean function of two inputs, linearly separable or not. For each of the 16 functions there are values of \(w_1, \dots, w_4\) for which the network's outputs match that function's truth table. It sounds astonishing for two reasons: some of the functions are not linearly separable, and the same template serves all 16. It is not astonishing once you see what the hidden layer does.

Each perceptron in the hidden layer fires for exactly one input, and no two fire for the same input. Take \(h_1\), whose weights are \((-1, -1)\). On the input \((-1, -1)\) its weighted sum is \((-1)(-1) + (-1)(-1) = 2\), and \(-2 + 2 = 0 \ge 0\), so it fires. On \((-1, +1)\) the sum is \(1 - 1 = 0\) and \(-2 + 0 < 0\); on \((+1, +1)\) it is \(-2\). So \(h_1\) fires only for \((-1, -1)\). The same check works for the others:

On any other input at least one product is \(-1\), so the weighted sum is 0 or \(-2\), below the required 2. Each hidden unit specializes in one input.

Let \(w_0\) be the bias of the output neuron, in the sense that it fires if \(\sum_{i=1}^{4} w_i h_i \ge w_0\). For each input exactly one \(h_i\) equals 1, so the output's aggregate is just that unit's weight. For XOR (0 here stands for False, that is \(-1\)):

So XOR needs \(w_1 < w_0\), \(w_2 \ge w_0\), \(w_3 \ge w_0\) and \(w_4 < w_0\). Unlike the single perceptron's four inequalities, these contain no contradiction, because each weight appears in exactly one condition and can be set on its own. For example \(w_0 = 1\), \(w_2 = w_3 = 1\) and \(w_1 = w_4 = 0\) (one choice among many; the slides leave the values open). Each \(w_i\) is responsible for one of the four possible inputs: four inputs, four degrees of freedom, and the weights never interact.

The same network represents the remaining 15 boolean functions too. Each gives a different set of four non-contradicting inequalities, one per weight, which can be satisfied by setting \(w_1, \dots, w_4\) appropriately.

With three inputs there are 8 possible inputs, so the hidden layer gets 8 perceptrons (bias \(-3\) on the slides), each firing for exactly one of them. Each of the 8 layer-2 weights \(w_1, \dots, w_8\) is responsible for one input and can be adjusted to give the desired output for it; again no two conditions share a weight. With \(n\) inputs the hidden layer needs \(2^n\) perceptrons, one per input combination, and the video gives each a bias of \(-n\). (A hidden unit whose \(\pm1\) weights match its input exactly gets a weighted sum of \(n\); on any other input at least one sign is wrong and the sum is at most \(n - 2\).)

• Sufficient, not necessary. A network of \(2^n + 1\) perceptrons always works, but it is not always needed: AND, for example, needs just one perceptron.
• The catch. As \(n\) increases, the number of hidden perceptrons grows exponentially. At \(n = 100\) the construction would need \(2^{100}\) of them.

Why care about boolean functions here? Return to predicting whether we like a movie. We are given our past movies, each with the values of the factors \(x_1, \dots, x_n\) we base the decision on and the label \(y\) (like or dislike). The \(p_i\) are the movies with output 1 and the \(n_j\) those with output 0:

With boolean factors this table is a truth table, and real data of this kind will usually not be linearly separable. The theorem says that a network of perceptrons exists, and its weights can be learned, such that for every given \(p_i\) or \(n_j\) the network's output equals its label: the positive and negative points are separated. The catch is again the exponential number of perceptrons.

Two problems remain open at the end of the lecture: the exponential number of perceptrons, and the hard linear decision each perceptron makes. Lecture 3 takes them up, starting with sigmoid neurons.

Artificial neuron

The basic unit of a neural network, inspired by the biological neuron: it aggregates its inputs and makes a decision.

McCulloch–Pitts (MP) neuron

Boolean inputs and output. Fires when \(g(\mathbf{x}) = \sum_i x_i \ge \theta\) and no inhibitory input is 1 (1943).

Thresholding logic

Deciding by comparing an aggregate of the inputs with a threshold \(\theta\), the thresholding parameter.

Inhibitory input

An input that forces the output to 0 whenever it is 1. All other inputs are excitatory.

Linearly separable

A line, plane or hyperplane puts every input with output 1 on one side and every input with output 0 on the other.

Positive half-space

The side of the line \(\mathbf{w}^\top\mathbf{x} = 0\) where \(\mathbf{w}^\top\mathbf{x} \ge 0\); the unit outputs 1 there. "Above the line" is not a reliable substitute.

Perceptron

Real inputs with weights: \(y = 1\) exactly when \(\sum_{i=0}^{n} w_i x_i \ge 0\). Rosenblatt 1958; refined by Minsky and Papert 1969.

Bias \(w_0\)

\(w_0 = -\theta\), the weight of the constant input \(x_0 = 1\). It encodes the prior, or prejudice, of the decision.

Error surface

The number of misclassified inputs as a function of the weights: integer-valued, with flat steps.

Perceptron learning algorithm

On a mistake, \(\mathbf{w} \leftarrow \mathbf{w} + \mathbf{x}\) for a positive point and \(\mathbf{w} \leftarrow \mathbf{w} - \mathbf{x}\) for a negative one, until every input is classified correctly.

Absolutely linearly separable

Some \(w_0, \dots, w_n\) put every point of \(P\) at \(\sum_{i=1}^n w_i x_i \ge w_0\) and every point of \(N\) below \(w_0\). Under it the perceptron makes finitely many corrections.

Hidden layer

A layer between the input and the output layer; its outputs \(h_1, h_2, \dots\) are not seen from outside.

Multilayer perceptron (MLP)

A network of perceptrons with an input layer, one or more hidden layers and an output layer.

Representation power

Which functions a network can represent. One hidden layer of \(2^n\) perceptrons suffices for every boolean function of \(n\) inputs.