The Brain, the Perceptron, and Why We Need Curves

Biological inspiration & the artificial neuron

A perceptron is a comically simplified neuron. Real neurons fire when chemistry tips them past a threshold; perceptrons fire when a weighted sum tips past a number. Same shape, different ingredients.

Your brain has roughly 86 billion neurons. Each one is a tiny decision-maker. Dendrites are the input branches — they're covered in receptors that catch chemical messengers (neurotransmitters) released by other neurons, and they translate those chemical hits into electrical signals. Those signals all flow into the soma (the cell body), where they're summed up. The nucleus, sitting inside the soma, holds the cell's genetic material. If the total signal crosses some threshold, the neuron fires — sending an electrical pulse down the axon, which terminates in branches called axon terminals. Those terminals release their own neurotransmitters, which crash into the next neuron's dendrites, and the chain continues.

The artificial neuron — the perceptron — is a cartoon of this. We replace the dendrites with input numbers, the synapse strengths with weights, the threshold with a bias, and the firing decision with an activation function.

For a single perceptron, let's pick the simplest possible activation function: the binary step.

Think of it as: if the weighted vote crosses the threshold $b$, output 1; otherwise output 0.

The perceptron as a classifier

A perceptron with binary activation is a line-drawer. Give it 2D data and it will draw a straight line, calling everything on one side "1" and everything on the other "0".

Suppose you have data on apples and oranges. (Don't try this with pears — you can't compare them with apples.) For each fruit you measure two things: sweetness and size. Apples have sweetness 4–7 and size 6–10cm; oranges have sweetness 6–9 and size 8–12cm. The two clouds overlap in the middle, but mostly they sit in different parts of the 2D plane. We'd like a perceptron that, given a new fruit's $(x_1, x_2)$, tells us "apple" or "orange".

The decision boundary

For binary activation, the perceptron outputs $1$ exactly when $x_1 w_1 + x_2 w_2 - b \geq 0$. The set of points where $x_1 w_1 + x_2 w_2 - b = 0$ is a straight line in 2D (a hyperplane in higher dimensions). On one side of this line, the output is 1; on the other, 0.

The "game": pick the right weights

You don't get to control $x_1$ and $x_2$ — they're given by nature. Your job is to pick $w_1$, $w_2$, and $b$ so that the line they define correctly separates the two classes. Think of it as a game: nature throws you a point, you've already committed to your weights, and you want to be right as often as possible.

For the toy example where the ideal boundary is $x_1 + x_2 = 1$:

• We want $x_1 w_1 + x_2 w_2 \geq 1$ on the positive side.
• The cleanest choice is $w_1 = 1$, $w_2 = 1$, $b = 1$: then the perceptron fires iff $x_1 + x_2 \geq 1$. ✓

When binary breaks: sigmoid & more neurons

A binary perceptron is brutally confident. It says "1" or "0", with no shades of grey. Two upgrades fix this: smooth the activation (sigmoid), or stack many perceptrons together (multi-layer).

The binary step is computationally cheap and biologically pleasing — but it has a fatal flaw for learning: it has no useful derivative. The derivative is zero almost everywhere, with an infinite spike at the threshold. This means we can't use gradient-based methods (backprop) to update the weights; the gradient signal vanishes.

Fix #1: replace binary with the sigmoid. It looks like a smoothed step. Now near the boundary the perceptron output is $0.5$ish — a soft "I'm not sure" — and only out at the extremes does it commit to $\sim 0$ or $\sim 1$. The boundary becomes a fuzzy stripe instead of a knife-edge.

Fix #2: when even a smooth boundary can't separate the classes (e.g., a red blob surrounded by blue), one neuron isn't enough. Stack them: feed the outputs of several neurons into another neuron. The output region becomes the intersection or union of half-planes — and you can carve out arbitrary shapes (triangles, polygons, smooth blobs) by combining straight cuts.

The Perceptron Learning Algorithm

We've talked about which weights would classify correctly. Now: how does the perceptron find them on its own from data? Rosenblatt's 1958 answer is shockingly simple — and it's guaranteed to work, given certain conditions.

First, a small change of convention. From here on, the labels are not $\{0, 1\}$ but $\{-1, +1\}$. This makes the algebra cleaner — we'll multiply predictions by labels, and a wrong answer turns out to flip the sign. This change is part of the lecturer's handwritten annotation on slide 20: the activation is now

Setting up: augmenting the bias into the weights

It's cleaner to fold the bias into the weight vector. Define the augmented input and weights:

Then the activation potential becomes:

From now on we'll just write $\mathbf{x}^\top\mathbf{w}$, with the understanding that the bias is hidden inside.

The "classified correctly" trick

With labels in $\{-1, +1\}$ there's a beautiful single-line check for whether a point is classified correctly.

Assumptions for learning

The algorithm itself

Algorithm 1: Perceptron Algorithm
─────────────────────────────────
Initialize weights w ← 0
repeat
    Done ← True
    for each training sample (xⱼ, yⱼ):
        compute prediction ŷⱼ = sign(w · xⱼ)
        if ŷⱼ ≠ yⱼ:
            update weights w ← w + yⱼ · xⱼ
            Done ← False
        end if
    end for
until Done

Geometry — distance to a hyperplane

Before we can prove the algorithm converges, we need one geometric fact: how far is a given point from a given hyperplane? The answer is a single tidy formula, and the proof is two lines of algebra.

Proof (slide 27)

Consider the line through $\mathbf{x}_0$ in the direction of $\mathbf{w}$ (which is the normal to $H$):

This line $L$ meets the hyperplane $H$ at a single point — call it $\mathbf{x}'$ — at parameter $t = t'$. To find $t'$, we plug $\mathbf{x}_0 + t\mathbf{w}$ into the hyperplane equation:

Solving:

Now the distance is just $\|\mathbf{x}' - \mathbf{x}_0\|$:

Novikoff's Convergence Theorem

This is the punchline of the perceptron story. If the data is linearly separable with margin γ, the perceptron makes at most 1/γ² mistakes. The algorithm cannot get stuck: it always halts, and the bound on mistakes depends only on the geometry of the data — not on its size, dimension, or order.

This theorem is due to Albert Novikoff (1962) and is stated and proved entirely in the lecturer's handwritten annotations on slides 23–24. It's a foundational result in learning theory, and it's the kind of thing that absolutely could appear on the exam — the statement, the proof technique, and the role of every assumption.

Setup: the margin

The theorem

Proof (slide 24, full)

The proof is a beautiful sandwich argument. We'll bound $\mathbf{w}^{(m)\top}\mathbf{w}^*$ from below (it grows) and $\|\mathbf{w}^{(m)}\|$ from above (it doesn't grow too fast); together, these force $m$ to be small.

Notation: at step $k$, we make a mistake on point $(\mathbf{x}, y)$ (dropping the index for clarity) and update $\mathbf{w}' \leftarrow \mathbf{w} + y\mathbf{x}$. Two facts about the misclassified point:

• (i) Misclassification means $y(\mathbf{x}^\top\mathbf{w}) \leq 0$ (we got it wrong with current weights).
• (ii) Since $\mathbf{w}^*$ separates with margin γ: $y(\mathbf{x}^\top\mathbf{w}^*) \geq \gamma$.

Step 1: $\mathbf{w}^\top\mathbf{w}^*$ grows by at least γ each update

So after $m$ updates, starting from $\mathbf{w}^{(0)} = \mathbf{0}$:

Step 2: $\|\mathbf{w}\|^2$ grows by at most 1 each update

The middle term is $\leq 0$ (by fact (i): $y(\mathbf{w}^\top\mathbf{x}) \leq 0$, so $2y(\mathbf{w}^\top\mathbf{x}) \leq 0$). The last term is $y^2 \|\mathbf{x}\|^2 = \|\mathbf{x}\|^2 \leq 1$ (since $y \in \{-1,+1\}$ and $\|\mathbf{x}\| \leq 1$). So:

After $m$ updates, starting from $\mathbf{w}^{(0)} = \mathbf{0}$:

Step 3: combine via Cauchy–Schwarz

By Cauchy–Schwarz (recall: $\mathbf{a}^\top\mathbf{b} \leq \|\mathbf{a}\|\|\mathbf{b}\|$):

Combining with $(\star)$:

So $m\gamma \leq \sqrt{m}$, i.e., $m^2\gamma^2 \leq m$, i.e.,

Feed-forward in a multi-layer network

Now we move from one neuron to many, organised in layers. Feed-forward is the recipe for taking an input and producing an output, layer by layer. It's just plugging numbers into formulas — but with bookkeeping.

A multi-layer perceptron has an input layer (the raw features), one or more hidden layers (each layer's outputs become the next layer's inputs), and an output layer (whose outputs are the predictions $\hat{y}$). Each connection has a weight, each neuron has a bias and an activation function. Feed-forward is the act of pushing inputs through the layers in order to compute the output.

For this section, we'll use the ReLU activation: $f(x) = \max(0, x)$. We'll see why ReLU (or some non-linear function) matters in Part VIII.

Worked example (slides 34–36)

Let $x_1 = 2$, $x_2 = 3$, with weights:

• $w_1 = 0.11$ ($x_1 \to h_1$), $w_2 = 0.12$ ($x_2 \to h_1$),
• $w_3 = 0.21$ ($x_1 \to h_2$), $w_4 = 0.08$ ($x_2 \to h_2$),
• $w_5 = 0.14$ ($h_1 \to \hat{y}$), $w_6 = 0.15$ ($h_2 \to \hat{y}$).

(For simplicity all biases are zero in this example.)

Step 1. Compute the hidden activations.

Step 2. Compute the output.

Why we need non-linearity

Here's a striking fact: if every neuron uses a linear activation function, the entire network — no matter how many layers — collapses to a single linear neuron. Any expressive power has to come from non-linearity. Here's the algebra that proves it.

Take the same 2–2–1 network as before, with biases this time: $b_1$ and $b_2$ on the hidden neurons, $b_3$ on the output neuron, and a linear activation (i.e., $f(x) = x$). The output is

Now substitute $h_1 = x_1 w_1 + x_2 w_2 - b_1$ and $h_2 = x_1 w_3 + x_2 w_4 - b_2$:

Distribute everything:

Group by $x_1$, $x_2$, and constants:

Define new weights and a new bias:

Then:

This is exactly the formula for a single perceptron with weights $(w', w'')$ and bias $b'$. The hidden layer has vanished into the equivalent single neuron.

The activation-function zoo

There are six activation functions on the slides, each with a personality. Here they are, ranked by their pros, cons, and a one-line summary of when you'd use them.

1. Binary step

$$f(x) = \begin{cases} 1 & x \geq 0 \\ 0 & x < 0 \end{cases}$$

Advantages:

• Output bounded in $\{0, 1\}$.
• Computationally efficient (one comparison).
• Closer to biological neurons (used in spiking neural networks).

Disadvantages:

• No usable derivative — it's zero almost everywhere with an infinite spike at zero. Cannot be used with backpropagation.

2. Sigmoid (logistic)

$$f(x) = \frac{1}{1 + e^{-x}}.$$

Advantages:

• Smooth gradient — no jumps or kinks.
• Output bounded in $[0, 1]$ (interpretable as a probability).
• Little noise: for $x \geq 5$ and $x \leq -5$, the sigmoid is essentially 1 or 0.

Disadvantages:

• Vanishing gradient at extreme values of $x$. The derivative is $f(x)(1 - f(x))$, which is near zero when $f(x)$ is near 0 or 1 — meaning the network's gradient signal dies and it stops learning.
• Computationally expensive (involves exponentiation).

3. Hyperbolic tangent (tanh)

$$f(x) = \tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}.$$

Like sigmoid, but its outputs sit in $[-1, +1]$ instead of $[0, 1]$. So tanh is zero-centred on the y-axis.

Why this helps: if all hidden activations are positive (as with sigmoid), the gradient updates for the next layer's weights all share a sign — they all push up, or all push down — which slows learning. Centring around zero means activations can be positive or negative, and gradient updates can move in either direction independently.

4. ReLU (Rectified Linear Unit)

$$f(x) = \max(0, x).$$

Advantages:

• Computationally efficient — just one comparison and a possible zero-replacement.
• Non-linear, but has a simple derivative ($1$ on positives, $0$ on negatives) — fine for backpropagation.
• Doesn't saturate for positive inputs (no vanishing gradient on the positive side).

Disadvantages:

• The Dying ReLU problem. If a neuron's input becomes negative for all training points (e.g., due to a large negative bias), the gradient is zero, the network can't update those weights, and the neuron is permanently silent. It "dies." Whole regions of the network can shut down this way.

5. Leaky ReLU

$$f(x) = \max(0.1\,x,\, x).$$

Same as ReLU but with a small slope on the negative side (here, 0.1). This means even on negatives, the gradient is non-zero (it's 0.1), so a "dying" neuron can still receive update signals and recover.

Advantages:

• Prevents the Dying ReLU problem.
• Keeps all the other ReLU advantages.

Disadvantages:

• The leak coefficient $0.1$ is predefined and not learned — the network doesn't decide it. (Variants like Parametric ReLU let the network learn it.)

6. Swish

$$f(x) = \frac{x}{1 + e^{-x}} \;=\; x \cdot \text{sigmoid}(x).$$

A smooth, non-monotonic function that looks similar to ReLU but with a small dip below zero before flattening out. Empirically, Swish performs a tiny bit better than ReLU on many tasks (introduced by Google researchers via neural architecture search).

Summary table