Backpropagation in multilayer networks

◈ 5 cards

The exam Q9 answer: forward pass → output error → chain-rule blame to hidden weights → gradient-descent update; sigmoid f’ = f(1−f); solves XOR.

Why we need a differentiable neuron

The perceptron’s hard $sign$ has no usable gradient — the output jumps from $- 1$ to $+ 1$ with no sense of how wrong it is. Replace it with the smooth sigmoid (logistic) activation

$f (net) = \frac{1}{1 + e ^{- net}}, f^{'} (net) = f (net) (1 - f (net)) .$

The derivative is computable from the output alone, which is exactly what makes gradient descent practical. With a hidden layer in between, the network can carve the curved boundaries that a single perceptron cannot — it can solve XOR.

Backpropagation — the exam Q9 answer

Backprop trains a multilayer net by descending the squared-error surface. One pattern, four steps:

Forward pass. Feed the input forward; each layer computes $O = f (\sum w x)$ , layer by layer, to the output.
Output error. Compare output $O_{i}$ with the target $d_{i}$ . The output-node delta is
$δ_{i} = (d_{i} - O_{i}) O_{i} (1 - O_{i}),$
the error scaled by the sigmoid slope $O_{i} (1 - O_{i})$ .
Backward pass — assign blame by the chain rule. A hidden node has no target of its own. Its responsibility is the weighted sum of the deltas of the output nodes it feeds, scaled by its own slope:
$δ_{h} = O_{h} (1 - O_{h}) j \sum δ_{j} w_{hj} .$
That $\sum_{j} δ_{j} w_{hj}$ is the error propagated backward across the weights — the move that names the algorithm.
Update every weight by gradient descent, output and hidden alike:
$Δ w_{k i} = c δ_{i} x_{k} .$

Repeat over the training set for many epochs; the error falls toward a (local) minimum. The exercise trains a 2-2-1 sigmoid net on XOR and prints the error dropping to ~0 with final outputs ≈ $[0, 1, 1, 0]$ .

The net the exercise trains. Every hidden unit feeds the output, so on the backward pass every hidden unit is partly to blame for the output’s error — the share it receives is δ scaled by the weight on that very connection. Note that no hidden unit has a target of its own; the layer exists only to build intermediate features, which is what makes the recombined XOR problem separable.

The exam Q9 answer as a pipeline. Only step 3 is peculiar to backprop: a hidden unit has no target, so its error is manufactured from the deltas of the units it feeds, scaled by its own sigmoid slope. Steps 1, 2 and 4 are ordinary gradient descent. Repeat the whole cycle for every pattern, for many epochs.