Backpropagation in multilayer networks

Why we need a differentiable neuron

The perceptron’s hard $\text{sign}$ has no usable gradient — the output jumps from $-1$ to $+1$ with no sense of *how* wrong it is. Replace it with the smooth sigmoid (logistic) activation

$$f(\text{net}) = \frac{1}{1 + e^{-\text{net}}}, \qquad f'(\text{net}) = f(\text{net})\,\big(1 - f(\text{net})\big).$$

The derivative is computable from the output alone, which is exactly what makes gradient descent practical. With a hidden layer in between, the network can carve the curved boundaries that a single perceptron cannot — it can solve XOR.

Backpropagation — the exam Q9 answer

Backprop trains a multilayer net by descending the squared-error surface. One pattern, four steps:

1. Forward pass. Feed the input forward; each layer computes $O = f(\sum w x)$, layer by layer, to the output.
2. Output error. Compare output $O_i$ with the target $d_i$. The output-node *delta* is
3. $$\delta_i = (d_i - O_i)\,O_i(1 - O_i),$$
4. the error scaled by the sigmoid slope $O_i(1-O_i)$.
5. Backward pass — assign blame by the chain rule. A hidden node has no target of its own. Its responsibility is the weighted sum of the deltas of the output nodes it feeds, scaled by its own slope:
6. $$\delta_h = O_h(1 - O_h)\sum_j \delta_j\, w_{hj}.$$
7. That $\sum_j \delta_j w_{hj}$ is the error *propagated backward* across the weights — the move that names the algorithm.
8. Update every weight by gradient descent, output and hidden alike:
9. $$\Delta w_{ki} = c\,\delta_i\,x_k.$$

Repeat over the training set for many epochs; the error falls toward a (local) minimum. The exercise trains a 2-2-1 sigmoid net on XOR and prints the error dropping to ~0 with final outputs ≈ $[0,1,1,0]$.