Competitive & associative networks (breadth)

Beyond supervised gradient descent

Backprop is supervised (it needs target outputs). Three other connectionist styles you should recognize learn differently.

Hebbian coincidence learning

Hebb (1949): cells that fire together wire together. When neuron A repeatedly helps fire neuron B, strengthen their connection. Formalized as $\Delta w=c\cdot \text{sign}(o_i{o}_j)$ — weight up when two units agree in sign, down when they disagree. It is local (no global error signal) and the most biologically plausible rule. Its supervised variant builds the linear associator — an associative memory whose weight matrix is the sum of outer products $W={\sum }_i{Y}_i{X}_i^{\mathsf{T}}$.

Kohonen competitive learning (winner-take-all / SOM)

Unsupervised clustering: present an input, find the node whose weight vector is closest (minimum Euclidean distance), and move only that winner toward the input, $\Delta W=c(X-W)$, with $c$ shrinking over time. Over many inputs the nodes spread out to become prototypes of the data clusters — a self-organizing map (SOM). No labels are used; the network discovers structure.

Hopfield attractor networks

Feedforward nets are one-pass functions; attractor (feedback) networks loop their output back as input and settle into a stable state. A Hopfield network (1982) is fully connected with symmetric weights ($w_{ij}=w_{ji}$) and no self-connections. Hopfield proved an energy function $H(X)=-{\sum }_i{\sum }_j{w}_{ij}{x}_i{x}_j+2{\sum }_i{T}_i{x}_i$ never increases as nodes update, so the network must converge to a local energy minimum. Those minima are attractors — stored patterns — giving content-addressable memory: start from a noisy/partial pattern and the net settles to the clean stored one. The same energy machinery can encode optimization problems.