AI as empirical enquiry: PSSH, the paradigms & the generalization problem

AI as an experiment

Luger's mature definition reframes the field: AI is the study of the mechanisms underlying intelligent behavior through the construction and evaluation of artifacts designed to enact those mechanisms. That makes AI an empirical methodology: each program is an *experiment* that poses a question to nature, and the program's behavior is nature's answer. AI's unique advantage over psychology or biology is that we can inspect, halt, and modify our subjects at will — examining internal representations and search strategies directly.

The physical symbol system hypothesis, restated

The PSSH (Newell & Simon, 1976) claims a physical symbol system is necessary and sufficient for general intelligent action. *Sufficient*: an appropriately organized symbol system can achieve intelligence. *Necessary*: any intelligent agent — human, alien, or machine — must implement operations on symbol structures. It carries four methodological commitments: symbols as the medium, heuristic search as the mechanism, disembodiment (intelligence as formal structure, independent of physical implementation), and empiricism/constructivism (understand intelligence by building working models).

The sharpest critique is the symbol-grounding problem: symbols in a formal system have no *intrinsic* meaning — they are pre-interpreted by their designers, not grounded in embodied experience. When a human says "summer," the word is grounded in warmth and light and vacation; when a program uses the token SUMMER, it has only formal relations to other tokens. Searle's Chinese Room sharpens this — syntactic symbol manipulation can pass for understanding without *being* understanding. The "sufficient" clause is well-supported by practice; the "necessary" clause remains the central open question, with neural networks (knowledge in distributed weights, not discrete symbols) as the standing counter-example.

Four paradigms

1. Symbolic (PSSH/GOFAI) — logic + heuristic search; transparent and provable, but brittle and ungrounded.
2. Connectionist — distributed representation in weights, learning from examples, graceful degradation; but real cortex (~$10^{10}$ neurons, fan-out ~$10^5$, many neuron types) is far richer than our uniform artificial neurons.
3. Emergent / agent-based — global intelligence from many simple local agents under Darwinian selection (Holland's "no baker knows the city's bread needs, yet bakers collectively supply it"). The empiricist's dilemma: if intelligence merely *emerges*, by what criterion do we know it has been achieved?
4. Stochastic / probabilistic — Bayes and Markov models; principled and practical (speech, tagging, diagnosis), but corpus-bound (may fail on truly novel input) and current BBNs are propositional.

The generalization problem (the load-bearing exercise)

The deepest empirical question is generalization: does a learned model predict *new* cases, not just the training data? This is Hume's problem of induction in computational form.

- Underfitting — the model is too simple to capture the pattern; *both* training and test error are high. - Good fit — the model captures the essential pattern without memorizing noise; low training *and* low test error. - Overfitting — the model fits the training data almost exactly but fails on new inputs, because it memorized noise rather than the underlying pattern; training error is near zero while test error is large.

The Python exercise makes this concrete: fit polynomials of degree 1, 3, and 9 to the same noisy points. Degree 1 underfits (train and test error both high); degree 3 fits well (both near zero); degree 9 overfits — it threads every training point (train ≈ 0) but its wild wiggles miss the test points (test error large). The moral: *"success on training data" is a weak criterion* — generalization is what matters, and it cannot be read off training error alone.

Inductive bias, No Free Lunch, and constructivism

Why can't one algorithm just learn everything? Because every learner has an inductive bias — the prior commitments baked into its representation and search (ID3's preference for short trees, a network's topology, a GA's mutation/crossover operators). All experience is theory-laden; all learning is bias-laden. The No Free Lunch theorems (Wolpert & Macready, 1995) make this formal: averaged over *all* possible problems, no learning algorithm beats random — every algorithm's wins on some problems are paid for by losses on others. The practical art is matching the inductive bias to the problem structure.

Constructivism (Piaget) reconciles the old rationalist–empiricist fight: agents assimilate experience using current schemata and accommodate (revise) those schemata when they fail — cognitive *equilibration*. Rationalists are right that prior structure is necessary for *any* interpretation (Meno's paradox: with no schema, nothing is even perceived); empiricists are right that the structure is shaped by experience. For the AI practitioner the lesson is humility: models are progressive approximations, not final truths, and research is continual model revision through experiment. That is precisely the empirical loop this whole course has been training you to run.