Observational studies, experiments, and confounding

The one question that decides the conclusion

Who decided which units got the treatment? If the units themselves, or circumstance, decided — the study is observational, and it can establish an association. If the investigator decided, by a chance mechanism — the study is an experiment, and it can establish causation. Everything about study design on this paper follows from that split.

Worked example — e-invoicing and days to pay

The observational finding. A software vendor reports that among 400 Ontario firms, those that adopted e-invoicing are paid on average 9 days sooner than those that did not. Does e-invoicing speed up collection?

Not necessarily. Firms chose whether to adopt. Larger, better-organised firms are more likely to adopt and more likely to have disciplined credit control, which speeds collection on its own. Firm size and organisation is a confounder: a third variable associated with both the explanatory variable (adoption) and the response (days to pay), which can manufacture the association without any effect of adoption at all. The finding is real; the causal reading is not licensed.

The experiment. The vendor persuades one distributor with 80 client accounts to run a pilot: 40 accounts are chosen at random to receive e-invoices for six months; the other 40 continue on paper. Days to pay are recorded for all 80.

Now adoption was assigned by a coin, not by the client. Large and small, organised and chaotic accounts are — on average — spread evenly across the two groups, and so is every confounder anyone could name and every one nobody thought of. If the e-invoiced group is paid sooner by more than chance would produce, e-invoicing caused it.

The vocabulary of an experiment

• Treatment — what the investigator applies (e-invoicing).
• Experimental unit — what receives it (a client account).
• Control group — units that receive no treatment, or the standard one, measured the same way over the same period, so the comparison isolates the treatment.
• Random assignment — the chance mechanism that allocates units to groups. This is the step that licenses causation.
• Blinding — hiding which group a unit is in from the unit (single-blind) or also from whoever measures the response (double-blind), to remove expectation effects. It is not always possible: a client can see whether an invoice arrived by email.

Lurking versus confounding

A lurking variable is an unmeasured third variable that drives the association. A confounder is a third variable, possibly measured, whose effect on the response cannot be separated from the treatment's because the two vary together. Either can produce an association without an effect; randomisation neutralises both because it breaks the link between the third variable and the treatment.

What the paper asks

Given a finding, say (1) observational or experimental, and why; (2) a plausible confounder, named concretely, with the direction it would push; (3) the minimal redesign that would license a causal claim — random assignment, a control group, the same measurement over the same period, blinding where possible.