p̂ and its confidence interval

A proportion is a mean in disguise

Code each unit 1 if it has the attribute and 0 if not. The sample mean of those 0/1 values is the sample proportion $\hat{p}=x/n$, where $x$ counts the 1s. Module 8’s CLT therefore applies to $\hat{p}$ exactly as it applies to $\bar{x}$: for large $n$,

$$\hat{p}\approx N\left(p,\frac{p(1-p)}{n}\right),\mathrm{SE}(\hat{p})=\sqrt{\frac{p(1-p)}{n}}$$

The variance $p(1-p)$ is the Bernoulli variance from Module 6 — there is no separate $\sigma$ to estimate, because once $p$ is known the spread is known too. That is why inference for a proportion is always z, never t: the t distribution exists to absorb the uncertainty in an estimated $\sigma$, and a proportion has none.

The population $p$ in the SE is unknown, so the interval substitutes $\hat{p}$:

$$\hat{p}\pm z_{\alpha /2}\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$$

Valid when the normal approximation holds: $n\hat{p}\ge 5$ and $n(1-\hat{p})\ge 5$ — at least five successes and five failures in the sample.

Worked example — Maple Ledger’s invoice error rate

An internal audit at Maple Ledger checks a random sample of 500 processed invoices and finds 60 with at least one error. Estimate the error rate for all invoices with a 95 % interval.

Conditions. $n\hat{p}=60\ge 5$ and $n(1-\hat{p})=440\ge 5$. Fine.

Estimate. $\hat{p}=60/500=0.12$.

Standard error.

$$\mathrm{SE}=\sqrt{\frac{0.12\times 0.88}{500}}=\sqrt{0.0002112}=0.01453$$

Margin. $1.96\times 0.01453=0.0285$.

Interval. $0.12\pm 0.0285=(0.0915,0.1485)$.

With 95 % confidence, between 9.2 % and 14.9 % of all Maple Ledger invoices contain an error. Not "95 % of invoices"; not "a 95 % probability that $p$ is in the interval" — the same wording rules as Module 9, with the parameter now a proportion.

What moves the width

The margin is $z\sqrt{\hat{p}(1-\hat{p})/n}$. Halving it needs four times the sample, as for a mean. And $\hat{p}(1-\hat{p})$ is largest at $\hat{p}=0.5$ (where it equals 0.25) and shrinks towards either end: a proportion near 0.12 is estimated more precisely than one near 0.5 from the same $n$, which is the fact L12.2 turns into the minimum-$n$ rule.

Reading the interval as a test

If the firm’s tolerable error rate is 10 %, the interval $(0.0915,0.1485)$ contains 0.10 — so a two-sided test at 5 % would not reject $p=0.10$. L12.3 runs that test properly, and shows that the test’s SE is built differently from the interval’s; the duality is approximate for proportions, not exact, but the reading is usually the same.