Conclusions in business context

The sentence that carries the last mark

Step 4 is two things: the decision — reject $H_0$ or do not reject $H_0$, at $\alpha$ — and a sentence a manager could act on. That sentence must name four things: the level, the parameter, the direction, and the setting. For Kitchener Ridge's lower-tailed test ($z=-1.90$, $p=0.0287$), four candidate sentences:

1. The new process works. — No level, no parameter, no direction. Zero.
2. We reject $H_0$. — A decision, not a conclusion; nothing a manager can use.
3. The mean days-to-pay is 43.1 days. — Reports the sample, asserts it of the population, mentions no test. Zero.
4. There is evidence, at the 5 % level, that the mean days-to-pay under the new process is below 45 days. — Level, parameter, direction, context. The mark.

The template for a rejection: there is evidence at α that ⟨parameter⟩ ⟨direction⟩ ⟨null value⟩ ⟨in this setting⟩. For a non-rejection: there is insufficient evidence at α that … — the same clause, with the word insufficient.

Failing to reject is not proving the null

At $\alpha =0.01$ the same test has $p=0.0287>0.01$: do not reject. The sentence is there is insufficient evidence at the 1 % level that the mean days-to-pay is below 45 days. It is not the mean is 45 days, and not we accept $H_0$. The data did not clear the bar; that is all. The mean may well be below 45 — the same sample rejected at 5 % — and a larger sample might show it. "Accept $H_0$" claims a proof the test never offers, and the paper deducts for the word.

Statistically significant, practically trivial

Kitchener Ridge later samples $n=10,000$ invoices and finds $\bar{x}=44.8$. With $\sigma =8$, $\mathrm{SE}=8/100=0.08$ and

$$z=\frac{44.8-45}{0.08}=-2.50,p=\Phi (-2.50)=0.0062$$

Reject at 1 %: strong evidence the mean is below 45. But the estimated reduction is 0.2 days on a 45-day cycle — nothing any manager would act on. The test answered is the mean below 45? with high confidence; it never answered by how much? That is what the confidence interval is for: $44.8\pm 2.576\times 0.08=(44.59,45.01)$ at 99 % — all of it within a quarter-day of 45. Statistical significance says an effect is detectable; practical significance says it matters. A large $n$ makes the first cheap and says nothing about the second. Report both: the p-value, and the size of the effect with its interval.

The checklist

Before leaving step 4, check the sentence has: reject / do not reject — at α = … — evidence / insufficient evidence — the parameter in words — the direction from $H_a$ — the null value — the setting. Seven items, one sentence, and the mark that most students leave on the table.