Sign, size, range, decision — and a pivot is a model too

◈ 9 cards

Four questions before a fitted line reaches the CFO: is the sign the one expected (positive — yes)? is the size plausible ($3.45k per $1k — yes)? does the range cover the decision ($8k is inside 2–10 — yes)? what decision changes? "R² is 0.99" is assess model (modelling); "the sign matches" and "$12k is outside our data" are evaluation; "re-fit each quarter" is deployment. aggregate(sales ~ channel, FUN = mean) — Mall 30.33, Street 37.67 — is a model with a categorical predictor; lm(sales ~ channel) reports the Mall mean and the Street − Mall difference, 7.33.

Four questions, in order

summary(fit) says the line fits. CRISP-DM's evaluation phase asks something different: does the result answer the business question, and should the CFO act on it? Four questions, asked of the stores line before it goes into the memo:

Sign — is it the direction the business expected? Positive: more advertising, more sales. Yes.
Size — is the magnitude plausible? $3.45k of sales per $1k of ad spend. Plausible for a retailer with healthy margins; if it were $345k, something in the data is wrong before anything in the model is right.
Range — does the data cover the decision? The CFO is choosing between $6k and $8k per store. Both inside 2–10. Yes.
Decision — what changes? At $3.45k per $1k, moving from $6k to $8k predicts about $6.9k more sales per store. If that clears the margin hurdle, the recommendation is raise it; if the sign had come out negative, the recommendation is nothing — and the project goes back to business understanding to ask why the data say the opposite of what everyone believed. A wrong-signed slope is not a modelling failure; it is a finding, and the finding is that the question was framed on a false premise.

None of the four is about $R^{2}$ . A curved scatter can carry a high $R^{2}$ and still be the wrong model; a line with $R^{2} = 0.3$ can still change a decision if the sign and size are right and the range is covered.

Worked example — which phase said that?

Statements from the memo, tagged by phase — the figure below is the table.

" $R^{2}$ is 0.9972, the residual SE is $0.59k." → assess model — modelling. Statistics about the fit.
"The slope has the sign the CFO expected, at a plausible size." → evaluate results — evaluation. The fit against the question.
"Ad spend above $10k is outside our data; we cannot say what happens there." → evaluation. A limit on what the result answers.
"We will re-fit the line each quarter as new store data arrive." → deployment. Monitoring and maintenance.

The tell: a sentence about the model's numbers is modelling; a sentence about the decision, the expectation, or the limits of the answer is evaluation; a sentence about what happens next is deployment.

A pivot is a model too

The channel column has been sitting there. A group mean by channel is Module 7's pivot — and it is also a model, with a categorical predictor:

> aggregate(sales ~ channel, data = stores, FUN = mean)
  channel    sales
1    Mall 30.33333
2  Street 37.66667

The "prediction" for a Mall store is $30.33k, for a Street store $37.67k — one fitted value per level instead of a line. lm() can fit the same thing, and reports it in a shape worth recognising:

> coef(lm(sales ~ channel, data = stores))
  (Intercept) channelStreet 
    30.333333      7.333333

(Intercept) is the mean of the first level (Mall, alphabetically); channelStreet is the difference Street − Mall, 7.33 — not the Street mean. Recognition only: the pivot gives the means directly, and AFM 113 says whether 7.33 is more than noise from three stores a side.

Type one call, then phase-tag four statements

CRISP-DM: this lesson is the evaluation phase — evaluate results, review process, determine next steps.

Which phase said that. Numbers about the fit are modelling; the fit against the question and its limits are evaluation; what happens next is deployment.