Outliers, influential points, and refitting

◈ 4 cards

Extreme in y or extreme in x — the 2s rule for outliers, the remove-and-refit check for influence, and reporting both fits.

Two different kinds of unusual point

An outlier in regression is a point with a large residual — far from the line vertically, unusual in $y$ given its $x$ . An influential point is one whose removal changes the fitted line substantially — typically a point far out in $x$ , at the edge of the data, where it acts as a lever on the slope. A point can be either, both or neither, and the two are diagnosed differently.

Worked example — a ninth client

Northfield adds a ninth client: income $150,000, score 600 — a high earner with a poor score. Refit on all nine (sums: $\sum x = 705 $,$ \sum y = 6{,}100 $,$ \sum x^2 = 65{,}535 $,$ \sum y^2 = 4{,}161{,}150 $,$ \sum xy = 480{,}925$):

	Eight clients	Nine clients
$r$	0.986	0.186
slope $\hat{β}_{1}$	2.066	0.300

One point took a very strong association and reduced it to almost nothing, and cut the slope by a factor of seven. That is influence, and it comes from the point's position in $x$ : 150 is well beyond the others' 35–110, so the line pivots toward it. The scatter plot shows it at once — eight points in a tight rising band and one far to the right and low.

The diagnostic is the one just performed: fit with and without, and compare. If the slope, $r$ or the intercept move materially, the point is influential.

The 2s rule for outliers

For a large residual the yardstick is the residual standard error

$s = \frac{S S E}{n - 2}$

— the typical size of a residual, on $n - 2$ degrees of freedom because two parameters were estimated. For the original eight, $s = 557.8/6 = 9.64$ and $2 s = 19.3$ . A point whose residual exceeds $2 s$ in absolute value is flagged as an outlier. The largest of the eight residuals is $∣ - 11.6∣$ , client 3: no outliers in the original fit.

Note that the ninth client is not mainly an outlier in this sense. Once it is in the fit, the line has moved toward it, and its own residual is not the largest. That is exactly why the 2s rule alone misses influential points — they pull the line to themselves — and why refitting is the test.

What to do about it

Investigate first: is the 600 a keying error for 690? Is the client a special case (a recent bankruptcy, a business owner with reported income that does not reflect cash flow)? Then report both fits, state which point was set aside and why, and let the reader see the difference. Silently dropping the point produces a line that describes eight clients while claiming to describe nine; silently keeping it produces a slope of 0.30 that describes none of them well.