Spread, sample vs population, and comparing across scales

Spread around the mean

The variance is the average squared distance from the mean; the standard deviation (SD) is its square root, back in the original units — dollars here. It is the spread that pairs with the mean, as the IQR pairs with the median. The only real decision is the denominator.

Sample versus population

Divide the sum of squared deviations by n − 1 and you have the sample SD; divide by n and you have the population SD. The n − 1 is not a fudge: once the mean is known, only n − 1 of the deviations are free — the last is fixed by the others summing to zero — so n − 1 is the number of independent pieces of information. The sample version is also the one that, on average, gets the population's spread right when you only have a sample.

The spreadsheet offers both:

• =STDEV.S(D2:D13) → 362.82 (sample, n − 1 = 11)
• =STDEV.P(D2:D13) → 347.37 (population, n = 12)

The legacy STDEV equals STDEV.S; VAR.S and VAR.P are the variances. The default is sample. A quiz that says "the standard deviation of these 12 orders" with no claim that the twelve are the entire population wants STDEV.S. R makes the choice for you:

> sd(orders$amount)
[1] 362.8193
> var(orders$amount)
[1] 131637.9

sd() and var() are sample-only. There is no population version and no argument to request one; multiply by (n − 1)/n by hand if a population SD is ever needed. Python's statistics module names the two: stdev is sample, pstdev population.

Comparing spread across scales

A SD of $362.82 is large or small only relative to the mean. The coefficient of variation is the ratio:

$$CV=\frac{SD}{\text{mean}}=\frac{362.82}{511.67}=0.71$$

It is unit-free, so it compares the spread of order amounts (CV 0.71) with the spread of, say, units per order or delivery days, on different scales. Its limit: when the mean is near zero or negative the ratio blows up or flips sign, and it stops meaning anything.

Locating one value: the z-score

How unusual is the $1,320 order? Measure its distance from the mean in SDs:

$$z=\frac{x-\bar{x}}{s}=\frac{1320-511.67}{362.82}=2.23$$

The spreadsheet has a function for it — =STANDARDIZE(1320, AVERAGE(D2:D13), STDEV.S(D2:D13)) — and R is the formula itself, (1320 - mean(orders$amount))/sd(orders$amount), which prints [1] 2.227922. A z of 2.23 says the order sits 2.23 standard deviations above the mean; the $90 order has z = −1.16, a little over one SD below. R's scale(orders$amount) computes every z at once but returns a matrix, not a vector — wrap it in as.numeric() before using it as a column.

Type three calls, then fill the worksheet

Type the sample SD, the STANDARDIZE call and the R sd(). The worksheet asks for the sample SD, the CV and the z of 1320; the code block prints both Python SDs side by side so the n − 1 gap is visible.

CRISP-DM: spread and z-scores are data understanding → explore data.