Measures, metrics, indicators, and software analytics — COMP 410

The thinking move: a number is not yet information

The exam description names software analytics as a definition term, and the reason it is worth 7 marks is that almost everyone defines it as "advanced metrics". It is not. The route to a good answer is to climb a four-rung ladder first, because analytics is what sits above the top rung, and the ladder is the single tidiest piece of vocabulary in the whole course.

The four terms, deliberately distinguished

A measure is a single data point about an attribute of a product or process — fourteen errors found in the reservation component. Measurement is the act of collecting one or more data points: reviewing the errors found across every component review and unit test is a measurement. A metric relates individual measures — an average, a ratio, a density — which is what makes it comparable across things of different size: 3.2 errors per thousand lines of code across the release. An indicator is a metric, or a combination of metrics, that provides insight into the process, the project, or the product: error density is falling release over release while review coverage rises, so the reviews are doing their job.

Notice what changes as you climb: nothing new is measured, but each rung adds interpretation. That is the shape of the answer the examiner is looking for. Notice too that the ladder is what makes "a metric" comparable and "an indicator" decidable — a raw count of 14 errors tells you nothing until you know across how much code, and a density tells you nothing until you know which way it is moving.

What makes a metric worth collecting

Hundreds of software metrics have been proposed and most are never used, so it is worth knowing why. An effective metric is easy to learn and to compute (if it needs dozens of counts and a complex formula, nobody will collect it), intuitively satisfying (a cohesion metric should go up as cohesion goes up), unambiguous in its results, dimensionally sensible (multiplying team members by language keywords produces a number in nonsense units), based on the requirements model, the design model or the program structure rather than on the quirks of a language's syntax, and finally actionable — it must lead to a decision that improves the product. That last attribute is the bridge to analytics.

Software analytics

Software analytics is the systematic computational analysis of software engineering data to give managers and engineers meaningful insight and let their teams make better decisions. The two words that carry the definition are timely and actionable. Knowing the number of defects in the product today is much less valuable than knowing that the number is five percent higher than last month, because only the second statement implies an action. Analytics predicts how many defects to expect, where to look for them, and how long they will take to fix, which is what lets a manager build an incremental schedule on evidence rather than on hope.

It is applied to a recognisable set of decisions: targeted testing (where to aim regression and integration effort), targeted refactoring (where technical debt will cost most), release planning, understanding customers (how the product is actually used in the field), judging stability (is the evolving product settling down), and targeting inspection (which inspections are worth their cost, how often, how deep). And it has a practical precondition: automated tooling over large, dynamic data sets. A metric can be counted by hand from one release; analytics cannot.

A key performance indicator (KPI) sits deliberately on the boundary — it is a metric used to track performance and to trigger remedial action when its value falls outside a predetermined range. A KPI is where a metric acquires an action, which is why it is the useful bridge term in a compare answer.

Worked example — the same BorrowBox data, four ways

BorrowBox's release 3 defect log contains one entry for a double-charged late fee. That entry is a measure. Collecting every entry across the release, from the review logs and the test runs, is measurement. Dividing the total by the size of the release gives 4.1 defects per KLOC — a metric, and now comparable with release 2's 5.6. Setting that beside review coverage, which rose from 40% to 75% of changed components over the same period, gives an indicator: the reviews appear to be paying for themselves.

Now the analytics step, which no rung of the ladder reaches on its own. Feed three releases of defect records, commit history and test results through tooling that localises and predicts, and the output is: defect arrivals are 5% above last month; 60% of them are in the late-fee component, which is also where both of the last two releases regressed; the component's cyclomatic complexity has doubled since release 1; expect roughly nine more defects there before the code stabilises. Nothing there is a new measurement. The difference is that the last sentence tells you to aim next sprint's regression suite and inspection effort at the late-fee component — and that is a decision, not a number.

Nothing new is measured as you go down the table — each row adds interpretation to the row above. Analytics is not a fifth rung of the same ladder: it is a different instrument applied to the same data, and its output is an action rather than a number.

Analytics is defined by the decisions it supports, so learning the application areas is most of learning the definition. Every entry in the right-hand column is an action, not a number — which is the test to apply to your own answer.