Product, process, project and quality metrics — COMP 410

The thinking move: choose metrics from a goal, never from a catalogue

Hundreds of software metrics exist. A candidate who lists twenty of them has demonstrated a good memory and poor judgement, which is the opposite of what an applied question rewards. The professional move is to start from a goal, derive the questions whose answers would tell you whether you are meeting it, and only then pick the metrics that answer those questions. That is the goal/question/metric paradigm, and it is also the shape of a good exam answer.

The families, and what each one is for

Product metrics measure the artefacts. At the requirements stage, function points size the system from what it does — external inputs, external outputs, external inquiries, internal logical files and external interface files, weighted and adjusted — which means you can estimate before any code exists, and compare systems written in different languages. Specification quality can be measured too: specificity is the fraction of requirements that exactly one reader interpretation fits. At the design stage there are structural measures, and for object-oriented designs the CK suite: weighted methods per class, depth of inheritance tree, number of children, coupling between object classes, response for a class, and lack of cohesion in methods. Note that two of those are our old friends coupling and cohesion made countable — a design concept turned into a metric. At the code level sit Halstead's measures and cyclomatic complexity, which doubles as a test-design driver and a restructuring flag.

Process metrics are collected across many projects over long periods, and their purpose is strategic: long-term process improvement. Project metrics are collected inside one project and are tactical: assess status, track risks, expose problems before they go critical, adjust workflow, and judge the team's ability to control quality. Many of the same measures serve both — the difference is the time horizon and who acts on them.

Quality metrics aim at the delivered product. Correctness is commonly defects per KLOC; maintainability is often mean time to change; integrity and usability have their own instruments. The most useful of the family is defect removal efficiency, DRE = E / (E + D), where E is the number of errors found before delivery and D the number of defects found after. DRE approaches 1 as more problems are caught inside the process, and it can be computed per framework activity, in which case a low value tells you exactly which activity is leaking errors downstream. Maintenance has its own metric, the software maturity index, which rises toward 1 as the number of modules added, changed and deleted per release falls — the product is stabilising.

The ethic, which is examinable

Process metrics can be misused so badly that they cause more harm than they cure, and there is a short etiquette worth reproducing: interpret data with organisational sensitivity; give regular feedback to the people who collect it; never use metrics to appraise individuals; set goals with the practitioners rather than for them; never use metrics to threaten a team; treat data that shows a problem as an indicator for improvement rather than as evidence against someone; and do not obsess over a single metric. The third of those is the one worth a mark in a written answer, because it is the one a naive answer violates.

Worked example — a metrics set for BorrowBox, chosen by GQM

BorrowBox's business goal for release 3 is stated as reduce the number of members who abandon a collection at the locker. That is a goal, not a metric. The questions it implies are: how often does a collection fail at the locker? where in the flow does it fail? are failures getting rarer? and are we catching these problems before members do?

Only now do metrics get chosen, one per question: field failure rate per 100 collections (project, tactical, from telemetry); failures split by stage — token expired, door timeout, wrong locker (product, localising); the same rate release over release (process, strategic); and DRE for the locker component per release (quality). Four metrics, each traceable to a question, each traceable to the goal. What is deliberately not collected is as informative: no lines of code per engineer, no defect counts attributed to named individuals — both violate the etiquette and both would corrupt the data within one sprint, because people optimise what they are measured on.

Families first, instances second. The point of learning the taxonomy is not recitation: it is that "which metrics would you use?" is answerable only if you can say which family a candidate metric belongs to and therefore what decision it can support.

GQM read left to right on the BorrowBox goal. Every metric in the right-hand column is defensible because a question in the middle column asked for it, and every question is defensible because the goal implied it. Anything with no path back to the goal is not collected.