SQA, statistical SQA, reliability, availability and safety

What the SQA function actually does

The SQA group is best understood as the customer's in-house representative: it looks at the software from the buyer's point of view and asks whether the adopted standards were followed and whether the practices were really conducted. Its tasks are concrete — prepare the SQA plan during project planning and have all stakeholders review it; participate in describing the process; review engineering activities for conformance to it; audit work products; ensure deviations are documented and resolved rather than quietly dropped; record and report. Modern SQA is data-driven: stakeholders define goals and quality measures, indicators are measured, and a decision follows about whether the process must change — goals spanning requirements, design and code quality plus quality-control effectiveness. Around all of it sit the standards frameworks: ISO 9001 registration, audited by third parties, and the SQA plan in the shape IEEE 730 describes.

Statistical SQA — five steps, and the Pareto move is the method

Statistical software quality assurance makes the quality conversation quantitative. Five steps. One, collect errors and defects and categorise them. Two, trace each to its underlying cause — incomplete specification, misinterpreted customer communication, violated standards, ambiguous interface. Three, apply the Pareto principle to isolate the vital few: roughly 20% of causes produce roughly 80% of defects. Four, correct those causes. Five, measure again and repeat. Step three is the whole method — a team that categorises but never isolates has a reporting habit, not a quality programme. Six Sigma packages the same idea: define, measure, analyse, improve, control — DMAIC — for an existing process, and DMADV (define, measure, analyse, design, verify) when the process is being created rather than repaired.

Reliability is a probability with two qualifiers

Software reliability carries the standard statistical definition used across the field and the standards: the probability of failure-free operation of a computer program in a specified environment for a specified time. Both qualifiers are part of the definition, not decoration: a figure without an environment and a duration is meaningless. A program with reliability 0.999 over eight processing hours runs correctly 999 times in a thousand such runs. Here failure means nonconformance to requirements, and failures grade from irritating to catastrophic.

Three measures. MTBF = MTTF + MTTR — mean time between failures is mean time to failure plus mean time to repair. FIT counts failures per billion hours. Availability = MTTF / (MTTF + MTTR) × 100%. Two nuances earn marks. MTBF beats a raw defect count because faults do not have equal failure rates: a program may hold defects that will not surface for tens of thousands of hours, and removing those changes what users experience by nothing. And availability is more sensitive to MTTR than MTBF is — MTTR being an indirect measure of maintainability, which is how a design property shows up in an operational number. Scheduled downtime also costs availability while involving no failure at all.

ISO 9000 and ISO 9001 — the process, audited by an outsider

Everything above is what an organisation does for itself. ISO 9000 is what it does when somebody outside wants evidence. It is a family of generic quality-management standards — generic in the strict sense that one standard applies to any organisation making any product, from turbines to insurance policies — and its subject is the quality system, not the product. It says what elements a quality system must contain; it never says how a particular organisation must implement them, which is why it can be generic in the first place.

The member of the family that matters for software is ISO 9001, the standard describing the quality system used to maintain and improve quality across design, development, production, installation and servicing. Its requirements group into a small number of themes: management responsibility, a documented quality system, contract and design control, control of documents and of purchasing, product identification and traceability, process control, inspection and testing, corrective and preventive action, control of quality records, internal audits, training, and statistical technique. Read that list and the point becomes clear — almost every item is something SQA already does. What the standard adds is the requirement that it be documented, followed and auditable by a third party, which is what turns a good practice into a certificate a customer or a regulator can rely on.

The honest limitation is the standard’s own: registration certifies that a process is defined and followed, not that the product is good. An organisation can be ISO 9001 certified and ship poor software, provided it does so in a documented and repeatable way. That is not an argument against the standard; it is the reason certification is a floor and not a claim.

Safety asks a different question

Software safety is an SQA activity that identifies and assesses potential hazards — conditions that could cause the entire system to fail — early enough that design features can eliminate or control them. The process: identify hazards, categorise them by criticality and risk, assign each a severity and a probability, and analyse the chains that lead to them with fault tree analysis, real-time logic or Petri nets. The output is a set of safety-related requirements — undesirable events plus the responses the system must make.

The distinction from reliability is the mark-bearing sentence. Reliability uses statistics to estimate whether a failure occurs. Safety examines what a failure does — the ways a failure produces conditions that lead to a mishap — evaluating failures not in a vacuum but in the context of the entire computer-based system and its environment, humans included. A rare failure can therefore be the most dangerous thing in the system, and "more reliable" does not imply "safer".

Worked example — the BorrowBox locker service on all three axes

Reliability. The locker service has run 3,000 processor-hours since the last release, with MTTF 700 hours and MTTR 2 hours: MTBF is 702 hours and availability 700/702 ≈ 99.7%, inside the 99.5% the requirements ask for during opening hours. What improves that fastest is not removing more defects but cutting MTTR — here, a watchdog that restarts the door controller without a site visit.

Statistical SQA. A year of BorrowBox data yields 120 defects across five categories; traced to cause, 61% come from just two — incomplete requirements on the fee rules, and ambiguous locker-vendor documentation. Neither is fixed by more testing. The corrections are process changes: acceptance criteria for every fee rule, and a vendor interface contract reviewed before construction.

Safety. The locker firmware has a fault that releases all latches on an unexpected power-cycle. Its probability is very low — good reliability, and it has never fired — but its hazard is catastrophic: the entire tool bank open on a public street. The fault tree shows the mishap needs three things together, the fault plus a mains outage plus no staff on site, so the mitigation is not more firmware testing but a mechanical fail-secure latch that holds without power. Reliability analysis, which ranks by likelihood, would never have produced that decision.