Testing strategy: V&V, organizing for testing, scaffolding, and stopping

A strategy, not a pile of tests

A testing strategy is a template you fill with tactics, and every serious one shares the same shape. Technical reviews come first, so that defects which never needed a running program to find are gone before anyone spends machine time on them. Testing then begins at the component level and works outward — one component, then the interfaces between components, then the software against its requirements, then the software inside the complete computer-based system. Testing is performed by the developer and, on anything large, by an independent group as well. And testing is not debugging: testing finds symptoms, debugging finds causes, and a strategy that does not budget separately for the second will overrun.

The outward order is not an arbitrary convention. Each ring can only be entered when the ring inside it is trustworthy, because an integration failure with untested components underneath is uninterpretable — you cannot tell an interface defect from a component defect, and localisation is the entire value of testing incrementally.

Verification and validation

Verification is the set of tasks that confirm the software correctly implements a specific function; Boehm's phrasing is are we building the product right? Validation is the different set of tasks that confirm what has been built is traceable to what the customer actually needs: are we building the right product? The distinction is about what you check against. Verification checks a work product against the specification that preceded it — code against design, design against requirements — and it can be done entirely inside the team. Validation checks the built system against the customer's real need, and it can never fully succeed without the customer, because the team is not the authority on what was wanted.

Two consequences matter for the exam. First, V&V is much larger than testing: quality and configuration audits, technical reviews, feasibility studies, performance monitoring, algorithm analysis, documentation and database review, simulation, usability testing, acceptance testing and installation testing all sit inside V&V. Second, quality cannot be inserted at the end. Verification can pass perfectly on a product nobody wants, so a strategy that runs only verification is systematically blind to the most expensive class of defect there is.

Who tests, and the conflict of interest

Developers should test their own components — nobody else understands them as fast. But a developer has a psychological stake in the code passing, and a subtle interest in not constructing the test that breaks it. That is why larger projects add an independent test group, which is paid to find defects rather than to ship. The ITG is not a replacement for developer testing and it is not a separate department that owns quality; it is a second set of incentives applied to the same artifacts.

Scaffolding, and a criterion for done

Testing a component in isolation usually needs scaffolding: a driver is a throwaway main program that calls the component under test and feeds it data; a stub is a throwaway stand-in for a component that the one under test calls. Both are real code with real cost, which is one of the arguments in the integration-strategy choice later in this module. Finally, when do we stop? — never when the calendar says so. A defensible criterion is statistical: model the defect arrival rate, stop when it has flattened and the projected residual defect density is below an agreed threshold, with every acceptance criterion met. That is a claim someone can audit; we ran out of time is not.

Worked example — V&V on the BorrowBox three-second budget

BorrowBox has a non-functional requirement: the locker door must open within three seconds of a valid scan. Verification activities: review the architecture against the budget and confirm the reservation service is not on the critical path; instrument the component that signs the open command and assert its worst-case latency in a unit test; audit the configuration so the shipped build has debug logging off, since it added 900 ms. All of that can happen with no customer in the room and all of it can pass. Validation is the different question — and it is the one that found the real defect. Watching six members collect tools in a car park showed that the three seconds is measured from a scan that has already succeeded, whereas what a member experiences begins when they take their glove off to wake the phone. The system met its specification exactly and missed the requirement the members actually had. Verification could not have found that, because the specification itself was wrong.