Compile-time and run-time defences: canaries, NX, and ASLR

Two families, decided by one question

Defences against buffer overflow split by when they act, and the split is forced by a practical fact: there is an enormous installed base of vulnerable binaries that nobody is going to recompile.

Compile-time defences harden new programs. They require a rebuild, and often a source change.

Run-time defences protect programs that already exist, as operating-system updates, with no recompilation. That is the whole reason they were invented.

Compile-time: four approaches

Choice of programming language. A strongly typed language whose compiler inserts range checks automatically removes the entire class. The cost is real: extra memory and processor time for the checks, and loss of access to some instructions and hardware, which is why device drivers and kernels resist the change. And the protection is not total — such a program may still be vulnerable through system libraries written in unsafe languages.

Safe coding techniques. C's assumption that the programmer is always right is exactly what makes C fast and exactly what makes it dangerous. The discipline is to code with constant awareness of how things can go wrong and to fail gracefully; the rule that carries the most weight is that any code writing into a buffer must first check that space is available.

Language extensions and safe libraries. Automatic bounds checking is easy for statically allocated arrays and hard for dynamic allocations whose size is not known at compile time. A replacement standard library that reimplements the dangerous routines safely can be loaded before the system libraries, and therefore protects existing binaries without recompiling them — a rare compile-time idea that is deployable at run time.

Stack protection. The canary, below.

The stack canary

Instrument every function so that on entry it writes a canary value between the saved frame pointer and the local variables, and on exit checks that value before returning. Any classic stack overflow — one that walks from a local buffer up to the return address — must cross the canary and alter it, so the check fails and the program aborts instead of returning into attacker-controlled code.

The critical requirement is that the canary be unpredictable and different on every system, typically randomised per process. A fixed canary is no defence at all: the attacker simply includes the correct value at the right offset in the payload.

Costs: every protected program must be recompiled, and the changed frame layout can break debuggers. A variant avoids the second cost by keeping a copy of the return address in a separate protected region and comparing on exit, leaving the frame layout untouched.

And note the scope limit, which is the examinable part: canaries detect stack-frame corruption and nothing else. Heap overflows, global-data overflows and the adjacent-local-variable corruption from L11.1 all pass a canary check untouched.

Run-time: three mechanisms

Non-executable memory (NX / DEP). Tag the stack's pages non-executable in the memory management unit, so that even if the attacker lands the program counter in the buffer, the processor refuses to execute what is there. It breaks legitimate uses of executable stack memory — just-in-time compilers, nested functions, some signal handlers — which is why it needs an opt-out.

Address space layout randomization (ASLR). Randomise where things are, so that the attacker's guessed address is wrong. Three distinct targets, and knowing all three is the exam point: randomise the stack's location per process; randomise dynamic memory allocation, which defeats heap attacks relying on successive allocations being adjacent; and randomise the order and addresses at which shared libraries load.

Guard pages. Place unmapped gaps between regions of the address space and flag them illegal, so any access aborts the process. A large overflow crosses one and dies before it reaches anything worth corrupting.

The ladder — each attack is a reply to a defence

Read the attack techniques as responses, not as a second list.

Canary → the replacement stack frame attack. Overwrite only the saved frame pointer, leaving the canary and the return address untouched, and point it at a dummy frame the attacker built inside the buffer. The current function returns normally; then the caller restores the doctored frame pointer and returns into the attacker's code. The canonical enabler is an off-by-one error — a <= where < was meant — which changes exactly one byte, and one byte is enough to move the frame pointer to a dummy frame a few tens of bytes away. Note this is harder than a classic overflow, not easier: an exact address goes into the frame pointer, so there is no NOP-sled slack.

Non-executable stack → return-to-libc. Do not inject code at all. Overwrite the return address with the address of an existing library function such as system(), and lay out a placeholder return address followed by the parameters, so the library function finds a stack that looks exactly as it would after a normal call. Chaining several such calls — classically using strcpy() to copy bytes somewhere executable and then jumping there — is the direct ancestor of return-oriented programming.

ASLR → an information leak first. Randomization does not fix the bug; it makes the guess wrong, so the program crashes instead of being exploited. Two honest limits follow. An attacker who can leak a single address — through a format-string bug, an over-read, or a verbose error message — recovers the whole layout, because the regions move together. And an attacker who can make many attempts, each with a different guess, defeats it by brute force, which the exercise below quantifies.