Buffer overflow: the stack frame and the return address

What a buffer overflow actually is

A buffer overflow — equivalently a buffer overrun — occurs when the input accepted into a buffer exceeds the capacity set aside for it, so whatever was stored alongside is written over. Nothing exotic happens at the machine level. The write is an ordinary store instruction, issued by the program itself, at an address the programmer never intended it to reach. The processor has no idea a boundary was crossed, because in C there is no boundary recorded anywhere for it to check.

Buffers live in three places, and an overflow can happen in any of them:

• the stack, where a function's local arrays live for the duration of a call;
• the heap, where malloc hands out dynamically allocated blocks;
• the global/static data area, where variables declared outside any function live for the life of the process.

The consequences run on a scale. At the mild end, corrupted data and a value that is silently wrong. In the middle, a memory access violation and a crash — an accidental denial of service. At the far end, execution of arbitrary code with the privileges of the attacked process. It is the same bug in every case. What separates a nuisance from a full compromise is not the flaw but the attacker's degree of control over what gets written, and where.

To exploit one deliberately an attacker needs two things: (1) a buffer overflow reachable by externally sourced data under their control, and (2) an understanding of how that buffer is laid out in process memory. The second requirement is what makes real exploits architecture- and operating-system-specific, and it is why an overflow that merely crashes on one machine can be exploitable on another. Vulnerabilities are found by source inspection, by tracing execution with oversized inputs, and by fuzzing.

The call and return protocol — the prerequisite you cannot skip

You cannot reason about a stack overflow without being able to draw a stack frame. So draw one. When function A calls function B:

1. A pushes the arguments, conventionally in reverse declaration order.
2. A executes the call instruction, which pushes the return address — the address in A at which execution resumes.
3. B pushes the current frame pointer (A's), so it can be restored later.
4. B sets the frame pointer to the stack pointer, fixing a stable base for its own storage.
5. B allocates its local variables by moving the stack pointer down.

On return B reverses all of it: restore the stack pointer from the frame pointer, pop the saved frame pointer, and return — which pops the return address straight into the program counter. A then pops the arguments it pushed.

Two facts about that layout decide everything that follows. The stack grows downward, toward lower addresses. But an array is indexed upward, toward higher addresses. So a local buffer is filled from its own lowest address toward higher ones — and higher, inside a stack frame, is precisely where the saved frame pointer sits, and above that the return address. That is why an overflowing local array can reach the return address at all, and it is the single fact learners most often draw upside down. Get the direction wrong and the whole chapter stops making sense.

Worked example — 24 bytes into an 8-byte buffer

Take a 32-bit frame for a function whose only local is char buf[8]. Pointers are four bytes wide. Counting offsets from the start of the buffer, upward:

Now write 24 bytes starting at buf[0]. Byte by byte:

• bytes 1–8 fill the buffer exactly, and nothing is wrong yet;
• bytes 9–12 land on the saved frame pointer;
• bytes 13–16 land on the return address;
• bytes 17–24 run on into the caller's argument area.

So the distance from the buffer's start to the return address is 12 bytes, and the first byte that touches it is byte 13. An attacker who wants to redirect control needs exactly twelve bytes of padding, then four bytes of address — and on a little-endian machine that address is written least significant byte first. The padding must be exact; the address may be approximate, which is what the NOP sled in the next lesson is for.

Two details are routinely forgotten. First, the saved frame pointer must itself be a plausible stack value: a garbage value there crashes the program on restore, before the doctored return address is ever used. Second, a string copy writes a terminating NUL past the last character, so a 24-character string actually corrupts 25 bytes.

The case with no control-flow hijack at all

Here is the variant that gets missed, and it is the one Assignment 3 Q4 is built on. Suppose a function has two eight-byte local arrays laid out consecutively — one holding a value the program expects, one holding a value the user supplied — and then compares them. An oversized read into the second can overwrite the first. If the attacker supplies sixteen well-chosen characters, both arrays end up holding the same bytes, the comparison succeeds, and the program takes the privileged branch.

No return address was touched. No code was injected. The attacker controlled program logic purely by corrupting an adjacent local variable. This case matters far beyond its novelty, because none of the defences in the next-but-one lesson stop it: a stack canary sits between the locals and the return address, so a write from one local into another never crosses it; a non-executable stack is irrelevant when no code is injected; address randomization is irrelevant when no address is guessed. Only correct coding prevents this one.