IPsec: security associations and the ESP packet

Why put security at the IP layer at all

Every protocol in this module so far secures one application: S/MIME secures mail, TLS secures a transport connection an application deliberately opened. But some concerns cut across protocol layers, and an application that was never written with security in mind gets no benefit from any of them. Putting security at the IP layer protects the security-ignorant applications too, because everything travels over IP whether it asked to or not.

IPsec's three functional areas are authentication (this packet really came from the source named in its header, and nothing in it changed en route), confidentiality, and key management. Its five benefits follow from where it sits:

1. In a firewall or router it applies strong security to all traffic crossing the perimeter, while traffic that stays inside the workgroup avoids the overhead entirely.
2. It is resistant to bypass when the firewall is the only entrance from the outside.
3. It is below the transport layer, and therefore transparent to applications — no software changes anywhere.
4. It can be transparent to end users: no training, no per-user keying material to issue and revoke.
5. It can still secure individual users where that is what you need — a remote worker, or a secure virtual subnetwork inside an organisation.

Four RFCs are worth memorising as a set: 4301 (architecture) · 4302 (AH) · 4303 (ESP) · 4306 (IKEv2).

A security association is one-way, and that is the whole point

A security association (SA) is IPsec's unit of protection: a single agreement, running in one direction only, that fixes the algorithms, keys and counters applied to everything a sender sends to one receiver. Read "one-way" literally and the most-tested consequence falls straight out: a two-way secure exchange requires two security associations, one per direction, each with its own keys, its own sequence counter and its own database entry.

An SA is uniquely identified by three parameters:

• the Security Parameter Index (SPI) — a bit string with local significance only, carried in the ESP header so the receiver can pick the right SA to process the packet under;
• the IP destination address — which may be an end system or a firewall or router;
• the protocol identifier — a field in the outer IP header saying whether this is an AH or an ESP association.

Worked example. Host A at a branch office and host B at head office run one bidirectional session over IPsec. That is two SAs: A→B with SPI 0x3A91 in A's outbound entry and B's inbound entry, and B→A with a completely unrelated SPI, 0x00C4, chosen by B. Neither SPI means anything outside the machine that assigned it. If A's inbound processing looked up B's traffic under 0x3A91, it would find nothing — the two directions share nothing but the endpoints.

Beyond the three identifiers, the security association database (SAD) entry holds a 32-bit sequence number counter, a sequence counter overflow flag, the anti-replay window, the AH information and the ESP information (algorithms, keys, lifetimes), the lifetime of the SA as a time interval or byte count, the IPsec protocol mode (tunnel, transport or wildcard) and the observed path MTU.

The ESP packet, field by field

ESP provides confidentiality of message contents and limited traffic-flow confidentiality, plus an optional authentication service. Its packet has seven fields:

1. Security Parameters Index — 32 bits. Identifies the SA.
2. Sequence Number — 32 bits. A monotonically increasing counter, and the input to the anti-replay window.
3. Payload Data — variable. A transport-level segment in transport mode, or a whole IP packet in tunnel mode.
4. Padding — 0 to 255 bytes. Needed when the encryption algorithm wants the plaintext to be a multiple of some number of octets.
5. Pad Length — 8 bits. How many pad bytes immediately precede this field.
6. Next Header — 8 bits. Identifies the first header inside the payload — an IPv6 extension header, or an upper-layer protocol such as TCP.
7. Integrity Check Value — variable, an integral number of 32-bit words. Computed over the ESP packet excluding the authentication data itself.

The groupings matter as much as the widths. Fields 1–2 are the ESP header; fields 4–6 are the ESP trailer; field 7 is the authentication data, appended after the trailer. And the two protective spans are different lengths: encryption covers the payload plus the trailer, while authentication covers the ciphertext plus the ESP header. The ICV excludes itself, for the same reason a certificate's signature is verified by hashing the certificate without its signature — a value cannot be an input to its own computation.

That asymmetry is why the SPI and Sequence Number sit outside the encryption: a receiver must be able to read them before it knows which key to decrypt with.

The anti-replay window

A sequence number alone does not stop replay if the receiver merely checks "is this bigger than the last one" — the network legitimately reorders packets, and a strict counter would discard honest traffic. IPsec uses a sliding window of size W, default 64, whose right edge is the highest sequence number received so far on a valid, authenticated packet, with a marked slot for every correctly received packet inside the window. Three cases, and only three:

• To the right of the window — accept, authenticate, and advance the window.
• Inside the window and unmarked — accept, authenticate, and mark the slot. This is the case that tolerates reordering.
• Inside the window and already marked, or to the left of the window entirely — discard as a replay.

This works only because the sequence number is inside the ICV's coverage. An attacker who could edit it freely would simply renumber the replay.