AH versus ESP, and transport versus tunnel mode

The coverage rule, one line each

Everything examinable about the two IPsec modes reduces to a single question: what does the protection cover?

• Transport mode protects the IP packet's payload — a TCP or UDP segment — but not the IP header. It is the host-to-host, end-to-end case: two machines that both run IPsec and talk directly.
• Tunnel mode protects the entire original IP packet, including its IP header, by making that whole packet the payload of a fresh outer IP packet with a new outer header. It is used when one or both ends of the SA is a security gateway — a firewall or router running IPsec on behalf of the hosts behind it, which then need no IPsec of their own.

The paper asks this in both directions: protection that stops at the payload and leaves the original IP header exposed is transport mode, and protection that covers the original packet in full, its header included, is tunnel mode. A learner who has memorised only "tunnel means VPN" will get one of the two right by luck. Memorise what gets wrapped instead — the payload, or the whole packet — and both directions fall out.

Worked example — one packet through a tunnel, in five steps. Host A on the branch network sends an ordinary IP packet to host B at head office. Neither host knows anything about IPsec.

1. A emits a normal IP packet, source A, destination B.
2. It reaches the firewall at the boundary of A's network, which filters all outgoing packets to decide which need IPsec processing. This one is going to head office, so it does.
3. The firewall performs the IPsec processing and encapsulates the packet in an outer IP header whose source is the firewall itself and whose destination is B's firewall.
4. The packet crosses the Internet. Every intermediate router reads only the outer header. The inner header — A's real address and B's real address — is inside the protection and unreadable, which is how tunnel mode hides the real endpoints. The outer packet may carry completely different addresses, and that is security gained for free.
5. B's firewall strips the outer header and delivers the recovered inner packet to B, which sees an ordinary packet from A.

Transport mode cannot do any of that, because the original IP header is the thing it does not cover.

AH versus ESP

IPsec offers a combined authentication-and-encryption function, ESP, and an authentication-only function, the Authentication Header (AH). Because message authentication is already provided by ESP, AH is deprecated: it stays in IPsecv3 for backward compatibility and should not be used in new work. The 5e says so explicitly and declines to discuss it further.

It is still examinable, because the standard question "what are the two ways of providing authentication in IPsec?" has exactly two answers: ESP's optional authentication service, and AH.

AH's header, from RFC 4302, is six fields:

• Next Header — 8 bits, naming the protocol that follows.
• Payload Length — 8 bits, the AH header's own length in 32-bit words, minus two.
• Reserved — 16 bits, sent as zero.
• Security Parameters Index — 32 bits, the same role it plays in ESP.
• Sequence Number — 32 bits, feeding the same anti-replay window.
• Integrity Check Value — variable, a whole number of 32-bit words.

That is a 12-byte fixed portion plus the ICV, and — the structural difference from ESP — AH is a header only. There is no AH trailer, because there is nothing to pad: AH never encrypts.

Why AH breaks through NAT and ESP does not

AH authenticates the payload and the immutable fields of the IP header. A NAT device works by rewriting the source address in the IP header. Those two facts cannot both hold: the rewrite changes bytes AH covered, so the ICV recomputed at the far end no longer matches and the packet is discarded as forged. ESP does not authenticate the outer IP header at all, so NAT can rewrite it freely and ESP still verifies. This is not a quirk — it is the single most practical reason ESP won.