Anomaly detection, signature detection, sensors, and Snort

Two approaches, and the same attack seen twice

Every intrusion detector analyses its data one of two ways, and the difference is what model it holds.

Signature detection — also called misuse detection, and the two words mean exactly the same thing — holds a model of what attacks look like. It compares observed behaviour against a database of known malicious patterns and rules. It is cheap, it is fast, and its verdicts are specific enough to act on. Its limitation is total and structural: it cannot detect anything nobody has written a signature for.

Anomaly detection holds a model of what normal looks like. It runs in two phases — a training phase in which the model is built from observed legitimate operation, and a detection phase in which departures from that model are flagged. It is the only approach that can catch an attack that did not exist yesterday. Its limitations are equally structural: the cost of collecting enough good training data, and a false alarm rate that the previous lesson should have made frightening.

Watch the same attack through both. A worm exploiting a vulnerability disclosed this morning crosses the monitored segment. The signature detector sees traffic matching none of its rules, and passes it. The anomaly detector sees a host that normally talks to three machines suddenly talking to three hundred, on a port it has never used, at a bandwidth it has never reached — and raises an alarm, alongside a dozen other alarms it raised this hour for a backup job and a new starter.

That is the trade, stated honestly: signature detection knows attacks and misses the new ones; anomaly detection knows normal and pays for the new ones in false alarms.

The three anomaly categories, and the model ladder inside the first

Anomaly detection sorts into three categories:

1. Statistical — analyse the captured data with univariate, multivariate or time-series models. Simple, cheap, no assumptions about known behaviour, but hard to choose good metrics for.
2. Knowledge-based — an expert system of rules describing legitimate behaviour, developed by hand. Robust and flexible; expensive in expert time.
3. Machine learning — build the model automatically from labelled normal training data. Flexible and adaptive, captures interdependencies between metrics — and pays for it with high resource cost and an often unacceptable false alarm rate.

Inside the statistical category is the ladder that gets examined:

• Univariate — each metric modelled on its own, as an independent random variable. Too crude to discriminate well.
• Multivariate — models the correlations between two or more metrics, on the reasoning that intrusive behaviour shows up in the relationship between measures even when each measure alone looks ordinary.
• Time-series — uses the order of events and the intervals between them, flagging a sequence that is improbable in time.

One deep observation about all of this, and it applies to the machine-learning category most sharply: an IDS anomaly model is trained only on legitimate data, unlike anomaly detection almost everywhere else, which trains on both classes. That absence is not an oversight — it follows directly from wanting to detect attacks that do not exist yet — and it caps what any technique in the category can achieve.

Where the sensors go

A NIDS examines traffic packet by packet in near real time. Its sensors come in two modes. An inline sensor sits in the traffic path, so it can block — that is intrusion prevention. A passive sensor monitors a copy of the traffic; it is more efficient because it adds no delay to the packets, and it taps the medium using a network interface with no IP address configured at all, so it participates in no protocol and is very nearly unattackable.

There are four classic placements, and each one answers a different question:

• Outside the external firewall. Sees all traffic, unfiltered — which documents the volume and type of attacks the organisation attracts. It carries the highest processing burden of any location, and its matches include everything the firewall was always going to block, so it measures attack volume, not exposure.
• Just inside the external firewall. Sees what got through the perimeter, which makes every match meaningful: either the firewall policy has a hole or its enforcement has a problem. It can also sometimes spot the outbound traffic of an already-compromised internal server even when the inbound attack was missed.
• On major backbone networks. High traffic volume, tunable to the protocols in use, and — the reason it exists — it detects unauthorised activity by authorised insiders, who never cross the perimeter at all.
• On departmental LANs and critical subsystems. Focuses limited detection resources on the highest-value assets.

The modern caveat the book raises and does not solve: with pervasive encryption, a NIDS has lost access to most payload content. It can still see who talks to whom, how much and when — which is exactly why the behavioural worm signals above survive encryption and content matching does not.

Honeypots

A honeypot is a decoy system with no production role, filled with fabricated but attractive-looking information. It has three goals: divert an attacker from systems that matter, collect information about how they operate, and keep them engaged long enough for defenders to respond.

Its value comes from one syllogism, worth carrying in both directions. Nothing legitimate has any reason to touch it, so any attempt to communicate with a honeypot is a probe, a scan or an attack. And if a honeypot ever initiates outbound communication, it has already been compromised.

Low-interaction honeypots emulate services convincingly enough for a first contact, which catches intruders in the early stages of the attack methodology and makes excellent early warning for a distributed IDS. High-interaction honeypots are real systems with real operating systems: far more realistic, far more expensive, and — the risk that decides the placement question — if compromised, they can be used to attack other people.

Worked example — reading a Snort rule

Snort is a lightweight, open-source IDS, deployable host-based or network-based, passive or inline. It has four components: a packet decoder (isolates the protocol headers at the data-link, network, transport and application layers; its primary work is setting pointers), a detection engine (checks each packet against the rules — the first rule that matches triggers that rule's action, and a packet matching nothing is discarded), a logger, and an alerter.

A rule is a seven-field header followed by options in parentheses. The header fields, in order, are action · protocol · source address · source port · direction · destination address · destination port. Addresses and ports allow lists, ranges and negation; the direction operator may be one-way or bidirectional, which tells Snort to watch both sides of a conversation.

Take this rule apart field by field:

alert tcp any any -> 10.1.0.0/16 80 (msg:"..."; content:"union select"; nocase;)

alert is the action — generate an alert and then log the packet. tcp is the protocol (Snort recognises tcp, udp, icmp and ip). any any is the source address and port: from anywhere, from any port. -> is the direction, one-way. 10.1.0.0/16 and 80 are the destination address and port. Everything after the parenthesis is options, separated by semicolons, each a keyword and its argument separated by a colon.

The options sort into four categories. Meta-data — msg, reference, classtype — carries information about the rule and has no effect on detection at all. Payload options look inside the packet: content searches for text or binary, and depth, offset and nocase modify the content keyword immediately before them, not the rule as a whole, so their position is load-bearing. Non-payload options test header fields: ttl, id, flags, seq, and dsize, which tests payload size and is genuinely useful for spotting buffer overflows. Post-detection options such as logto and session fire after a match.

The eight actions are alert, log, pass, activate (alert, then enable a dynamic rule), dynamic (idle until activated, then behaves as a log rule), drop, reject (drop and log, then send a TCP reset or an ICMP port-unreachable), and sdrop (drop without logging). The last three are inline-only — that is the detection/prevention boundary made concrete, and a drop rule in a passive deployment does exactly nothing.

Two traps to carry into the assignment. Rule order changes behaviour, because the first match wins: a broad pass rule early in the file silently disables everything below it. And the same rule means different things at different sensor locations — outside the firewall a match is an attempt, inside the firewall the same match is a penetration.