The base-rate fallacy and NIDS alarm probability

◈ 6 cards

Why an IDS that catches 99% of intrusions with a 1% false alarm rate still produces almost nothing but false alarms — and how to condition correctly on a multi-node threat table.

The founding assumption, and the overlap it admits

Intrusion detection rests on one assumption: an intruder behaves differently from a legitimate user, in ways that can be quantified but not crisply. The two behaviour distributions overlap. That is not a flaw in any particular product; it is the shape of the problem.

Because they overlap, every detector sits on a slider. Interpret loosely and you catch more intruders and produce more false positives. Interpret tightly and you cut the false positives and produce more false negatives — missed intrusions. The two quantities are:

detection rate = intrusions detected ÷ total intrusions;
false alarm rate = normal events misclassified ÷ total normal events.

The whole art is a high detection rate at a low false alarm rate. What follows is why "low" has to mean very low.

Worked example — the base-rate fallacy, with numbers

Take an IDS with genuinely good figures. It detects 99% of intrusions. Its false alarm rate is 1%. On the face of it, an alarm means an intrusion.

Now add the fact everybody leaves out: intrusions are rare. Suppose 1 audited event in 10,000 is an intrusion. Watch what happens to ten thousand events.

Intrusions: 1. The detector catches 99% of them, so it raises about 1 true alarm.
Normal events: 9,999. It misclassifies 1% of them, so it raises about 100 false alarms.

A hundred and one alarms. One is real.

Bayes says the same thing formally. With $I$ for "this event is an intrusion" and $A$ for "the IDS raised an alarm",

$Pr (I ∣ A) = \frac{Pr ( A ∣ I ) Pr ( I )}{Pr ( A ∣ I ) Pr ( I ) + Pr ( A ∣ \neg I ) Pr ( \neg I )}$

$Pr (I ∣ A) = \frac{0.99 \times 0.0001}{0.99 \times 0.0001 + 0.01 \times 0.9999} \approx 0.0098$

About 0.98%. Ninety-nine alarms in a hundred are false, from a detector most people would describe as excellent.

The part that changes how you think about products is what happens when you try to fix it. Push the detection rate from 99% to a perfect 100% and the posterior moves from 0.98% to 0.99% — nothing. Cut the false alarm rate by a factor of a hundred, from 1% to 0.01%, and the posterior jumps to about 50%. The false alarm rate is the only lever that matters, because the false alarms are drawn from the enormous normal population and the true alarms are drawn from the tiny intrusion population.

The consequence is operational, not merely statistical. Too few detections give a false sense of security. Too many false alarms and the analysts start ignoring the alarms, at which point the detection rate is irrelevant because nobody is reading the output. You will run the second code exercise below and watch both sweeps; the numbers are worth seeing once, because after that you never mis-rank a detector again.

Worked example — conditioning on a multi-node NIDS

Now the second calculation, which is the one an assignment asks for.

A decentralised network IDS has three nodes. Each node independently reports exactly one of four signatures, $S_{1}$ through $S_{4}$ , and — this is a modelling assumption you must state, because the question does not — each signature is equally likely, so each has probability $\frac{1}{4}$ . The three reports go to a central node which classifies the threat from the combination it receives:

Threat level	Combination
Low	two $S_{1}$ and one $S_{3}$
Medium	two $S_{2}$ and one $S_{3}$
High	two $S_{4}$ and one $S_{2}$
Very High	three $S_{4}$

Question. Given that at least one node reports $S_{2}$ , what is the probability the classification is High?

Step 1 — state the sample space. Three nodes, four signatures each, ordered because the nodes are distinguishable: $4^{3} = 64$ equally likely outcomes.

Step 2 — count the High outcomes. High is the unordered combination two $S_{4}$ and one $S_{2}$ . The table names a multiset, so every arrangement of it counts: the lone $S_{2}$ can be at node 1, node 2 or node 3. That is $(1 3) = 3$ outcomes, so $Pr (High) = \frac{3}{64}$ . This is the first place marks are lost — reading the table as a fixed order gives 1 instead of 3.

Step 3 — compute the conditioning event. "At least one $S_{2}$ " is easiest by complement. The probability that a given node does not report $S_{2}$ is $\frac{3}{4}$ , and the nodes are independent, so

$Pr (\geq 1 S_{2}) = 1 - (\frac{3}{4})^{3} = 1 - \frac{27}{64} = \frac{37}{64}$

Step 4 — intersect. Every High outcome contains exactly one $S_{2}$ , so every High outcome already satisfies the condition. The intersection is the whole of High: $\frac{3}{64}$ .

Step 5 — divide.

$Pr (High ∣\geq 1 S_{2}) = \frac{3/64}{37/64} = \frac{3}{37} \approx 0.0811$

About 8.1%. The second place marks are lost is skipping step 3 entirely and answering $\frac{3}{64} \approx 0.047$ — that is $Pr (High)$ , not the conditional probability that was asked for.

The first code exercise below enumerates all 64 outcomes and confirms every count, using exact fractions rather than simulation, because a probability question deserves an exact answer.

The same detector, drawn as a population rather than as two percentages. One hundred and one alarms; one of them is real. Nothing about the detector is bad — the base rate is doing all the damage.

The table names unordered combinations, so every arrangement counts. Three, three, three, one — and the remaining fifty-four outcomes fall into no level at all.