OS hardening: four steps to secure it, five to keep it that way

A system is insecure the moment it is installed

Start from an uncomfortable fact: operating systems and applications ship with vulnerabilities already in them. Not through negligence — through scale. Between the day an image is cut and the day you install it, researchers and attackers have both been reading the same code, and worms are scanning the address space continuously looking for exactly what they found. So there is a window, between the machine is on the network and the machine is patched and hardened, in which the machine can be compromised during its own installation. That is not a hypothetical; it is why building and deploying a system has to be a planned process rather than an afternoon's work with an installer.

It helps to see the system as a stack, because that is how attackers see it: physical hardware at the bottom; then UEFI or BIOS firmware and other code that sits outside the kernel and is largely invisible to it; then the base operating system — privileged kernel code, system APIs, services; then user applications and utilities on top. Each layer needs hardening, and each is vulnerable to attack from the layer below it. A rootkit that gets under the kernel does not need to defeat the kernel's access controls; it simply lies to them. That is the same ladder Module 8 climbed from the attacker's side, drawn here from the defender's.

The four steps, and the five that follow

NIST's Guide to General Server Security (SP 800-123) frames securing a system as four basic steps:

1. Install and patch the operating system.
2. Harden and configure it for the security needs identified during planning — remove unnecessary services, applications and protocols; configure users, groups and permissions; configure resource controls.
3. Install and configure additional security controls where needed — anti-virus, a host-based firewall, host IDS, application allow-listing.
4. Test the security of the base operating system to confirm the first three actually did what you intended.

Security is not finished at step 4, because the environment does not hold still: new vulnerabilities are disclosed, new threats appear, and the system's own configuration drifts. The same guide gives five maintenance steps: monitoring and analysing log data · performing regular backups · recovering from security compromises · regularly re-testing system security · using proper software maintenance processes to patch and update all critical software and to revise the configuration as needed. Notice that steps 1 and 4 of the build reappear as maintenance steps: the process is a loop, not a line.

The empirical argument, and where the numbers come from

The Australian Signals Directorate's Strategies to Mitigate Cyber Security Incidents is worth knowing by number because it is the rare security recommendation backed by an incident-response population rather than by opinion. ASD reported that implementing just four strategies — allow-list approved applications, patch third-party applications, patch operating-system vulnerabilities and run current versions, and restrict administrative privileges — would have prevented at least 85% of the targeted intrusions it investigated. In 2017 it extended those four to the Essential Eight by adding configuration of Microsoft Office macro settings, user application hardening, multifactor authentication, and regular backups. The eight together are a defence-in-depth set, and they align closely with the Critical Security Controls developed by other agencies.

Worked example — hardening the Rideau Library catalogue server

Rideau Library is a municipal library service. Its catalogue server holds borrower records and runs a public web catalogue plus a staff circulation application. Here is the build, step by step, with the threat each step closes.

Plan. Before any media is touched, write down what the system is for, what data it holds, which categories of user exist (public, circulation staff, cataloguers, administrators), how each is authenticated, what access it needs to other hosts, who will administer it and how, and what extra controls are required. This is not paperwork for its own sake: every later decision — which packages, which permissions, which logs — is derived from it, and retro-fitting security is far more expensive than planning it in.

Install minimally, on a protected network. Build it on a network with no inbound access and outbound access only to the distribution and patch mirrors, or on a fully isolated network with the image and patches carried on verified media. Install the base system and the two applications; do not install the compilers, the sample web scripts, the database GUI or the mail server that the installer offers, on the reasoning that software never installed cannot be exploited, re-enabled, or left behind by a failed uninstall script. Set a password on UEFI configuration changes and restrict the boot media, so an attacker cannot boot a system of their own choosing and read the disk around the operating system's access controls.

Patch, then configure. Enable automatic security updates. Remove the distribution's default accounts, disable interactive login on the service accounts, and change every default credential — a step so routinely skipped that scanning for unchanged defaults is a viable attack strategy on its own. Give the catalogue web server its own unprivileged account. Enforce the credential policy (length, complexity, age) and require multifactor authentication for administrators. Then set resource controls: borrower records readable by the circulation application and by nobody else; the web root readable but not writable by the account the web server runs as, with only the upload directory and the log files writable. That last line is the single most practical control in the chapter — the reason web defacements are so common is that the served files are owned by the account the server runs as, so any flaw in the server or one of its scripts rewrites all of them.

Add controls, then test. Install a host-based firewall permitting only ports 443 and the circulation application's port from the staff subnet; install anti-malware; turn on application allow-listing, which is feasible here precisely because a catalogue server's set of programs barely changes. Then test: run the hardening checklist, run a vulnerability scanner, and confirm each configuration step actually took effect. Testing is not a formality — it is how you learn that the firewall rule you wrote does not match the interface the service is bound to.

Maintenance: logging, and the backup/archive pair

Logging is, in SP 800-123's phrase, a cornerstone of a sound security posture — while being honestly reactive: it tells you about bad things that have already happened. Its value is that it lets you establish what happened quickly and accurately enough to focus remediation. Two practical constraints follow: logs consume real space, so allocate it and configure automatic rotation and archiving; and manual log review is tedious and unreliable, so analysis should be automated, which is exactly what an IDS does.

And then the definitional pair the examiners like, because candidates blur it. A backup makes copies at regular intervals so that lost or corrupted data can be recovered over a relatively short horizon — hours to weeks. An archive retains copies over an extended horizon — months to years — to meet legal and operational requirements to reach past data. They are often implemented by the same tooling and managed together, but they answer different questions, and the answers differ in retention, in medium and in who is allowed to delete them. Two decisions belong to planning: online or offline, and local or transported off site. The reason is a real case: an attacker who compromised an Australian hosting provider destroyed not only thousands of customers' live sites but all of the online backups too — customers without independent copies lost everything. Substitute the word ransomware for attacker and you have the modern argument for offline, immutable copies, unchanged.