Every line is a row of fields

The data model

Where grep sees lines and sed sees a stream, awk sees a table. It reads one record at a time — a line, by default — and splits that record into fields on runs of blanks. You then write rules of the form

pattern { action }

and awk runs the action on every record the pattern selects. Omit the pattern and the action runs on every record; omit the action and the default action is to print the record. That is the entire language you need for this course, and it is why one line of awk replaces a cut plus a grep plus a loop.

The built-in names are few enough to memorise:

• a dollar sign followed by a number is a field: the first field, the second field, and so on. A dollar sign followed by zero is the whole record.
• NF is the number of fields in the current record, so the last field is the field numbered NF.
• NR is the record number — how many records have been read so far.
• -F on the command line sets the field separator: -F: splits on colons.
• BEGIN { } runs once before the first record, END { } once after the last.

Worked example — building up a pipeline

The fixture is the output of ps -ef, whose columns are UID, PID, PPID, C, STIME, TTY, TIME and CMD.

ada    2071     1  0 09:40 pts/1  00:00:00 sleep 1000

Step 1 — one column. ps -ef | awk '{print $2}' prints the second field of every record: the PID column. There is no pattern, so every record is selected; the action names one field.

Step 2 — select rows first. ps -ef | awk '/sleep/ {print $2}' prints the PID only of the records whose text contains sleep. The pattern is a regular expression between slashes, matched against the whole record. This is grep and cut in a single program, and awk did the field splitting without being told where the columns start — which is the point, because ps output is not fixed width.

Step 3 — a different separator. The password file is colon-separated, so blanks are the wrong split. awk -F: '{print $1,$7}' /etc/passwd prints the login name and the login shell of every account. The comma between the two field references inserts a space; had you written them adjacent with no comma they would be concatenated with nothing between.

Step 4 — a whole-file answer. awk 'END {print NR}' /etc/passwd prints one number: how many records were read. NR keeps counting through the file and the END block runs once, after the last record, when the count is final. That is wc -l written as an awk program, and it generalises — END {print NR, NF} also tells you how wide the last record was.

The two traps in the assignment pipeline

Assignment 3 asks you to start a background sleep and then find and terminate it using ps, grep, awk and kill in a single pipe. Both halves of that have a trap.

Trap one: grep finds itself. Every command in a pipeline runs as a process, so by the time ps has produced its output the grep process is in it, and its command line contains the very pattern you are searching for. ps -ef | grep 'sleep 1000' therefore reports two rows: the sleep you started, and the grep looking for it. Two ways out. grep -v grep filters the extra row afterwards. Better, grep '[s]leep 1000' sidesteps it: the bracket set matches exactly the character s, so the pattern still matches the text sleep 1000, but grep's own command line contains the literal characters [s]leep, which the pattern does not match. Same result, one process, no extra stage.

Trap two: kill does not read standard input. kill takes process IDs as arguments. Piping a PID into it does nothing at all — the PID lands on kill's standard input, which kill never reads, and kill complains about missing operands. Two ways out again: xargs turns standard input into arguments, and command substitution does the same job with different punctuation.

ps -ef | grep '[s]leep 1000' | awk '{print $2}' | xargs kill

That is the model answer. pkill -f 'sleep 1000' does the whole thing in one command, and is worth naming — but the assignment asks for the pipeline, because the pipeline is what teaches you where each tool's boundary lies.