stdout of one into stdin of the next

The temporary file you no longer need

Suppose you want to page through a long directory listing. Without pipes you write three commands:

ls -l > tmp
more tmp
rm tmp

Count what that costs. A file is created, written and closed; the same file is opened and read; the file is unlinked. The listing has made a round trip to storage on its way from a program that had just produced it to a program that was standing by to consume it. You also had to invent a name that collides with nothing, and remember to clean it up — and if you forget, or the middle command fails, the debris stays. Now:

ls -l | more

One line, no name, nothing to clean up, and nothing touches the disk. The vertical bar tells the shell to create a pipe, then start both commands at the same time, with ls stdout and more stdin attached to the two ends of it.

What a pipe actually is

A pipe is not a temp file with the housekeeping hidden. It is a bounded buffer in kernel memory with a reader and a writer running concurrently, and the differences are all observable:

• Data moves first in, first out, byte for byte. There is no name, no directory entry and no inode you can list.
• The buffer has a fixed capacity. When the writer outruns the reader and fills it, the writer blocks until room appears; when the reader empties it, the reader blocks until more data arrives. Nobody writes any coordination code — the kernel does the flow control. (POSIX guarantees a minimum capacity through PIPE_BUF, which also bounds how much may be written atomically; Module 13 makes that guarantee do real work.)
• A pipe carries stdout only. Descriptor 2 of each command is untouched, which is why a failing build still prints its errors to your screen even when its output has gone into less.
• It is one-way. A two-way conversation needs two pipes and cannot be written at the shell level at all — it needs the pipe() system call from C, which is Module 13.
• The joined processes must be related. The shell forks them from itself, so they share an ancestor and inherit the descriptors. Two unrelated processes cannot find each other's pipe; that limitation is what the next lesson removes.

Worked example: scaling the argument up

Suppose you want a sorted list of the people currently logged in whose lines match a pattern, mailed to yourself. Written with temporary files:

who > tmp1
sort tmp1 > tmp2
grep pts tmp2 > tmp3
mail -s users me < tmp3

Four commands, three temporary files, and six file operations against storage — three creations plus three reads — before anything useful has happened. Every one of those files is a name you invented and must remove. As one pipeline:

who | sort | grep pts | mail -s users me

The intermediate results never exist as files. The four programs run concurrently, each consuming its predecessor's output as it is produced. And the pipeline is readable: each stage does one thing, and the line reads left to right as a sentence about what happens to the data.

Filters, and why they compose

The reason this works at all is a convention, not a mechanism. A filter is a command that reads stdin, transforms what it reads, and writes stdout. Because every filter has the same shape, any filter can follow any other, and a pipeline of n stages needs no glue between them. The composability of UNIX is entirely this one convention, kept.

tee — the branch

A pipeline can do one thing redirection cannot, and redirection can do one thing a pipeline cannot: keep the intermediate. tee is the answer to needing both. It copies its stdin to its stdout and to each file you name:

ps -e | tee procs.txt | grep sshd | wc -l

The count still comes out on your screen, and procs.txt now holds the full process list for the next step of the script — one run of ps, two consumers. Without tee you would have to run ps twice and hope the machine had not changed in between.