Three transforms that are not each other

Three transforms, routinely confused

Students merge these three because all of them turn a file into a different-looking file. They are not variations on a theme; they answer three different questions.

Compression changes the representation. The information is identical; the encoding of it is shorter. Lossless compression — gzip, compress, zip — recovers the original byte for byte, and it is the only kind you may use on text, source code or an executable. Lossy compression — JPEG, MP3 — discards information you are betting nobody will miss, and applying it to a program produces a program that does not run. Compression buys disk space and transmission time and pays for them in CPU time, which is a good trade whenever the file will be sent more than once.

Encoding changes the alphabet. It takes arbitrary bytes and re-expresses them using only printable characters, because some transport in the middle — classically e-mail, which was defined for seven-bit text — cannot carry arbitrary bytes. It is fully reversible by anyone, needs no key, and makes the file bigger: base64 inflates by about 33%, and the older uuencode by about 38% once its line framing is counted.

Encryption changes the content under a key. It is reversible only by someone holding the key, and it is the only one of the three that keeps a secret. It does not shrink anything; done well, ciphertext is statistically indistinguishable from random bytes, which is exactly why it does not compress afterwards.

Compression ratio is a property of the file, not the tool

Run gzip -9 — maximum effort — over three files and the results are nothing like each other:

$ gzip -9 access.log report.txt photo.jpg
$ gzip -l access.log.gz report.txt.gz photo.jpg.gz
 compressed  uncompressed  ratio  uncompressed_name
     181203       2612094  93.1%  access.log
     139560        436124  68.0%  report.txt
    1904118       1911774   0.4%  photo.jpg

The log is 93% smaller because it is thousands of near-identical lines and a compressor's whole business is repetition. The plain-text report loses about two thirds, which is typical for English prose. The JPEG barely moves, because JPEG is compression — the redundancy was already spent, and there is nothing left for gzip to find. Compressing an already-compressed file is at best free and often slightly negative.

The commands themselves are short. gzip file writes file.gz and deletes the original; gunzip file.gz reverses it; -1 through -9 trade speed for ratio; -l lists the sizes and ratio without decompressing; zcat file.gz writes the decompressed contents to standard output, leaving the archive alone, so it composes into a pipeline. The classic compress/uncompress pair behaves the same way with a .Z suffix and an older algorithm.

Archiving: tar

gzip compresses one file. To package a whole directory you first need to turn the directory into a single stream, and that is what tar — tape archive — does. It is universally paired with a compressor, and GNU tar will drive one for you with a single option letter:

$ tar -czf proj.tar.gz proj
$ tar -tzf proj.tar.gz
proj/
proj/alpha.txt
proj/.hidden
proj/src/
proj/src/main.c
$ mkdir -p /tmp/restore
$ tar -xzf proj.tar.gz -C /tmp/restore

Read the option letters as three independent choices. Exactly one of c (create), t (list) or x (extract) says what you are doing. z says the stream passes through gzip on the way — that is the letter doing the compression, and dropping it on extraction of a .tar.gz is the most common tar mistake there is. f says the next argument is the archive filename rather than a tape device. -C dir changes directory before extracting, which is how you restore somewhere other than where you are standing.

Two things tar does that the glob-based commands of the last lesson do not: it includes dotfiles, because it walks the directory itself rather than being handed an expansion, and it preserves the tree — permissions, modification times and the directory structure all come back on extraction.

Worked example — getting a binary through a text-only channel

You must send a 4 MB executable through something that accepts only printable text. Two transforms are needed, and their order is not a matter of taste.

Compress first, then encode. Compression on the original gets whatever ratio the content allows — call it 60%, leaving 1.6 MB — and encoding then inflates that by a third, to about 2.1 MB. Do it the other way and encoding inflates 4 MB to 5.3 MB, and compressing that recovers very little: base64 output is a near-uniform stream of 64 characters, with the file's original redundancy already smeared across it. You end up with a much larger message and more CPU spent producing it.

$ gzip -9 tool.bin
$ base64 tool.bin.gz > tool.b64

The inverse pipeline is the exact mirror, decode then decompress:

$ base64 -d tool.b64 > tool.bin.gz
$ gunzip tool.bin.gz

The general rule, and it is worth carrying past this course: compress before you encode, and compress before you encrypt. Both encoding and encryption destroy the redundancy that a compressor lives on, so anything applied after them compresses badly or not at all.