Python 3 when the script outgrows the shell

The point at which the shell stops paying

A shell script is unbeatable for gluing commands together. It gets expensive the moment it has to understand what those commands printed, because the only tool it has for that is splitting text into fields, and field positions are not a stable interface. Python 3 is the usual next step, and the translation is mechanical enough to do line by line.

Here is a Bourne script assembled from two Module 9 pieces — the for loop over the arguments and the -d file test:

#!/bin/sh
for d in "$@"
do
    if [ -d "$d" ]; then
        echo "$d: directory"
    fi
done

And here it is in Python 3:

import os
import sys

for d in sys.argv[1:]:
    if os.path.isdir(d):
        print(f"{d}: directory")

Arguments. The shell's positional parameters become one list, sys.argv, whose element zero is the program name. So $@ is sys.argv[1:] and $# is len(sys.argv[1:]). One list replaces $1 through $9, the brace form beyond nine, #`, `* and $@ together, and it slices.

File tests. [ -d "$d" ] is os.path.isdir(d), [ -f ] is os.path.isfile, [ -e ] is os.path.exists, [ -s ] is os.path.getsize(d) > 0. They return real booleans, so there is no exit-status convention to remember and no quoting to get wrong — an empty or space-containing filename is just a string.

Reading input. read w1 w2 w3 becomes input().split(), which returns a list you can unpack or index. The shell's rule that surplus words all pile into the last variable is not reproduced by split(), and if you want it you ask for it: input().split(maxsplit=2).

The strongest argument: stop parsing ls

A shell script that needs a file's inode number runs ls -il and takes the first field. A script that needs its size takes the fifth field of ls -l. Both work until they do not: field positions move between systems, a filename with spaces shifts everything after it, a long user name changes the column widths, and locale changes the date format.

Python asks the kernel instead:

import os

st = os.stat("report.txt")
print(st.st_ino, st.st_size, oct(st.st_mode & 0o777))

os.stat is a thin wrapper over the stat system call, so st_ino, st_size, st_nlink, st_mode, st_uid and st_mtime come back as numbers, from the source of truth, with no text in between. There is no field to count and nothing to break. That single substitution is the most persuasive reason in this lesson to move a script out of the shell.

Running other commands, safely

A Python script still needs to run commands. The modern call is subprocess.run:

import subprocess

result = subprocess.run(["wc", "-l", "notes.txt"], capture_output=True, text=True)
print(result.returncode, result.stdout.strip())

Two details matter for the rest of this module. First, capture_output=True collects standard output and standard error, and text=True decodes them to strings instead of bytes; without those you get bytes and no capture. Second, and this is the security point, pass a list. A list goes straight to exec with no shell involved, so a filename containing a semicolon or a space is one argument and nothing else. Building a single string and passing shell=True hands the whole thing to a shell that will happily act on any metacharacter in it. The older os.system and os.popen calls always go through a shell, which is why they are the wrong default now.

The returncode is Python's version of the shell exit status: zero for success, and you test it yourself rather than relying on an if that branches on it implicitly.