Right tool for the job people, right tool for the job

What am I talking about?

I’m referring to all the use of sed -i spread wantonly on the internet, sans appropriate caveats.

Background: most implementations of the -i flag allow sed to edit files in place; by default, it edits standard in and writes to standard out, so you end up having to do <file sed ... >file.new && mv file.new file. Trying to redirect back into the file (<file sed ... >file) fails because the file is truncated before sed gets to read it!

The -i flag is not specified by POSIX, which makes it non-portable. This isn’t such a big deal, except that different seds require different arguments! GNU sed edits in place, no argument provided. BSD sed, such as the one on macOS, requires a suffix:

• if the suffix is non-empty, it places the original file at file.suffix and the new contents in file;
• if the suffix is empty, no backup is made.

So now, every sed -i solution to a problem needs to at least

• mention non portability, and
• mention two different implementations and how they handle it.

I don’t know about you, but I wonder if the solution to editing files in place is a lot simpler…

sed for streams, ed for files

sed stands for “stream editor.” It was designed based on the text-editor ed’s commands! ed is built to edit files.

• If you need to edit a file in place, use POSIX ed (or the improved POSIX-specified ex);
• if you need to transform a stream, sed is an option.

I cannot emphasize this enough. Trying to use sed -i in any kind of portable anything is likely to break; worse was the thought that sed should ever be used for editing files! Use a text-editor: if you thought “I’m automating, I don’t want to run a text-editor to automate,” well, remember:

• it is entirely possible to automate (read: script) ed and ex, and
• people today fire up entire web browsers in their desktop applications in order to automate things—running a tiny text-editor is the least of your worries.

Addendum: scripting ed

ed takes commands on standard in. It can be made silent with -s, and the argument is a file to edit. I ignore -p for now because it isn’t useful for scripting/automation. So, to script ed, generate a list of commands and pipe it in!

printf '%s\n' g/abc/d w q | ed -s file

That’s the most portable, but bash users can probably do

ed -s file <<EOF
g/abc/d
w
q
EOF

There is quite a bit of flexibility here; the left-hand side could actually be another program that generates ed commands on standard out! It turns out, this is what diff does: in its default output mode, we have

These lines resemble ed subcommands to convert file1 into file2. The line numbers before the action letters shall pertain to file1; those after shall pertain to file2. Thus, by exchanging a for d and reading the line in reverse order, one can also determine how to convert file2 into file1. As in ed, identical pairs (where num1=num2) are abbreviated as a single number.

And when we ask for the -e output mode:

With the -e option, a script shall be produced that shall, when provided as input to ed, along with an appended w (write) command, convert file1 into file2.

So now, you can probably write a patch command you parse the output from diff, with maybe slight modification, and then pass it as input to ed!

Categories: Blog