SED Tutorial

SED Tutorial

* The sed utility is an "editor"
* It is also noninteractive. This means you have to insert commands to be executed on the data at the command line or in a script to be processed.
* sed accepts a series of commands and executes them on a file (or set of files).
* sed fittingly stands for stream editor.
* It can be used to change all occurrences of "SAD" to "SED" or "New York" to "Newport."
* The stream editor is ideally suited to performing repetitive edits that would take considerable time if done manually.

How sed Works

The sed utility works by sequentially reading a file, line by line, into memory. It then performs all actions specified for the line and places the line back in memory to dump to the terminal with the requested changes made. After all actions have taken place to this one line, it reads the next line of the file and repeats the process until it is finished with the file. As mentioned, the default output is to display the contents of each line on the screen. Two important factors come into play here—first, the output can be redirected to another file to save the changes; second, the original file, by default, is left unchanged. The default is for sed to read the entire file and make changes to each line within it. It can, however, be restricted to specified lines as needed.

The syntax for the utility is:

sed [options] '{command}' [filename]

In this tutorial we will walk through the most commonly used commands and options and illustrate how they work and where they would be appropriate for use.

The Substitute Command

One of the most common uses of the sed utility, and any similar editor, is to substitute one value for another. To accomplish this, the syntax for the command portion of the operation is:

's/{old value}/{new value}/'

Thus, the following illustrates how "lion" can be changed to "eagle" very simply:

$ echo The lion group will meet on Tuesday after school | sed
's/lion/eagle/'
The eagle group will meet on Tuesday after school
$

Notice that it is not necessary to specify a filename if input is being derived from the output of a preceding command—the same as is true for awk, sort, and most other LinuxUNIX command-line utility programs.

Multiple Changes

If multiple changes need to be made to the same file or line, there are three methods by which this can be accomplished. The first is to use the "-e" option, which informs the program that more than one editing command is being used. For example:

$ echo The lion group will meet on Tuesday after school | sed -e '
s/lion/eagle/' -e 's/after/before/'
The eagle group will meet on Tuesday before school
$

This is pretty much the long way of going about it, and the "-e" option is not commonly used to any great extent. A more preferable way is to separate command with semicolons:

$ echo The lion group will meet on Tuesday after school | sed '
s/lion/eagle/; s/after/before/'
The eagle group will meet on Tuesday before school
$

Notice that the semicolon must be the next character following the slash. If a space is between the two, the operation will not successfully complete and an error message will be returned. These two methods are well and good, but there is one more method that many administrators prefer. The key thing to note is that everything between the two apostrophes (' ') is interpreted as sed commands. The shell program reading in the commands will not assume you are finished entering until the second apostrophe is entered. This means that the command can be entered on multiple lines—with Linux changing the prompt from PS1 to a continuation prompt (usually ">")—until the second apostrophe is entered. As soon as it is entered, and Enter pressed, the processing will take place and the same results will be generated, as the following illustrates:

$ echo The lion group will meet on Tuesday after school | sed '
> s/lion/eagle/
> s/after/before/'
The eagle group will meet on Tuesday before school
$

Global Changes

Let's begin with a deceptively simple edit. Suppose the message that is to be changed contains more than one occurrence of the item to be changed. By default, the result can be different than what was expected, as the following illustrates:

$ echo The lion group will meet this Tuesday at the same time
as the meeting last Tuesday | sed 's/Tuesday/Thursday/'
The lion group will meet this Thursday at the same time
as the meeting last Tuesday
$

Instead of changing every occurrence of "Tuesday" for "Thursday," the sed editor moves on after finding a change and making it, without reading the whole line. The majority of sed commands function like the substitute one, meaning they all work for the first occurrence of the chosen sequence in each line. In order for every occurrence to be substituted, in the event that more than one occurrence appears in the same line, you must specify for the action to take place globally:

$ echo The lion group will meet this Tuesday at the same time
as the meeting last Tuesday | sed 's/Tuesday/Thursday/g'
The lion group will meet this Thursday at the same time
as the meeting last Thursday
$

Bear in mind that this need for globalization is true whether the sequence you are looking for consists of only one character or a phrase.

sed can also be used to change record field delimiters from one to another. For example, the following will change all tabs to spaces:

sed 's/ / /g'

where the entry between the first set of slashes is a tab, while the entry between the second set is a space. As a general rule, sed can be used to change any printable character to any other printable character. If you want to change unprintable characters to printable ones—for example, a bell to the word "bell"—sed is not the right tool for the job (but tr would be).

Sometimes, you don't want to change every occurrence that appears in a file. At times, you only want to make a change if certain conditions are met—for example, following a match of some other data. To illustrate, consider the following text file:

$ cat sample_one
one 1
two 1
three 1
one 1
two 1
two 1
three 1
$

Suppose that it would be desirable for "1" to be substituted with "2," but only after the word "two" and not throughout every line. This can be accomplished by specifying that a match is to be found before giving the substitute command:

$ sed '/two/ s/1/2/' sample_one
one 1
two 2
three 1
one 1
two 2
two 2
three 1
$

And now, to make it even more accurate:

$ sed '
> /two/ s/1/2/
> /three/ s/1/3/' sample_one
one 1
two 2
three 3
one 1
two 2
two 2
three 3
$

Bear in mind once again that the only thing changed is the display. If you look at the original file, it is the same as it always was. You must save the output to another file to create permanence. It is worth repeating that the fact that changes are not made to the original file is a true blessing in disguise—it lets you experiment with the file without causing any real harm, until you get the right commands working exactly the way you expect and want them to.

The following saves the changed output to a new file:

$ sed '
> /two/ s/1/2/
> /three/ s/1/3/' sample_one > sample_two

The output file has all the changes incorporated in it that would normally appear on the screen. It can now be viewed with head, cat, or any other similar utility.

Script Files

The sed tool allows you to create a script file containing commands that are processed from the file, rather than at the command line, and is referenced via the "-f" option. By creating a script file, you have the ability to run the same operations over and over again, and to specify far more detailed operations than what you would want to try to tackle from the command line each time.

Consider the following script file:

$ cat sedlist
/two/ s/1/2/
/three/ s/1/3/
$

It can now be used on the data file to obtain the same results we saw earlier:

$ sed -f sedlist sample_one
one 1
two 2
three 3
one 1
two 2
two 2
three 3
$

Notice that apostrophes are not used inside the source file, or from the command line when the "-f" option is invoked. Script files, also known as source files, are invaluable for operations that you intend to repeat more than once and for complicated commands where there is a possibility that you may make an error at the command line. It is far easier to edit the source file and change one character than to retype a multiple-line entry at the command line.

Restricting Lines

The default is for the editor to look at, and for editing to take place on, every line that is input to the stream editor. This can be changed by specifying restrictions preceding the command. For example, to substitute "1" with "2" only in the fifth and sixth lines of the sample file's output, the command would be:

$ sed '5,6 s/1/2/' sample_one
one 1
two 1
three 1
one 1
two 2
two 2
three 1
$

In this case, since the lines to changes were specifically specified, the substitute command was not needed. Thus you have the flexibility of choosing which lines to changes (essentially, restricting the changes) based upon matching criteria that can be either line numbers or a matched pattern.

Prohibiting the Display

The default is for sed to display on the screen (or to a file, if so redirected) every line from the original file, whether it is affected by an edit operation or not; the "-n" parameter overrides this action. "-n" overrides all printing and displays no lines whatsoever, whether they were changed by the edit or not. For example:

$ sed -n -f sedlist sample_one
$

$ sed -n -f sedlist sample_one > sample_two
$ cat sample_two
$

In the first example, nothing is displayed on the screen. In the second example, nothing is changed, and thus nothing is written to the new file—it ends up being empty. Doesn't this negate the whole purpose of the edit? Why is this useful? It is useful only because the "-n" option has the ability to be overridden by a print command (-p). To illustrate, suppose the script file were modified to now resemble the following:

$ cat sedlist
/two/ s/1/2/p
/three/ s/1/3/p
$

Then this would be the result of running it:

$ sed -n -f sedlist sample_one
two 2
three 3
two 2
two 2
three 3
$

Lines that stay the same as they were are not displayed at all. Only the lines affected by the edit are displayed. In this manner, it is possible to pull those lines only, make the changes, and place them in a separate file:

$ sed -n -f sedlist sample_one > sample_two
$

$ cat sample_two
two 2
three 3
two 2
two 2
three 3
$

Another method of utilizing this is to print only a set number of lines. For example, to print only lines two through six while making no other editing changes:

$ sed -n '2,6p' sample_one
two 1
three 1
one 1
two 1
two 1
$

All other lines are ignored, and only lines two through six are printed as output. This is something remarkable that you cannot do easily with any other utility. head will print the top of a file, and tail will print the bottom, but sed allows you to pull anything you want to from anywhere.

Deleting Lines

Substituting one value for another is far from the only function that can be performed with a stream editor. There are many more possibilities, and the second-most-used function in my opinion is delete. Delete works in the same manner as substitute, only it removes the specified lines (if you want to remove a word and not a line, don't think of deleting, but think of substituting it for nothing—s/cat//).

The syntax for the command is:

'{what to find} d'

To remove all of the lines containing "two" from the sample_one file:

$ sed '/two/ d' sample_one
one 1
three 1
one 1
three 1
$

To remove the first three lines from the display, regardless of what they are:

$ sed '1,3 d' sample_one
one 1
two 1
two 1
three 1
$

Only the remaining lines are shown, and the first three cease to exist in the display. There are several things to keep in mind with the stream editor as they relate to global expressions in general, and as they apply to deletions in particular:

# The up carat (^) signifies the beginning of a line, thus

sed '/^two/ d' sample_one

would only delete the line if "two" were the first three characters of the line.
# The dollar sign ($) represents the end of the file, or the end of a line, thus

sed '/two$/ d' sample_one

would delete the line only if "two" were the last three characters of the line.

The result of putting these two together:

sed '/^$/ d' {filename}

deletes all blank lines from a file. For example, the following substitutes "1" for "2" as well as "1" for "3" and removes any trailing lines in the file:

$ sed '/two/ s/1/2/; /three/ s/1/3/; /^$/ d' sample_one
one 1
two 1
three 1
one 1
two 2
two 2
three 1
$

A common use for this is to delete a header. The following command will delete all lines in a file, from the first line through to the first blank line:

sed '1,/^$/ d' {filename}

Appending and Inserting Text

Text can be appended to the end of a file by using sed with the "a" option. This is done in the following manner:

$ sed '$a
> This is where we stop
> the test' sample_one
one 1
two 1
three 1
one 1
two 1
two 1
three 1
This is where we stop
the test
$

Within the command, the dollar sign ($) signifies that the text is to be appended to the end of the file. The backslashes () are necessary to signify that a carriage return is coming. If they are left out, an error will result proclaiming that the command is garbled; anywhere that a carriage return is to be entered, you must use the backslash.

To append the lines into the fourth and fifth positions instead of at the end, the command becomes:

$ sed '3a
> This is where we stop
> the test' sample_one
one 1
two 1
three 1
This is where we stop
the test
one 1
two 1
two 1
three 1
$

This appends the text after the third line. As with almost any editor, you can choose to insert rather than append if you so desire. The difference between the two is that append follows the line specified, and insert starts with the line specified. When using insert instead of append, just replace the "a" with an "i," as shown below:

$ sed '3i
> This is where we stop
> the test' sample_one
one 1
two 1
This is where we stop
the test
three 1
one 1
two 1
two 1
three 1
$

The new text appears in the middle of the output, and processing resumes normally after the specified operation is carried out.

Reading and Writing Files

The ability to redirect the output has already been illustrated, but it needs to be pointed out that files can be read in and written out to simultaneously during operation of the editing commands. For example, to perform the substitution and write the lines between one and three to a file called sample_three:

$ sed '
> /two/ s/1/2/
> /three/ s/1/3/
> 1,3 w sample_three' sample_one
one 1
two 2
three 3
one 1
two 2
two 2
three 3
$

$ cat sample_three
one 1
two 2
three 3
$

Only the lines specified are written to the new file, thanks to the "1,3" specification given to the w (write) command. Regardless of those written, all lines are displayed in the default output.

The Change Command

In addition to substituting entries, it is possible to change the lines from one value to another. The thing to keep in mind is that substitute works on a character-for-character basis, whereas change functions like delete in that it affects the entire line:

$ sed '/two/ c
> We are no longer using two' sample_one
one 1
We are no longer using two
three 1
one 1
We are no longer using two
We are no longer using two
three 1
$

Working much like substitute, the change command is greater in scale—completely replacing the one entry for another, regardless of character content, or context. At the risk of overstating the obvious, when substitute was used, then only the character "1" was replaced with "2," while when using change, the entire original line was modified. In both situations, the match to look for was simply the "two."

Change All but...

With most sed commands, the functions are spelled out as to what changes are to take place. Using the exclamation mark, it is possible to have the changes take place everywhere but those specified—completely reversing the default operation.

For example, to delete all lines that contain the phrase "two," the operation is:

$ sed '/two/ d' sample_one
one 1
three 1
one 1
three 1
$

And to delete all lines except those that contain the phrase "two," the syntax becomes:

$ sed '/two/ !d' sample_one
two 1
two 1
two 1
$

If you have a file that contains a list of items and want to perform an operation on each of the items in the file, then it is important that you first do an intelligent scan of those entries and think about what you are doing. To make matters easier, you can do so by combining sed with any iteration routine (for, while, until).

As an example, assume you have a text file named "animals" with the following entries:

pig
horse
elephant
cow
dog
cat

And you want to run the following routine:

#mcd.ksh
for I in $*
do
echo Old McDonald had a $I
echo E-I, E-I-O
done

The result will be that each line is printed at the end of "Old McDonald has a." While this is correct for the majority of the entries, it is grammatically incorrect for the "elephant" entry, as the result should be "an elephant" rather than "a elephant." Using sed, you can scan the output from your shell file for such grammatical errors and correct them on the fly, by first creating a file of commands:

#sublist
/ a a/ s/ a / an /
/ a e/ s/ a / an /
/a i/ s / a / an /
/a o/ s/ a / an /
/a u/ s/ a / an /

and then executing the process as follows:

$ sh mcd.ksh 'cat animals' | sed -f sublist

Now, after the mcd script has been run, sed will scan the output for anywhere that the single letter a (space, "a," space) is followed by a vowel. If such exists, it will change the sequence to space, "an," space. This corrects the problem before it ever prints on the screen and ensures that editors everywhere sleep easier at night. The result is:

Old McDonald had a pig
E-I, E-I-O
Old McDonald had a horse
E-I, E-I-O
Old McDonald had an elephant
E-I, E-I-O
Old McDonald had a cow
E-I, E-I-O
Old McDonald had a dog
E-I, E-I-O
Old McDonald had a cat
E-I, E-I-O

Quitting Early

The default is for sed to read through an entire file and stop only when the end is reached. You can stop processing early, however, by using the quit command. Only one quit command can be specified, and processing will continue until the condition calling the quit command is satisfied.

For example, to perform substitution only on the first five lines of a file and then quit:

$ sed '
> /two/ s/1/2/
> /three/ s/1/3/
> 5q' sample_one
one 1
two 2
three 3
one 1
two 2
$

The entry preceding the quit command can be a line number, as shown, or a find/matching command like the following:

$ sed '
> /two/ s/1/2/
> /three/ s/1/3/
> /three/q' sample_one
one 1
two 2
three 3
$

You can also use the quit command to view lines beyond a standard number and add functionality that exceeds those in head. For example, the head command allows you to specify how many of the first lines of a file you want to see—the default number is ten, but any number can be used from one to ninety-nine. If you want to see the first 110 lines of a file, you cannot do so with head, but you can with sed:

sed 110q filename

Handling Problems

The main thing to keep in mind when dealing with sed is how it works. It works by reading one line in, performing all the tasks it knows to perform on that one line, and then moving on to the next line. Each line is subjected to every editing command given.

This can be troublesome if the order of your operations is not thoroughly thought out. For example, suppose you need to change all "two" entries to "three" and all "three" to "four":

$ sed '
> /two/ s/two/three/
> /three/ s/three/four/' sample_one
one 1
four 1
four 1
one 1
four 1
four 1
four 1
$

The very first "two" read was changed to "three." It then meets the criteria established for the next edit and becomes "four." The end result is not what was wanted—there are now no entries but "four" where there should be "three" and "four."

When performing such an operation, you must pay diligent attention to the manner in which the operations are specified and arrange them in an order in which one will not clobber another. For example:

$ sed '
> /three/ s/three/four/
> /two/ s/two/three/' sample_one
one 1
three 1
four 1
one 1
three 1
three 1
four 1
$

This works perfectly, since the "three" value is changed prior to "two" becoming "three."

Labels and Comments

Labels can be placed inside sed script files to make it easier to explain what is transpiring, once the files begin to grow in size. There are a variety of commands that relate to these labels, and they include:

# : The colon signifies a label name. For example:

:HERE

Labels beginning with the colon can be addressed by "b" and "t" commands.

# b {label} Works as a "goto" statement, sending processing to the label preceded by a colon. For example,

b HERE

sends processing to the line

:HERE

If no label is specified following the b, processing goes to the end of the script file.

# t {label} Branches to the label only if substitutions have been made since the last input line or execution of a "t" command. As with "b," if a label name is not given, processing moves to the end of the script file.

# # The pound sign as the first character of a line causes the entire line to be treated as a comment. Comment lines are different from labels and cannot be branched to with b or t commands.