The stream editor (sed) is a very popular noninteractive stream editor. Normally, whenever we edit files using the vi editor, we need to open the file using the vi command, then we interact with the file, such as to see the content of the file on screen, then, edit it, and save the file. Using sed, we can type commands on the command line and sed will make the changes to the text file. The sed is a nondestructive editor. The sed makes the changes to the file and displays the content on screen. If we want to save the changed file, then we need to redirect the output of the sed to the file.
The procedure to install sed is shown here.
For Ubuntu or any Debian-based distributions enter the following command:
$ apt-get install sed
For Red Hat or any rpm-based distribution enter the following command:
$ yum install sed
To check the version of sed, enter the following command:
$ sed –V
Otherwise, enter this command:
$ sed --version GNU sed version 3.0 2
Whenever you use sed commands on a text file, sed reads the first line of the file and stores it in a temporary buffer called pattern space. The sed processes this pattern space buffer as per commands given by the user. Then, it prints the output on screen. This line from the pattern space is then removed and the next line of the file is loaded in the pattern space. In this way, it processes all the lines one by one. This line-by-line processing is continued till the last line of the file. As the sed commands are processed on the temporary buffer or pattern space, the original line is not modified. Therefore, we say sed is a nondestructive buffer.
While using sed, regular expressions are enclosed in forward slashes. As grep and sed use regular expressions and metacharacters for searching patterns in the file. For example:
sed -n '/Regular_Expression/p' filename sed -n '/Mango/p' filename
This will print lines matching the Mango pattern:
sed -n 's/RE/replacement string/' filename sed -n 's/Mango/Apple/' filename
This will find the line containing the Mango pattern and then the Mango pattern will be replaced by the Apple text. This modified line will be shown on screen and the original file will be unchanged.
The following is a summary of various metacharacters and it's usage in sed:
|
Metacharacter |
Function |
|---|---|
|
^ |
This is the beginning-of-line anchor |
|
$ |
This is the end-of-line anchor |
|
. | |
|
* |
This matches zero or more characters |
|
[ ] |
This matches one character in the set |
|
[^ ] |
This matches one character not in the set |
|
(..) |
This saves matched characters |
|
& |
This saves the search string so it can be remembered in the replacement string |
|
< |
This is the beginning-of-word anchor |
|
> |
This is the end-of-word anchor |
|
x{m} |
This is the repetition of the character |
|
x{m,} |
This means at least |
|
x{m,n} |
This means between |
We can specify which line or number of lines the pattern search and commands are to be applied on while using the sed commands. If line numbers are not specified, then the pattern search and commands will be applied on all lines of the input file.
The line numbers on which commands are to be applied are called address. Address can be a single line number or range of lines in which the starting number of the line and the ending number of the range will be separated by commas. Ranges can be composed of numbers, regular expressions or a combination of both.
The sed commands specify actions such as printing, removing, replacing, and so on.
The syntax is as follows:
sed 'command' filename(s )
Example:
$ cat myfile | sed '1,3d'
Otherwise it can be:
sed '1,3d' myfile
This will delete lines 1 to 3:
sed -n '/[Aa]pple/p' item.list
If the Apple or apple pattern is found in the item.list file, then those lines will be printed on screen and the original file myfile will be unchanged.
To negate the command, the exclamation character (!) can be used.
Example:
sed '/Apple/d' item.list
This tells sed to delete all the lines containing the Apple pattern.
Consider the following example:
sed '/Apple/!d' item.list
This will delete all the lines except the line containing the Apple pattern.
The sed is a nondestructive editor. This means the output of sed is displayed on screen; but the original file is unchanged. If we want to modify the file, then we can redirect the output of the sed command to the file. Deleting lines is illustrated in the following examples:
$ sed '1,3d' datafile > tempfile $ mv tempfile newfile
In this example, we have deleted lines 1 to 3 and stored the output in tempfile. Then, we have to rename tempfile to newfile.
By default, the action of the sed command is to print the pattern space, such as every line which is copied in buffer, and then print the result of processing on it. Therefore, the sed output will consist of all lines along with the processed line by sed. If we do not want the default pattern space line to be printed, then we need to give the –n option. Therefore, we should use the –n option and the p command together to see the result of the sed processed output.
Here is an example:
$ cat country.txt
The output is as follows:
Country Capital ISD Code USA Washington 1 China Beijing 86 Japan Tokyo 81 India Delhi 91 $ sed '/USA/p'country.txt
The output is as follows:
Country Capital ISD Code USA Washington 1 USA Washington 1 China Beijing 86 Japan Tokyo 81 India Delhi 91
All the lines from the file are printed by default and the lines with the USA pattern are also printed:
$ sed –n '/USA/p' country.txt
The output is as follows:
USA Washington 1
As we have given the –n option, sed has suppressed default printing of all lines from the country file; but has printed the line that contains the text pattern USA.
The d command is used to delete lines. After sed copies a line from a file and puts it into a pattern buffer, it processes commands on that line, and finally, displays the contents of the pattern buffer on screen. When the d command is issued, the line currently in the pattern buffer is removed, not displayed which is shown as follows:
$ cat country.txt Country Capital ISD Code USA Washington 1 China Beijing 86 Japan Tokyo 81 India Delhi 91 $ sed '3d' country.txt
The output is as follows:
Country Capital ISD Code USA Washington 1 Japan Tokyo 81 India Delhi 91
Here is the explanation.
The output will contain all the lines except the third line. The third line is deleted by the following command:
$ sed '3,$d' country.txt
Country Capital ISD Code USA Washington 1
This will delete third line to the last line. The dollar sign in the address indicates the last line. The comma is called ra nge operator.
$ sed '$d' country.txt
The output is as follows:
Country Capital ISD Code USA Washington 1 China Beijing 86 Japan Tokyo 81
Here is the explanation.
This deletes the last line. All lines except lines will be displayed.
Here is an example:
$ sed '/Japan/d' country.txt
The output is as follows:
Country Capital ISD Code USA Washington 1 China Beijing 86 India Delhi 91
The line containing the Japan pattern is deleted. All other lines are printed:
$ sed '/Japan/!d' country.txt
The output is as follows:
Japan Tokyo 81
This has deleted all the lines that do not contain Japan.
Let's see a few more examples with the delete command.
This will delete line 4 and the next five lines:
$ sed '4,+5d'
This will keep lines 1 to 5 and delete all the other lines:
$ sed '1,5!d'
This will delete lines 1, 4, 7, and so on:
$ sed '1~3d'
Starting from 1, every third line step increments. The number that follows the tilde is what is called the step increment. The step increment indicates the following:
$ sed '2~2d'
This will delete every other line starting with line 2 to be deleted.
If we want to substitute the text by new text, then we can use commands. After the forward slash, the regular expression is enclosed and then the text to be substituted is placed. If the g option is used, then substitution will happen globally, meaning that it will be applied in the full document. Otherwise, only the first instance will be substituted:
$ cat shopping.txt
The output is as follows:
Product Quantity Unit_Price Total_Cost Apple 2 3 6 Orange 2 .8 1.6 Papaya 2 1.5 3 Chicken 3 5 15 Cashew 1 10 10 $ sed 's/Cashew/Almonds/g' shopping.txt
The output is as follows:
Product Quantity Unit_Price Total_Cost Apple 2 3 6 Orange 2 .8 1.6 Papaya 2 1.5 3 Chicken 3 5 15 Almonds 1 10 10
The s command has replaced Cashew by Almonds. The g flag at the end indicates that the substitution is to be applied globally. Otherwise, it will be applied to the first pattern match only.
The following substitution command will replace two digit numbers at the end of the line with .5 appended to them:
$ sed 's/[0–9][0–9]$/&.5/' shopping.txt
The output is as follows:
Product Quantity Unit_Price Total_Cost Apple 2 3 6 Orange 2 .8 1.6 Papaya 2 1.5 3 Chicken 3 5 15.5 Cashew 1 10 10.5
The ampersand in the search pattern represents the exact pattern found. This will be replaced by the exact pattern with .5 appended to it.
To use the sed effectively, we should be clear about how to define range. Range is typically two addresses in a file as follows:
- Range with numbers:
'6d': range of line 6 '3,6d': range from line 3 to 6
- Range with pattern:
'/pattern1/,/pattern2/
This will specify the range of all the lines between the
pattern1andpattern2patterns. We can even specify th range with a combination of both, that is,'/pattern/,6'. This will specify the range of lines between the pattern and line 6.
As mentioned, we can specify the range as numbers, pattern, or a combination of both.
For example:
$ cat country.txt Country Capital ISD Code USA Washington 1 China Beijing 86 Japan Tokyo 81 India Delhi 91 $ sed -n '/USA/,/Japan/p' country.txt
The output is as follows:
USA Washington 1 China Beijing 86 Japan Tokyo 81
In this example, all the lines between addresses starting from USA and until the pattern Japan will be printed on screen.
For example:
$ sed -n '2,/India/p' country.txt
The output is as follows:
USA Washington 1 China Beijing 86 Japan Tokyo 81 India Delhi 91
In this example, line 2 to the pattern India, are printed on screen.
For example:
$ sed '/Apple/,/Papaya/s/$/** Out of Stock **/' shopping.txt
The output is as follows:
Product Quantity Unit_Price Total_Cost Apple 2 3 6** Out of Stock ** Orange 2 .8 1.6** Out of Stock ** Papaya 2 1.5 3** Out of Stock ** Chicken 3 5 15 Cashew 1 10 10
In this example. for all the lines between the Apple and Papaya patterns, the end of line will be replaced by the ** Out of Stock ** string.
If we need to perform multiple editing by the same command, then we can use the –e command. Each edit command should be separated by the –e command. The sed will apply each editing command separated by –e on the pattern space before loading the next line in the pattern space:
$ cat shopping.txt
The output is as follows:
Product Quantity Unit_Price Total_Cost Apple 2 3 6 Orange 2 .8 1.6 Papaya 2 1.5 3 Chicken 3 5 15 Cashew 1 10 10
For example:
sed -e '5d' -e 's/Cashew/Almonds/' shopping.txt
The output is as follows:
Product Quantity Unit_Price Total_Cost Apple 2 3 6 Orange 2 .8 1.6 Papaya 2 1.5 3 Almonds 1 10 10
Initially, the command for deleting the fifth line is called, then, the next substitution command to replace Cashew by Almonds is processed.
If we need to insert text from another file into the file, which is processed by sed, then we can use the r command. We can insert text from another file to the specified location:
For example:
$ cat new.txt
The output will be:
********************************* Apples are out of stock $ sed '/Apple/r new.txt' shopping.txt
The output is as follows:
Product Quantity Unit_Price Total_Cost Apple 2 3 6 ********************************* Apples are out of stock ********************************* Orange 2 .8 1.6 Papaya 2 1.5 3 Chicken 3 5 15 Cashew 1 10 10
The explanation is that, this command has added the content of the new.txt file after the line containing the Apple pattern.
The sed command for write is w. Using this command, we can write lines from one file to another file.
For example:
$ cat new.txt
The output is as follows:
new is a empty file $ sed -n '/Chicken/w new.txt' shopping.txt $ cat new.txt Chicken 3 5 15
After the w command, we specify the file to which we will perform the write operation. In this example, the line containing the Chicken pattern is written to the new.txt file.
The a command is used for appending. When the append command is used, it appends the text after the line in the pattern space, in which the pattern is matched. The backslash should be placed immediately after the a command. On the next line, the text to be appended is to be placed.
For example:
$ cat shopping.txt
The output is as follows:
Product Quantity Unit_Price Total_Cost Apple 2 3 6 Orange 2 .8 1.6 Papaya 2 1.5 3 Chicken 3 5 15 Cashew 1 10 10 $ sed '/Orange/a **** Buy one get one free offer on this item ! ****' shopping.txt
The output is as follows:
Product Quantity Unit_Price Total_Cost Apple 2 3 6 Orange 2 .8 1.6 **** Buy one get one free offer on this item ! **** Papaya 2 1.5 3 Chicken 3 5 15 Cashew 1 10 10
The new text **** Buy one get one free offer on this item ! **** is appended after the line containing the Orange pattern.
The i command is used for inserting text above the current pattern space line. When we use the append command, new text is inserted after the current line which is in the pattern buffer. In this similar-to-append command, the backslash is inserted after the i command.
For example:
$ cat shopping.txt Product Quantity Unit_Price Total_Cost Apple 2 3 6 Orange 2 .8 1.6 Papaya 2 1.5 3 Chicken 3 5 15 Cashew 1 10 10 $ sed '/Apple/i New Prices will apply from Next month ! ' shopping.txt
The output is as follows:
Product Quantity Unit_Price Total_Cost New Prices will apply from Next month ! Apple 2 3 6 Orange 2 .8 1.6 Papaya 2 1.5 3 Chicken 3 5 15 Cashew 1 10 10
In this example, the new text New Prices will be applied from next month! is inserted before the line containing the Apple pattern. Please check the i command and the backslash following it.
The c command is the change command. It allows the sed to modify or change existing text with new text. The old text is overwritten with the new:
$ cat shopping.txt
The output is as follows:
Product Quantity Unit_Price Total_Cost Apple 2 3 6 Orange 2 .8 1.6 Papaya 2 1.5 3 Chicken 3 5 15 Cashew 1 10 10
For example:
$ sed '/Papaya/c Papaya is out of stock today !' shopping.txt
The output is as follows:
Product Quantity Unit_Price Total_Cost Apple 2 3 6 Orange 2 .8 1.6 Papaya is out of stock today ! Chicken 3 5 15 Cashew 1 10 10
In this example, the line containing the expression Papaya is changed by the new line Papaya is out of stock today!.
The command transform is similar to the Linux tr command. The characters are translated as per character sequence given. For example, y/ABC/abc/ will convert lowercase abc into uppercase ABC.
For example:
$ cat shopping.txt
The output will be:
Product Quantity Unit_Price Total_Cost Apple 2 3 6 Orange 2 .8 1.6 Papaya 2 1.5 3 Chicken 3 5 15 Cashew 1 10 10 $ sed '2,4y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/' shopping.txt
The output will be:
Product Quantity Unit_Price Total_Cost APPLE 2 3 6 ORANGE 2 .8 1.6 PAPAYA 2 1.5 3 Chicken 3 5 15 Cashew 1 10 10
In this example, on lines 2, 3 and 4, all the lowercase letters are converted to uppercase letters.
The q command is used for quitting the sed processing without proceeding to the next lines:
$ cat shopping.txt
The output will be:
Product Quantity Unit_Price Total_Cost Apple 2 3 6 Orange 2 .8 1.6 Papaya 2 1.5 3 Chicken 3 5 15 Cashew 1 10 10
For example:
$ sed '3q' shopping.txt
The output will be:
Product Quantity Unit_Price Total_Cost Apple 2 3 6 Orange 2 .8 1.6
In this example, after printing the first to third lines, sed quits further processing.
We have already seen that the sed has pattern buffer. The sed has one more type of buffer called holding buffer. By the h command, we can inform sed to store the pattern buffer in the holding buffer. And whenever we need the line that is stored in the pattern buffer, we can get it by the g command, that is, get the buffer.
For example:
$ sed -e '/Product/h' -e '$g' shopping.txt
The output is as follows:
Product Quantity Unit_Price Total_Cost Apple 2 3 6 Orange 2 .8 1.6 Papaya 2 1.5 3 Chicken 3 5 15 Cashew 1 10 10 Product Quantity Unit_Price Total_Cost
In this example, the line containing the Product pattern is stored in the holding buffer by the h command. Then, the next editing command indicates to the sed to get the line from the holding buffer when the last line of the file is reached and appends the line from the holding buffer after the last line of the file.
The x is an exchange command. By using this command, we can exchange the holding buffer with the current line in the pattern buffer.
For example:
$ sed -e '/Apple/h' -e '/Cashew/x' shopping.txt
The output is as follows:
Product Quantity Unit_Price Total_Cost Apple 2 3 6 Orange 2 .8 1.6 Papaya 2 1.5 3 Chicken 3 5 15 Apple 2 3 6
In this example, the line with the Apple pattern is stored in the holding buffer. When the pattern with Cashew is found, that line will be exchanged by the holding buffer.
The sed script file contains a list of sed commands in a file. To inform the sed about our script file, we should use the –f option before the script file name. If the sed commands are not separated by a new line, then every command should be separated by a colon ":". We have to take care that there should not be any training white space after every command in the sed script file; otherwise, the sed will give an error. sed takes each line in the pattern buffer and then, it will process all commands on that line. After this line is processed, the next line will be loaded in the pattern buffer. For the continuation of any sed command, which cannot be fitted in one line, we need to add one backslash at the end of the line to inform about continuation.
For example:
$ cat shopping1.txt
The output is as follows:
Product Quantity Unit_Price Apple 200 3 Orange 200 .8 Papaya 100 1.5 Chicken 65 5 Cashew 50 10 April, third week $ cat stock
The output is as follows:
# This is my first sed script by : 1i Stock status report /Orange/a Fresh Oranges are not available in this season. Fresh Oranges will be available from next month /Chicken/c ********************************************************** We will not be stocking this item for next few weeks. ********************************************************** $d
Enter the next command as follows:
$ sed -f stock shopping1.txt
The output is as follows:
Stock status report Product Quantity Unit_Price Apple 200 3 Orange 200 .8 Fresh Oranges are not available in this season. Fresh Oranges will be available from next month Papaya 100 1.5 ********************************************************** We will not be stocking this item for next few weeks. ********************************************************** Cashew 50 10
In this script, the following processing has taken place:
- The comment line starts with the pound (
#) sign. - The command
1iinforms sed to insert the next text before line number 1. - The command
/Orange/ainforms sed to append the next text after the line containing theOrangepattern. - The command
/Chicken/cinforms sed to replace the line containing theChickenpattern by the next line. - The last command,
$d, tells sed to delete the last line of the input file.