Chapter 5
Working with Text Files
When the UNIX system, on which Linux was based, was created, almost all information was managed on the system in plain-text files. Thus, it was critical for users to know how to use tools for searching for and within plaintext files and to be able to change and configure those files.
Today, most configuration of Linux systems is still done by editing plaintext files. Even when a graphical tool is available for working with a configuration file, the graphical tool doesn't provide a way to do everything you might want to do in that file. As a result, you need to use a text editor to configure the file. Likewise, some document file types, such as HTML and XML, are also plaintext files that can be edited manually.
Before you can become a full-fledged system administrator, you need to be able to use a plain-text editor. The fact that most professional Linux servers don't even have a graphical interface available makes manual editing of plaintext configuration files necessary.
Once you know how to edit text files, you still might find it tough to figure out where the files are located that you need to edit. With commands such as find, you can search for files based on various attributes (filename, size, modification date, and ownership, to name a few). With the grep command, you can search inside of text files to find specific search terms.
It's almost impossible to use Linux for any period of time and not need a text editor because, as noted earlier, most Linux configuration files are plaintext files that you will almost certainly need to change manually at some point.
If you are using a desktop, you can run gedit (select Applications ⇒ Accessories ⇒ gedit Text Editor), which is fairly intuitive for editing text. There's also a simple text editor you can run from the shell called nano. However, most Linux shell users use either the vi or emacs command to edit text files.
The advantage of vi or emacs over a graphical editor is that you can use the command from any shell, character terminal, or character-based connection over a network (using telnet or ssh, for example) — no graphical interface is required. They also each contain tons of features, so you can continue to grow with them.
This section provides a brief tutorial on the vi text editor, which you can use to manually edit a text file from any shell. It also describes the improved versions of vi called vim. (If vi doesn't suit you, see the sidebar “Exploring Other Text Editors” for other options.)
The vi editor is difficult to learn at first, but once you know it, you never have to use a mouse or a function key — you can edit and move around quickly and efficiently within files just by using the keyboard.
Most often, you start vi to open a particular file. For example, to open a file called /tmp/test, type the following command:
$ vi /tmp/test
If this is a new file, you should see something similar to the following:
χ ∼ ∼ ∼ ∼ ∼ "/tmp/test" [New File]
A blinking box at the top represents where your cursor is. The bottom line keeps you informed about what is going on with your editing (here, you just opened a new file). In between, there are tildes (∼) as filler because there is no text in the file yet. Now, here's the intimidating part: There are no hints, menus, or icons to tell you what to do. To make it worse, you can't just start typing. If you do, the computer is likely to beep at you. And some people complain that Linux isn't friendly.
First, you need to know the two main operating modes: command and input. The vi editor always starts in command mode. Before you can add or change text in the file, you have to type a command (one or two letters and an optional number) to tell vi what you want to do. Case is important, so use uppercase and lowercase exactly as shown in the examples!
To get into input mode, type an input command. To start out, type any of the following commands. When you are done inputting text, press the Esc key (sometimes twice) to return to command mode. Remember the Esc key!
Type a few words and then press Enter. Repeat that a few times until you have a few lines of text. When you're finished typing, press Esc to return to command mode. Now that you have a file with some text in it, try moving around in your text with the keys or letters described in the next section.
To move around in the text, you can use the up, down, right, and left arrows. However, many of the keys for moving around are right under your fingertips when they are in typing position:
The only other editing you need to know is how to delete, copy, or change text. The x, d, y, and c commands can be used to delete and change text. All of them can be used along with movement keys (arrows, PgUp, PgDn, letters, and special keys) and numbers to indicate exactly what you are deleting, copying, or changing. Consider the following examples:
The <?> after each letter in the preceding list identifies the place where you can use a movement command to choose what you are deleting, changing, or yanking. For example:
Any of the commands just shown can be further modified using numbers, as you can see in the following examples:
After text has been copied to the buffer (by deleting, changing, or yanking it), you can place that text back in your file using the letter p or P. With both commands, the text most recently stored in the buffer is put into the file in different ways.
After you delete, change, or paste text, you can repeat that action by typing a period (.). For example, with the cursor on the beginning of the name Joe, you type cw and type Jim to change Joe to Jim. You search for the next occurrence of Joe in the file, position the cursor at the beginning of that name, and press a period. The word changes to Jim and you can search for the next occurrence.
To wrap things up, use the following commands to save or quit the file:
You have learned a few vi editing commands. I describe more commands in the following sections. First, however, consider the following tips to smooth out your first trials with vi:
Besides the few movement commands described earlier, there are other ways of moving around a vi file. To try these out, open a large file that you can't do much damage to. (Try copying /var/log/messages to /tmp and opening it in vi.) Here are some movement commands you can use:
To search for the next or previous occurrence of text in the file, use either the slash (/) or the question mark (?) character. Follow the slash or question mark with a pattern (string of text) to search forward or backward, respectively, for that pattern. Within the search, you can also use metacharacters. Here are some examples:
After you have entered a search term, simply type n or N to search forward or backward for the same term again, respectively.
The vi editor was originally based on the ex editor, which didn't let you work in full-screen mode. However, it did enable you to run commands that let you find and change text on one or more lines at a time. When you type a colon and the cursor goes to the bottom of the screen, you are essentially in ex mode. The following are examples of some of those ex commands for searching for and changing text. (I chose the words Local and Remote to search for, but you can use any appropriate words.)
To learn more about the vi editor, try typing vimtutor. The vimtutor command opens a tutorial in the vim editor that steps you through common commands and features you can use in vim.
Even a basic Linux installation can have thousands of files installed on it. To help you find files on your system, you can use commands such as locate (to find commands by name), find (to find files based on lots of different attributes), and grep (to search within text files to find lines in files that contain search text).
On most Linux systems (Fedora and RHEL included), the updatedb command runs once per day to gather the names of files throughout your Linux system into a database. By running the locate command, you can search that database to find the location of files stored in that database.
Here are a few things you should know about searching for files using the locate command:
PRUNE_BIND_MOUNTS = "yes" PRUNEFS = "9p afs anon_inodefs auto autofs bdev binfmt_misc cgroup cifs coda configfs cpuset debugfs devpts ecryptfs exofs fuse fusectl gfs gfs2 hugetlbfs inotifyfs iso9660 jffs2 lustre mqueue ncpfs nfs nfs4 nfsd pipefs proc ramfs rootfs rpc_pipefs securityfs selinuxfs sfs sockfs sysfs tmpfs ubifs udf usbfs" PRUNENAMES = ".git .hg .svn" PRUNEPATHS = "/afs /media /net /sfs /tmp /udev /var/cache/ccache /var/spool/cups /var/spool/squid /var/tmp"
Here are some examples of using the locate command to search for files:
$ locate .bashrc /etc/skel/.bashrc /home/cnegus/.bashrc # locate .bashrc /etc/skel/.bashrc /home/bill/.bashrc /home/joe/.bashrc /root/.bashrc
When run as a regular user, locate only finds .bashrc in /etc/skel and the user's own home directory. Run as root, the same command locates .bashrc files in everyone's home directory.
$ locate muttrc /usr/share/doc/mutt-1.5.20/sample.muttrc ... $ locate -i muttrc /etc/Muttrc /etc/Muttrc.local /usr/share/doc/mutt-1.5.20/sample.muttrc ...
Using locate -i, filenames are found regardless of case. In the previous example, Muttrc and Muttrc.local were found with -i whereas they weren't found without that option.
$ locate services /etc/services /usr/share/services/bmp.kmgio /usr/share/services/data.kmgio
Unlike the find command, which uses the -name option to find filenames, the locate command locates the string you enter if it exists in any part of the file's path. For example, if you search for services using the locate command, you find files and directories that contain the “services” string of text.
The find command is the best command for searching your filesystem for files, based on a variety of attributes. Once files are found, you can act on those files as well (using the -exec or -okay options) by running any commands you want on them.
When you run find, it searches your filesystem live, which causes it to run slower than locate, but gives you an up-to-the-moment view of the files on your Linux system. However, you can also tell find to start at a particular point in the filesystem, so the search can go faster by limiting the area of the filesystem being searched.
Nearly any file attribute you can think of can be used as a search option. You can search for filenames, ownership, permission, size, modification times, and other attributes. You can even use combinations of attributes. Here are some basic examples of using the find command:
$ find $ find /etc # find /etc $ find $HOME -ls
Run on a line by itself, the find command finds all files and directories below the current directory. If you want to search from a particular point in the directory tree, just add the name of the directory you want to search (such as /etc). As a regular user, find does not give you special permission to find files that have permissions that make them readable only by the root user. So, the find will produce a bunch of error messages. Run as the root user, find /etc will find all files under /etc.
A special option to the find command is -ls. A long listing (ownership, permission, size, and so on) is printed with each file when you add -ls to the find command. This option will help you in later examples when you want to verify that you have found files that contain the ownership, size, modification times, or other attributes you are trying to find.
To find files by name, you can use the -name and -iname options. The search is done by base name of the file; the directory names are not searched by default. To make the search more flexible, you can use file-matching characters, such as asterisks (*) and question marks (?), as in the following examples:
# find /etc -name passwd /etc/pam.d/passwd /etc/passwd # find /etc -iname '*passwd*' /etc/pam.d/passwd /etc/passwd- /etc/passwd.OLD /etc/passwd /etc/MYPASSWD /etc/security/opasswd
Using the -name option and no asterisks, the first example above lists any files in the /etc directory that are named passwd exactly. By using -iname instead, you can match any combination of upper and lower case. Using asterisks, you can match any filename that includes the word passwd.
If your disk is filling up and you want to find out where your biggest files are, you can search your system by file size. The -size option enables you to search for files that are exactly, smaller than, or larger than a selected size, as you can see in the following examples:
$ find /usr/share/ -size +10M $ find /mostlybig -size -1M $ find /bigdata -size +500M -size -5G -exec du -sh {} ; 4.1G /bigdata/images/rhel6.img 606M /NotBackedUp/Fedora-16-i686-Live-Desktop.iso 560M /NotBackedUp/dance2.avi
The first example in the preceding code finds files larger than 10MB. The second finds files less than 1MB. In the third example, I'm searching for ISO images and video files that are between 500MB and 5GB. This includes an example of the -exec option (which I describe more later) to run the du command on each file to see its size.
You can search for a particular owner (-user) or group (-group) when you try to find files. By using -not and -or, you can refine your search for files associated with specific users and groups, as you can see in the following examples:
$ find /home -user chris -ls 131077 4 -rw-r--r-- 1 cnegus cnegus 379 Jun 29 2010 ./.bashrc # find /home -user chris -or -user joe -ls 131077 4 -rw-r--r-- 1 cnegus cnegus 379 Jun 29 2010 ./.bashrc 181022 4 -rw-r--r-- 1 joe joe 379 Jun 15 2010 ./.bashrc # find /etc -group ntp -ls 131438 4 drwxrwsr-x 3 root ntp 4096 Mar 9 22:16 /etc/ntp # find /var/spool -not -user root -ls 262100 0 -rw-rw---- 1 rpc mail 0 Jan 27 2011 /var/spool/mail/rpc 278504 0 -rw-rw---- 1 joe mail 0 Apr 3 2011 /var/spool/mail/joe 261230 0 -rw-rw---- 1 13599 mail 0 Dec 18 14:17 /var/spool/mail/bill 277373 2848 -rw-rw---- 1 chris mail 8284 Mar 15 2011 /var/spool/mail/chris
The first example outputs a long listing of all files under the /home directory that are owned by the user chris. The next lists files owned by chris or joe. The find command of /etc turns up all files that have ntp as their primary group assignment. The last example shows all files under /var/spool that are not owned by root. You can see files owned by other users in the sample output.
Searching for files by permission is an excellent way to turn up security issues on your system or uncover access issues. Just as you changed permissions on files using numbers or letters (with the chmod command), you can likewise find files based on number or letter permissions along with the -perm options. (Refer to Chapter 4, “Moving Around the Filesystem,” to see how to use numbers and letters with chmod to reflect file permissions.)
If you use numbers for permission, as I do below, remember that the three numbers represent permissions for the user, group, and other. Each of those three numbers varies from no permission (0) to full read/write/execute permission (7), by adding read (4), write (2), and execute (1) bits together. With a hyphen (-) in front of the number, all three of the bits indicated must match; with a plus (+) in front of it, any of the numbers can match for the search to find a file. The full, exact numbers must match if neither a hyphen or plus is used.
Consider the following examples:
$ find /bin -perm 755 -ls 788884 28 -rwxr-xr-x 1 root root 28176 Mar 10 2011 /bin/echo $ find /home/chris/ -perm -222 -type d -ls 144503 4 drwxrwxrwx 8 cnegus cnegus 4096 Jun 23 2011 /home/chris/OPENDIR
By searching for -perm 755, any files or directories with exactly rwxr-xr-x permission is matched. By using -perm -222, only files that have write permission for user, group, and other are matched. Notice that, in this case, the -type d is added to match only directories.
$ find /myreadonly -perm +222 -type f 685035 0 -rw-rw-r-- 1 cnegus cnegus 0 Dec 30 16:34 /tmp/write/abc $ find . -perm -002 -type f -ls 266230 0 -rw-rw-rw- 1 cnegus cnegus 0 Dec 30 16:28 ./LINUX_BIBLE/abc
Using -perm +222, you can find any file (-type f) that has write permission turned on for the user, group, or other. You might do that to make sure that all files are read-only in a particular part of the filesystem (in this case, beneath the /myreadonly directory). The last example, -perm +002, is very useful for finding files that have open write permission for “other,” regardless of how the other permission bits are set.
Date and time stamps are stored for each file when it is created, when it is accessed, when its content is modified, or when its meta data is changed. Meta data includes owner, group, time stamp, file size, permissions, and other information stored in the file's inode. You might want to search for file data or meta data changes for any of the following reasons:
$ find /etc/ -mmin -10
$ find /bin /usr/bin /sbin /usr/sbin -ctime -3
$ find /var/ftp /var/www -atime +300
As you can glean from the examples, you can search for content or meta data changes over a certain number of days or minutes. The time options (-atime, -ctime, and -mtime) enable you to search based on the number of days since each file was accessed, changed, or had its meta data changed. The min options (-amin, -cmin, and -mmin) do the same in minutes.
Numbers that you give as arguments to the min and time options are preceded by a hyphen (to indicate a time from the current time to that number of minutes or days ago) or a plus (to indicate time from the number of minutes or days ago and older). With no hyphen or plus, the exact number is matched.
With the -not and -or options, you can further refine your searches. There may be times when you want to find files owned by a particular user, but not assigned to a particular group. You may want files larger than a certain size, but smaller than another size. Or, you might want to find files owned by any of several users. The -not and -or options can help you do that. Consider the following examples:
$ find /var/allusers ( -user joe -o -user chris ) -ls 679967 0 -rw-r--r-- 1 chris chris 0 Dec 31 12:57 /var/allusers/myjoe 679977 1812 -rw-r--r-- 1 joe joe 4379 Dec 31 13:09 /var/allusers/dict.dat 679972 0 -rw-r--r-- 1 joe sales 0 Dec 31 13:02 /var/allusers/one
$ find /var/allusers/ -user joe -not -group joe -ls 679972 0 -rw-r--r-- 1 joe sales 0 Dec 31 13:02 /var/allusers/one
$ find /var/allusers/ -user joe -and -size +1M -ls 679977 1812 -rw-r--r-- 1 joe root 1854379 Dec 31 13:09 /var/allusers/dict.dat
One of the most powerful features of the find command is the capability to execute commands on any files you find. With the -exec option, the command you use is executed on every file found, without stopping to ask if that's okay. The -ok option will stop at each matched files and ask if you want to run the command on it or not.
The advantage of using -ok is that, if you are doing something destructive, you can make sure that you okay each file individually before the command is run on it. The syntax for using -exec and -ok is the same:
$ find [options] -exec command {} ;
With -exec or -ok, you run find with any options you like to find the files you are looking for. Then enter the -exec or -ok option, followed by the command you want to run on each file. The set of curly braces indicates where on the command line to read in each file that is found. Each file can be included in the command line multiple times, if you like. To end the line, you need to add a backslash and semicolon (;). Here are some examples:
$ find /etc -iname iptables -exec echo "I found {}" ; I found /etc/bash_completion.d/iptables I found /etc/sysconfig/iptables I found /etc/rc.d/init.d/iptables
$ find /usr/share -size +5M -exec du {} ; | sort -nr 101608 /usr/share/icons/oxygen/icon-theme.cache 42636 /usr/share/virtio-win/virtio-win-1.3.3.iso 20564 /usr/share/fonts/cjkuni-uming/uming.ttc
# find /var/allusers/ -user joe -ok mv {} /tmp/joe/ ; < mv ... /var/allusers/dict.dat > ? y < mv ... /var/allusers/five > ? y
Notice in the preceding code that you are prompted for each file that is found before it is moved to the /tmp/joe directory. You would simply type y and press Enter at each line to move the file, or just press Enter to skip it.
For more information on the find command, type man find.
If you want to search for files that contain a certain search term, you can use the grep command. With grep, you can search a single file or search a whole directory structure of files recursively.
When you search, you can have every line containing the term printed on your screen (standard output) or just list the names of the files that contain the search term. By default, grep searches text in a case-sensitive way, although you can do case-insensitive searches as well.
Instead of just searching files, you can also use grep to search standard output. So, if a command turns out a lot of text and you want to find only lines that contain certain text, you can use grep to filter just want you want.
Here are some examples of grep command lines, used to find text strings in one or more files:
$ grep desktop /etc/services desktop-dna 2763/tcp # Desktop DNA desktop-dna 2763/udp # Desktop DNA $ grep -i desktop /etc/services sco-dtmgr 617/tcp # SCO Desktop Administration Server sco-dtmgr 617/udp # SCO Desktop Administration Server airsync 2175/tcp # Microsoft Desktop AirSync Protocol ...
In the first example, a grep for the word desktop in the /etc/services file turned up two lines. Searching again, using the -i to be case-insensitive (as in the second example), there were 24 lines of text produced.
To search for lines that don't contain a selected text string, use the -v option. In the following example, all lines from the /etc/services file are displayed except those containing the text tcp (case-insensitive):
$ grep -vi tcp /etc/services
To do recursive searches, use the -r option and a directory as an argument. The following example includes the -l option, which just lists files that include the search text, without showing the actual lines of text. That search turns up files that contain the text peerdns (case-insensitive).
$ grep -rli peerdns /usr/share/doc/ /usr/share/doc/dnsmasq-2.48/setup.html /usr/share/doc/initscripts-9.03.23/sysconfig.txt
The next example recursively searches the /etc/sysconfig directory for the term root. It lists every line in every file beneath the directory that contains that text. To make it easier to have the term root stand out on each line, the --color option is added. By default, the matched term appears in red.
$ grep -ri --color root /etc/sysconfig/
To search the output of a command for a term, you can pipe the output to the grep command. In this example, I know that IP addresses are listed on output lines from the ip command that include the string inet. So, I use grep to just display those lines:
$ ip addr show | grep inet inet 127.0.0.1/8 scope host lo inet 192.168.1.231/24 brd 192.168.1.255 scope global wlan0
Being able to work with plaintext files is a critical skill for using Linux. Because so many configuration files and document files are in plaintext format, you need to become proficient with a text editor to effectively use Linux. Finding file names and content in files are also critical skills. In this chapter, you learned to use the locate and find commands for finding files and grep for searching files.
The next chapter covers a variety of ways to work with processes. There, you learn how to see what processes are running, run processes in the foreground and background, and change processes (send signals).
Use these exercises to test your knowledge of using the vi (or vim) text editor, commands for finding files (locate and find), and commands for searching files (grep). These tasks assume you are running a Fedora or Red Hat Enterprise Linux system (although some tasks will work on other Linux systems as well). If you are stuck, solutions to the tasks are shown in Appendix B (although in Linux, there are often multiple ways to complete a task).
# Note that it is presently the policy of IANA to assign a single well-known # port number for both TCP and UDP; hence, most entries here have two entries # even if the protocol doesn't support UDP operations. # Updated from RFC 1700, "Assigned Numbers" (October 1994). Not all ports # are included, only the more common ones.