Wrestling with Your Jobs
Every program you run is a job according to the UNIX system, and although it might not be obvious, you have the ability to start and stop them at any time, and more.
Task 16.1: Job Control in the Shell: Stopping Jobs
 | Whether you're requesting a man page, listing files with ls, starting vi, or running just about any UNIX command, you're starting one or more processes. In UNIX, any program that's running is a process. You can have multiple processes running at once. The pipeline ls -l | sort | more invokes three processes: ls, sort, and more. Processes in both the C and Korn shells are also known as jobs, and the program you're running is known as the current job. |
Any job or process can have various states, with “running” being the most typical state. In both shells, you can stop a job by pressing ^z. To restart it, enter fg when you are ready.
Earlier I was perusing the man page entry for sort. I had reached the bottom of the first screen:% man sort SORT(1) DYNIX Programmer's Manual SORT(1) NAME sort - sort or merge files SYNOPSIS sort [ -mubdfinrtx ] [ +pos1 [ -pos2 ] ] ... [ -o name ] [ -T directory ] [ name ] ... DESCRIPTION Sort sorts lines of all the named files together and writes the result on the standard output. The name `-' means the standard input. If no input files are named, the standard input is sorted. The default sort key is an entire line. Default ordering is lexicographic by bytes in machine collating sequence. The ordering is affected globally by the following options, one or more of which may appear. b Ignore leading blanks (spaces and tabs) in field com- --More-- _I'd like to try using the -b flag mentioned at the bottom of this screen, but I want to read the rest of the man page, too. Instead of typing q to quit and then restarting the man program later, I can stop the program. I press ^z and see this:
ordering is affected globally by the following options, one or more of which may appear. b Ignore leading blanks (spaces and tabs) in field com- --More-- Stopped %At this point, I can do whatever I'd like:
% ls -s | sort -b | head -4 1 Archives/ 1 InfoWorld/ 1 Mail/ 1 News/ 1 OWL/I can resume at any time. I enter fg, the program reminds me where I was, and man (which is actually the more program invoked by man) returns to its prompt:
% fg man sort --More-- _ %
Screen-oriented programs are even smarter about stopping and starting jobs. For example, -vi refreshes the entire screen when you return from it having been stopped. If I were in vi working on the dickens.note file, the screen would look like this:
A Tale of Two Cities Preface When I was acting, with my children and friends, in Mr Wilkie Collins's drama of The Frozen Deep, I first conceived the main idea of this story. A strong desire came upon me then, to embody it in my own person; and I traced out in my fancy, the state of mind of which it would necessitate the presentation to an observant spectator, with particular care and interest. As the idea became familiar to me, it gradually shaped itself into its present form. Throughout its execution, it has had complete possession of me; I have so far verified what is done and suffered in these pages, as that I have certainly done and suffered it all myself. Whenever any reference (however slight) is made here to the condition of the Danish people before or during the Revolution, it is truly made, on the faith of the most trustworthy witnesses. It has been one of my hopes to add something to the popular and picturesque means of "dickens.note" 28 lines, 1123 charactersPressing ^z would result in this:
witnesses. It has been one of my hopes to add something to the popular and picturesque means of "dickens.note" 28 lines, 1123 characters Stopped % _I can check to see whether someone is logged in and then return to vi with the fg command.
% who | grep marv % fg A Tale of Two Cities Preface When I was acting, with my children and friends, in Mr Wilkie Collins's drama of The Frozen Deep, I first conceived the main idea of this story. A strong desire came upon me then, to embody it in my own person; and I traced out in my fancy, the state of mind of which it would necessitate the presentation to an observant spectator, with particular care and interest. As the idea became familiar to me, it gradually shaped itself into its present form. Throughout its execution, it has had complete possession of me; I have so far verified what is done and suffered in these pages, as that I have certainly done and suffered it all myself. Whenever any reference (however slight) is made here to the condition of the Danish people before or during the Revolution, it is truly made, on the faith of the most trustworthy witnesses. It has been one of my hopes to add something to the popular and picturesque means of "dickens.note" 28 lines, 1123 characters
 | There are many aspects to processes and jobs in UNIX, particularly regarding the level of control offered by the shell. The rest of this hour explains how to exploit these capabilities to make your work easier and faster. |
Task 16.2: Foreground/Background and UNIX Programs
 | Now that you know how to suspend programs in their tracks, it's time to learn how to have them keep running in the background (by using the bg command) while you're doing something else, and how to have programs instantly go into the background (by using the & notation). |
In the first hour, you learned that one of the distinguishing characteristics of UNIX is that it's a true multitasking operating system. It is capable of running hundreds of programs at the same time. The best part is that you're not limited to just one process! If you want to save a couple of man pages to a file, for example, you can run those processes in the background while you are working on something else.
Once a job is stopped, you can enter fg to restart it as the program you're working with. (The fg command takes its name from foreground, which refers to the program that your display and keyboard are working with.) If the process will continue without any output to the screen and without any requirement for input, you can use bg to move it into the background, where it runs until it is done. If the program needs to write to the screen or read from the keyboard, the system will stop its execution automatically and inform you. You then can use fg to bring the program into the foreground to continue running.
 | If you find that background jobs are just writing information to your screen, try the stty tostop command to fix the problem. |
You can also use job control to start a couple of programs and then use the fg command, with the job ID as an argument, to start the job you want to work with. Not entering the job ID will bring the most recently stopped job back to the foreground. If your system takes a long time to start big applications (such as emacs or vi), this could save you lots of time.
 | Although a job may be stopped, it still consumes resources, so you should be careful not to have too many stopped programs around, in deference to the other users of your machine. To free resources, kill or terminate the jobs instead. |
A different strategy is to start a program in the background, letting UNIX manage it. If the program needs some input or output, it stops, just like processes you've put into the background with bg after they've already started running. To have a program (or pipeline!) automatically start in the background, simply type an & at the end of the command line.
Here's an example of a command that processes files without needing any input or offering any output:% awk -F: '{ print $1" = "$5} ' < /etc/passwd | awk -F, '{ print $1} '| awk '{ if (NF > 2) print $0 } ' | sort > who.is.who
After about 20 seconds, the % prompt returns; it takes that long to feed the password file through the three-part awk filter, sort the entire output, and save it to the file who.is.who.
With this new file, I easily can look up an account to see the full name of that user:
% alias lookup 'grep -i !* who.is.who' % who | head root console Dec 6 18:02 maritanj ttyAa Dec 8 21:20 efb ttyAb Dec 8 12:12 wifey ttyAc Dec 8 19:41 phamtu ttyAe Dec 8 21:14 curts ttyAf Dec 8 21:14 seifert ttyAg Dec 8 21:11 taylor ttyAh Dec 8 21:09 halcyon ttyAi Dec 8 18:34 jamilrr ttyAj Dec 8 20:25 Broken pipe % lookup maritanj maritanj = Jorge Maritan % lookup efb efb = Edward F. Billiard %
 | When you're working with long commands, it's useful to know that you always can move to the next line—even in the middle of entering something—by ending the current line with a single backslash. Note that the backslash must be the very last character on the line. |
To have this process run in the background, I can stop the process immediately after I start it, by using ^z:
% !awk awk -F: '{ print $1" = "$5} ' < /etc/passwd | awk -F, '{ print $1} ' | awk '{ if (NF > 2) print $0 } ' | sort > who.is.who Stopped %At this point, bg will continue the program, running it in the background:
% bg [1] awk -F: { print $1" = "$5} < /etc/passwd | awk -F, { print $1} | awk { if (NF > 2) print $0 } | sort > who.is.who & %The number in square brackets is this job's control number in the shell. In a moment, you learn why this is a handy number to note.
On some systems a completed background job will notify you immediately that it's done, but on most systems, after a completed background job has finished running, it waits until you press Return to get a new system prompt before it lets you know. After about 30 or 40 seconds, I press Return and see this:
% [1] Done awk -F: { print $1" = "$5} < /etc/passwd | awk -F, { print $1} | awk { if (NF > 2) print $0 } | sort > who.is.who %
 | Notice that the command I repeated using the history mechanism was listed as being all on a single line. |
Alternatively, a better strategy for moving a program into the background is to move the process to the background automatically by adding a & to the end:
% !awk & awk -F: '{ print $1" = "$5} ' < /etc/passwd | awk -F, '{ print $1} ' | awk '{ if (NF > 2) print $0 } ' | sort > ! who.is.who & [1] 27556 27557 27558 27559 %This is more interesting. This command is shown with a control number of 1, but the four numbers listed after that are the actual process ID numbers of each piece of the pipeline: 27556 is the first awk process, 27557 is the second awk process, 27558 is the third awk process, and 27559 is the sort program.
Again, when it's complete, pressing Return lets me know:
% [1] Done awk -F: { print $1" = "$5} < /etc/passwd | awk -F, { print $1} | awk { if (NF > 2) print $0 } | sort > who.is.who %What happens if I try to automatically move to the background a program that has input or output?
% vi & [1] 28258 %This looks fine. Pressing Return indicates otherwise, though:
% [1] + Stopped (tty output) vi %
You can see that this program has stopped because of some information (output) it wants to display. If the program expected input, the message would be Stopped (tty input) program name.
I can use fg to bring this program into the foreground and work with it, or even just to quit vi.
 | Because so much of the UNIX design focuses on running streams of data through filters and saving the output to a file, there are various commands you could be running in the background, freeing you up to do other work in the meantime. Remember also that you can put in the background jobs that take a fair amount of processing time and then display information on the screen. When it's time to write something to the screen, the program will stop automatically until you enter fg to pull it into the foreground again. |
Task 16.3: Finding Out What Tasks Are Running
 | There are two ways to keep tabs on what programs are flying around in the UNIX operating system. The easier way, jobs, shows what processes you've stopped and moved into the background in the shell. Enter jobs, and csh (or ksh) tells you what programs, if any, are stopped or running. |
The alternative is a complex command called ps, which shows the processor status for the entire computer. The processor is another name for the computer itself. Fortunately, without any arguments, it shows the active or stopped programs associated with your terminal only. The ps program actually has more flags even than ls, I think. The vast majority of them, however, are never going to be of value to you or any normal UNIX user. Worse, the flags are very different between BSD systems and System V. The flags that are most helpful are summarized in Table 16.1.
| Flag | Meaning |
| -a | Shows all processes associated with terminals attached to the system. |
| -g | Shows all interesting processes on the system (that is, all processes other than those required by the operating system). |
| -l | Gives the long listing format for each line. |
| -t xx | Lists only processes associated with the specified ttyxx. |
| -u | Produces user-oriented output. |
| -w | Uses wide output format. If repeated (-ww), it will show as much of each command as possible. |
| -x | Shows all processes in the system. |
The -a, -g, and -x flags all affect how much information is displayed by ps. To use either the -g or the -x command, you also must use the -a command. On most machines, -ax yields considerably more output than -ag. The most commonly used flags (and flag combinations) are -u, to have only your processes listed in a friendly format; -aux, to see everything on the machine (you almost always want to pipe this to grep or more, lest you be overrun with hundreds of lines of information); and -wtxx, to show all the processes associated with ttyxx, in wide format.
 | The ps program varies from System V to Berkeley UNIX more than any other command. Fortunately, the two or three most common flags are similar across the two systems. To explore more about the ps command on your system, you should start by reading the man page. |
To begin, I'm going to start vi in the background:% vi dickens.note & [1] 4352 %I'll start that awk job again too:
% !awk awk -F: '{ print $1" = "$5} ' < /etc/passwd | awk -F, '{ print $1} ' | awk '{ if (NF > 2) print $0 } ' | sort > ! who.is.who & [2] 4532 4534 4536 4537 %The jobs command will show what processes I have running:
% jobs [1] + Stopped (tty output) vi dickens.note [2] - Running awk -F: { print $1" = "$5} < /etc/passwd | awk -F, { print $1} | awk { if (NF > 2) print $0 } | sort > who.is.who %Now that you know the job numbers (the numbers in square brackets here), you can easily move specific jobs into the foreground or the background by specifying the job number prefixed by %. To show what I mean, I'll put a couple more vi jobs in the background:
% vi buckaroo.confused & [2] 13056 % vi awkscript csh.man cheryl mbox & [3] 13144 %
Now I'll use the jobs command to see what's running:
% jobs [1] Stopped (tty output) vi dickens.note [2] - Stopped (tty output) vi buckaroo.confused [3] + Stopped (tty output) vi awkscript csh.man cheryl mbox %To edit the buckaroo.confused note, I need only to enter fg %2 to pull the file into the foreground. To terminate these processes (something you learn more about later in this hour), I can use the kill command:
% kill %2 %3 %Nothing happened. Or did it? Pressing Return reveals what occurred in the operating system:
% [3] - Done vi awkscript csh.man cheryl mbox [2] - Done vi buckaroo.confused %
 | Notice that the awk job finished. |
Restart the awk command with !awk. Contrast the output of jobs with the output of the Berkeley (BSD) ps command:
% ps PID TT STAT TIME COMMAND 4352 Ah T 0:00 vi dickens.note 4532 Ah R 0:03 awk - : { print $1" 4534 Ah R 0:02 awk - , { print $1} 4536 Ah S 0:01 - k { if (NF > 2) print $0 } (awk) 4537 Ah S 0:00 sort 4579 Ah R 0:00 ps %You can see here that there really are four unique processes running for that pipeline: three awk processes and one sort process. In addition, vi and ps are listed as running. Note that my login shell (csh) isn't in this listing.
Figure 16.1 explains each field, and Table 16.2 lists possible values for the STAT program status columns.
Figure 16.1. The ps default process output.
There are other process states, but they rarely show up for most users. A zombie process is one that has ended but hasn't freed up its resources. Usually, it takes a second or two for the system to completely recover all memory used by a program. Sometimes, zombies are stuck in the process table for one reason or other. UNIX folk refer to this as a wedged process, which stays around until the system is rebooted. Sometimes it's listed as <defunct> in process listings. Any process that is preceded by a sleep command is noted as sleeping.
| Value | Meaning |
| R | Running |
| S | Sleeping (20 seconds or less) |
| I | Idle (sleeping more than 20 seconds) |
| T | Stopped |
| Z | Zombie process |
Adding some flags can change the output of ps quite dramatically:
% ps -x PID TT STAT TIME COMMAND 4352 Ah T 0:00 vi dickens.note 6171 Ah R 0:02 awk - : { print $1" 6172 Ah R 0:01 awk - , { print $1} 6173 Ah S 0:01 - k { if (NF > 2) print $0 } (awk) 6174 Ah S 0:00 sort 6177 Ah R 0:00 ps -x 19189 Ah S 0:06 -csh (csh) 19649 Ah I 0:02 newmail %Two new processes show up here: -csh (the shell), which is, finally, my login shell; and newmail, a program that automatically starts in the background when I log in to the system (it's located at the end of my .login).
 | The shell process is shown with a leading dash to indicate that it's a login shell. Any other copies of csh that I run won't have that leading dash. That's one way the C shell knows not to read through the .login file every time it's run. |
To see more about what's happening, I add yet another flag, -u, to expand the output on the display:
% ps -xu USER PID %CPU %MEM SZ RSS TT STAT ENG TIME COMMAND taylor 7011 10.4 0.2 184 100 Ah R 6 0:02 awk - : { print $1" taylor 7012 6.3 0.1 160 92 Ah S 0:01 awk - , { print $1} taylor 7013 5.9 0.1 160 92 Ah R 3 0:01 - k { if (NF > 2) print taylor 19189 1.1 0.2 256 148 Ah S 0:07 -csh (csh) taylor 7014 1.0 0.1 316 64 Ah S 0:00 sort taylor 7022 0.1 0.2 180 116 Ah R 0 0:00 ps -xu taylor 4352 0.0 0.3 452 168 Ah T 0:00 vi dickens.note taylor 19649 0.0 0.1 124 60 Ah I 0:02 newmail %Figure 16.2 explains these fields.
Figure 16.2. The -u user-oriented output of ps.
I won't show the output from the -aux flags, but you should look at the number of lines produced by the -ag and -ax flags:
% ps -ag | wc -l 377 % ps -ag | head PID TT STAT TIME COMMAND 1403 co IW 0:01 -csh (csh) 2200 p3 IW 0:18 server 6076 p6 I 0:13 rlogin sage -l hirschna 6082 p6 I 0:11 rlogin sage -l hirschna 25341 p8 IW 0:06 -tcsh (tcsh) 681 pa IW 0:05 -tcsh (tcsh) 10994 pa IW 2:10 ghostview pop5.ps 11794 pa IW 0:12 pwlookup 13861 pa I 0:56 gs Broken pipe %
You can see here that each process is owned by a specific terminal but that these processes are all idle (that is, they've been sleeping for more than 20 seconds). This probably means that these users have turned away for a little while. Look back at the output generated by ps -xu, and you will see that newmail is also idle. That's because the program runs in a loop: It sleeps for five minutes, checks for new mail, goes back to sleep again, and so on. Processes that have the W after the I in the status column are processes that have been moved out of main memory and are swapped out to disk. This is not a problem, and the users might not even realize that anything has happened; the only symptom of this is that when the users wake up their programs, the programs will take an additional second or two to return.
What is the output from ps -ax?
% ps -ax | wc -l 765 % ps -ax | head PID TT STAT TIME COMMAND 0 ? D 8:58 swapper 1 ? S 14:45 (init) 2 ? D 20:43 pagedaemon 27 ? I 0:00 rpc.rquotad 59 ? S 6:36 /etc/syslogd -m480 70 ? I 0:02 /etc/portmap 74 ? IW 0:00 (biod) 75 ? IW 0:00 (biod) 76 ? IW 0:00 (biod) Broken pipe %
These are some of the “guts” of the UNIX operating system. Notice that none of these processes is actually associated with a terminal. Also notice that some of these processes have incredibly low process ID numbers! Any one-digit process ID is a program that is a part of the core UNIX system and must be running for UNIX to be alive. Any two-digit process is also started by the system itself but is probably optional. The D status for some of these processes indicates that they're waiting for disk resources of some sort. Finally, note how much time these processes have taken. I venture that you will never have a process that takes 20 minutes of CPU time—ever!
On a Sun workstation, the output of the ps commands is a bit different:
% ps PID TT STAT TIME COMMAND 8172 qb S 0:00 -csh (csh) 8182 qb T 0:00 vi 8186 qb R 0:00 ps %In many ways, though, these different workstations have very similar output from the ps commands. For example, compare this Sequent output from ps -xu to the ps -xu output on the Sun that I already showed:
% ps -xu USER PID %CPU %MEM SZ RSS TT STAT START TIME COMMAND taylor 8191 7.7 0.4 284 536 qb R 19:16 0:00 ps -xu taylor 8182 0.0 0.4 140 432 qb T 19:16 0:00 vi taylor 8172 0.0 0.3 68 400 qb S 19:16 0:00 -csh (csh) taylor 8180 0.0 0.1 52 144 qb S 19:16 0:00 newmail %The ENG column of the previous examples is replaced by a START column on the Sun workstation. The numbers in the START column indicate the exact time when the processes were started on the computer.
 | UNIX works with processes. Your login shell, the edit session you run, and even the ls program listing your files are all processes in the operating system. This means that you can work with processes. You can stop programs temporarily to do something else, restart them as you choose, and even look at all the programs you're running at any time, including otherwise hidden processes such as your login shell itself. |
Task 16.4: Terminating Processes with kill
 | Now that you know how to create multiple processes, tuck some into the background, and find stray processes, you need some way to permanently stop them from running, as needed. The command to accomplish this in UNIX is kill. For the most part, to use kill, you specify the process ID numbers of those programs you want to terminate. Both the C shell and the Korn shell have a convenient shorthand you've already seen: the percent-job-number notation. |
The kill command can send various signals to a process. To specify a job control action, you need to specify to kill one of the various signals. Table 16.3 lists signals you can use with kill.
| Number | Name | Meaning |
| 1 | SIGHUP | Hang up |
| 2 | SIGINT | Interrupt |
| 9 | SIGKILL | kill (cannot be caught or ignored) |
| 15 | SIGTERM | Software termination signal from kill |
UNIX knows about more than 30 signals, but Table 16.3 lists the ones that are most helpful. The SIGHUP signal is what's sent to every process you are running just before you hang up (log out of the system). SIGINT is the signal sent when you press ^c; many programs respond in specific ways when this signal is received. SIGKILL is the “Terminator” of the UNIX signals: Programs cannot ignore it and cannot process it. The process is terminated immediately, without even a chance to clean up after itself. SIGTERM is the more graceful alternative: It requests an immediate termination of the program, but it allows the program to remove temporary files it might have created.
By default, kill sends a SIGTERM to the processes specified. You can specify other signals, however, by using either the number or the name of the signal (minus the SIG prefix, that is). On many systems, you also can specify the -l flag to kill to see what signals are available.
 | The kill command should be used with caution. It can get you into a lot of trouble. For example, do you want to log out rather suddenly? To do that, find the process ID of your login shell and terminate it. Learn to use kill, but learn to use it cautiously. |
The simplest way to use the kill command is from the shell. First, start a job in the background:% vi & [1] 6016 %I can terminate this process now by using either kill %1 or kill 6016, but if I try both of them, the second command will fail because the first already will have terminated the process:
% kill %1 % kill 6016 6016: No such process [1] Done vi %
Just as if I had dropped a process into the background and it instantly stopped because it needed to produce output, the kill process also had no feedback and took a second or two to occur. In the interim, I entered the second kill command, which then output the error message No such process. Following that, I get an indication from the shell itself that the job ended.
Using the ps command, I can find that pesky newmail program that's always running in the background:
% ps -ux | grep newmail taylor 6899 0.1 0.1 52 28 Av S 0:00 grep newmail taylor 25817 0.0 0.1 124 60 Av I 0:01 newmail %I want to send that process a hang-up signal (SIGHUP):
% kill -HUP 25817 % !ps ps -ux | grep newmail taylor 7220 0.0 0.1 52 28 Av S 0:00 grep newmail %
Because the newmail program isn't in this listing, I can conclude that the SIGHUP signal stopped newmail.
 | Because kill tells you if a process cannot be found, the typical UNIX solution to finding out whether the command worked is to enter !! immediately to repeat the kill command a second time. If kill worked, you see No such process. |
Some processes are pesky and can resist the less powerful signals SIGTERM and SIGHUP. (In UNIX, this is called “catching” a signal. In some processes, you need to send and catch signals to perform certain actions.) That's when you need to use what I call the “Big Guns,” or SIGKILL. You see this referred to sometimes as the terminate-with-extreme-prejudice command; the format is kill -9 processID, and it's not for the faint of heart!
I strongly recommend that you just let kill send the SIGTERM signal and see whether that does the job. If it doesn't, try SIGHUP, and if that also fails, use SIGKILL as a last resort.
What happens if you try to use kill on jobs that aren't yours? Fortunately, it doesn't work:
% ps -aux | head -5 USER PID %CPU %MEM SZ RSS TT STAT ENG TIME COMMAND news 7460 97.7 0.4 336 252 ? R N 4 4:33 sort -u /tmp/nnsubj6735a phaedrus 8693 18.1 1.1 1260 720 rm S 0:03 nn root 8741 14.4 0.4 416 252 ? R 9 0:03 nntpd root 8696 13.9 0.4 416 252 ? S 0:03 nntpd Broken pipe % kill 7460 7460: Not owner %
Finally, if you forget and leave stopped jobs in the background and try to log out, here's what happens:
% logout There are stopped jobs. %You must either use fg to bring each job into the foreground and terminate each normally, or use kill to terminate each of the jobs and then log out.
 | In this task, you have been introduced to the kill command and some of the signals associated with it. |