Have you ever wondered what modifications a web or mail administrator makes to her servers? Maybe you’re curious about what policies other administrators use to implement bandwidth control? How do busy administrators manage the log data from a server farm?
Perhaps you’ve contemplated using the Expect scripting language.
However, there’s a good chance you’ve never thought of using eesh
, a totally undocumented but useful
scripting utility.
This chapter also includes two hacks on the emergency repair process, as many users prefer to hope that they’ll never need an emergency repair kit. Instead, learn to overcome your fear of the inevitable and master the art of repairing before the emergency.
Know how to tune and what to tune on your FreeBSD system
As an administrator, you want to tune your server systems so they work at peak efficiency. How do you know what to tune? The answer depends heavily upon the system’s function. Will the system perform a lot of small network transactions? Will it perform a small number of large transactions? How will disk operations factor in?
How you answer these and other questions determines what you need to do to improve the performance of your systems. This hack starts with general optimizations and then looks at function-specific tunables.
A good place to start is with software compiling, as you want to compile software and updates as efficiently as possible. Whenever you compile, your compiler makes assumptions about your hardware in order to create binaries. If you have an x86-compliant CPU, for example, your compiler will create binaries that can run on any CPU from a 386 onward. While this allows portability, it won’t take advantage of any new abilities of your CPU, such as the extended MMX, SSE, SSE2, or 3DNow! instruction sets. This is also why using precompiled binaries on your system is a surefire way to reduce your overall performance.
To ensure that software will be compiled efficiently, update your compiler flags in /etc/make.conf . This file does not exist on new systems, but you can copy it from /usr/share/examples/etc/defaults/make.conf.
Start by editing the CPUTYPE=
line to reflect your CPU type; you’ll find supported types listed as
comments just before this line. While this will take advantage of your
CPU’s features, the disadvantage is that your compiled binaries may
not run on different CPU types. However, if all of your systems run
the same CPU platform, any optimizations you make to shared binaries
will affect all of your systems equally well.
Next, change the CFLAGS
line
to CFLAGS= -O2 -pipe
-funroll-loops
. The -pipe
option can significantly decrease the amount of time it takes to
compile software, by using pipes to communicate between compiler
processes instead of temporary files, but at the expense of using
slightly more memory. The -funroll-loops
saves one CPU register that
would otherwise be tied up in tracking the iteration of the loop, but
at the expense of making a slightly larger binary.
In your kernel configuration, add the following line after
the machine i386
line:
makeoptions COPTFLAGS="-O2 -pipe -funroll-loops -ffast-math"
This is similar to the CLAGS
option in /etc/make.conf, except that it optimizes
kernel compilation.
See [Hack #54] for instructions on how to strip and compile a kernel.
You can also add this line:
TOP_TABLE_SIZE=number
where number
is a prime number that
is at least twice the number of lines in /etc/passwd. This statement sets the size
of the hash that top
uses.
Set the following option if you have an AMD K5/K6/K6-2 or Cyrix 6x86 chip. It enables cache write allocation for the L1 cache, which is disabled by default for these chips.
options CPU_WT_ALLOC
This option will disable NFS server code, so include it when you know that you will not be acting as an NFS server:
options NFS_NOSERVER
Another way of saving kernel memory is to define the maximum
number of swap devices, as shown in the next example. Your kernel
needs to allocate a fixed amount of bitmapped memory so that it can
interleave swap devices. I set the number to 1
on my workstation and 2
on my servers. If I need to add more to a
server, I can easily create another partition.
options NSWAPDEV=number
If you plan on compiling all your requisites into the kernel (NIC driver, IPF/IPFW, etc.) and won’t be loading any of these options as modules, you can include this line to skip module compiling. This saves significantly on the time taken to compile a kernel (sometimes reducing it by two-thirds).
makeoptions MODULES_OVERRIDE=""
By default, all kernel options are compiled as modules. This
allows you to use kldload
to load a
module even though it isn’t specified in your kernel configuration
file.
The advantage of MODULES_OVERRIDE
is the decrease in kernel compilation time. The
disadvantage is that you’ll need to recompile your kernel if you ever
need to add additional functionality, since you will have lost the
ability to load the kernel module separately.
Most modern network cards and switches support the ability to auto-negotiate the communication speed. While this reduces administration, it comes at the cost of network throughput. If your switch, server, or workstation is set to use auto-negotiation, it will stop transferring network traffic every few moments to renegotiate its speed.
If your network driver supports it, you can set network speed
with ifconfig
at runtime or in /etc/rc.conf at boot time. Here is an
example:
% grep fxp0 /etc/rc.conf
ifconfig_fxp0="inet x.x.x.x netmask x.x.x.x media 100BaseTX mediaopt
full-duplex"
Read the manpage for your NIC driver to see whether it
supports mediaopt
. For example,
if your NIC is rl0
, read man 4 rl
.
Next, you can enable DEVICE_POLLING
in your kernel, which changes the method by which data
travels from your network card to the kernel. Without this setting,
frequent interrupt calls may never free the kernel. This is known as
livelock and can leave your machine unresponsive. Those of us
unfortunate enough to be on the wrong side of certain
denial-of-service attacks know about this.
The DEVICE_POLLING
option
causes the kernel to poll the network card at certain predefined
times, during idle loops, or on clock interrupts. This allows the
kernel to decide when it is most efficient to poll a device for
updates and for how long, and ultimately results in a significant
increase in performance.
To take advantage of DEVICE_POLLING
, you need to compile two
options into your kernel: options
DEVICE_POLLING
and options
HZ=1000
. The latter option slows the clock interrupts to 1,000 times
per second, which prevents the kernel from polling too often.
Once you’ve recompiled your kernel, you’ll still need to enable the feature. Add this line to /etc/sysctl.conf:
kern.polling.enable=1
The DEVICE_POLLING
option
does not work with SMP-enabled kernels by default. If you are
compiling an SMP kernel with DEVICE_POLLING
, first remove the following
lines from /usr/src/sys/kern/kern_poll.c:
#ifdef SMP #include "opt_lint.h" #ifndef COMPILING_LINT #error DEVICE_POLLING is not compatible with SMP #endif #endif
Mail servers typically have a very large number of network connections, during which they transfer a small amount of data for a short period of time before closing the connection. In this case, it is useful to have a large number of small network buffers.
Network connections have two buffers, one for sending and one for receiving. The size of the buffer dictates how quickly data will funnel through the network and, in the event of a network delay, how much data can back up the server for that connection before there is a problem. Having a network buffer that is too small will cause a data backlog as the CPU waits for the network to clear, which causes greater CPU overhead. Having a network buffer that is too large wastes memory by using the buffer inefficiently. Finding a balance is the key to tuning.
I find that multiplying the number of established connections by 32 leaves me with room to breathe in the event that I see an abnormally high surge of traffic. I’ve come to this number over time through trial and error. So, if you expect to have a peak of 128 servers sending you mail, having 8,192 network buffer clusters would be good (128 2 per connection 32). Also, remember that connections can take up to two full minutes or more to close completely. If you expect more than 128 emails in any given two-minute period, increase the number accordingly.
Another important value to control is the maximum number of sockets. Start with the same number of sockets as there are network buffers, and then tune as appropriate.
You can find out how many network buffer clusters are in use
with the command netstat -m
. You
can specify the values you want in /boot/loader.conf. For example:
kern.ipc.nmbclusters=8192 kern.ipc.maxsockets=8192
As with any performance tuning, monitor your system after making changes. Did you go overboard or underestimate what you would need? Always check and adjust accordingly.
File servers generally have longer-lived and less frequent network connections than those on mail servers. They usually transfer larger files.
To determine the optimal number of network buffer clusters, consider how many clients you have. Multiplying the number of network buffers by two is good practice, though some admins prefer to multiply by four to accommodate multiple file transfers. If you have 128 clients connecting to the file server, set the number of network buffer clusters to 1,024 (128 2 per connection 4).
If you have more than one element on your web page (for example, multiple images or frames), expect web browsers to make multiple connections to your web server. It’s common to see four connections per page served. Also count any database or network connections made in server-side scripting.
Web servers go through periods of highs and lows. While you might serve 100 pages per minute on average, at your low you might serve 10 pages per minute and at peak over 1,000 pages per minute. At a peak of 1,000 pages per minute, your clusters and sockets should be around 16,384 (1,000 pages 2 per connection 4 connections 2 for growth).
man tuning
man gcc
(the GCC manpage,
which explains CPU compiling optimizations)
man ifconfig
“Tuning FreeBSD for different applications” (http://silverwraith.com/papers/freebsd-tuning.php)
“Optimizing FreeBSD and its kernel” (http://silverwraith.com/papers/freebsd-kernel.php)
Notes on tuning Apache servers at http://www.bolthole.com/uuala/webtuning.txt
Allocate bandwidth for crucial services.
If you’re familiar with your network traffic, you know that it’s possible for some systems or services to use more than their fair share of bandwidth, which can lead to network congestion. After all, you have only so much bandwidth to work with.
FreeBSD’s dummynet
may provide a viable method of getting the most out of
your network, by sharing bandwidth between departments or users or by
preventing some services from using up all your bandwidth. It does so by
limiting the speed of certain transfers on your network—also called
traffic shaping.
To take advantage of the traffic shaping functionality of your FreeBSD system, you need a kernel with the following options:
options IPFIREWALL options DUMMYNET options HZ=1000
dummynet
does not require the
HZ
option, but its manpage strongly recommends it. See
[Hack
#69] for more about HZ
and
[Hack #54] for detailed
instructions about compiling a custom kernel.
The traffic-shaping mechanism delays packets so as not to exceed
the transfer speed limit. The delayed packets are stored and sent
later. The kernel timer triggers sending, so setting the frequency to
a higher value will smooth out the traffic by providing smaller
delays. The default value of 100 Hz will trigger sends every 10
milliseconds, producing bursty traffic. Setting HZ=1000
will cause the trigger to happen
every millisecond, resulting in less packet delay.
Traffic shaping occurs in three stages:
Configuring the pipes
Configuring the queues
Diverting traffic through the queues and/or pipes
Pipes are the basic elements of the traffic shaper. A pipe emulates a network link with a certain bandwidth, delay, and packet loss rate.
Queues implement weighted fair queuing and cannot be used without a pipe. All queues connected to a pipe share the bandwidth of that pipe in a certain configurable proportion.
The most important parameter of a pipe configuration is its bandwidth. Set the bandwidth with this command:
# ipfw pipe 1 config bw 120kbit/s
This is a sample command run at the command prompt. However,
as the hack progresses, we’ll write the actual dummynet
policy as rules within an
ipfw
rulebase.
This command creates pipe 1 if it does not already exist, assigning it 120 kilobits per second of bandwidth. If the pipe already exists, its bandwidth will be changed to 120 Kbps.
When configuring a queue, the two most important parameters are the pipe number it will connect to and the weight of the queue. The weight must be in the range 1 to 100, and it defaults to 1. A single pipe can connect to multiple queues.
# ipfw queue 5 config pipe 1 weight 20
This command instructs dummynet
to configure queue 5
to use pipe 1, with a weight of 20. The weight parameter allows you to
specify the ratios of bandwidth the queues will use. Queues with
higher weights will use more bandwidth.
To calculate the bandwidth for each queue, divide the total bandwidth of the pipe by the total weights, and then multiply each weight by the result. For example, if a 120 Kbps pipe sees active traffic (called flows) from three queues with weights 3, 2, and 1, the flows will receive 60 Kbps, 40 Kbps, and 20 Kbps, respectively.
If the flow from the queue with weight 2 disappears, leaving only the flows with weights 3 and 1, those will receive 90 Kbps and 30 Kbps, respectively. (120 / (3+1) = 30, so multiply each weight by 30.)
The weight concept may seem strange, but it is rather simple. Queues with equal weights will receive the same amount of bandwidth. If queue 2 has double the weight of queue 1, it has twice as much bandwidth. Queues that have no traffic are not taken into account when dividing traffic. This means that in a configuration with two queues, one with weight 1 (for unimportant traffic) and the other with weight 99 (for important business traffic), having both queues active will result in 1%/99% sharing, but if there is no traffic on the 99 queue, the unimportant traffic will use all of the bandwidth.
Another very useful option is to create a mask by adding
mask
mask-specifier
at the end your config
line. Masks allow you to turn one
flow into several flows; the mask will distinguish the different
flows.
The default mask is empty, meaning all packets fall into the
same flow. Using mask all
would
make all connections significant, meaning that every TCP or UDP
connection would appear as a separate flow.
When you apply a mask to a pipe, each of that pipe’s flows acts as a separate pipe. Yet, each of those flows is an exact clone of the original pipe, in that they all share the same parameters. This means that the three active flows from our example pipe will use 360 Kbps, or 120 Kbps each.
For a queue, the flows will act as several queues, each with the same weight as the original one. This means you can use the mask to share a certain bandwidth equally. For our example with three flows and the 120 Kbps pipe, each flow will get a third of that bandwidth, or 40 Kbps.
This hack assumes that you will integrate these rules in your
firewall configuration or that you are using ipfw
only for traffic shaping. In the latter
case, having the IPFIREWALL_DEFAULT_TO_ACCEPT
option in the kernel will greatly simplify your
task.
In this hack, we sometimes limit only incoming or outgoing bandwidth. Without this option, we would have to allow traffic in both directions, traffic through the loopback interface, and through the interface we will not limit.
However, you should consider disabling the IPFIREWALL_DEFAULT_TO_ACCEPT
option, as it
will drop packets that your policy does not specifically allow.
Additionally, enabling the option may cause you to accept potentially
malicious traffic you hadn’t considered. The example configurations in
this hack were tested with an ipf
-based firewall that had an explicit deny
rule at the end.
When integrating traffic shaping into an existing ipfw
firewall, keep in mind that an ipfw pipe
or ipfw
queue
rule is equivalent to “ipfw accept after slow down . .
. " if the sysctl
net.inet.ip.fw.one_pass
is set to 1 (the
default). If the sysctl
is set to
0, that rule is just a delay in a packet’s path to the next rule,
which may well be a deny or another round of shaping. This hack
assumes that the default behavior of the pipe and queue commands is to
accept
or an equivalent
action.
There are several ways of limiting bandwidth. Here are some examples that assume an
external interface of ed0
:
# only outgoing gets limited ipfw pipe 1 config bw 100kbits/s ipfw add 1 pipe 1 ip from any to any out xmit ed0
To limit both incoming and outgoing to 100 and 50 Kbps, respectively:
ipfw pipe 1 config bw 100kbits/s ipfw pipe 2 config bw 50kbits/s ipfw add 100 pipe 1 ip from any to any in recv ed0 ipfw add 100 pipe 2 ip from any to any out xmit ed0
To set a limitation on total bandwidth (incoming plus outgoing):
ipfw pipe 1 config bw 100kbits/s ipfw add 100 pipe 1 ip from any to any in recv ed0 ipfw add 100 pipe 1 ip from any to any out xmit ed0
In this example, each host gets 16 Kbps of incoming bandwidth (outgoing is not limited):
ipfw pipe 1 config bw 16kbits/s mask dst-ip 0xffffffff ipfw add 100 pipe 1 ip from any to any in recv ed0
Here are a couple of real-life examples. Let’s start by limiting a web server’s outgoing traffic speed, which is a configuration I have used on one of my servers. The server had some FreeBSD ISO files, and I did not want it to hog all the outgoing bandwidth. I also wanted to prevent people from gaining an unfair advantage by using download accelerators, so I chose to share the total outgoing bandwidth equally among 24-bit networks.
# pipe configuration, 2000 kilobits maximum ipfw pipe 1 config bw 2000kbits/s # the queue will be used to enforce the /24 limit mentioned above ipfw queue 1 config pipe 1 mask dst-ip 0xffffff00 # with this mask, only the first 24 bits of the destination IP # address are taken into consideration when generating the flow ID # divert outgoing traffic from the web server (at 1.1.1.1) ipfw add queue 1 tcp from 1.1.1.1 80 to any out
Another real-life example involves limiting incoming traffic by department. This configuration limits the incoming bandwidth for a small company behind a 1 Mbps connection. Before this was applied, some users were using peer-to-peer clients and download accelerators, and they were hogging almost all the bandwidth. The solution was to implement some weighted sharing between departments and let the departments take care of their own hogs.
# Variables we will use # External interface EXTIF=fxp0 # My IP address ME=192.168.1.1 # configure the pipe, 95% of total incoming capacity ipfw pipe 1 config bw 950kbits/s # configure the queues for the departments # departments 1 and 2 heavy net users ipfw queue 1 config pipe 1 weight 40 ipfw queue 2 config pipe 1 weight 40 # accounting, they shouldn't use the network a lot ipfw queue 3 config pipe 1 weight 5 # medium usage for others ipfw queue 4 config pipe 1 weight 20 # incoming mail (SMTP) to this server, HIGH priority ipfw queue 10 config pipe 1 weight 100 # not caught by the previous categories - VERY LOW bandwidth ipfw queue 11 config pipe 1 weight 1 # classify the traffic # only incoming traffic is limited, outgoing is not affected. ipfw add 10 allow ip from any to any out xmit via $EXTIF # department 1 ipfw add 100 queue 1 ip from any to 192.168.0.16/28 in via $EXTIF # department 2 ipfw add 200 queue 2 ip from any to 192.168.0.32/28 in via $EXTIF # accounting ipfw add 300 queue 3 ip from any to 192.168.0.48/28 in via $EXTIF # mail ipfw add 1000 queue 10 ip from any to $ME 25 in via $EXTIF # others ipfw add 1100 queue 11 ip from any to any in via $EXTIF
The incoming limit is set to 95% of the true available bandwidth. This will allow the shaper to delay some packets. If this were not the case and the pipe had the same bandwidth as the physical link, all of the delay queues for the pipe would have been empty. The extra 5% of bandwidth on the physical link fills the queues. The shaper chooses packets from the queues based on weight, passing through packets from queues with a higher weight before packets from queues with lower weight.
man dummynet
man ipfw
man ipf
“Using Dummynet for Traffic Shaping on FreeBSD” (http://www.bsdnews.org/02/dummynet.php)
The Boy Scout and system administrator motto: “Be prepared!"
As a good administrator, you back up on a regular basis and periodically perform a test restore. You create images [Hack #23] of important servers so you can quickly recreate a system that is taken out of commission.
Are you prepared if a system simply refuses to boot?
Some parts of your drives are as important as your data, yet few backup programs back them up. I’m talking about your partition table and your boot blocks. Pretend for a moment that these somehow become corrupted. The good news is that your operating system and all of your data still exist. The bad news is that you can no longer access them.
Fortunately, this is recoverable, but only if you’ve done some preparatory work before the disaster. Let’s see what’s required to create an emergency repair kit.
When you install a system, particularly a server, invest some time preparing for an emergency. On a FreeBSD system, your kit should include:
The original install CD (or two floppies containing kern.flp and mfsroot.flp or one floppy containing boot.flp)
A floppy containing additional drivers, drivers.flp
A fixit floppy, fixit.flp (or a CD containing the live filesystem; this will be the second, third, or fourth CD in a set, but not the first CD)
A printout of your partition table, /etc/fstab, and /var/run/dmesg.boot
Place these items in an envelope and store it in a secure location with your backup tapes. Make a note on the envelope of the system to which this kit should apply, along with the version of the operating system. Ideally, you should have two copies of both your emergency kit and backup media. Store the second copy off-site.
Regardless of how you install a system, take a few minutes to
download the *.flp files found in
the floppies directory. This is especially important if you use
cvsup
to upgrade a system, as you
can go months or years without the installation CD-ROM or floppy
media. Your aim is to test these floppies on your system
before a disaster strikes. The last thing you
want to be doing in an emergency is scurrying around creating floppies
only to find that an essential driver is missing.
Here, I’ll connect to the main FreeBSD FTP server and download the files for an i386, 5.1-RELEASE system:
#ftp ftp.freebsd.org
Trying 62.243.72.50... Connected to ftp.freebsd.org. <snip banner> 220 Name (ftp.freebsd.org:dlavigne6):anonymous
331 Guest login ok, send your complete e-mail address as password. Password: ftp>cd pub/FreeBSD/releases/i386/5.1-RELEASE/floppies
250 CWD command successful. ftp>binary
200 Type set to I. ftp>mget *.flp
mget boot.flp [anpqy?]?a
Prompting off for duration of mget. <snip transfer of five files> ftp>bye
221 Goodbye.
I find it convenient to create a floppies directory with subdirectories for each version of FreeBSD I have running in my network. I then download the appropriate *.flp files to the appropriate subdirectory so they are available when I wish to create an emergency repair kit for a new system.
Once you have all five files, you can decide which ones you’ll need for your particular system. To perform an emergency repair, you’ll need some way to load your version of the operating system into memory so you can access the utilities on the fixit floppy and restore whatever damage has happened to your own operating system. There are several ways to load an operating system.
The first approach is to boot directly from the install CD-ROM, assuming it is bootable and your BIOS supports this. If this is your scenario, you don’t need boot.flp, kern.flp, or mfsroot.flp.
If booting from the CD-ROM isn’t an option, you can use either boot.flp or both kern.flp and mfsroot.flp. boot.flp is basically the contents of the other two floppies placed onto one floppy. The kicker is that you need a floppy capable of holding 2.88 MB of data.
Depending upon your hardware, you may or may not need drivers.flp. If the installer detected all of your hardware, you won’t need this floppy. Otherwise, you will. Finally, if you don’t have a CD containing the live filesystem, you’ll need fixit.flp, as this floppy contains the actual repair utilities.
Use dd
to transfer these
files to floppies. Repeat this for each *.flp file you require, using a different
floppy for each file:
# dd if=fixit.flp of=/dev/fd0
Label each floppy with its name and version of FreeBSD and write protect the floppies.
Before testing your floppies, print some important system information—you won’t remember all of these details in an emergency. First, you’ll want a copy of your filesystem layout:
# more /etc/fstab
# Device Mountpoint FStype Options Dump Pass#
/dev/ad0s1b none swap sw 0 0
/dev/ad0s1a / ufs rw 1 1
/dev/ad0s1e /tmp ufs rw 2 2
/dev/ad0s1f /usr ufs rw 2 2
/dev/ad0s1d /var ufs rw 2 2
/dev/acd0 /cdrom cd9660 ro,noauto 0 0
proc /proc procfs rw 0 0
linproc /compat/linux/proc linprocfs rw 0 0
/dev/fd0 /floppy msdos rw,noauto 0 0
Here, I’ve just sent the output to a pager for viewing.
Depending upon how printing is set up on your system, redirect that
output either directly to lpr
or to
a file that you can send to a printer.
Notice that all of my hard drive partitions start with /dev/ad0s1. The name of your hard drive is needed in order to view the partition table, or what FreeBSD calls the disklabel:
# bsdlabel ad0s1
# /dev/ad0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 524288 0 4.2BSD 2048 16384 32776
b: 1279376 524288 swap
c: 30008097 0 unused 0 0 # "raw" part, don't edit
d: 524288 1803664 4.2BSD 2048 16384 32776
e: 524288 2327952 4.2BSD 2048 16384 32776
f: 27155857 2852240 4.2BSD 2048 16384 28512
Once you have a printout of your disklabel, complete your kit by printing the contents of /var/run/dmesg.boot. This file contains your startup messages, including the results of the kernel probing your hardware.
Now you’re ready to test that your kit works before sealing
the envelope and sending it off for secure storage. First, boot the
system using either your CD-ROM or the emergency floppies. Once the
kernel has loaded and probed your hardware, the screen will ask:
Would you like to load kernel
modules from the driver floppy?
If
you choose yes, you will be asked to insert the drivers.flp floppy and will be presented
with a list of modules to choose from:
cd9660.ko if_awi.ko if_fwe.ko if_sk.ko if_sl.ko if_sn.ko <snip>
Taking a look at those modules, aren’t you glad you’re testing
your kit before an emergency? While the modules
don’t have the most descriptive names, it’s easy to find out what each
module represents if you have access to a working system. For example,
the modules that begin with if
are
interfaces. To see what type of interface if_awi.ko
is:
% whatis awi
awi(4) - AMD PCnetMobile IEEE 802.11 PCMCIA wireless network driver
You can whatis
each name;
just don’t include the beginning if
or the trailing .ko
. If you do need
any of these drivers, save yourself some grief and write yourself a
note explaining which drivers to choose off of the drivers.flp. The lucky bloke who has to
repair the system will thank you for this bit of homework.
Once you exit from this menu, you’ll be prompted to remove the
floppy. You’ll then be presented with the sysinstall Main Menu
screen. Choose Fixit
from the menu and insert fixit.flp. You should be prompted to press
Alt F4
, and you should then see a
Good Luck!
screen with a Fixit#
prompt. Excellent, your floppy is
good and your repair kit is complete. Type exit
to return to the menu and exit your way
out of the install utility.
If this had been an actual emergency, you’d definitely want to read the next hack [Hack #72] .
man bsdlabel
The Emergency Restore Procedure section of the FreeBSD Handbook (http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/backup-basics.html)
Learn how to use your emergency repair kit before the emergency.
Now that you have an emergency repair kit, it’s worth your while to do a dry run so you know ahead of time what options will be available to you. You may even decide to modify your kit as a result of this test.
Let’s go back to that sysinstall Main Menu screen [Hack
#71] and see what happens when you choose Fixit
. You’ll be presented with the following
options:
Please choose a fixit option
There are three ways of going into "fixit" mode: - you can use the live filesystem CDROM/DVD, in which case there will be full access to the complete set of FreeBSD commands and utilities, - you can use the more limited (but perhaps customized) fixit floppy, - or you can start an Emergency Holographic Shell now, which is limited to the subset of commands that is already available right now.X
Exit Exit this menu (returning to previous)2
CDROM/DVD Use the "live" filesystem CDROM/DVD3
Floppy Use a floppy generated from the fixit image4
Shell Start an Emergency Holographic Shell
If you choose the Shell
option,
you’ll find that they weren’t kidding when they warned you’d be limited
to a subset of commands. Nearly all of the commands you know and love
will result in a not found error
message. This is why you went to the trouble of either creating that
fixit floppy or purchasing/burning
a CD-ROM/DVD that contains the live filesystem.
Let’s see what you can repair with the fixit floppy.
When you choose that option, follow the prompts: insert the floppy,
then press Alt F4
. Do make note of
the message you receive:
+-----------------------------------------------------------------------+ | You are now running from FreeBSD "fixit" media. | | --------------------------------------------------------------------- | | When you're finished with this shell, please type exit. | | The fixit media is mounted as /mnt2. | | | | You might want to symlink /mnt/etc/*pwd.db and /mnt/etc/group | | to /etc/ after mounting a root filesystem from your disk. | | tar(1) will not restore all permissions correctly otherwise! | | | | Note: you might use the arrow keys to browse through the | | command history of this shell. | +-----------------------------------------------------------------------+ Good Luck! Fixit#
It’s not a bad idea to create those symlinks now, before you forget. You’ll have to mount your root slice first, so refer to your /etc/fstab printout for the proper name of that slice. In this example, / is on /dev/ad0s1a. I’ll mount it with the read-write option:
Fixit# mount -o rw /dev/ad0s1a /mnt
Fixit#
If your command is successful, you’ll receive the prompt back. A
quick ls
through /mnt should convince you that you now have
access to the hard disk’s root filesystem.
If your command is not successful, run fsck_ffs
until the filesystem is clean, then mount the
filesystem:
Fixit#fsck_ffs /dev/ad0s1
** /dev/ad0s1 ** Last Mounted on /mnt ** Phase 1 - Check blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 821 files, 27150 used, 99689 free (985 frags, 12338 blocks, 0.8% fragmentation) Fixit#mount -u -o rw /dev/ad0s1 /mnt
Now for those symlinks:
Fixit#ln -f -s /mnt/etc/*pwd.db /etc
Fixit#ln -f -s /mnt/etc/group /etc
Note that you need to include the force (-f
) switch when you make your symbolic
(-s
) links. You need to overwrite
the existing link that links mnt2
,
or the fixit floppy, to /etc. You
instead want to link the files on your hard drive (/mnt) to /etc.
You’ll also notice that while in the Fixit#
prompt, the up arrow will recall
history, but tab completion does not work.
At that Fixit#
prompt, you
have two command sets available to you. The first is that limited
command set that comes with the sysinstall
utility. Note that these are the
only commands available at that holographic shell prompt:
Fixit# ls stand
-sh* gunzip* route*
[* gzip* rtsol*
arp* help/ sed*
boot_crunch* hostname* sh*
camcontrol* ifconfig* slattach*
cpio* minigzip* sysinstall*
dhclient* mount_nfs* test*
dhclient-script* newfs* tunefs*
etc/ ppp* usbd*
find* pwd* usbdevs*
fsck_ffs* rm* zcat*
The second command set is on the floppy itself, mounted as mnt2:
Fixit# ls mnt2/stand
bsdlabel* dd* fixit_crunch* mount_cd9660* sleep*
cat* df* ftp* mount_msdosfs* swapon*
chgrp* disklabel* kill* mv* sync*
chmod* dmesg* ln* reboot* tar*
chown* echo* ls* restore* telnet*
chroot* ex* mkdir* rm* umount*
clri* expr* mknod* rmdir* vi*
cp* fdisk* mount* rrestore* view*
You’ll also find a minimal set of notes in:
Fixit# ls stand/help
One of the first things you’ll notice, especially if you try to
read one of those help documents, is the lack of a pager. You won’t
have any luck with more
or less
. However, cat
and view
are available for viewing files. If
you’ve never used view
before,
remember to type :q
to quit the
viewer.
Also note that all of the restore utilities are on hand, unless
you’ve used pax
as your backup
utility.
Let’s pause here for a moment and compare the fixit floppy to the live filesystem. There’s one CD marked as live in a purchased set. If you burn your own ISO images, the second image for your release will contain the live filesystem. For example, here is the listing for ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/ISO-IMAGES/5.1/:
5.1-RELEASE-i386-disc1.iso 630048 KB 06/05/03 00:00:00 5.1-RELEASE-i386-disc2.iso 292448 KB 06/05/03 00:00:00 5.1-RELEASE-i386-miniinst.iso 243488 KB 06/05/03 00:00:00 CHECKSUM.MD5 1 KB 06/05/03 00:00:00
disc1.iso
is the install CD,
and disc2.iso
is the live
filesystem CD.
There are several advantages to using the live filesystem.
First, you don’t have to make any floppies. In fact, your entire kit
can be as simple as this one CD and your printouts specific to that
system. Second, the CD is bootable, so you can reach that Fixit#
prompt in under a minute.
Third, you have the entire built-in command set available to
you. When you enter the Fixit
screen, you’ll see the same welcome message as before. This time, it
is the CD that is mounted as /mnt2, which is really a link to /dist:
Fixit#ls -l /mnt2
lrwxr-xr-x 1 root wheel 5 Dec 8 08:22 /mnt2@ -> /dist Fixit#ls /dist
.cshrc boot/ etc/ root/ tmp/ .profile boot.catalog floppies/ rr_moved/ usr/ COPYRIGHT cdrom.inf mnt/ sbin/ var/ bin/ dev/ proc/ sys@
A quick ls /dist/bin
and
ls /dist/sbin
will display all of
the commands that come with a FreeBSD system. There isn’t a limited
command set with the live filesystem.
Now that I’ve shown you the various ways to enter the Fixit
facility, you’re probably wondering
what you should be doing at that prompt. FreeBSD is quite robust and
is usually capable of booting your hard drive to some sort of prompt.
However, if the disk fails completely or is somehow incapable of
booting to a prompt, the fixit
facility is one of your options.
From here, you can run fsck
on your various filesystems, which may fix the problem. You can see
which filesystems are still mountable, allowing you to assess the
extent of the damage. If some files were damaged, you can restore
those files from backup.
If it turns out that the drive is damaged beyond repair, you can rest easy in the fact that you have a printout of your hardware and partitioning scheme, a floppy containing any necessary drivers, and a backup of all of your data. Above all, you were prepared.
The Backup Basics section of the FreeBSD Handbook (http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/backup-basics.html)
You don’t have to be a programmer to use a debugger.
As an end user, you may not realize that you have the ability to analyze security exploits. After all, the organization that distributes your operating system of choice or the provider of a given application will deal with security issues and make updates available.
However, keep in mind that Security Officers apply the same tools and techniques that end users use for debugging programs. Knowing how to analyze a problem will help you to troubleshoot any misbehaving process in a Unix environment.
Analyzing a malfunctioning process starts with basic information, such as error messages and return values. Sometimes those aren’t enough, though. Some error messages are unclear. In the case of security vulnerabilities, there may not be an error code or return value, because the program may crash or misbehave silently.
The BSDs provide several tools to analyze a program’s execution.
You can monitor system calls with ktrace
and resources with fstat
. You can run a debugger such as GDB,
the GNU Debugger, and watch your operating system’s internal
operation.
In some cases, a program must run in a particular environment,
which may make it difficult to analyze due to the limitations of some
tools. For example, a telnetd
advisory from 2001 (http://www.cert.org/advisories/CA-2001-21.html)
affected most Unix operating systems. This particular vulnerability
came to light when a group called TESO released an example exploit for
it.
On Unix systems, telnetd
runs
as root, so that once the system authenticates the user, the process
has the privileges required to set the user ID of the login shell to
that of the user who logged in. This means that a remote entity who
can cause telnetd
to misbehave by
sending it carefully designed input could execute processes as root on
your system.
On most Unix systems, telnetd
does not run as a standalone daemon. Since logins are relatively
infrequent (on the system timescale compared to thousands of
interrupts per second), the inetd
service starts telnetd
as
needed.
This is a simple example of the data stream sufficient to crash
vulnerable telnetd
s using perl
and nc
(netcat):
% perl -e 'print "377366"x512' | nc testhost telnet
This was the example I used to diagnose the problem and test the fix. If you run this command against an impervious Telnet daemon, you’ll see the following output:
% perl -e 'print "377366"x512' | nc testhost telnet
[Yes]
[Yes]
[Yes]
The [Yes]
message will repeat
512 times because the characters you sent, 377366
, represent the Telnet protocol’s
“ARE YOU THERE” control message, and you asked the question 512
times.
If you run this command against a vulnerable telnetd
, the output can vary. In some cases,
your connection may close before you get 512 [Yes]
responses because telnetd
crashed. In other cases, you may
receive seemingly random output from portions of the telnetd
memory space. These both indicate
that the program did something it was not supposed to, due to the
specific input you gave it.
In order to fix the problem, we need to find out where the
executable did something incorrectly. We would like to run the program
under the control of GDB, but we cannot start telnetd
from the command line the way we
usually would when debugging most executables. Normally, GDB is
invoked in one of three ways.
First, to run a program and debug it, type:
%gdb
programname GNU gdb 5.3nb1 Copyright 2002 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386--netbsdelf"...(no debugging symbols found)... (gdb)run
If this is your first time using gdb
, type help
at the (gdb)
prompt. Type quit
when you are finished using the
debugger.
Second, to examine the core file of a program that has already crashed, use:
%gdb
programnameprogramname
.core
Third, to examine a program that is already running, type:
%gdb
programnameprocessid
In the case of telnetd
, we
cannot use the first method, because inetd
must start telnetd
in order to attach it to a network
socket and operate properly. We cannot use the second method, because
processes that run with root privileges do not leave core files, since
the program’s memory image could contain sensitive data.
That leaves the third method. Attaching to a running process is
problematic because telnetd
isn’t
running until someone connects. We’ll need to modify our attack
script:
% perl -e 'sleep 30; print "377366"x512' | nc testhost telnet
Now nc
opens a socket to the
testhost, inetd
spawns a telnetd
in response, and perl
waits for 30 seconds before sending the
attack string.
In another terminal, on the testhost, we say:
%ps -ax | grep telnetd
27857 ?? S 0:00.05 telnetd 27859 pd S+ 0:00.02 grep telnetd %gdb /usr/libexec/telnetd 27857
GNU gdb[...] Attaching to program `/usr/libexec/telnetd', process 27857
From here we can allow telnetd
to crash and observe the exact type
of error that caused the crash. If we’ve built telnetd
with debugging information, GDB will
even display the line of source code the program was executing when it
crashed. Now we can use our favorite debugging techniques and either
insert debugging messages or use GDB and set breakpoints and
watchpoints to discover at what point the program went off course. We
can then determine what changes to make to correct the error and
prevent the exploit.
If you’re not a programmer, you can save the information and send it to the developers.
We were fortunate in this example because we had details of the exploit. That made it easy to experiment and try different approaches. In many cases, however, you won’t know the details of an exploit, and you may only know that there is a problem because of error messages in your logs.
You can use tcpdump
to
capture the traffic on the relevant port. Once you can correlate the
timestamp of the log’s error message with some of your tcpdump
traffic, you can take the data sent
in an attack and create a Perl script to resend it. You can then apply
the techniques already described to analyze and correct the
problem.
man ktrace
man fstat
man gdb
The Netcat web site; see the Read Me file (http://netcat.sourceforge.net)
The “Debugging with GDB” tutorial (http://www.delorie.com/gnu/docs/gdb/gdb_toc.html)
Automate log processing on a web farm.
As the administrator of multiple web servers, I ran across a
few logging problems. The first was the need to collect logs from
multiple web servers and move them to one place for processing. The
second was the need to do a real-time tail
on multiple logs so I could watch for
specific patterns, clients, and URLs.
As a result, I wrote a series of Perl scripts collectively known
as logproc
. These
scripts send the log line information to a single log host where some other log analysis tool can work on
them, solving the first problem. They also multicast the log data,
letting you watch live log information from multiple web servers without
having to watch individual log files on each host. A primary goal is
never to lose log information, so these scripts are very careful about
checking exit codes and such.
The basic model is to feed logs to a program via a pipe. Apache supports this with its standard logging mechanism, and it is the only web server considered in this hack. It should be possible to make the system work with other web servers—even servers that can only write logs to a file—by using a named pipe.
I’ve used these scripts on production sites at a few different companies, and I’ve found that they handle high loads quite well.
Download logproc
from http://www.peterson.ath.cx/~jlp/software/logproc.tar.gz.
Then, extract it:
%gunzip logproc.tar.gz
%tar xvf logproc.tar
%ls -F logproc
./ ../ logserver.bin/ webserver.bin/ %ls -F logserver.bin
./ apache_rrd* cleantmp* logwatch* mining/ ../ arclogs* collect* meter* %ls -F webserver.bin
./ ../ batcher* cleantmp* copier*
As you can see, there are two parts. One runs on each web server and the other runs on the log server.
The logs are fed to a process called batcher
that runs on the web server and writes the log lines to
a batch file as they are received. The batch file stays small,
containing only five minutes’ worth of logs. Each completed batch file
moves off to a holding area. A second script on each web server, the
copier
, takes the completed batch files and copies them to the
centralized log host. It typically runs from cron
. On the log host, the collect
process, also run from cron
, collects the batches and sorts the log
lines into the appropriate daily log files.
The system can also monitor log information in real time. Each
batcher
process dumps the log lines
as it receives them out to a multicast group. Listener processes can
retrieve those log lines and provide real-time analysis or monitoring.
See the sample logwatch
script
included with logproc
for
details.
First, create a home directory for the web server user. In this
case, we’ll call the user www
. Make
sure that www
’s home directory in
/etc/master.passwd points to that
same location, not to /nonexistent. If necessary, use vipw
to modify the location in the password
file.
#mkdir ~www
#chown www:www ~www
Next, log in as the web server user and create a public/private SSH keypair:
#su www
%ssh-keygen -t dsa
Create the directories used by the log processing tools, and copy the scripts over:
%cd ~www
%mkdir -p bin logs/{work,save}/0 logs/tmp logs/work/1
%cp $srcdir/logproc/webserver.bin/* bin/
Examine those scripts, and edit the variables listed in Table 7-1 to reflect your situation.
Script | Variable | Value |
| | The name of the web server user |
| The name of the interface that can reach the log host | |
| The home directory of the web server user | |
| | The home directory of the web server user |
copier | | The name of the host where the logs will collect |
| The home directory of the web server user | |
| The directory on the collector host where the logs will be collected | |
$loghost_loguser | The user on the log host who owns the logs | |
$scp_prog | The full path to the | |
$ssh_prog | The full path to |
Then, make sure you have satisfied all of the dependencies for these programs:
# perl -wc batcher; perl -wc cleantmp; perl -wc copier
The only dependency you likely won’t have is IO::Socket::Multicast
. Install it via the
/usr/ports/net/p5-IO-Socket-Multicast port
on FreeBSD systems or from the CPAN site (http://www.cpan.org/).
Next, configure httpd.conf
to log to the batcher
in parallel
with normal logging. Note that the batcher
command line must include the
instance (site
, virtual
, secure
) and type (access
, error
, ssl
) of logging:
LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" "%{Cookie}i" %v" full CustomLog "|/home/www/bin/batcher site access" full ErrorLog "|/home/www/bin/batcher site error"
You can adjust the LogFormat
directive as necessary to log the information you or your log
summarization software needs.
Finally, restart Apache and verify that the batchers are creating batches:
#apachectl configtest
#apachectl graceful
#cd $wwwhome/logs/
#ls tmp
Should list error log files for each batcher instance #ls work/0
Should list the working batches for each batcher instance #ls save/0
Verify that batches have moved into the save directory after a five-minute batch interval #ls work/0
and that new batches are currently being created
Start by creating a log
user to receive the logs, complete with a home directory. Become the
log
user and copy the public key
from the web server into ~log/.ssh/authorized_keys2. Then, as the
log
user, create the directories
the log collection tools use:
#su log
%cd ~log
%mkdir -p bin web/{work,save}/{0,1} web/tmp web/{current,archive}
From a web server (as the web server’s user), ssh
to the log host manually to verify the
configuration of the authorized_keys2
:
#su www
%ssh loghost -l loguser date
Then, run copier
manually to
verify that the log files actually make it to the log server. Watch
your run output on the web server, then check that save/0 on the log server contains the newly
copied logs.
Once you’re satisfied with these manual tests, schedule a
cron
job that copies and cleans up
log files. These jobs should run as the web server user:
# crontab -e -u www
----------------------------- cut here -----------------------------
# copy the log files down to the collector host every 15 minutes
0,15,30,45 * * * * /home/www/bin/copier
# clean the tmp directory once an hour
0 * * * * /home/www/bin/cleantmp
----------------------------- cut here -----------------------------
Finally, wait until the next copier
run and verify that the batches
appear on the log host.
You should now have several batches sitting in save/0 in the log tree. Each batch contains
the log lines collected over the batch interval (by default, five
minutes) and has a filename indicating the instance (site
, virtual
, secure
), type (access
, error
, ssl
), web server host, timestamp indicating
when the batch was originally created, and PID of the batcher
process that created each
batch.
Now, install the log processing scripts into bin/:
# cp $srcdir/collector/{arclogs,cleantmp,collect} bin/
Edit them to have valid paths for their new location and any OS dependencies, as shown in Table 7-2.
Again, make sure all dependencies are satisfied:
# perl -wc arclogs; perl -wc cleantmp; perl -wc collect
If you don’t have Time::ParseDate
, then install it from the
/usr/ports/devel/p5-Time-modules
port on FreeBSD or from CPAN.
Run collect
manually as the
log user to verify that the log batches get collected and that log
data ends up in the appropriately dated log file. Once you’re
satisfied, automate these tasks in a cron
job for the log user:
# crontab -e -u log
----------------------------- cut here -----------------------------
# run the collector once an hour
0 * * * * /home/log/bin/collect
# clean the tmp directory once an hour
0 * * * * /home/log/bin/cleantmp
----------------------------- cut here -----------------------------
Wait until the next collect
run and verify that the batches are properly collected.
Compare the collected log files with the contents of your old logging mechanism’s log file on the web servers. Make sure every hit makes it into the collected log files for the day. You might want to run both logging mechanisms for several days to get a good feel that the system is working as expected.
The log server programs provide additional tools for
monitoring and summarizing live log data. On a traditional single web
server environment, you can always tail
the log file to see what’s going on.
This is no longer easy to do, because the logs are now written in
small batches. (Of course, if you have multiple web servers, multiple
tail
processes would have to run on
each web server.)
The
batcher
process helps with this by
multicasting the logs out to a multicast group. Use the logwatch
tool on the log server to view the
live log data:
%cd ~log/bin
%./logwatch
<lines of log data spew out here>
On a high-volume web site, there is likely to be too much data
to scan manually. logwatch
accepts
arguments to specify which type of log data you want to see. You can
also specify a Perl regular expression to limit the output.
The meter
script watches the
log data on the multicast stream, in real time, and summarizes some
information about the log data. It also stores information in an
RRDTool (http://www.rrdtool.org/)
database.
The mining directory
contains a checklog
script that
produces a “top ten clients” and “top ten vhosts” report.
Alternatively, you can feed the collected log files to your existing
web server log processing tools.
The logproc
web site (http://www.peterson.ath.cx/~jlp/software/logproc.tar.gz)
Use an expect script to help users generate GPG keys.
There are occasions when you can take advantage of Unix’s flexibility to control some other tool or system that is less flexible. I’ve used Unix scripts to update databases on user-unfriendly mainframe systems when the alternative was an expensive mainframe-programming service contract. You can use the same approach in reverse to let the user interact with a tool, but with a constrained set of choices.
The Expect scripting language is ideal for creating such
interactive scripts. It is available from NetBSD pkgsrc
as pkgsrc/lang/tcl-expect or pkgsrc/lang/tk-expect, as well as from the
FreeBSD ports and OpenBSD packages collections. We’ll use the
command-line version for this example, but keep in mind that expect-tk
allows you to provide a GUI frontend
to a command-line process if you’re willing to write a more complex
script.
In this case, we’ll script the generation of a GPG key. Install GPG from either pkgsrc/security/gnupg or the appropriate port or package.
During the process of generating a GPG key, the program asks the user several questions. We may wish to impose constraints so that a set of users ends up with keys with similar parameters. We could train the users, but that would not guarantee correct results. Scripting the generation makes the process easier and eliminates errors.
First, let’s look at a typical key generation session:
%gpg --gen-key
gpg (GnuPG) 1.2.4; Copyright (C) 2003 Free Software Foundation, Inc. This program comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the file COPYING for details. Please select what kind of key you want: (1) DSA and ElGamal (default) (2) DSA (sign only) (4) RSA (sign only) Your selection?4
What keysize do you want? (1024)2048
Requested keysize is 2048 bits Please specify how long the key should be valid. 0 = key does not expire <n> = key expires in n days <n>w = key expires in n weeks <n>m = key expires in n months <n>y = key expires in n years Key is valid for? (0)0
Key does not expire at all Is this correct (y/n)?y
You need a User-ID to identify your key; the software constructs the user id from Real Name, Comment and Email Address in this form: "Heinrich Heine (Der Dichter) <[email protected]>" Real name:
Let’s pause there to consider the elements we can constrain.
You probably want to specify the cryptographic algorithm and key length for all users consistently, based on your security and interoperability requirements. I’ll choose RSA signing and encryption keys, but GPG doesn’t provide a menu option for that. I’ll have to create the signing key first and then add the encryption subkey.
Here’s an expect
script that
would duplicate the session shown so far:
#!/usr/pkg/bin/expect -f set timeout -1 spawn gpg --gen-key match_max 100000 expect "(4) RSA (sign only)" expect "Your selection? " send "4 " expect "What keysize do you want? (1024) " send "2048 " expect "Key is valid for? (0) " send -- "0 " expect "Key does not expire at all" expect "Is this correct (y/n)? " send -- "y " expect "Real name: "
The script begins by set
ting
timeout
to infinite, or -1
, so expect
will wait forever to match the
provided input. Then we spawn
the
process that we’re going to control, gpg
--gen-key
. match_max
sets
some buffer size constraints in bytes, and the given value is far more
than we will need.
After the initial settings, the script simply consists of
strings that we expect
from the
program and strings that we send
in
reply. This means that the script will answer all of the questions GPG
asks until Real
name
: , without waiting for the user’s
input.
Note that in several places we expect things besides the prompt.
For example, before responding to the Your
selection?
prompt, we verify that the version of GPG we have
executed still has the same meaning for the fourth option, by
expecting that the text of that menu choice is still RSA (sign
only)
. If this were a real, production-ready
script, we should print a warning message and terminate the script if
the value does not match our expectations, and perhaps include a check
of the GPG version number. In this simple example, the script will
hang, and you must break out of it with Ctrl-c.
There are several ways of handling the fields we do want the
user to provide. For the greatest degree of control over the user
experience, we could use individual expect
commands, but here we will take a
simpler approach. Here’s some more of the script:
interact " " return send " " expect "Email address: " interact " " return send " " expect "Comment: " interact " " return send " " expect "Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? " interact " " return send " " expect "Enter passphrase: " interact " " return send " " expect "Repeat passphrase: " interact " " return send " "
The interact
command allows
the user to interact directly with the spawned program. We place a
constraint that the user’s interaction ends as soon as the user
presses the Enter key, which sends the carriage return character,
. At that point, the interact
command returns and the script
resumes. Note that we have to send
the
from the script; expect
intercepted the carriage return and
GPG did not see it.
Again, a correct script would have a more complex flow of
execution and allow for cases where the spawned program rejects the
user’s input with an error message. For example, the Real Name
field must be more than five
characters long. If a user types less than five characters, GPG will
prompt him to retype his username. However, the expect
script just shown will not accept the
new user input, because it is now waiting for the Email address
: prompt.
Alternatively, we could replace these three lines:
interact " " return send " " expect "Email address: "
with:
interact -o "Email address: " return send_user "Email address: "
Instead of stopping interaction when the user presses return
, we stop interaction when the program
outputs the Email address
: prompt.
That’s the difference between interact
and interact -o
; the former stops interaction
based on input from the user, and the latter on output from the
program. This time, we don’t need to send the carriage return, because
the user’s keypress is passed through to GPG. However, we do need to
echo the prompt, because expect
has
consumed it. This method lets GPG handle the error conditions for
us:
Real name: abc Name must be at least 5 characters long Real name: abcde Email address:
After GPG receives the information it needs to generate the key,
it might not be able to find enough high-quality random data from the
system. The script ought to handle that by spawning a process to
generate more system activity, such as performing a lot of disk
activity by running a find
across
the entire disk.
After generating the signing key, the script could spawn a new
instance of GPG with the --edit-key
option, to generate the desired RSA encryption key.
Although the final script may end up executing three processes,
the whole process is seamless to the user. You can hide even more of
the guts by using expect
’s log_user
setting to hide the output of the
programs at points where the user does not need to see them.
You can use a script like this in conjunction with any Unix
command-line program. By combining expect
with telnet
or ssh
, you can control non-Unix systems,
thereby leveraging the flexibility of Unix into a non-Unix domain.
This even works with programs for which you do not have source code,
such as control utilities for commercial databases or application
software.
In the case of GPG, we do have source code, so we
could modify the program, but writing an expect
script is easier. A carefully
designed expect
script may not
require changes when a new version of GPG is released. Source code
changes to GPG would require integration with any new version of
GPG.
man expect
The expect
web site,
which includes sample scripts (http://expect.nist.gov/)
Exploring Expect , by Don Libes, the author of expect
(http://www.oreilly.com/catalog/expect/)
I frequently represent NetBSD at trade shows. It’s challenging to attract attention because there are many booths at a show—people will walk by quickly unless something catches their eye. You also need to balance eye-candy with functionality so that you can attract and keep a visitor’s attention. I needed an enticing demo to run on one of the computers in the booth.
I wanted to show off several applications, such as office productivity tools, video, and games, and have music playing, but there’s only so much screen real estate. Cramming all of those things on the screen at once would clutter the screen, and the point would be lost.
Most X window managers have some concept of virtual desktops, separate work spaces that you can flip between. For example, Enlightenment (pkgsrc/wm/enlightenment) not only has the concept of virtual desktops, but as an added bonus for the trade show environment offers a nice sliding effect as you transition from one desktop to the next.
Normally in Enlightenment, to switch from one virtual desktop to
the next, you move the mouse pointer to the edge of the screen and
then push past it, or you use a key sequence to move to an adjacent
desktop. For an unattended demo, we need to automate this process.
Enlightenment provides an undocumented utility called eesh
that can control most aspects of the
Enlightenment window manager. You can write scripts to move windows,
resize them, or flip between desktops.
Note that eesh
isn’t a
friendly utility; it doesn’t even produce a prompt when you run it.
Type help
for the menu or exit
to quit:
%eesh
help
Enlightenment IPC Commands Help commands currently available: use "help all" for descriptions of each command use "help <command>" for an individual description actionclass active_network advanced_focus sfa autosave background border button button_show colormod configpanel copyright current_theme tc cursor default_theme dialog_ok dok dock dump_mem_debug exit q focus_mode sf fx general_info geominfo_mode sgm goto_area sa goto_desktop sd group gc group_info gl group_op gop help ? imageclass internal_list il list_class cl list_remember list_themes tl module move_mode smm nop
Unfortunately, the eesh
utility seems to be untested. It sometimes behaves inconsistently by
not accepting commands until you enter them a second time or by
withholding output until you press Enter again. As an example, there
are actually more commands than those indicated in the help
listing. Look in the Enlightenment
source’s ipc.c file for a
complete list.
We’ll start our script by making sure that Enlightenment is
configured the way we want for our demo. We want six work spaces (3 by
2) to display our programs. Within eesh
, try the following commands:
num_areas ?
Number of Areas: 2 2help num_areas
Enlightenment IPC Commands Help : num_areas (sna) -------------------------------- Change the size of the virtual desktop Use "num_areas <width> <height>" to change the size of the virtual desktop. Example: "num_areas 2 2" makes 2x2 virtual destkops Use "num_areas ?" to retrieve the current settingnum_areas 3 2
Now we have the number of areas we want. areas
is the Enlightenment name for virtual
desktops, since Enlightenment also supports multiple desktops, but
that’s different. Now we’d like our screen to display the first area,
so that the programs our script runs will open there:
goto_area 0 0
If your terminal wasn’t on the first area, it just moved off the screen. Use the mouse to return to that area.
eesh
also lets us write
commands on the command line with the -e
(execute command) flag:
% eesh -e "goto_area 0 0"
Now we know enough to write a simple demo script:
#!/bin/sh eesh -e "num_desks 1" eesh -e "num_areas 3 2" sleep 1 eesh -e "goto_area 0 0" # Configure the default gqmpeg playlist to play your desired music gqmpeg # Show an interesting avi file. xanim -geometry +50x+10 netbsd3.avi & # Give the programs time to start, to make sure they # open on the correct area. # Also, lets people watching see what started up. sleep 3 eesh -e "goto_area 1 0" # Word Processing abiword sampledoc.abw & sleep 2 eesh -e "goto_area 2 0" # Spreadsheet gnumeric samplesheet.gnumeric & sleep 2 eesh -e "goto_area 0 1" # A lively game battleball & sleep 2 eesh -e "goto_area 1 1" # Web Browsing (of a local hierarchy, in case you don't have net # connectivity at a trade show) firebird file://index.html & sleep 3 eesh -e "goto_area 2 1" sleep 1 # Insert your favorite application here # Leave screen back at page 1. eesh -e "goto_area 0 0"
When you run the script, the screen will slide around to the various areas and pause a few seconds between program launches. We have most of the things we wanted: music, video, and applications. The next step is to keep it moving. Try the following script:
#!/bin/sh while [ 1 ] do eesh -e "goto_area 0 0" sleep 2 eesh -e "goto_area 1 0" sleep 2 eesh -e "goto_area 2 0" sleep 2 eesh -e "goto_area 0 1" sleep 2 eesh -e "goto_area 1 1" sleep 2 eesh -e "goto_area 2 1" sleep 2 done
To stop the moving display, you have to get your keyboard focus
into the xterm
where the script is
running so that you can press Ctrl-c. That can be difficult, but we’ll
address it shortly.
For a complex demonstration, you can have different sets of these scripts that visit different sets of areas. You can also change the delay so that complex areas display for a longer period. I also made a script that clears all of the viewing areas. That way, when visitors to the booth play around with the machine, I can easily reset to a clean state and then start the demo again.
Since many of the utilities you’ll demonstrate don’t create
.pid files, I find it easiest to
use pkill
, the “kill process by
name” utility. (FreeBSD provides killall
.)
I’ll also leave you with two example scripts that show how to extract information about Enlightenment’s current settings for use in a more complex script.
The first script is retitle
:
#!/bin/sh WIN=`eesh -ewait "set_focus ?" | sed 's/^focused: //' ` xterm -geometry 47x7+227+419 -fn -*-courier-*-o-*-*-34-*-*-*-*-*-*-* -e /home/david/bin/retitle2 $WIN
The second is retitle2
:
#!/bin/sh WIN=$1 echo "enter new title:" read TITLE eesh -e "win_op $WIN title $TITLE"
With these scripts and e16keyedit
, you can bind a key combination to change the title of
any window. This makes it much easier to keep track of xterm
s, if you prefer task-oriented
titles.
Now back to the control issue. When I first wrote this demo, I used a switch wired to a serial port to start and stop the demo so that keyboard focus did not matter. However, wiring switches is more work than configuring software, so I found a better way.
The e16keyedit
utility,
written by Geoff “Mandrake” Harrison and Carsten “Raster” Haitzler (the primary developers of Enlightenment),
allows you to bind function keys and Meta keys to run programs or
perform the same functions that you can with eesh
. Using e16keyedit
, you can define function keys to
set up the demo, clean up the demo, and start and stop the area
rotations. Since the function keys can be bound to work anywhere
within Enlightenment, keyboard focus no longer matters. You’re ready
to give a fantastic demo!
e16keyedit
is not part of the
main Enlightenment distribution. Download it from SourceForge (http://sourceforge.net/project/showfiles.php?group_id=2).
The Enlightenment web site (http://www.enlightenment.org/)