IN THIS CHAPTER
“And memories, he knew, were not glass treasures to be kept locked within a box. They were bright ribbons to be hung in the wind.” | ||
| --The Talismans of Shannara—Terry Brooks | ||
When a user requests a directory without specifying a particular filename in that directory, there are a few different things that Apache can do to serve that request. For example, if a user types in a URL that looks like http://www.example.com/directory/, Apache will have to somehow figure out which file from the directory directory is to be sent to the user.
Apache can respond by sending the default document, defined by the DirectoryIndex directive. If certain options are enabled, and if there is no default document, it can generate a directory index. Or it can just return an error message, and not give the user anything at all.
The DirectoryIndex directive tells Apache what file to serve when a directory is requested but no particular file in that directory is specified. The default value of this directive is index.html, so when you request a URL ending in a slash (/), or one that happens to specify a directory and not a file, Apache will attempt to give you the file index.html out of that directory.
For example, we'll consider a site configured with a DocumentRoot of /usr/local/apache/htdocs, with a hostname of boxofclue.com. If a client requests the URL http://www.boxofclue.com/, Apache will attempt to serve the file index.html out of the directory /usr/local/apache/htdocs.
If there is no such file in that directory (assuming that there have been no additional configuration changes), Apache will return a 404 error—file not found.
You can, however, set DirectoryIndex to any filename at all if you want to call your index file something else, or if you want to have somewhat different behavior for a particular directory. And you can set DirectoryIndex to several things at once, in order to give Apache several choices about what files it serves.
Example:
DirectoryIndex index.html index.htm default.html default.htm index.cgi
This example kills a variety of birds with one stone. If you have customers who are used to providing content for Microsoft IIS servers, they may be used to calling their default document default.htm, since that is the default document on an IIS server. By providing .htm and .html alternatives for each of the most common default filenames, you will catch most of your users and enable the server to do what they expect.
The last entry in the list, index.cgi, demonstrates that the default document does not necessarily need to be an HTML document, but can be a CGI program. If you have CGI execution turned on for the given directory, Apache will execute the CGI program and provide the output as the default document.
Finally, it is not even necessary that the default document be in the specified directory. By providing an absolute (but local) URL, rather than a relative one, you can specify any content on your server as the default content.
DirectoryIndex index.html /errors/notfound.html
Note that this same behavior could also be supplied with an ErrorDocument directive.
ErrorDocument 404 /errors/notfound.html
Things become somewhat more interesting if you allow Apache to generate content for you. This can be done at a variety of levels of complexity, from a simple bulleted list of filenames to a highly customized directory listing.
The automatic index generation is handled by a module called mod_autoindex. Most of the rest of this chapter discusses directives that are provided by that module. mod_autoindex is part of the core set of Apache modules, and is compiled into Apache by default.
Note
In order to use any of the directives and options discussed here, you must enable Options +Indexes.
The simplest level of auto-indexing is achieved by turning on the Indexes option:
Options +Indexes
For any directory where this option is turned on, if the default file is not found in that directory, Apache will generate a file listing. By default, this is a very simple bulleted list of filenames, with each filename linked to the appropriate file so that clicking it downloads the file. Any additional frills are provided using the IndexOptions directive.
For anything more than a simple bulleted list of filenames, you need to use the IndexOptions directive, which customizes the way that the directory listing is generated.
The IndexOptions directive, like the Options directive, enables you to turn several options on or off by preceding them with a + or - sign.
IndexOptions +option -option
Specifying a list of options without the + and - signs turns on only the specified options.
IndexOptions +FancyIndexing
The very first option that you will want to turn on is FancyIndexing. In fact, you will almost certainly discover that it is turned on already in the default configuration file that came with Apache when you installed it. So when you tried out the previous example, adding an Options +Indexes to your server configuration,[1] you probably saw not a bulleted list of filenames, but something that looked much more like a directory listing in a GUI file manager or on an FTP site viewed in a browser (see Figure 11.1).
This is the view that you are probably most accustomed to seeing on Web sites when there are file listings.
For each common file type, the filename will have a small (20×22 pixels) icon next to it, which will indicate that file type. The particular icon that is displayed is configured by the various AddIcon directives, and can be replaced by any icon or other image of your choosing.
There are three AddIcon directives—AddIcon, AddIconByEncoding, and AddIconByType—that you can use to associate a particular icon either with specific files or with types of files. Additionally, the DefaultIcon directive determines what icon will be displayed if none of the AddIcon directives matches the file.
IndexOptions DescriptionWidth=30
or
IndexOptions DescriptionWidth=*
The description column is, by default, 23 characters wide. If your description is longer than that, it will be truncated. You can, with this option, make that column larger or smaller by specifying a column width in characters. Using an argument value of * will cause the column to be the width of the longest description that it needs to contain.
See the “AddDescription” section for more information on file descriptions.
AddIcon is the first, and most specific, of the three directives. You will see several examples of the AddIcon directive in your default Apache configuration file, where the mapping of files to icons is set up. There are no default values for these mappings, and they must be set up in the configuration files in order for any images to be displayed in the directory listing.
The syntax of the AddIcon directive is as follows:
AddIcon icon_url name [name]
The icon_url is the local URL for an image file. The name is a particular filename, some portion of a filename (such as the file extension), or a wildcard that could match a filename.
Files that match something in the argument list will be displayed in directory listings with that icon next to them.
Apache comes with about 70 icons, which are located in the icons directory, which is usually located in your ServerRoot directory at the same level as the htdocs directory. These icons, or any that you supply yourself, can be used for the icon URL.
Example:
AddIcon /icons/movie.gif .mov .avi .mpg
AddIcon optionally takes a somewhat different argument syntax if you want to provide alternate text for the image, in the event that the client has image loading turned off or is using a text-only browser. Figure 11.2 shows a directory listing customized by the addition of on icon associated with .tex files, which is accomplished with the following directive:
AddIcon (LaTeX,/icons/tex.gif) .tex
You can also use the AddAlt directive to add alternate text.
AddIconByType is slightly less specific concerning the files you are adding icons for, and so should be used to pick up the files for which you don't have specific AddType directives.
As the name suggests, this directive specifies the icon to be displayed for files of a particular MIME type. Like AddIcon, this directive is only effectual when the FancyIndexing option is turned on in IndexOptions.
Example:
AddIconByType /icons/image3.gif image/*
As you can see in this example, wildcards can be used in the type argument, in order to match larger classes of content type.
This final AddIcon directive associates a particular icon with an encoding type. For example, you can display the “compressed” icon for all files that have an encoding type of x-compressed:
AddIconByEncoding (Compressed,/icons/compressed.gif) x-compress
In this example, I am again using the (ALT,/image/url) form of the arguments to provide alternate text for the image in the event that the client does not load this image. This is an important part of keeping your Web site accessible to people using alternate technology. Blind people using screen readers, people reading the site on a handheld computer, and people who for whatever reason prefer to use a text-based browser will thank you for using this argument style and making their lives easier.
When all else fails, auto-indexed directories will show the DefaultIcon next to images that don't match any of the AddIcon directives. In the default Apache configuration file, this is /icons/unknown.gif, which is a question mark. If you intend to use auto-generated directory listings as a major interface on your site, you should attempt to provide icons for as many of your common file types as you can so that people can very quickly ascertain what file types they are looking at.
IndexOptions +FoldersFirst
If you are used to using a graphical interface to your file system, such as the Windows file explorer, you may initially find the layout of the default directory indexing a little confusing, as the directories are mixed in with the files, in alphabetical order. In most graphical directory and file management tools, the directories appear at the top, and are followed by the actual files below.
The FoldersFirst option reorders the auto-index in this manner, as shown in Figure 11.3.
IndexOptions +HTMLTable
This option formats the index listing as an HTML table, rather than as a preformatted list.
IndexOptions +IconHeight=20 +IconWidth=22
The IconHeight and IconWidth directives determine the size of the icons displayed in the directory listing. They are, by default, 20×22 pixels, as this is the size of the images that are shipped with Apache. However, it will use whatever images you ask it to use, and they will be displayed at whatever their actual size is. By setting these directives, you can force all the icons to be displayed at the same size, for more uniformity in the listing. Additionally, using these directives causes the page to appear to load faster, since each image tag will include the height and width attributes, which permits the browser to precalculate the page layout and display the text items on the page while the images are still loading.
IconHeight and IconWidth should be used together—that is, you should not use one without the other.
IndexOptions +IconsAreLinks
By default, while the text name of the file is a link to download that file, the icon is not. Because people seem to intuitively click the icon rather than the name, this option makes the icon a link also.
IndexOptions +IgnoreClient
This option causes mod_autoindex to ignore all query variables from the client. This disables the ability to re-sort the file listing by columns, and so implies SuppressColumnSorting.
IndexOptions +NameWidth=*
Filenames will be truncated, as shown in Figure 11.4, if they exceed 20 characters in total filename length.
The NameWidth option lets you set the maximum length, or, by using an asterisk (*) rather than a number as the argument, you can have Apache automatically use the length of the longest filename in the directory as the maximum.
IndexOptions +ScanHTMLTitles
This option is not a particularly nice thing to do to your server, since it is extremely CPU- and disk-intensive. When a directory listing is generated, each HTML file will be opened, and whatever appears in the <TITLE> tag will be used as the file description. This is handy for a number of reasons during Web site development. It gives you a quick way to figure out what the various filenames mean, and to see whether you have put sensible <TITLE> tags in your files. However, once your site is live, leaving this option on will cause a significant performance hit, resulting in slow load times for this directory, and will also do unfriendly things to your server's hard drives. Remember that the HTML files are opened and read each time the directory is requested, as there is no caching mechanism built into this process.
IndexOptions +SuppressColumnSorting
The names of the columns in the directory listing (Name, Last Modified, Size, and Description) are all links. Clicking a link re-sorts the view by that attribute.
The SuppressColumnSorting option turns off this behavior, and you are stuck with the default ordering. This is a good thing to do for performance reasons. Users like to tinker. Once they discover that they can reorder the directory listing, some people will sit for several minutes reordering the listing by the various options. This causes a drain on the server, because, as you might recall, it is not caching results, and so has to read and sort the directory listing each time, causing a large amount of disk I/O for no particularly good reason.
IndexOptions +SuppressDescription
This is a great option to set in order to recover some screen space. Most of the time, the file descriptions are blank, because most people are not aware that they can add a description for a file.[2] So you have a blank column taking up space over on the right side of the screen. By setting the SuppressDescription option, the description column is dropped, and the rest of the columns expand into the available space.
See the section “AddDescription” for information about adding descriptions to files.
IndexOptions +SuppressHTMLPreamble
This option tells mod_autoindex not to generate the standard HTML header when displaying the file listing. You should use this directive if you are also using the HeaderName directive (See the “Headers and Footers” section) to place a customized HTML header on the index.
IndexOptions +SuppressIcon
SuppressIcon entirely turns off the displaying of icons in the directory listing. This results in a more lightweight, faster-loading directory index.
IndexOptions SuppressLastModified
By default, the Last Modified column takes up 19 characters horizontally. The SuppressLastModified option prevents this column from being displayed at all, and these 19 characters are given to the Description column. Consider whether the date and time each file was last modified is a useful piece of information to your audience. If you are distributing software, this may be crucial information. If, on the other hand, you have a directory of MP3 files for download, this information is probably fairly unimportant.
IndexOptions +SuppressRules
At the top and bottom of the auto-generated listing are horizontal rules, created with the HTML <hr> tag. Use of this option removes those rules. Technically, the HTML 3.2 specification does not permit either images or horizontal rules to appear in a preformatted block created with the HTML <pre> tag, which this directory listing is. Therefore, use of this option, and the SuppressIcon option, make this generated document HTML 3.2 compliant.
IndexOptions +SuppressSize
As with the other Suppress options, this option causes one of the columns to be dropped from the index listing. The column containing the file size is not displayed when this option is in effect. However, the size of a file is usually a very important piece of information when you are providing files for download.
IndexOptions +TrackModified
I've mentioned more than once in this chapter that the directory index is generated each time a user requests the directory, and that this information is not cached for the next client. The TrackModified option goes partway to solving this problem.
If your operating system provides accurate information about the status of files (the stat command), mod_autoindex will include a Last-Modified HTTP header with the response so that a HEAD request can determine whether it actually needs to retrieve the document. This will also tell caching proxy servers that they can just return the copy of the document they already have, rather than forcing the server to regenerate the listing if none of the files in the directory have been modified since the last time the index was requested, or if no files have been added or removed.
Note that many operating systems don't return the necessary status information, and on some platforms, the Last-Modified header will not be changed if files change size or time stamp.
IndexOptions +VersionSort
Although filenames are usually sorted alphabetically,[3] this sorting scheme will not give the desired results when you have file version numbers as part of the filenames. For example, a file called Date-ISO-1.14.tar.gz will be listed before a file called Date-ISO-1.3.tar.gz, even though it was an earlier version of the file.
The VersionSort option does the right thing with regard to versioned files, listing the files in version order rather than alphabetical order.
By default, the Description column will be empty, since there are no default file descriptions. This is not to say that the description feature is somehow turned off, just that there are no file descriptions set.
The AddDescription directive allows you to set the description for a particular file, or for a set of files that match a wildcard pattern.
AddDescription "Sarah at the lake" sarah_lake.jpg AddDescription "gzipped tar file" *.tar.gz
This directive can be used in any context, but only has an effect if FancyIndexing is turned on. If you are not using this in a directory-specific context (that is, a <Directory> section or a .htaccess file), make sure you provide a full path to the file that you're talking about.
AddDescription "Mayan calendar" /home/www/images/maya.gif
A fairly small amount of space is available for your file description string. The documentation says the following:
The typical, default description field is 23 bytes wide. 6 more bytes are added by the
IndexOptions SuppressIconoption, 7 bytes are added by theIndexOptionsSuppressSizeoption, and 19 bytes are added by theIndexOptions SuppressLastModifiedoption. Therefore, the widest default the description column is ever assigned is 55 bytes.
Judicious use, therefore, of the various Suppress commands will gain you a decent amount of space for your file descriptions. However, you may not want to sacrifice these other fields just so that you can say more about your files.
The DescriptionWidth option of IndexOptions (discussed earlier in the chapter) enables you to further influence the size of this column.
It is also important to know that the file description can contain HTML markup. If you've permitted AllowOverride Indexes so that users can add their own file descriptions in generated index listings (among other index-related things), you may find that they are using this feature and getting strange results. If a description string containing HTML markup is truncated (because it is longer than the allotted space in the column) before an HTML tag is closed, this tag can influence the rest of the generated document.
Say, for example, you have the following:
AddDescription "<b>Long, bolded description goes here</b>" file.name
If the description were truncated after description, before the HTML bold tag was closed, the rest of the HTML document from that point on would be rendered in a bold font.
HeaderName /header.html ReadmeName /footer.html
The HeaderName and ReadmeName directives give you the ability to add something to the top and bottom (respectively) of the auto-generated HTML page. Since these files can be HTML, you can make your auto-generated page fit in with the rest of your site look by simply inserting the list of files into your regular site template. And if you have Options Includes on for this directory, these files can even contain SSI directives (see Chapter 16, “Server-Side Includes”).
If the file referenced by your HeaderName directive does in fact contain the start of an HTML page (<HTML>, <HEAD>, and so on) you will probably also want to enable IndexOptions SuppressHTMLPreamble so that these tags are not repeated in the generated HTML document. Also, you should make sure that there is a complementary ReadmeName file to close those HTML tags that you opened in the header.
Note also that the values for the HeaderName and ReadmeName directives are not necessarily just filenames in the current directory, but can be URLs, so you can use a single header and footer file for your entire site and merely load them into the various places where you need them.
This URL can even refer to a CGI program, which can produce the header and/or footer dynamically. However, these directives require that the file have a major content type of text/* (something like text/html or text/plain, for example). So to meet this requirement, one only has to add a content type of text/html to the CGI program in question:
AddType text/html .cgi
See Chapter 8, “MIME and File Types,” for more information on this directive.
There are certain files that you never want showing up in directory listings. Things that come to mind are .htaccess files, .tmp files (or whatever you use to designate a temporary file), .swp files (something vi leaves lying around when you edit files), and a variety of other files that the end user never needs to know about.
The IndexIgnore directive lets you list those files that you don't want showing up. By default, the list contains the current directory (.), but you will need to add everything else yourself.
IndexIgnore README HEADER *.swp .htaccess
Unless you have used the SuppressColumnSorting option, there is a link at the top of each column in your directory index that, when clicked, will redisplay the index sorted by that column rather than by name (the default).
mod_autoindex accomplishes this with very simple parsing of URL arguments, which are appended to the URL with these links.
Note
These arguments completely changed in version 2.0, so you need to make sure that you are reading the portion of this section that refers to the version you are running.
In version 1.3, there are four columns by which you can sort, and each column can be sorted in either ascending or descending order. This is accomplished by appending a query string to the URL.[4]
The query string can have one key, which is N, M, S, or D, and this key can have one of two values—either A or D. The key represents the column by which you want to sort, with the A meaning ascending, and the D meaning descending.
| key | value |
|---|---|
| N | Name |
| M | Last Modified |
| S | File Size |
| D | Description |
When any key other than the name is selected, the secondary sort key is always the filename, sorted in ascending order (with the A's at the top, and the Z's at the bottom). When the page is sorted strictly by the filenames, there is no ambiguity, because filenames are guaranteed to be unique, so there is no need for a secondary sort key.
For example, if the URL http://apache.rcbowen.com/apache-admin/ generates a directory listing, the URL http://apache.rcbowen.com/apache-admin?S=D will generate the same directory listing, but in descending order by the size of the files, with the largest files at the top.
Note
When sorting by file size, the actual file size in bytes is used, which is not necessarily what is displayed in the listing, since file sizes are rounded to the nearest number of kilobytes. Hence, two files that are displayed with file size 1K are actually sorted by the number of bytes, which may be, say, 1,203 bytes and 978 bytes.
The links at the top of each column are automatically generated by mod_autoindex to reorder the listing by that column. When the view is already ordered in one direction (say, ascending) by a particular column, the link will then be generated to re-sort the list in the opposite order (now descending).
In version 2.0.23, directory sorting was overhauled. The list of possible arguments and values has been expanded, and may be combined to produce a larger matrix of results. The following tables show the various options:
| Argument C | Column by Which to Order |
|---|---|
| C=N | Order by the filename |
| C=M | Order by last modified date |
| C=S | Order by file size |
| C=D | Order by description |
| Argument O | Order in Which the Column Is to Be Sorted |
|---|---|
| O=A | Sort in ascending order |
| O=D | Sort in descending order |
| Argument F | Format in Which the Directory Should Be Displayed |
|---|---|
| F=0 | A simple list. No fancy indexing used |
| F=1 | FancyIndexing list
|
| F=2 | Displayed as an HTML table |
| Argument V | Ordered by Version Information |
|---|---|
| V=0 | Version ordering disabled |
| V=1 | Version ordering enabled |
| Argument p | Pattern for Which to Search |
|---|---|
| p=pattern | Only files that match the specified pattern will be shown. |
Note that the pattern search supplied by the p argument is applied after the list of files to be displayed has already been filtered through anything you have put in IndexIgnore, and so cannot be used to get at files that you have explicitly excluded from directory listings.
If mod_autoindex encounters an argument with which it is not familiar, it will immediately stop processing the arguments.
The Apache documentation supplies the following HTML snippet, which you can paste directly into a file referred to by HeaderName, in order to provide an easy-to-use way for users to select how they want the directory to be ordered.
<FORM METHOD="GET">
Show me a <SELECT NAME="F">
<OPTION VALUE="0"> Plain list
<OPTION VALUE="1" SELECTED> Fancy list
<OPTION VALUE="2"> Table list
</SELECT>
Sorted by <SELECT NAME="C">
<OPTION VALUE="N" SELECTED> Name
<OPTION VALUE="M"> Date Modified
<OPTION VALUE="S"> Size
<OPTION VALUE="D"> Description
</SELECT>
<SELECT NAME="O">
<OPTION VALUE="A" SELECTED> Ascending
<OPTION VALUE="D"> Descending
</SELECT>
<SELECT NAME="V">
<OPTION VALUE="0" SELECTED> in Normal order
<OPTION VALUE="1"> in Version order
</SELECT>
Matching <INPUT TYPE="text" NAME="P" VALUE="*">
<INPUT TYPE="submit" NAME="X" VALUE="Go">
</FORM>
Directory indexes are, by default, ordered by the names of the files. However, this default ordering can be changed with the IndexOrderDefault directive. This directive can specify which of the other fields you want to order by, and whether you want this ordering to be ascending or descending.
IndexOrderDefault Ascending Size
The first argument can be one of Ascending or Descending, and the second argument can be one of Name, Date, Size, or Description. When an option other than Name is chosen, the secondary key is always the filename. When the filename is the primary sort criterion, there is no ambiguity, as filenames are guaranteed to be unique.
If used in conjunction with the SuppressColumnSorting option, this directive will guarantee a particular ordering, no matter what the client asks for.
The following examples give a few ways in which directory index directives and options may be combined to provide customized directory index listings.
<Directory /usr/local/apache/vhosts/apache/htdocs/apache-admin> Options +Indexes IndexOptions +IconHeight=20 +IconWidth=22 IndexOptions +FoldersFirst +SuppressLastModified +SuppressHTMLPreamble HeaderName HEADER.html AddDescription 'zip file' *.zip AddDescription 'PDF format' *.pdf IndexOptions +NameWidth=* </Directory>
In the first line (the <Directory> directive) I specify that the directives here pertain only to the particular directory in which I am keeping these files, and any subdirectories thereof.
Next, I turn on Options Indexes so that the rest of the directives will be honored.
I specify a particular height and width for the icons to be displayed. This is the same as the default values for these sizes, but these sizes enable the browser to display the page before the images have loaded. I make the directories appear at the top of the directory listing, rather than scattered throughout the listing. I turn off the last modified column to give me sufficient room for the descriptions I add later. And I turn off the HTML preamble so that I can use the HTML header file specified in the next line with the HeaderName directive.
With two AddDescription directives, I add descriptions to the zip files and to Portable Document Format (PDF) files. Files with a .zip extension will be displayed with a description of “zip file”, and files with a .pdf extension are displayed with a description of “PDF format.”
Finally, with the NameWidth option, I make sure that the name field can expand to compensate for my tendency towards rather verbose filenames.
<Directory /usr/local/apache/htdocs/manual> Options +Indexes IndexOptions +SuppressSize +SuppressLastModified +ScanHTMLTitles HeaderFile header.html ReadmeFile footer.html </Directory>
In this example, you end up with a listing of your HTML files, with the title of each HTML document in the description field. This is very time- and CPU-intensive, but it is useful during the development of your Web site, showing you what files you have and whether they have well-formed and descriptive titles, and giving quick access to the various files without requiring that you navigate through the usual channels.
In one sense, there is a security risk involved in permitting auto-generated directory indexes. People can see all the files in a given directory and download whatever files they want from that directory. If there are files in the directory that you did not explicitly link to from anywhere on your site, they can still get those files. However, assuming that people cannot download files from your site merely because you have not linked to them from anywhere is what I refer to as cosmetic security, or security through the advanced technique of “hoping nobody notices.”[5] It is not real security, and if it gives you a false feeling of security, it is doing more harm than good. You should not put files that you don't want in the hands of the general public in any unauthenticated location on your public Web site.
The DirectoryIndex directive specifies a file to be displayed by default when a directory is requested without a particular file being requested.
Turning on Options Indexes enables the mod_autoindex module to automatically generate a directory index, with links to download each file. This listing is highly customizable with respect to the ordering of the files, which files are displayed, and what information is displayed about each file.
With these two sets of techniques, you can provide automatically generated file listings with a minimum amount of work, and provide users with an intuitive interface for downloading files.
[1] You are trying out all the examples as you come across them, right?
[2] Of course, now that you are aware of this capability, you might want to use it and ignore this advice!
[3] Or, more accurately, what is sometimes called “ASCII-betically,” since names are sorted by the ASCII value of the characters. This is why all the filenames that start with capital letters appear before all the files that start with lowercase letters.
[4] A query string is a series of one or more key/value pairs, appended after a question mark on the end of the URL. For example, ?name=Rich&occupation=author is a query string with two variables. See Chapter 15, “CGI Programs,” for more treatment of query strings and how to deal with them.
[5] Thank you, Scott Adams.