Where the access_log is located is actually a configuration option. If you look in your configuration file, httpd.conf, you should see a line that looks like the following:

CustomLog /usr/local/apache/logs/access\_log common

The CustomLog directive specifies where a particular log file should be stored, and what format that log should be in. The log format described previously is the common log format, which has been in use as the standard since the beginning of Web servers. That's why it still contains the ident information field, even though almost no clients actually pass that information to the server.

The path specified there is the location of the log file.

LogFormat "%h %l %u %t "%r" %>s %b" common

LogFormat %404{ Referer} i BrokenLinks

LogFormat %!200U SomethingWrong

CustomLog /var/log/httpd/bogus_log SomethingWrong
CustomLog /usr/local/apache/logs/broken BrokenLinks
CustomLog /usr/local/apache/logs/access_log common

Your error log file should be in the same location as your access log file. It will be called error_log, or, on Windows machines, error.log.

The location of your error log can be configured with the ErrorLog directive:

ErrorLog logs/error.log

The location, unless it has a leading slash, is assumed to be relative to the ServerRoot directory.

In a default Apache installation the log file is located in /usr/local/apache/logs. As with the access log, if you installed with one of the various package managers out there, you might find it just about anywhere.

The error log, as the name suggests, contains a record of everything that went wrong while your server was running. It also contains general diagnostic messages, such as a notification of when your server was restarted, or shut down.

You can set your log level higher or lower to control the amount, and type, of messages that appear in your log file. This is configured with the LogLevel directive. The default setting of this directive is error, which tells you about error conditions. The complete list of possible settings is contained in Table 24.7.

In most cases, the things that you see in your log file will be in two categories: document errors and CGI errors. You will also occasionally see configuration errors, server start, and server stop messages.

Document errors are things that are in the 400 series of server response codes. The most common of these is 404—Document Not Found. 404's are followed in frequency, on most servers, by authentication errors.

A 404 error occurs whenever someone requests a resource—a URL—that is not on your server. Either they have mistyped something, there was a typo in a link somewhere, or you moved or deleted a document that used to be on your server.

When a client is unable to locate a document on your server, you'll see an entry like this in your logs:

[Fri Aug 18 22:36:26 2000] [error] [client 192.168.1.6] File does not exist:
/usr/local/apache/bugletdocs/Img/south-korea.gif

Note that, as in the case of the access_log file, this record is broken down into several fields.

First, we have the date/time stamp. The first thing that you might notice is that the format is not the same as the format in the access_log. The format that we called the standard-English format. This is merely an accident of history.

Next, we have the level of the message. This will be one of the levels specified in the documentation for LogLevel (see previous). error is right between warn and crit. This simply indicates how serious the problem is. A 404 error means that you irritated someone, but it's not actually a critical condition affecting the health of your server.

The next field indicates the address of the client machine that made the request. In this case, it is a machine on my local network.

The last part of the log entry is the actual error message. In the case of a 404, it gives you the full path of the file that the server tried to serve. This is particularly useful when you're getting a 404 on a file that you just know is there. Frequently you have a configuration wrong, or the file is on a different virtual host than you thought, or some other strangeness.

Note that document errors, because they are a direct result of a client request, will be accompanied by an entry in access_log as well.

Authentication errors will look very much the same:

[Tue Apr 11 22:13:21 2000] [error] [client 192.168.1.3] user rbowen
authentication failure for "/cgi-bin/hirecareers/company.cgi": password
mismatch

Perhaps the most useful purpose of the error is troubleshooting misbehaved CGI programs (or other content generation programs, such as mod_perl). Anything that a CGI program emits to STDERR (Standard Error) gets appended directly to the error log for your perusal. This means that (well-written) CGI programs, when they have problems, will tell you, via the log file, exactly what that problem is.

The downside to this is that you end up with stuff in the error log that is not in any well-defined format, and so it makes it very hard to have any automatic error-log parsing to get useful information out.

What follows is an example of an entry in an error log from problematic Perl CGI code:

Wed Jun 14 16:16:37 2000] [error] [client 192.168.1.3] Premature
end of script headers: /usr/local/apache/cgi-bin/TestProg/announcement.cgi
Global symbol "$rv" requires explicit package name at
/usr/local/apache/cgi-bin/TestProg/announcement.cgi line 81.
Global symbol "%details" requires explicit package name at
/usr/local/apache/cgi-bin/TestProg/announcement.cgi line 84.
Global symbol "$Config" requires explicit package name at
/usr/local/apache/cgi-bin/TestProg/announcement.cgi line 133.
Execution of /usr/local/apache/cgi-bin/TestProg/announcement.cgi
aborted due to compilation errors.

Although this entry actually does follow the same format as the previous 404 error, in that it has a date, error level, and a client address, the error message itself is several lines long, which tends to confuse some log-parsing software.

Even if you don't know Perl, you should be able to look at the previous error messages and glean some useful information about what went wrong. At the very least, you can tell on what lines the program had problems. Perl is very good about telling you where you made a mistake. Your mileage may vary based on what language you are using.

Without logs, it would be very difficult to troubleshoot most CGI programs because running it from the command line is a rather different environment than running it from a Web server.

When actively developing a CGI program, or a content-generation application using some other technology, such as mod_perl, it's a good idea to actively watch the error log, so that when error conditions happen, you have immediate feedback.

This is done using the utility called tail. tail is a standard part of any Unix operating system, and tail clones are available for other operating systems.

At the command prompt, type the following:

tail -f /usr/local/apache/logs/error_log

This will show the last few lines of your log file, and, as lines are added to the file, it will show you those as they happen. The f stands for “follow” because it follows the log as it grows.

If you want to use tail (and other Unix utilities) on Windows, you may want to try AINTX, which is a collection of AIX utilities for NT. You can find these at http://maxx.mc.net/jlh/nttools/html/nttools.htm.

It's extremely good practice to keep several terminal windows open, with your error log tailing in one window, and your access log tailing in the other, while you work on your site. This will tell you what is going on as it happens, so you know about problems before the customer has a chance to call you about them.

A lot of information is available to put in your log files, including the following:

HTTP is a stateless, anonymous protocol. This is by design, and is not, at least in my opinion, a shortcoming of the protocol. If you want to know more about your visitors, you have to be polite, and actually ask them. And be prepared to not get reliable answers. This is amazingly frustrating for marketing types. They want to know the average income, number of kids, and hair color of their target demographic. And they don't like to be told that that information is not available in the log files. However, it is quite beyond your control to get this information out of the log files. Explain to them that HTTP is anonymous.

Even what the log files do tell you is occasionally suspect. For example, you can expect to have numerous entries in your log files indicating that a machine called something like cache-mtc-am05.proxy.aol.com has visited your Web site. This tells you that this is a machine that is on the AOL network. But because of the way that AOL works, this might be one person visiting your site many times, or it might be many people visiting my site one time each. All requests coming from the AOL network are proxied. A proxy server is one that one or more people sit behind. They type an address into their browser. It makes that request to the proxy server. The proxy server gets the page (generating the log file entry on my Web site). It then passes that page back to the requesting machine. This means that I never see the request from the originating machine, but only the request from the proxy.

Another implication of this is that if, 10 minutes later, someone else sitting behind that same proxy requests the same page a log file entry is not generated at all. They type the address, and that request goes to the proxy server. The proxy sees the request and thinks “I already have that document in memory. There's no point asking the Web site for it again.” So instead of asking my Web site for the page, it gives the copy that it already has to the client. So, not only is the address field suspect, but the number of requests is also suspect.

Most proxies will not cache dynamic content, such as the results of a CGI program, as that can change from one client to the next.

It might sound like the data that you receive is so suspect that it's useless. This is in fact not the case. It should just be taken with a grain of salt. The number of hits that your site receives is almost certainly not really the number of visitors that came to your site. But it's a good indication. And it still gives you some useful information. Just don't rely on it for exact numbers.

So, to the real meat of all this. How do you actually generate statistics from your Web server logs?

There are two main approaches that you can take here. You can either do it yourself, or you can get one of the existing applications that is available to do it for you.

Unless you have custom log files that don't look anything like the Common log format, you should probably get one of the available apps out there. There are some excellent commercial products, and some really good free ones, so you just need to decide what features you are looking for.

The following are some of the available programs on the market. This should not be considered a comprehensive list, and you should do your own research before choosing one for your site because they all have somewhat different feature sets and different reports.

The programs that I have chosen here either all have versions that will run on Unix and NT operating systems, or are in Perl, and will consequently run anywhere.

If you want to do your own log parsing and reporting, the best tool for the task is Perl. In fact, Perl's name (Practical Extraction and Report Language) is a tribute to its capability to extract useful information from logs and generate reports. (In reality, the name “Perl” came before the expansion of it, but I suppose that does not detract from my point.)

The Apache::ParseLog module, available from your favorite CPAN mirror, makes parsing log files simple, and, therefore, takes all the work out of generating useful reports from those logs.

For detailed information about how to use this module, install it and read the documentation. After you have installed the module, you can get at the documentation by typing perldoc Apache::ParseLog.

Trolling through the source code for WWWStat is another good way to learn about Perl log file parsing.

Because the log file format is so simple, writing code to chop it up into its component parts and do statistical analysis of it is rather simple.

If you want to do this yourself, you can use a Perl module (freely available from CPAN) called Logfile::Rotate. The following code, run periodically (perhaps once a week?) by cron, will rotate out your log file, keeping five previous log files at any given time. Each backup log file will be gzipped to conserve space.

#!/usr/bin/perl
use Logfile::Rotate;
foreach $log (qw(error_log access_log)) {
    $logfile = new Logfile::Rotate(
        File => "/usr/local/apache/logs/$log",
        Count => 5,
        Gzip => '/bin/gzip',
        Post => sub {
            `/usr/local/apache/bin/apachectl restart`;
        }
    }
);

The Perl module takes care of all the details. You'll end up with files called things such as access_log.1.gz, access_log.2.gz, and so on. Each file will get bumped up one number each time and the file that used to be access_log.5.gz will be deleted each time. The Count parameter specifies how many log files are kept.

This keeps you from running out of space on your log drive, and keeps as much of an archive as you like.

Additionally, Apache ships with a utility called logrotate, which enables you to rotate log files on a regular basis. You can use the logrotate utility by adding it to your Apache configuration file as a process to which you will log. The syntax will look like this:

CustomLog "|/usr/local/apache/bin/rotatelogs /var/log/archive/apachelog 86400" common

The parameter 86400 here is the number of seconds after which the log will be moved and a new log started.

The path /var/log/archive/apachelog specifies where the old log files will be put for archival purposes. More specifically, the log file will be backed up to a file named by taking this argument, and appending a timestamp.

86400 seconds, by the way, is 24 hours.