As you know, logs are crucial when it comes to troubleshooting. Linux generally does a very good job of logging almost everything you would want to know, but over time these logs can really add up. On a production server, a log file growing out of control and taking up literally all of your server's free space is a very real issue if left unchecked. In addition to disk space being consumed, a gigantic log file is very hard to open in a text editor in order to view the contents, which makes troubleshooting even harder. A log file of over 500 GB would not only take up a ridiculous amount of space; it would likely cause the system to hang if you try to open it, and transferring a log file to another server for analysis once it reaches a very large size isn't practical either.
For the most part, excessive log files are not as much of an issue on newer Linux distributions than those of the past. With syslog, there was no automatic maintenance. If you didn't either clean the logs yourself or set up something to rotate them for you, you would definitely need to keep an eye on them. Nowadays, journald handles this for us. But with Debian and CentOS, this can be somewhat of a mixed bag. This is because although the systemd journald takes care of logging for us on newer releases of most popular Linux distributions, syslog is still used for compatibility. Therefore, we still have to deal with log rotation even though all the pieces are in place for journald. The journald is the future, though syslog is still used on Enterprise Linux distributions today for compatibility.
Log rotation is the process of taking an existing log file, renaming it, and having the process write to a brand-new empty log file. The previous log files can all be kept, or you can keep only a few of them if you wish. It's not uncommon for Enterprise systems to have a specific retention policy. It's a common practice to compress previous logs, which saves a great amount of disk space. This is where logrotate comes in. It's a process that we can run on our server to automatically swap out our log files and (as an option) compress the backup copies.
While designing a Linux network, it's important to understand which processes each server needs to run and to take into account the logging requirements of those processes from the start. Having logrotate installed and configured before a server enters production is a good practice. Having a server run out of free space in the middle of production is never a good experience, and knowing first what log files a running process creates, and being prepared to handle them is a good idea. While configuring your logging, it's important to take into consideration the retention requirements of your company, if there are any.
On the CentOS system used in my lab, logrotate was installed by default. Debian had it installed out of the box as well. To verify this on your system, simply run the following command:
which logrotate
On CentOS, the logrotate binary is located in /usr/sbin, while Debian stores theirs in /usr/sbin. If the which command shows no output, you may need to use your distribution's package manager to install the logrotate package.
With the default installations of both Debian and CentOS, logrotate is already configured to run each day. When it does, it checks the /etc/logrotate.d directory for instructions and then executes them. The configuration for setting up logrotate rules is fairly straightforward. If you need example syntax, refer to your own system. By default, several logrotate scripts are created for you. An example of this is Debian's package manager apt. Whenever you install packages on a Debian system, it's logged in the following place:
/var/log/apt/history.log
If you view this file, you should see results of recent package installations that you or another user has performed. By default, the following file exists on Debian systems to handle the rotation of this log:
/etc/logrotate.d/apt
On Debian 8, this file contains the following:
/var/log/apt/term.log { rotate 12 monthly compress missingok notifempty } /var/log/apt/history.log { rotate 12 monthly compress missingok notifempty }
As you can see, this configuration file for logrotate handles not only the history.log we mentioned earlier, but also term.log as well. Each section of this configuration begins with a path for logrotate to check, followed by individual options within brackets.
Among the options, we can see rotate 12, which means that up to 12 backup log files will be kept. Next, we see monthly, which details how often the log will actually be rotated. Despite the fact that logrotate is configured by default to run daily, it will follow the instructions contained within the individual configurations and only rotate if it meets that criteria. The compress option tells logrotate to compress the backed up file, which is probably what you want in most cases. Compressed log files use up very little space compared to the uncompressed live log, so it's definitely something to consider. missingok tells logrotate to keep running even if it encounters a missing log file. Otherwise, it would've displayed an error. Finally, we have notifempty, which simply tells logrotate to not bother with a log file if it is empty.
While logrotate has some fairly decent default configuration for some of the services that ship with CentOS and Debian, you'll want to consider creating configuration for any new services that you set up. To do so, it's easiest to follow the format shown in example files that you'll already have stored in /etc/logrotate.d. It's as simple as beginning your configuration block with the path to a file you want logrotate to handle for you, followed by options within curly brackets. There's no service to restart or special command to make your new configuration active. The next time that logrotate runs, it will check the /etc/logrotate.d directory for new configurations and run them if there are no errors.