By default, Linux logs almost everything. This is important for developing a root cause analysis when things go wrong. When you're faced with a problem on a production server, all you should need to do is determine the time in which the problem started and then read the log files for the types of things that happened on the system during that time. Linux logging is very informative.

But nowadays, the way that Linux handles logging is changing. With the rise of systemd, which is now the default init system on most Linux distributions, it's taken over almost everything, including logging. In the past, you would venture into /var/log whenever you wanted to read your logs, which is a directory containing various log files in plain text format. On both Debian and CentOS, you can still find logs in /var/log, so you'll still be able to utilize them for troubleshooting the same as we always have. But it's not yet certain how much longer this will be kept around.

Many might think that systemd taking over logging is a bad thing. After all, having the init system take care of so much of the system's upkeep gives it more work to perform, which may stretch it too thin. But one issue with syslog (the previous approach) is that there was no consistency from one distribution to another in how the logs were created or named. For example, Debian systems include an auth.log, which CentOS doesn't. Both have dmesg and only CentOS has a boot.log file. This makes troubleshooting a mixed environment, a beast.

The systemd approach (which we'll discuss later) offers a more consistent approach between distributions. So while it may be true that systemd is being spread thin with the multitude of responsibilities it has on the system, consistency is definitely welcome.

Both Debian and CentOS have a log file that is used whenever a user logs into the system, even if she or he does so via SSH. On CentOS, this log is located in /var/log/secure. Debian uses /var/log/auth.log for this purpose. If you need to know who is logging into your system and when, you would want to look at these logs in order to find out. On both, you can find /var/log/messages, which includes a smorgasbord of useful information, such as output from processes, network activation, services starting up, and more. When it comes to troubleshooting hardware, /var/log/dmesg is a great place to look. In fact, /var/log/dmesg has its own command. Typing dmesg from anywhere on the system (even if your current working directory isn't /var/log) will present you with the same log.

The log files in /var/log are very easy to follow in near real time by using tail -f. The -f flag of tail isn't specifically limited to log files. It allows you to display the output of a log file, as it's being written to. When you're troubleshooting a system, tail -f is indispensable. For example, if you have a user that cannot log in to the system, you could run the following on a Debian system to watch the auth.log file as they make their attempt. That way, you can see what error message the system is registering for their failed attempts at logging in:

# tail -f /var/log/auth.log

From there, as the auth.log gets updated, the results will show in your terminal immediately. To end, simply press Ctrl + C to stop following the output. You can do this with any log, or any text file on your system. This is very useful for a multitude of troubleshooting tactics, as most processes you may want to investigate will log its activities to at least one log.