Monitoring with a quantitative time series of the system status enables us to quickly identify which components in our system have failed, but it still isn't capable of diagnosing the root cause of a problem. What we need is a logging system that gathers, persists, and searches logs, by means of correlating events with the anomalies detected. Surely, in addition to troubleshooting and postmortem analysis of system failures, there are also various business use cases that need a logging system.

In general, there are two main components in a logging system: the logging agent and the logging backend. The former is an abstract layer of a program. It gathers, transforms, and dispatches logs to the logging backend. A logging backend warehouses all logs received. As with monitoring, the most challenging part of building a logging system for Kubernetes is determining how to gather logs from containers to a centralized logging backend. Typically, there are three ways to send out the logs of a program:

• Dumping everything to stdout/stderr.
• Writing log files to the filesystem.
• Sending logs to a logging agent or logging to the backend directly. Programs in Kubernetes are also able to emit logs in the same manner, so long as we understand how log streams flow in Kubernetes.