So far, we've discussed various logging scenarios that we may encounter in the real world. It's now time to roll up our sleeves and fabricate a logging system. The architectures of logging systems and monitoring systems are pretty much the same in a number of ways: they both have collectors, storage, and consumers such as BI tools or a search engine. The components might vary significantly, depending on the needs. For instance, we might process some logs on the fly to extract real-time information, while we might just archive other logs to durable storage for further use, such as for batch reporting or meeting compliance requirements. All in all, as long as we have a way to ship logs out of our container, we can always integrate other tools into our system. The following diagram depicts some possible use cases:
In this section, we're going to set up the most fundamental logging system. Its components include Fluent Bit, Elasticsearch, and Kibana. The templates for this section can be found under 7-3_efk, and they are to be deployed to the logging namespace.
Elasticsearch is a powerful text search and analysis engine, which makes it an ideal choice for analyzing the logs from everything running in our cluster. The Elasticsearch template for this chapter uses a very simple setup to demonstrate the concept. If you'd like to deploy an Elasticsearch cluster for production use, using the StatefulSet controller to set up a cluster and tuning Elasticsearch with proper configurations, as we discussed in Chapter 4, Managing Stateful Workloads, is recommended. We can deploy an Elasticsearch instance and a logging namespace with the following template (https://github.com/PacktPublishing/DevOps-with-Kubernetes-Second-Edition/tree/master/chapter7/7-3_efk):
$ kubectl apply -f logging-ns.yml
$ kubectl apply -f elasticsearch
We know that Elasticsearch is ready if we get a response from es-logging-svc:9200.
The next step is to set up a node logging agent. As we'd run this on every node, we want it to be as light as possible in terms of node resource use; hence why we opted for Fluent Bit (https://fluentbit.io/). Fluent Bit features lower memory footprints, which makes it a competent logging agent for our requirement, which is to ship all the logs out of a node.
In our example, Fluent Bit is configured as the first logging pattern. This means that it collects logs with a logging agent per node and sends them to Elasticsearch directly:
$ kubectl apply -f logging-agent/fluentbit/
The ConfigMap for Fluent Bit is already configured to tail container logs under /var/log/containers and the logs of certain system components under /var/log. Fluent Bit can also expose its stats metrics in Prometheus format on port 2020, which is configured in the DaemonSet template.
To use Kubernetes events, we can use the eventrouter:
$ kubectl apply -f eventrouter.yml
This will start to print events in JSON format at the stdout stream, so that we can index them in Elasticsearch.
To see logs emitted to Elasticsearch, we can invoke the search API of Elasticsearch, but there's a better option: Kibana, a web interface that allows us to play with Elasticsearch. Deploy everything under kibana in the examples for this section with the following command:
$ kubectl apply -f kibana
Kibana, in our example, is listening to port 5601. After exposing the service from your cluster and connecting to it with any browser, you can start to search logs from Kubernetes. In our example Fluent Bit configuration, the logs routed by eventrouter would be under the index named kube-event-*, while logs from other containers could be found at the index named kube-container-*. The following screenshot shows what a event message looks like in Kibana:
