Logs contain rich information about the state and behavior of a system or an application. Each system/application generates logs whenever an event occurs, and the frequency, amount of information, and format of the information it logs varies from one system/application to another. With so much information at our disposal, collecting it, extracting the relevant information from it, and analyzing it in near real time can be a daunting task.

In the previous chapters, we have already explored how Elasticsearch, with its rich aggregation features, assists in analyzing huge amounts of data in near real time. Before analysis can be performed, we need a tool that can assist/ease the process of collecting logs, extracting the relevant information from them, and pushing them to Elasticsearch.

In this chapter, we will be exploring Logstash, another key component of the Elastic Stack that is mainly used as an ETL (Extract, Transform, and Load) engine. We will also be exploring the following topics:

• Log Analysis challenges
• Using Logstash
• The Logstash architecture
• Overview of Logstash plugins
• Ingest node