This section will show you how to configure Filebeat manually instead of using out-of-the-box preconfigured modules for shipping files/logs/events. This section contains a list of inputs that Filebeat uses to locate and process log files. Each input item begins with a dash (-) and contains input-specific configuration options to define the behavior of the input.

A sample configuration is as follows:

As of Filebeat 7.0, inputs supported are Log, Stdin, Redis, UDP, Docker, TCP, Syslog, and NetFlow. Depending on the type of input configured, each input has specific configuration parameters that can be set to define the behavior of log/file/event collection. You can configure multiple input types and selectively enable or disable them before running Filebeat by setting the enabled parameter to true or false.

Since logs are one commonly used input, let's look into some of the configurations that can be set to define the behavior of Filebeat to collect logs.

log input-specific configuration options are as follows:

• type: It has to be set to log in order to read every log line from the file. 
• paths: It is used to specify one or more paths to look for files that need to be crawled. One path needs to be specified per line, starting with a dash (-). It accepts Golang glob-based paths, and all patterns Golang glob (https://golang.org/pkg/path/filepath/#Glob) supports are accepted by the paths parameter.
• exclude_files: This parameter takes regex to exclude file patterns from processing.
• exclude_lines: It accepts a list of regular expressions to match. It drops the lines that match any regular expression from the list. In the preceding configuration example, it drops all the lines beginning with DBG. 
• include_lines: It accepts a list of regular expressions to match. It exports the lines that match any regular expressions from the list. In the preceding configuration example, it exports all the lines beginning with either ERR or WARN.

• tags: It accepts a list of tags that will be included in the tags field of every event Filebeat ships. tags aid conditional filtering of events in Kibana or Logstash. In the preceding configuration example, java_logs is appended to the tags list.
• fields: It is used to specify option fields that need to be included in each event Filebeat ships. Like tags, it helps with the conditional filtering of events in Kibana or Logstash. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify will be grouped under a fields sub-dictionary in the output document. In the preceding configuration example, a new field called env with the staging value would be created under the fields field.

• scan_frequency: It is used to specify the time interval after which the input checks for any new files under the configured paths. By default, scan_frequency is set to 10 seconds.
• multiline: It specifies how logs that are spread over multiple lines need to be processed. This is very beneficial for processing stack traces/exception messages. It is made up of a pattern that specifies the regular expression pattern to match; negate, which specifies whether or not the pattern is negated; and match, which specifies how Filebeat combines matching lines with an event. The values for the negate setting are either true or false; by default, false is used. The values for the match setting are either after or before. In the preceding configuration example, all consecutive lines that begin with the space pattern are appended to the previous line that doesn't begin with a space. 

Let's look into another frequently used input type, docker, which is used to read logs from docker containers. It also contain many overlapping configuration parameters for the log input type.

docker input-specific configuration options are as follows:

• type: It has to be set to docker in order to read container logs.
• containers.ids: This parameter is used to specify the list of containers to read logs from. In order to read logs from all containers, you can specify *. This is a required parameter.
• containers.path: The base path where logs are present so that Filebeat can read from them. If the location is not specified, it defaults to /var/lib/docker/ containers.
• containers.stream: The stream to read the file from. The list of streams available is: all, stdout, and stderr. all is the default option.

• exclude_lines: It accepts a list of regular expressions to match. It drops lines that match any regular expression from the list. In the preceding configuration example, it drops all lines beginning with DBG. 
• include_lines: It accepts a list of regular expressions to match. It exports lines that match any regular expressions from the list. In the preceding configuration example, it exports all lines beginning with either ERR or WARN.
• tags: It accepts a list of tags that will be included in the tags field of every event Filebeat ships. tags aids conditional filtering of events in Kibana or Logstash. In the preceding configuration example, java_logs is appended to the tags list.
• fields: It is used to specify option fields that need to be included in each event Filebeat ships. Like tags, it aids conditional filtering of events in Kibana or Logstash. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify will be grouped under a fields sub-dictionary in the output document. In the preceding configuration example, a new field called env with the staging value would be created under the fields field.

• scan_frequency: It is used to specify the time interval after which the input checks for any new files under the configured paths. By default, scan_frequency is set to 10 seconds.