Attacks can be manifested in different forms: a single, discrete action or a complex, connected set of actions over time, spanning different parts of the infrastructure and thus different datasets. The shape or the geometry of an attack is very variable and will depend on the goal of the attack, plus the structure of the IT environment.

Let's take the example of a DNS tunneling attack, which consists of a Compromised Server exfiltrating data from the organization using encrypted messages disguised as a series of legitimate-looking DNS requests. This behavior, which is described in the following diagram, shows that Compromised Server emits a series (thousands, even millions) of DNS requests over time, with a small payload of data encrypted into the subdomain part of the request to an unconventional domain name. On the other side, a compromised DNS server, run by a hacker, accepts and re-assembles the encrypted data:

This exfiltration technique often goes undetected because of the following points:

• DNS traffic from inside an organization to the outside world is not blocked or restricted by security policies; DNS is too fundamental to the operation of the internet.
• DNS traffic is often incredibly voluminous, so the collection, storage, and analysis of the data isn't possible or practical with non-big data platforms. Therefore, the data is often not collected and is unwatched, making it the perfect place to hide malicious behaviors.
• The actual technique that's employed in DNS tunneling is inefficient and painstakingly slow; it can take weeks or months to exfiltrate a significant amount of data. So, this low and slow approach is an asset when being covert.

Clearly, it should be obvious that if you do not collect and analyze this kind of data due to a lack of capability in your legacy security tools, then you've just admitted that not only are you vulnerable to this attack vector, you would never even know if it were occurring right now.

This DNS tunneling activity is a discrete, malicious behavior that surely any security analyst would love to be able to detect and thwart. However, this activity could be at or near the end of a long chain of related malicious activities; it could have started with a user's machine getting compromised days, weeks, or months earlier.

To get the full picture of how this threat morphs over time from the point of initial compromise to the point of data exfiltration, the security analyst would have to have a vast set of diverse data, and have weeks or months of it on hand. The evidence of the threat's activity could manifest itself in various ways in network data, authentication logs, endpoint logs, or other data.

To illustrate this, the following diagram represents the life cycle of this kind of advanced persistent threat (APT):

We'll use this scenario throughout this chapter and will see how Elastic ML helps to uncover the details of such an attack.

But first, a couple of words on the preceding diagram in terms of how the kill-chain is articulated:

• The first step (the initial compromise) occurs when a machine within the organization accesses a known threat via a malicious external website
• Right after the threat was touched and the machine was infected, an unusual process starts running on the same machine
• Port scanning activity is then invoked from that machine as the APT enters its reconnaissance phase
• Suspicious login activity later occurs against a server that we know hosts some valuable assets
• Those valuable assets are then exfiltrated via DNS to the outside world

ML can help us detect each of these different IoCs occurring over time and would allow the alerting of the security team to investigate and take action.

Because the attack is broken down in different steps, and because the evidence of each step exists in different parts of the IT system, different datasets such as Windows machine logs, network logs, and others must be indexed and searchable. Therefore, building the proper data acquisition architecture is fundamental for being able to retrace all of those events.