IT operations and IT security organizations are collecting massive amounts of data. Some of that data is collected and/or stored in specialized tools, but some may be collected in general-purpose data platforms such as the Elastic Stack. But the question still remains: what percentage of that data is being paid attention to? By this, we mean the percentage of collected data that is actively inspected by humans, or being watched by some type of automated means (defined alarms based on rules, thresholds, and so on). Even generous estimates might put the percentage in the range of single digits. So, with 90% or more data being collected going unwatched, what's being missed? The proper answer might be that we don't actually know. 

Before we admonish IT organizations for the sin of collecting piles of data, but not watching it, we need to understand the magnitude of the challenge associated with such an operation. A typical user-facing application may do the following:

• Span hundreds of physical servers
• Have dozens (if not hundreds) of microservices, each of which may have dozens or hundreds of operational metrics or log entries that describe its operation

The combinatorics of this can easily rise to a six or seven figure range of unique measurement points. Additionally, there may be dozens or even hundreds of such applications under the umbrella of management by the IT organization. It's no wonder that the amount of data being collected by these systems per day can easily be measured in terabytes.

Accordingly, it is incredibly daunting to jump head-first into a strategy of bringing proactive coverage to all (or at least a significant portion) of this data to get holistic application visibility. Where you should start and what data to prioritize is not a trivial task, but it usually starts with a focus on the KPIs of the application.