The results index

As we mentioned previously, ML analyzes data and summarizes the results into a results index. By default, this index is called .ml-anomalies-shared, unless the job was configured to use a dedicated index via ticking the box in the configuration user interface (UI) or via setting the results_index_name field when using the API. If that was the case, then the results index will be called .ml-anomalies-custom-myname, where myname is either the declared value of the results_index_name field when using the API, or the name of the job itself when using the UI. In either case, the results index can be quite small compared to the size of data that was used to feed the ML analysis. In addition, an index alias is also created of the .ml-anomalies-jobname form, where jobname is the name of the job. The definition of this alias can be seen by hitting the following API call:

GET ml-anomalies-*/_alias

The results will include entries that look as follows:

      ".ml-anomalies-farequote": {
"filter": {
"term": {
"job_id": {
"value": "farequote",
"boost": 1
}
}
}
},

Here, farequote is the name of the ML job. It's obvious that the purpose of the alias is to only show results for a specific job_id via a term filter. Otherwise, querying the .ml-anomalies-shared index (or a greedier index pattern of .ml-anomalies-*) will return records for many jobs. If the user wants to query for the results of only one ML job, it is up to the user to either use this index alias or to do their own term filter on the proper job_id.

Inside the result index, there are a variety of different documents, each with their own usefulness with respect to Alerting. The ones we will discuss are the ones that directly relate to the three levels of abstraction that we discussed previously. They are aptly named as follows:

  • result_type:bucket: To give bucket-level results
  • result_type:record: To give record-level results
  • result_type:influencer: To give influencer-level results

The distribution of these document types will depend on the ML job configuration and the characteristics of the dataset being analyzed. These document types are written with the following heuristic:

  • result_type:bucket: One document is written for every bucket span's worth of time. In other words, if the bucket span is 15m, then there will be one document of this type being written every 15 minutes. Its timestamp will be equal to the leading edge of the bucket. For example, for the time bucket that encompasses the range between 11:30 and 11:45, the result document of this type will have a timestamp of 11:30.
  • result_type:record: One document is written for every occurrence of an anomaly within a time bucket. Therefore, with big datasets encompassing many entities (IP addresses, hostnames, and so on), a particular bucket of time could have hundreds or even thousands of anomaly records in a bucket during a major anomalous event or widespread outage. This document will also have a timestamp equal to the leading edge of the bucket.
  • result_type:influencer: One document is written for every influencer that's found for each anomaly record. Because there can potentially be more than one influencer type found for each anomaly record, this type of document can be even more voluminous than record results. This document will also have a timestamp that's equal to the leading edge of the bucket.

The reason why understanding this is so important with respect to Alerting is because there will inevitably be a balance between alert detail (usually, more is preferable to less) with the number of individual alerts per unit time (usually, less is preferable to more). We will revisit this when we start writing actual alerts.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset