Reporting Telemetry

As a refresher from Chapter 2, Istio supports three forms of telemetry (metrics, logs, traces), which can transmit a diverse set of insights between them. Telemetry is reported from the data plane to the control plane. Service proxy reports contain attributes (see Chapter 9 for more on attributes). Context attributes provide the ability to distinguish between HTTP and TCP protocols within policies.

As attributes are generated by service proxies, telemetry reports are sent at three different points in time:

• When the connection is established (initial report)

• Periodically, while the connection is alive (periodical report)

• When the connection is closed (final report)

The default interval for periodical reports is 10 seconds. It’s recommended that this interval should not be in subseconds.

Configuring Mixer to Collect Metrics

Telemetry (and policy) is configured using three types of resources:

Handlers

These determine the set of adapters that are being used and how they operate. Providing a logging adapter with the IP address for the remote syslog server is an example of handler configuration.

Instances

These describe how to map request attributes (generated by the service proxy) to adapter inputs (where an adapter will receive the generated telemetry). Instances represent a chunk of data on which one or more adapters will operate. For example, an operator might decide to generate request_bytes metric instances from an attribute such as destination_workload.

Rules

These identify when a specific adapter is invoked and which instances it is given (what telemetry is funneled to it). Rules consist of a match expression and actions. The match expression controls when to invoke an adapter, whereas the actions determine the set of instances to give the adapter.

To use the Prometheus Mixer adapter, we need to have a Prometheus server deployed either on the same Kubernetes cluster or elsewhere that is capable of scraping metrics from the Prometheus Mixer adapter. There are many ways to deploy Prometheus, either on Kubernetes or outside, the details of which are beyond the scope of this book.

Setting Up Metrics Collection and Querying for Metrics

To configure and use the Prometheus Mixer adapter, we need to do the following:

1. Create a metric instance that configures metrics that Istio will generate and collect, and configure a Prometheus handler to collect the metrics, assign appropriate labels, and make it available for a Prometheus instance to scrape, as shown in Example 10-1 (network traffic; see the full configuration in this section of this book’s GitHub repository).

   Example 10-1. Step 1: An excerpt from the Prometheus handler and the requests_total metric it tracks across various labels

   apiVersion: "config.istio.io/v1alpha2"
   kind: handler
   metadata:
     name: prometheus
     namespace: istio-system
   spec:
     compiledAdapter: prometheus
     params:
       metrics:
       - name: requests_total
         instance_name: requestcount.metric.istio-system
         kind: COUNTER
         label_names:2
         - reporter
         - source_app
         - source_namespace
         - source_principal
         - source_workload
         - source_workload_namespace
         - source_version
         - destination_app
         - destination_namespace
         - destination_principal
         - destination_workload
         - destination_workload_namespace
         - destination_version
         - destination_service
         - destination_service_name
         - destination_service_namespace
         - request_protocol
         - response_code
         - connection_mtls

2. Update Prometheus to scrape metrics from the Prometheus Mixer adapter, and create Istio rules that will forward the metrics collected by Istio Mixer to the Prometheus Mixer adapter with the configured labels, as demonstrated in Example 10-2.

   Example 10-2. Step 2: A rule to match HTTP traffic and take action to forward metrics to the Prometheus handler

   apiVersion: "config.istio.io/v1alpha2"
   kind: rule
   metadata:
    name: promhttp
    namespace: istio-system
    labels:
      app: mixer
      chart: mixer
      heritage: Tiller
      release: istio
   spec:
    match: (context.protocol == "http" || context.protocol == "grpc") &&
           (match((request.useragent | "-"), "kube-probe*") == false)
    actions:
    - handler: prometheus
      instances:
      - requestcount.metric
      - requestduration.metric
      - requestsize.metric
      - responsesize.metric

Traces

Distributed traces are arguably the most insightful of the telemetry information gleaned from the service mesh, giving you insight into difficult-to-answer questions like “Why is my service slow?” Zipkin and Jaeger are both bundled into Istio releases and available as popular, open source distributed tracing systems for storing, aggregating, and interpreting trace data.

package main

import (
    "fmt"
    "log"
    "net/http"
)

func tracingMiddleware(next http.HandlerFunc) http.HandlerFunc {
    incomingHeaders := []string{
        "x-request-id",
        "x-b3-traceid",
        "x-b3-spanid",
        "x-b3-parentspanid",
        "x-b3-sampled",
        "x-b3-flags",
        "x-ot-span-context",
    }

    return func(w http.ResponseWriter, r *http.Request) {
        for _, th := range incomingHeaders {
            w.Header().Set(th, r.Header.Get(th))
        }
        next.ServeHTTP(w, r)
    }
}

func main() {
    http.HandleFunc("/", tracingMiddleware(func(w http.ResponseWriter,
                         r *http.Request) {
        fmt.Fprintf(w, "Hello headers, %v", r.Header)
    }))

    log.Fatal(http.ListenAndServe(":8081", nil))
}

Disabling Tracing

The sampling of request traces does incur cost in terms of performance overhead. As shown by Meshery in Figure 10-1, there is a significant difference between sampling traces at a rate of 1% versus a rate of 100%.

Figure 10-1. The difference in average node CPU use between two performance tests

The simplest way to run your Istio mesh without tracing being enabled at all is to not enable it when installing the service mesh. Thus, your Helm chart looks like the following:

--set tracing.enabled=false

The default and minimal Istio configuration profiles don’t enable tracing when installing Istio. If you deployed with tracing on and now wish to disable tracing, assuming that your control plane is installed in the istio-system namespace, run the following:

$TRACING_POD=`kubectl get po -n <istio namespace> | grep istio-tracing
              | awk `{print $1}`
$ kubectl delete pod $TRACING_POD -n <istio namespace>
$ kubectl delete services tracing zipkin   -n <istio namespace>

Remove references of the Zipkin URL from the Mixer deployment:

$ kubectl -n istio-system edit deployment istio-telemetry

Now, manually remove instances of trace_zipkin_url from the file and save it.

Metrics

Top-line service performance is easily gleaned from the metrics visualized through charting and dashboarding tools. Grafana is a popular, open source metrics visualization tool used to query, analyze, and alert on metrics. Grafana is not deployed as a Mixer adapter, but is included in the default Istio deployment as an add-on and is configured to read metrics from Prometheus. Prometheus is a time-series database and collection toolkit. The Istio deployment of Grafana comes with predefined dashboards. Metrics shown in the Istio dashboards in Grafana are dependent upon Prometheus running in the environment. Here are some of the items that packaged dashboards include:

Mesh Summary View

Provides global summary view of the service mesh and shows HTTP/gRPC and TCP workloads.

Individual Services View

Provides metrics about requests and responses for each individual service within the mesh (HTTP/gRPC and TCP). Also, gives metrics about client and service workloads for this service.

Individual Workloads View

Provides metrics about requests and responses for each individual workload within the mesh (HTTP/gRPC and TCP). Also, gives metrics about inbound workloads and outbound services for this workload.