Enforcing Policy

The check API exposed by istio-policy handles policies of different types such as that of authentication and quota. Performance and availability of the check API is important when you consider that the check API is consulted inline (synchronously) when the service proxy is processing each request. Based on the request attributes presented, the check API validates whether a given request is in or out of compliance with the active policies configured in Mixer. Ultimately, the Mixer adapters determine whether specific policies’ conditions are met. Some adapters validate policy conditions against backend systems, whereas some process checks within the adapter itself (e.g., blacklists, quotas).

As an attribute-processing engine, Mixer transforms attributes into requests to specific backends via adapters, which massage the attributes into a format specific to a backend system for evaluation. Backend systems that adapters can interface with could be a policy engine or API management system, for example. These systems evaluate the check request, responding affirmatively or negatively to the request based on various conditions. There is a growing list of third-party adapters created for Mixer, many times contributed by third parties who represent a specific backend system.

Let’s look again at Mixer’s architecture, this time in the context of how istio-policy goes about receiving check requests, evaluating policy, and responding with a result. istio-policy exposes a check API with a fixed set of parameters, as shown in Figure 9-2.

Figure 9-2. Mixer istio-policy architecture overview

Service proxies call Mixer before each request to perform precondition checks and after each request to report telemetry. Check results from Mixer are cached in service proxies. Acting as a first-level cache, service proxies are able to answer a relatively large percentage of precondition checks from their local cache. In turn, as much as anything else, Mixer acts as a cache. The role it plays as a second-level cache for policy results is critical to mitigating the volume of policy check traffic to and evaluation overhead in Mixer.

Well-designed caches are key to enabling conscientiousness to security practices in distributed systems. Ideally, requests between services are authenticated and authorized at every single hop through the chain of upstream services (one request implicating any number of requests to other services in the process of responding to the request). Traditionally, authentication and authorization is performed on the service edge by something like an API gateway. The common pattern is that after a request is authenticated and authorized at the edge, requests to other services in this chain of requests are assumed safe, not subsequently verified.

Ideally, distributed systems have policy applied at every point in the service chain, not just at the edge. This is how we attain consistency of security throughout our distributed system. This presents a real problem to Istio Mixer, our authentication service, however. If every single service is calling Mixer, one request that implicates eight services, for example, means that eight different authorization requests would be sent to Mixer for consideration. Mixer and service proxies need to effectively operate a distributed cache.

Reporting Telemetry

Telemetry is generated as and when a request (network traffic) is received by the data plane’s service proxies. For each request received, there is an array of metadata that can be captured. This metadata provides context and accountable details of each request and is captured in the form of attributes. Mixer continually receives and processes these request attributes (telemetry). In Figure 9-3, istio-telemetry exposes a report API with a potentially long list of attributes that varies by adapter.

Reports are generated as Envoy is processing requests and are sent asynchronously to Mixer’s report API (exposed by istio-telemetry) out of band of the request. Envoy buffers outgoing telemetry such that Mixer is called only after having processed many requests (a configurable buffer size). It is within istio-telemetry that Mixer synthesizes attributes and pushes to a telemetry backend via one or more adapters.

Figure 9-3. Mixer istio-telemetry architecture overview: every individual service proxy has a report buffer where it batches telemetry that it flushes periodically to Mixer’s report API.

As noted, telemetry reports are generated as a service proxy processes requests. Requests come in the form of client–server interactions. Consider that requests are created by a client and sent to a service, which in turn can initiate a request (as a client) to another service (server). Both client and server services are capable of sending telemetry reports about any given request that they have handled. The default configuration of Istio releases after version 1.1 is to only have the server send the telemetry report so that only one report will be sent in. This is configurable, however, to the extent that both client and server-side request reports are desired.

Sending Reports

Service proxies send information to Mixer about requests and responses in the form of attributes (typed key/value pairs). Mixer transforms attribute sets into structured values, according to an operator-supplied configuration. Mixer dispatches the derived values to a set of adapters, according to the operator-supplied configuration. Adapters publish telemetry data to backend systems, making it available for further consumption and analysis. Attributes are primarily generated by service proxies; however, Mixer adapters also can produce attributes.

Checking Caches

After Mixer makes an initial policy verdict, the attribute protocol comes into play. Envoy and Mixer use well-known attributes to describe the policy used to evaluate service requests. When Mixer makes a verdict on a request, it computes a hash key using these attributes. Envoy uses these same attributes, reducing the overhead on request latencies since both Mixer and Envoy operate as caches once verdicts are delivered. Mixer creates a hash key using the attribute keys. A goal of Envoy’s configuration is to strike the balance of cache hits with superfluous sets of hash keys.

Check caches include a TTL value indicating the maximum amount of time a cached check result should be trusted. Caches need to be refreshed as Mixer’s configuration is bound to change over time, or in turn, backend that Mixer consults when deciding a check result may equally as likely change and cause service proxies to need their check cache refreshed. Again, in this way, the service proxies function as first-level caches and Mixer functions as a second-level shared cache.

In-Process Adapters

In-process adapters are written in Go and compiled into the Mixer process. Again, whether the adapter is implicated based on whether they are configured to be enacted through a handler, instance, and rule(s). Each of the adapters that follow is built into the Mixer binary, hence each Istio release:

Precondition checks

denier, listchecker, memquota, opa, rbac, redisquota

Telemetry

circonus, cloudwatch, dogstatsd, fluentd, prometheus, solarwinds, stackdriver, statsd, stdio

Attribute generation

kubernetesenv

To create a new in-process adapter, vendors implement a set of Golang interfaces and submit their adapter for review and consideration of inclusion in the Istio project.

Out-of-Process Adapters

Adapters were initially built into the main as in-process adapters but are moving to an out-of-process model in which adapter code isn’t kept in the Istio project, but kept and managed separately with the vendor. Out-of-process adapters interface via a gRPC service that implements template infrastructure backend protocol and runs outside the Mixer process (can be coprocess adapter or backend service). In moving to an out-of-process adapter model, Istio eliminates the need for custom Mixer builds to include or exclude specific adapters. Adapter authors are empowered to write their adapters in the language of their choice given the abstraction through gRPC. In moving to an out-of-process model, no longer will Mixer share a common fate with its adapters, which will run independently in their own process(es).

Mixer Configuration

Service operators control all operational and policy aspects of a Mixer deployment by manipulating configuration resources. Configuring Mixer includes describing which adapters are to be used, how they should operate, which request attributes to map to which adapter’s inputs, and when a particular adapter is invoked with specific inputs. Mixer configuration is manipulated and represented through Kubernetes custom resources—rules, handlers, instances, and adapters.

adapters encapsulate the logic necessary to interface with backends. Adapter configuration schema is specified by adapter packages. Configuration contains operational parameters needed by the adapter code to do work. A handler is an instantiation of an adapter—a configured adapter. Handlers can receive data (attributes). An instance is an object full of request data. Instances are a set of structured request attributes with well-known fields. Instances map request attributes to values that are passed to adapter code. Attribute mapping is controlled via attribute expressions.

As an example of an attribute expression, consider the Prometheus instance for requestduration in Example 9-1. In this example, requestduration is configured to return a response.code of 200 in the absence of a response.code. If destination.service isn’t present, the report will simply fail.

$ kubectl -n istio-system get metrics requestduration -o yaml
apiVersion: config.istio.io/v1alpha2
kind: metric
metadata:
  name: requestduration
  namespace: istio-system
spec:
  dimensions:
    destination_app: destination.labels["app"] | "unknown"
    destination_principal: destination.principal | "unknown"
    destination_service: destination.service.host | "unknown"
    destination_service_name: destination.service.name | "unknown"
    request_protocol: api.protocol | context.protocol | "unknown"
    response_code: response.code | 200
...

Rules specify when a particular handler is invoked with a specific instance, and map handlers and instances to each other. Rules essentially enforce that when a given condition is true, it’s given a particular handler specific instances (request attributes).

Rules contain a match predicate attribute expression and a list of actions to perform if the predicate evaluates to true. Rules evaluate to true for all requests if a match isn’t specified. This behavior is valuable when accounting for quota enforcement, as demonstrated in Example 9-2, in this rule snippet for a memquota adapter (a precondition check type adapter). In the following example, a rule with no match condition perpetually evaluates to true each time it is evaluated, and therefore, increments the requestcount.quota:

...
spec:
  actions:
  - handler: handler.memquota
    instances:
    - requestcount.quota
...

Open Policy Agent Adapter

The Open Policy Agent (OPA) is a general-purpose policy engine used to offload authorization decisions. OPA uses Rego as its declarative policy language. It is implemented in Go and can be deployed as a library or as a daemon.

Mixer’s OPA adapter is a check type adapter. The Mixer security model for check type adapters is to fail closed to ensure security is maintained. Any random policy that you can do with OPA, you can do with the OPA Mixer adapter. The adapter bundles up the OPA runtime. Attributes get fed to the Rego language engine for processing by what is a full OPA instance in the adapter.

Consider this example set of configurations for an OPA adapter’s rule, handler, and instance, as presented in Examples 9-3 through 9-5.

apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
  name: authz
spec:
  actions:
  - handler: opa-handler
    instances:
    - authz-instance

apiVersion: config.istio/v1alpha2
kind: authz
metadata:
  name: authz-instance
spec:
  subject:
    user: source.uid | ""
  action:
    namespace: target.namespace | "default"
    service: target.service | ""
    path: target.path | ""
    method: request.method | ""

apiVersion: config.istio.io/v1alpha2
kind: opa
metadata:
  name: opa-handler
spec:
  checkMethod: authz.allow
  policy: |
    package authz
    default allow = false
    allow { is_read }
    is_read { input.action.method = "GET" }

Prometheus Adapter

The Prometheus adapter is built into the Mixer binary and enabled by default with a metrics expiration duration of 10 minutes. The Prometheus adapter defines a custom resource; metrics, as shown in Example 9-6.

$ kubectl -n istio-system get metrics
NAME                   AGE
requestcount           25h
requestduration        25h
requestsize            25h
responsesize           25h
tcpbytereceived        25h
tcpbytesent            25h
tcpconnectionsclosed   25h
tcpconnectionsopened   25h

The Prometheus handler needs to know the specific dimensions and type of metrics. This is called an instance in general, but there are a few special instances that have a proper name; metric is one of those. Typically, an adapter is built to expect certain instances. We can see in Example 9-7 the set of instances that the Prometheus adapter consumes.

$ kubectl -n istio-system get metrics tcpbytereceived -o yaml

apiVersion: config.istio.io/v1alpha2
kind: metric
metadata:
  labels:
    app: mixer
    release: istio
  name: tcpbytereceived
  namespace: istio-system
spec:
  dimensions:
    connection_security_policy: conditional((context.reporter.kind | "inbound") ==
      "outbound", "unknown", conditional(connection.mtls | false, "mutual_tls",
      "none"))
    destination_app: destination.labels["app"] | "unknown"
    destination_principal: destination.principal | "unknown"
    destination_service: destination.service.host | "unknown"
    destination_service_name: destination.service.name | "unknown"
    destination_service_namespace: destination.service.namespace | "unknown"
    destination_version: destination.labels["version"] | "unknown"
    destination_workload: destination.workload.name | "unknown"
    destination_workload_namespace: destination.workload.namespace | "unknown"
    reporter: conditional((context.reporter.kind | "inbound") == "outbound",
      "source", "destination")
    response_flags: context.proxy_error_code | "-"
    source_app: source.labels["app"] | "unknown"
    source_principal: source.principal | "unknown"
    source_version: source.labels["version"] | "unknown"
    source_workload: source.workload.name | "unknown"
    source_workload_namespace: source.workload.namespace | "unknown"
  monitored_resource_type: '"UNSPECIFIED"'
  value: connection.received.bytes | 0

Mixer needs to know when to generate this metric data and send it to Prometheus. This is defined as a rule. Every rule has a match condition that is evaluated; if the match is true, the rule is triggered. For example, we could use the match to receive only HTTP data, only TCP data, and so on. Prometheus does exactly this, as shown in Example 9-8, and defines a rule for each set of protocols for which it has metric descriptions.

$ kubectl -n istio-system get rules
NAME                      AGE
kubeattrgenrulerule       25h
promhttp                  25h
promtcp                   25h
promtcpconnectionclosed   25h
promtcpconnectionopen     25h
stdio                     25h
stdiotcp                  25h
tcpkubeattrgenrulerule    25h

And, again, let’s inspect one to see what it looks like:

$ kubectl -n istio-system get rules promtcpconnectionopen -o yaml

apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
  annotations:
  ...
  generation: 1
  name: promtcpconnectionopen
  namespace: istio-system
spec:
  actions:
  - handler: prometheus
    instances:
    - tcpconnectionsopened.metric
  match: context.protocol == "tcp" && ((connection.event | "na") == "open")

In Chapter 10, all of these metrics are exposed to backends for analysis, visualization, alerting and so on.

As our multiple personality control-plane component, Mixer enables service operator control over policy decisions and telemetry dispatch based on configuration and acts as the point of integration between Istio and infrastructure backends. Through two services (istio-policy and istio-telemetry), Mixer provides the following core features:

• Precondition checking (ACLs, authentication)

• Quota management (rate limits)

• Telemetry reporting (metrics, logs, traces)

Mixer’s performance overhead can be quite high depending on how you configure your Istio deployment. Mixer provides aggressive caching and reduces observed latencies, and offers mediation for service operators to control policy enforcement and telemetry collection. With its backend abstractions, it reduces systemic complexity and the adapter model enables backend mobility.