Custom Resource Definitions

With a CRD, we can extend Kubernetes to manage our domain concepts on the Kubernetes platform. Custom resources are managed like any other resource, through the Kubernetes API, and eventually stored in the backend store Etcd. Historically, the predecessors of CRDs were ThirdPartyResources.

The preceding scenario is actually implemented with these new custom resources by the CoreOS Prometheus operator to allow seamless integration of Prometheus to Kubernetes. The Prometheus CRD is defined as in Example 23-1, which also explains most of the available fields for a CRD.

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: prometheuses.monitoring.coreos.com     
spec:
  group: monitoring.coreos.com                 
  names:
    kind: Prometheus                           
    plural: prometheuses                       
  scope: Namespaced                            
  version: v1                                  
  validation:
    openAPIV3Schema: ....                      

An OpenAPI V3 schema can also be specified to allow Kubernetes to validate a custom resource. For simple use cases, this schema can be omitted, but for production-grade CRDs, the schema should be provided so that configuration errors can be detected early.

Additionally, Kubernetes allows us to specify two possible subresources for our CRD via the spec field subresources:

scale

With this property, a CRD can specify how it manages its replica count. This field can be used to declare the JSON path, where the number of desired replicas of this custom resource is specified: the path to the property that holds the actual number of running replicas and an optional path to a label selector that can be used to find copies of custom resource instances. This label selector is usually optional, but is required if you want to use this custom resource with the HorizontalPodAutoscaler explained in Chapter 24, Elastic Scale.

status

When we set this property, a new API call is available that only allows you to change the status. This API call can be secured individually and allows for status updates from outside the controller. On the other hand, when we update a custom resource as a whole, the status section is ignored as for standard Kubernetes resources.

Example 23-2 shows a potential subresource path as is also used for a regular Pod.

kind: CustomResourceDefinition
# ...
spec:
  subresources:
    status: {}
    scale:
      specReplicasPath: .spec.replicas 
      statusReplicasPath: .status.replicas 
      labelSelectorPath: .status.labelSelector 

Once we define a CRD, we can easily create such a resource, as shown in Example 23-3.

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
spec:
  serviceMonitorSelector:
    matchLabels:
      team: frontend
  resources:
    requests:
      memory: 400Mi

The metadata: section has the same format and validation rules as any other Kubernetes resource. The spec: contains the CRD-specific content, and Kubernetes validates against the given validation rule from the CRD.

Custom resources alone are not of much use without an active component to act on them. To give them some meaning, we need again our well-known Controller, which watches the lifecycle of these resources and acts according to the declarations found within the resources.

Controller and Operator Classification

Before we dive into writing our Operator, let’s look at a few kinds of classifications for Controllers, Operators, and especially CRDs. Based on the Operator’s action, broadly the classifications are as follows:

Installation CRDs

Meant for installing and operating applications on the Kubernetes platform. Typical examples are the Prometheus CRDs, which we can use for installing and managing Prometheus itself.

Application CRDs

In contrast, these are used to represent an application-specific domain concept. This kind of CRD allows applications deep integration with Kubernetes, which involves combining Kubernetes with an application-specific domain behavior. For example, the ServiceMonitor CRD is used by the Prometheus operator to register specific Kubernetes Services for being scraped by a Prometheus server. The Prometheus operator takes care of adapting the Prometheus’ server configuration accordingly.

In our categorization of Controller and Operator, an Operator is-a Controller that uses CRDs.¹ However, even this distinction is a bit fuzzy as there are variations in between.

One example is a controller, which uses a ConfigMap as a kind of replacement for a CRD. This approach makes sense in scenarios where default Kubernetes resources are not enough, but creating CRDs is not feasible either. In this case, ConfigMap is an excellent middle ground, allowing encapsulation of domain logic within the content of a ConfigMap. An advantage of using a plain ConfigMap is that you don’t need to have the cluster-admin rights you need when registering a CRD. In certain cluster setups, it is just not possible for you to register such a CRD (e.g., like when running on public clusters like OpenShift Online).

However, you can still use the concept of Observe-Analyze-Act when you replace a CRD with a plain ConfigMap that you use as your domain-specific configuration. The drawback is that you don’t get essential tool support like kubectl get for CRDs; you have no validation on the API Server level, and no support for API versioning. Also, you don’t have much influence on how you model the status: field of a ConfigMap, whereas for a CRD you are free to define your status model as you wish.

Another advantage of CRDs is that you have a fine-grained permission model based on the kind of CRD, which you can tune individually. This kind of RBAC security is not possible when all your domain configuration is encapsulated in ConfigMaps, as all ConfigMaps in a namespace share the same permission setup.

From an implementation point of view, it matters whether we implement a controller by restricting its usage to vanilla Kubernetes objects or whether we have custom resources managed by the controller. In the former case, we already have all types available in the Kubernetes client library of our choice. For the CRD case, we don’t have the type information out of the box, and we can either use a schemaless approach for managing CRD resources, or define the custom types on our own, possibly based on an OpenAPI schema contained in the CRD definition. Support for typed CRDs varies by client library and framework used.

Figure 23-1 shows our Controller and Operator categorization starting from simpler resource definition options to more advanced with the boundary between Controller and Operator being the use of custom resources.

Figure 23-1. Spectrum of Controllers and Operators

For Operators, there is even a more advanced Kubernetes extension hook option. When Kubernetes-managed CRDs are not sufficient to represent a problem domain, you can extend the Kubernetes API with an own aggregation layer. We can add a custom implemented APIService resource as a new URL path to the Kubernetes API.

To connect a given Service with name custom-api-server and backed by a Pod with your service, you can use a resource like that shown in Example 23-4.

apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1alpha1.sample-api.k8spatterns.io
spec:
  group: sample-api.k8spattterns.io
  service:
    name: custom-api-server
  version: v1alpha1

Besides the Service and Pod implementation, we need some additional security configuration for setting up the ServiceAccount under which the Pod is running.

Once it is set up, every request to the API Server https://<api server ip>/apis/sample-api.k8spatterns.io/v1alpha1/namespaces/<ns>/... is directed to our custom Service implementation. It’s up to this custom Service’s implementation to handle these requests, including persisting the resources managed via this API. This approach is different from the preceding CRD case, where Kubernetes itself completely manages the custom resources.

With a custom API Server, you have many more degrees of freedom, which allows going beyond watching resource lifecycle events. On the other hand, you also have to implement much more logic, so for typical use cases, an operator dealing with plain CRDs is often good enough.

A detailed exploration of the API Server capabilities is beyond the scope of this chapter. The official documentation as well as a complete sample-apiserver have more detailed information. Also, you can use the apiserver-builder library, which helps with implementing API Server aggregation.

Now, let’s see how we can develop and deploy our operators with CRDs.

Operator Development and Deployment

At the time of this writing (2019), operator development is an area of Kubernetes that is actively evolving, with several toolkits and frameworks available for writing operators. The three main projects aiding in the creation of operators are as follows:

• CoreOS Operator Framework

• Kubebuilder developed under the SIG API Machinery of Kubernetes itself

• Metacontroller from Google Cloud Platform

We touch on them very briefly next, but be aware that all of these projects are quite young and might change over time or even get merged.

Example

Let’s have a look at a concrete Operator example. We extend our example in Chapter 22, Controller and introduce a CRD of type ConfigWatcher. An instance of this CRD specifies then a reference to the ConfigMap to watch and which Pods to restart if this ConfigMap changes. With this approach, we remove the dependency of the ConfigMap on the Pods, as we don’t have to modify the ConfigMap itself to add triggering annotations. Also, with our simple annotation-based approach in the Controller example, we can connect only a ConfigMap to a single application, too. With a CRD, arbitrary combinations of ConfigMaps and Pods are possible.

This ConfigWatcher custom resource looks like that in Example 23-5.

kind: ConfigWatcher
apiVersion: k8spatterns.io/v1
metadata:
  name: webapp-config-watcher
spec:
  configMap: webapp-config    
  podSelector:                
    app: webapp

In this definition, the attribute configMap references the name of the ConfigMap to watch. The field podSelector is a collection of labels and their values, which identify the Pods to restart.

We can define the type of this custom resource with a CRD, as shown in Example 23-6.

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: configwatchers.k8spatterns.io
spec:
  scope: Namespaced                   
  group: k8spatterns.io               
  version: v1                         
  names:
    kind: ConfigWatcher               
    singular: configwatcher           
    plural: configwatchers
  validation:
    openAPIV3Schema:                  
      properties:
        spec:
          properties:
            configMap:
              type: string
              description: "Name of the ConfigMap"
            podSelector:
              type: object
              description: "Label selector for Pods"
              additionalProperties:
                type: string

For our operator to be able to manage custom resources of this type, we need to attach a ServiceAccount with the proper permissions to our operator’s Deployment. For this task, we introduce a dedicated Role that is used later in a RoleBinding to attach it to the ServiceAccount in Example 23-7.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: config-watcher-crd
rules:
- apiGroups:
  - k8spatterns.io
  resources:
  - configwatchers
  - configwatchers/finalizers
  verbs: [ get, list, create, update, delete, deletecollection, watch ]

With these CRDs in place, we can now define custom resources as in Example 23-5.

To make sense of these resources, we have to implement a controller that evaluates these resources and triggers a Pod restart when the ConfigMap changes.

We expand here on our Controller script in Example 22-2 and adapt the event loop in the controller script.

In the case of a ConfigMap update, instead of checking for a specific annotation, we do a query on all resources of kind ConfigWatcher and check whether the modified ConfigMap is included as a configMap: value. Example 23-8 shows the reconciliation loop. Refer to our Git repository for the full example, which also includes detailed instructions for installing this operator.

curl -Ns $base/api/v1/${ns}/configmaps?watch=true |      
while read -r event
do
  type=$(echo "$event" | jq -r '.type')
  if [ $type = "MODIFIED" ]; then                         

    watch_url="$base/apis/k8spatterns.io/v1/${ns}/configwatchers"
    config_map=$(echo "$event" | jq -r '.object.metadata.name')

    watcher_list=$(curl -s $watch_url | jq -r '.items[]') 

    watchers=$(echo $watcher_list |                      
               jq -r "select(.spec.configMap == "$config_map") | .metadata.name")

    for watcher in watchers; do                           
      label_selector=$(extract_label_selector $watcher)
      delete_pods_with_selector "$label_selector"
    done
  fi
done

As for the Controller example, this controller can be tested with a sample web application that is provided in our example Git repository. The only difference with this Deployment is that we use an unannotated ConfigMap for the application configuration.

Although our operator is quite functional, it is also clear that our shell script-based operator is still quite simple and doesn’t cover edge or error cases. You can find many more interesting, production-grade examples in the wild.

Awesome Operators has a nice list of real-world operators that are all based on the concepts covered in this chapter. We have already seen how a Prometheus operator can manage Prometheus installations. Another Golang-based operator is the Etcd Operator for managing an Etcd key-value store and automating operational tasks like backing up and restoring of the database.

If you are looking for an operator written in the Java programming languages, the Strimzi Operator is an excellent example of an operator that manages a complex messaging system like Apache Kafka on Kubernetes. Another good starting point for Java-based operators is the JVM Operator Toolkit, which provides a foundation for creating operators in Java and JVM-based languages like Groovy or Kotlin and also comes with a set of examples.