Chapter 8: Knowing the Trade-offs

In the previous few chapters, we learned a lot about Crossplane, from its basics to many advanced patterns. Also, we introduced the idea of a unified approach to both application and infrastructure automation. This chapter will step back to analyze and approach configuration management for unified automation holistically. The chapter is heavily influenced by the white paper Declarative application management in Kubernetes (https://goo.gl/T66ZcD) by Brian Grant, a Kubernetes Steering Committee emeritus. The white paper mainly covers the Kubernetes Resource Model (KRM). We will cover the topics in this chapter from the KRM and Crossplane Resource Model (XRM) perspectives. The chapter will define the scope of the unified automation approach. It will continue by looking into many more concepts, such as the tools available, common pitfalls along the way, and the trade-off in using different patterns. In a way, it’s a revisit of the API boundaries discussion from Chapter 6, in more detail.

The following are the topics covered in the chapter:

• Unified automation scope
• Complexity clock, requirements, and patterns
• Open Application Model
• Specialized and extendable abstraction
• Impact of change frequency

Unified automation scope

Most of us will have different perceptions of what we mean by unified application and infrastructure automation. This section will help us understand the scope with a bit more clarity. Any application/product that we deploy is mostly a combination of custom-developed bespoke applications and common off-the-shelf (COTS) components. The term bespoke application means custom-written software for our requirements with a specific purpose. Bespoke applications are generally stateless, containerized workloads. From the perspective of Kubernetes, they are workloads that run on Pods, the basic unit of computing in Kubernetes. COTS components are generally stateful infrastructure components, such as databases, cache systems, storage systems, and messaging systems. In the cloud-native era, most COTS components are black-box, fully managed software as a service (SaaS) or platform as a service (PaaS). COTS vendors expose CRUD APIs to work with the resource with precise configuration knobs for security, scaling, monitoring, and updating concerns. These specific configuration knobs will support different use cases from consumers. When we talk about unified application and infrastructure automation, it’s an approach to use the same set of tools, processes, and patterns to configure both bespoke applications and COTS deployment. In other words, bespoke applications can follow KRM standards, and COTS dependencies can comply with XRM standards, an extension of KRM. There are many advantages to such unification:

• In the modern software engineering discipline, the product teams are vertically sliced to bring maximum delivery velocity. These vertically sliced teams own both bespoke applications and their dependent COTS components. Unified tooling, processes, and patterns will significantly reduce the cognitive load.
• It can bring down the need for specialized teams to manage COTS components, accelerating the delivery velocity further.
• The configuration data for policies in bespoke applications and their COTS dependencies can be quite simple. We can validate policies such as architecture fitness functions, security concerns, and compliance standards in a single place and format.
• All COTS vendors can offer KRM-compliant APIs (MRs) as a universal application dependency and integration standard. It is already happening with Crossplane providers for all primary cloud resources. The list is growing to cover other external resources, such as Git Lab, Argo CD, Rook, and Cloudflare, to cover end-to-end automation.

The following section will cover a few requirements, patterns, and tools for approaching unified automation.

Complexity clock, requirements, and patterns

The configuration complexity clock is a concept that explains how configuration management can become complex over time. It explores different stages in the evolution of configuration management, its use cases, and its pitfalls. It was initially discussed from the perspective of application configuration in the blog post found here: http://mikehadlow.blogspot.com/2012/05/configuration-complexity-clock.html. We will look at the same concept from the Kubernetes configuration management perspective.

The configuration complexity clock

Let’s say we are deploying our first application workload into Kubernetes. To start with, every Kubernetes artifact, such as Deployment, Service, and Ingress, can be managed as individual configuration artifacts. It may not be an issue when we operate on a tiny scale. But soon we will realize that there is an issue with consistently performing releases and rollbacks as the application configuration is spread over multiple Kubernetes resources. We will start thinking about packaging and release management tools such as Helm. Helm is a template-based configuration management tool. We will parameterize values with variables to enable configuration abstraction and expose only limited attributes. A typical example is setting the replica count for every deployment target (production and staging). Soon, the organization will realize the power of Kubernetes and decide to deploy further similar workloads in the Kubernetes ecosystem. As the configuration data for the new application looks identical to the initial Helm template, we will choose to parametrize more variables to support multiple applications with the same Helm template. It will be an excellent way to push the reuse agenda and minimize the configuration to maintain. A typical example of such new parameters would be a namespace, labels, and an application name.

Again, the organization will realize more benefits from Kubernetes and decide to experiment with a few new workloads. These workloads may have similar configuration data to the initial Helm template with minor customization. We would decide to fork the main Helm template to do the required customization. Template-based abstractions are complex to reuse when customization is required. After a certain length of time of the configuration clock running, we will see too many forks that are difficult to keep in sync. On the other side, many individual templates would have introduced new parameters to support new local use cases. This is a leak in the abstraction we created with the Helm templates. We would have parameterized all values and completely eroded abstraction as the clock ticks further. Have a look at this WordPress chart as an example: https://github.com/helm/charts/tree/master/stable/wordpress. The number of parameters has increased slowly into multiple folds from the initial commit. Entirely eroded templates are complex to read, and end users will find it challenging to understand the usage of each parameter. Users will find it challenging to see parameters that are not their concern. For example, developers may not know how to define Ingress parameters in the preceding WordPress Helm chart.

As the configuration clock ticks further, we must harden the configurations for security. Infrastructure operators will want to own the configuration as the application owners do not know how to configure the security parameter. The Helm post rendering feature can rescue us from the situation and help inject the security configuration as late binding. But it will still be challenging for the infrastructure operators to manage the post-rendering step in the deployment pipeline because of too many customization forks and unexpected rendering outputs. Some developers may decide to use a DSL such as Terraform to manage configuration as it is simple to read. DSLs inherit the same parameterization problem in input variables, and the post-rendering customization step is also challenging with DSL. Additionally, DSLs have the issue of a complex learning curve for developers and limited tooling support for concerns such as testing. Vulnerability or compliance scanning is another area where we may face tooling issues. The Crossplane configuration can also face similar problems as we scale over time. The idea of discussing these limitations is not to discourage any specific tool usage. Each tool has its use cases and trade-offs. For example, Helm packing works well when operating on a small scale and a few other use cases, such as release management. In a way, we will be looking at how we can use a combination of tools and patterns to overcome this limitation.

Configuration management requirements

To manage configurations without being trapped by the issues discussed regarding the complexity clock, we should keep a few guiding principles in mind. These guiding principles will be the technical requirements to perform trade-off analysis when selecting tools and patterns for configuration management. Baking these guiding principles into our configuration management recipes would allow us to evolve and scale with ease:

• Keep the configuration data readable for humans and machines to mutate along the configuration pipeline. Requirements such as environment-specific patching, security configuration patches, and policy as the configuration are best implemented as an automated step in the pipeline. Avoid maintaining configuration as code as they are challenging to manipulate. Rendered output from code may not always be straightforward for the machines to read.
• Configuration scanning requirements, such as audit, compliance, and security, work well with clean configuration as data. Keeping the configuration as data throughout the pipeline as much as possible and separating code that manipulates the configuration is vital to meet the evolving requirements. Many tools evolved around the Kubernetes ecosystem because configuration data is kept separate (in etcd) from code (controller) and has a standard data format with KRM.
• Segregation of concerns is another critical aspect of configuration management. Not all configuration knobs required to automate are meant for a single persona to define. We should manage the configuration so that different people can collaborate with ease.
• Build specialized abstractions to support customization with the reuse of configuration. For example, Pod is a fundamental configuration abstract over which Deployment, ReplicationController, DaemonSet, and so on are built. Later in the chapter, we will see more solid examples from the Crossplane and complete application management perspective.
• Version control the source configuration and the final mutated configuration, representing the desired state. Avoid modification to the configuration post applying to the cluster. Tools such as mutating admission controllers should be avoided. Late configuration binding may fail in ways that are hard to predict (still, the admission controller is suitable for policy validation).
• Use composing to bind the application and its dependent resource into a single API while maintaining separation of concerns with nested sub-APIs. Facilitate release management concerns such as discovery, life cycle management, and dependency management for the entire application bundle.

We will deep dive into all these guiding principles in the upcoming sections. The following section will examine different patterns available for configuration management and the trade-offs with each pattern.

Patterns and trade-off

Reusing the configuration as we scale is challenging. Most of the reuse patterns were discussed in the The configuration complexity clock section. The section will cover these patterns in more detail with their advantages and disadvantages. The following are the well-known approaches for configuration reuse:

• Fork: This is one of the frequently used methods to reuse configuration, primarily because of the ease of customization. Rebase is the strategy used to sync the configuration as things evolve. The rebasing process cannot be automated as humans must address merge conflicts. The method offers a high level of flexibility and comfort of maintenance in the short term. As time passes by, a couple of challenges arise. The forks will diverge a lot with time, creating challenges with rebasing. With independent agile teams managing different forks, the advantages of reuse can be overlooked to keep up with the evolving speed. I had a similar experience a year back when I decided to fork a code repository for customized deployment requirements. Different agile teams owned both forks. As the clock ticked further, code merge activity was ignored entirely, yielding to the delivery pressure. Finally, we ended up in a state where the two forks could never sync with thousands of conflicts. The fork solution works best when you quickly try a proof of concept. I will not recommend using them beyond that.
• Template parameterization: We discussed template-based parameterization in Chapter 2, and earlier in this chapter. It works well on a small scale but suffers leaking abstraction and team collaboration issues as we scale. Additionally, it will also push us toward using the fork pattern by making customization complex. Tools such as Helm fall under this category. As a template-based parameterization tool, Helm is extremely popular because of its other capabilities, such as application discovery and release management.
• Patch/overlay: This method would have the base configuration as pure data points and use the patch file to overlay the required variables for customization. We can replace values for an existing configuration knob or add a new configuration to the overall base. We can also use the technique as a post-rendering step in template-based abstraction tools such as Helm. The leaks are avoided as additional parameterization required for customization can be managed as a post-rendering patch. It’s a popular method that is being used quite a bit at the moment, especially since the Kustomize tool came into existence. These tools do not require human intervention and can be automated as a step in the deployment pipeline.
• Composition: It is a technique where we compose the dependencies to build a higher-level API and multiple sub-APIs consumed by different persona. We have discussed this pattern well enough as Crossplane itself is a composing tool. Pull and push are two sub-patterns within the composition. In the pull model, the dependencies are directly referred to. Nested XR is a typical example from the Crossplane world. With the push model, the dependencies are referred across API indirectly through runtime binding. The push pattern is suitable for separating concerns and building extendable abstractions. Resource references between two independent XRs using labels is a push composition example. Composition and patch/overlay work well together to create customization. For instance, in Crossplane, we can use a patch to generate environment-specific composition (production/staging).

Composition, patch/overlay, and template-based tools can complement each other and combining them can help you to build a robust deployment pipeline. We will look at a few such hands-on examples in the following two chapters. The next section of this chapter will look at the Open Application Model (OAM), an open source standard to define application and team-centric configuration management.

Open Application Model

OAM is an open source standard developed by Microsoft and Alibaba to guide us in creating higher-level abstraction APIs for application deployment. In other words, it’s about creating a model for application configuration management using composition as a pattern. The standard focus on addressing the following three problems:

• Developer focus: Exposing developers directly to Kubernetes configuration management to deploy the applications will make them spend time figuring out infrastructure details rather than application focus. Hence, OAM attempts to keep developers focused on the application.
• Vendor dependency: Configuring applications usually tends to depend on the underlying infrastructure. Completely decoupling the application configuration from the underlying infrastructure can enable the portability of workloads. Kubernetes does this to an extent, but the area requires more work with cross-cutting concerns and COTS dependencies.
• Extendibility: Configuration management at scale can have many problems, especially while balancing reuse and customization. OAM proposed a model for reuse and customization for scale.

OAM proposes layered configuration management based on personas and composing abstractions accordingly to address the problems. OAM defines three personas to manage different application deployment concerns, as represented in the following figure:

Figure 8.1 – OAM persona

KubeVela, the OAM implementation

The OAM community has developed KubeVela, a project that implements the specification. It is a CNCF sandbox project and a composing tool like Crossplane. KubeVela concentrates only on composing bespoke application configurations. But Crossplane composes both bespoke applications and COTS infrastructure/external dependencies. Both KubeVela and Crossplane can complement each other using the following two different patterns:

• KubeVela composes Crossplane: We can use KubeVela to compose bespoke applications deployment abstractions, and it can rely on Crossplane as the COTS external dependency provider. This pattern requires the Crossplane control plane to be present in the application workload cluster.
• Crossplane composes KubeVela: We can use Crossplane to compose abstractions for both bespoke applications and COTS infrastructure/external dependencies. For bespoke applications, it can use KubeVela through Crossplane Provider for Kubernetes. This pattern can work with a centralized Crossplane control plane or a distributed Crossplane control plane.

The following figure represents the KubeVela composes Crossplane pattern where both the Crossplane control plane and application workload cluster are the same:

Figure 8.2 – KubeVela composes Crossplane

The preceding figure represents a web service workload with an RDS database as a COTS dependency. The workload construct of KubeVela defines the application workload type. Traits define all workload characteristics, such as route and traffic. Components help define the COTS dependencies. First, we will compose RDS into an XR API using Crossplane. Later, we can compose the XR as a component inside KubeVela. The following figure represents the second pattern where Crossplane composes KubeVela:

Figure 8.3 – Crossplane composes KubeVela

The figure represents application management both in the same cluster and in a remote cluster. Additionally, the figure represents the usage of the Helm provider to set up KubeVela and other cluster dependencies. We will cover a hands-on example of KubeVela in Chapter 9.

Information

OAM and KubeVela are exciting topics to explore further. Visit https://oam.dev/ for more information.

The following section will cover two exciting patterns that can help platform teams to build extendable and reusable platform APIs.

Specialized and extendable abstraction

As we scale the number of applications deployed in Kubernetes, there could be an exponential proliferation of configurations to manage. Managing a high volume of KRM/XRM configuration files is prone to human error, is challenging to keep in sync, and requires a highly skilled workforce. Reuse is the key to keeping the volume of configurations low. But customization requirements at the individual team level will not allow us to reuse unless we have an easy and quick way to do so. Also, agile and product engineering practices will add additional pressure from a minimal external dependency perspective. Specialized and extendable abstraction is vital to address these problems. Let’s start with specialized abstraction in the following section.

Specialized abstraction

Specialized abstraction is a technique where we build a basic abstraction and reuse the base abstraction to make specialized abstractions that handle custom requirements. It is nothing but the inheritance pattern from object-oriented programming. Also, this is a well-known pattern in core Kubernetes. Think of the Pod resource type as an example. Deployment, DemonSet, and Job are specialized abstractions that use Pod as their base. Let’s look at some of the specialized abstractions we could build as a platform team:

• Policy abstraction layer: We may have policies per cloud resource as an organization. These policies could be from diverse areas, such as security, architecture, and compliance. If this is how the cloud strategy works in your organization, consider creating an abstract layer above the Crossplane provider with an XR for each resource. The layer can act as a solid base for creating recipes.
• Database recipes: Different product teams will have different database requirements. Some product teams will demand a geographically distributed database, and others may be happy with availability zone fault tolerance in a single region. A third team may require a PCI-DSS-compliant database to store payment transactions. We could create a base database XR and create a specialized XR above the base to fit each need.
• Workload recipes: Not every workload is the same. Web applications, RESTful APIs, scheduled jobs, and data transformation are some examples of workloads, and each will have different dependency requirements. For example, a scheduled job workload does not require Ingress. We create a primary workload with all common cross-cutting concerns and develop a specialized abstraction over the primary workload. We can do this with Crossplane or KubeVela or a mix of both.

These are just some general examples that may apply to your environment. Finding the correct abstractions that suit your environment will require a closer look at the organization structure, technical maturity, governance model, workloads, and so on. The base and specialized abstractions are typically developed by the platform team to be consumed by the application operator and developers. We may have a rare scenario where the application operator is building a specialized abstraction over the base developed by the platform team. We can look at extendable abstraction in the following section.

Extendable abstraction

Extendable abstraction is a technique where we build a partial abstraction and allow other teams to implement it fully. It is nothing but the polymorphism pattern from object-oriented programming. Also, this is a well-known pattern in core Kubernetes. Ingress is an example from core Kubernetes. Each managed Kubernetes provider will come up with its implementation for Ingress. Extendable abstraction is meant for both platform teams and application operators. Let’s look at some examples of the pattern’s usage:

• Shared resources: Let’s consider VPC as an example of a shared resource. Multiple variants of the resources sometimes need to be created and used for different scenarios. We can have a standard label name/value strategy for such resources, and other XR recipes can choose one VPC with appropriate label references. This is polymorphic behavior and provides extendibility through dependency injection.
• Nested XR: The platform teams can create an XR recipe with dependencies missed identified. The missing dependencies can be implemented separately. Both the pieces can be composed with a nested XR pattern. Choose this pattern when the requirement for that missed dependency changes with every product team. An application recipe that leaves the database choice open is an example of nested XR behaving in a polymorphic way. The application operator can complete the recipe based on the specific product team’s requirements.

Again, these are indicative examples. This concludes the abstraction discussion. In the following section of the chapter, we will look at configuration change frequency in detail.

Impact of change frequency

Looking at the configuration knobs from the perspective of frequency of change will help us categorize them between personas defined in the OAM. The change frequency detailing will bring in the discussion of ownership as these perspectives are linked. This section covers change frequency from the perspective of bespoke applications (KRM compliant) and COTS external dependencies (XRM compliant). We can classify configuration change frequency into the following three categories:

• Change very often: Generally, application KRM configuration parameters, such as image tags, release version, environment variables, secrets reference, and application configuration, change very frequently. These configurations are application-centric and owned primarily by the application developers. If we use a template- or DSL-based abstraction, they are good candidates to be exposed as variables. Suppose a plain configuration YAML is used instead of a template- or DSL-based abstraction, developers can own a version-controlled patch/overlay file. When composing is used as the solution, these configurations should be exposed to the developer with a high-level API.
• Less frequent change: Configurations such as application context-related information (namespaces, labels, references, and so on), resource constraints (memory, volume, and so on), and release-related knobs (replica count and rollout method) are examples of less frequently changing configurations. The preceding-mentioned configurations mainly vary per environment or change when there is a new operational requirement. For example, the number of replicas, namespace, or labels can change based on the deployment environment (production, staging, and so on). Irrespective of using template-based or DSL-based, or a plain YAML or composing, it’s best to use patching as a mechanism to manage less frequently changing configuration values and choose the patch file based on the target environment in the pipeline. Patching is a choice because application operators own these configurations independently. Exposing them to developers will increase the cognitive load.
• Rarely changes: Rarely customized configurations are the basic structure of the core application. Platform developers should own these configurations. The basic structure is the core abstractions to minimize the cognitive load for application operators and developers. Also, it will summarize the policy requirements inside the recipe. Generally, we use overlay, specialized, and extendable abstractions to achieve multiple variants of core configurations required by the different product teams and workloads.

XRM change frequency

The XRM-compliant COTS external infrastructure dependencies mostly do not change frequently. For example, changes to the external infrastructure, such as region scaling, autoscaling setup, tuning security configuration, upgrade, and migration, will happen at a low phase after the initial setup. Building these infrastructure abstractions with XRs should be owned by platform teams. The application operator could do a few XR composing exercises to make a new, specialized recipe. In other words, they could create workload-specific abstraction using the nested XR pattern. As discussed earlier, XRM goes beyond infrastructure dependencies to manage the external application using providers such as Helm and SQL. These configurations could change frequently. For example, every application release could change the SQL schema of the database. Hence, the application operator persona can extend the existing recipes to meet the product team requirement.

Summary

This chapter discussed various aspects of unified configuration management of bespoke applications and COTS components at scale. We covered concepts such as different tools available for configuration management, common pitfalls along the way when we scale, and the trade-off in using different patterns. Also, we discussed how different combinations of tools and practices could complement each other under different circumstances.

The following two chapters will look at hands-on examples to try out a few recipes discussed in this chapter. The recipes will include KRM and XRM configuration management as we move toward end-to-end automation of the whole application and its COTS dependencies.