4 Patterns for infrastructure dependencies
- Writing loosely coupled infrastructure modules using dependency patterns
- Identifying ways to decouple infrastructure dependencies
- Recognizing infrastructure use cases for dependency patterns
An infrastructure system involves a set of resources that depend on each other. For example, a server depends on the existence of a network. How do you know the network exists before creating a server? You can express this with an infrastructure dependency. An infrastructure dependency happens when a resource requires another one to exist before creating or modifying the first one.
Definition An infrastructure dependency expresses a relationship in which an infrastructure resource depends on the existence and attributes of another resource.
Usually, you identify the server’s dependency on the network by hardcoding the network identifier. However, hardcoding more tightly binds the dependency between server and network. Anytime you change the network, you must update the hardcoded dependency.
In chapter 2, you learned how to avoid hardcoding values with variables to promote reproducibility and evolvability. Passing the network identifier as a variable better decouples the server and network. However, a variable works between resources only in the same module. How can you express dependencies between modules?
The preceding chapter grouped resources into modules to enhance composability. This chapter covers patterns for managing infrastructure dependencies to enhance evolvability (change). You can more easily replace one module with another when they have a loose dependency.
In reality, infrastructure systems can be pretty complex, and you can’t swap modules without some disruption. Loosely coupled dependencies offer mitigation for change failure but don’t guarantee 100% availability!
4.1 Unidirectional relationships
Different dependency relationships affect infrastructure change. Imagine you add a firewall rule each time you create a new application. The firewall rule has a unidirectional dependency on the application IP address to allow traffic. Any change to the application gets reflected in the firewall rule.
Definition A unidirectional dependency expresses a one-way relationship in which only one resource refers to another.
You can express unidirectional dependencies between any set of resources or modules. Figure 4.1 describes the unidirectional relationship between the firewall rule and the application. The rule depends on the application, which makes it higher up the infrastructure stack than the lower-level application.
When you express a dependency, you have a high-level resource like the firewall that depends on the existence of a low-level resource like the application.
Definition A high-level resource depends on another resource or module. A low-level resource has high-level resources depending on it.
Let’s say a reporting application needs a list of rules for the firewall. It sends the rules to an audit application. However, the firewall needs to know the IP address of the reporting application. Should you update the IP address of the reporting application or the firewall rule first? Figure 4.2 shows the conundrum of deciding which application you should update first.
Figure 4.2 The reporting application and the firewall have a circular dependency on each other. Changes block connectivity to the application.
This example encounters a circular dependency, which introduces a chicken-or-egg problem. You cannot change one resource without affecting the other. If you first change the address of the reporting application, the firewall rule must change. However, the reporting application fails because it can’t connect. You might have blocked its request!
Circular dependencies cause unexpected behaviors during changes, which ultimately affect composability and evolvability. You don’t know which resource to update first. By contrast, you can identify how a low-level module’s change might affect the high-level one. Unidirectional dependency relationships make changes more predictable. After all, a successful infrastructure change depends on two factors: predictability and isolation.
4.2 Dependency injection
Unidirectional dependencies help you engineer ways to minimize the impact of low-level module changes to high-level modules. For example, network changes should not disrupt high-level resources like queues, applications, or databases. This section applies the software development concept of dependency injection to infrastructure and further decouples unidirectional dependencies. Dependency injection involves two principles: inversion of control and dependency inversion.
4.2.1 Inversion of control
When you enforce unidirectional relationships in your infrastructure dependencies, your high-level resource gets information about the low-level resource. Then it can run its changes. For example, a server gets information about the network’s ID and IP address range before it claims an IP address (figure 4.3).
Figure 4.3 With inversion of control, the high-level resource or module calls the low-level module for information and parses its metadata for any dependencies.
The server calls the network, naturally applying a software development principle called inversion of control. The high-level resource calls for information about the low-level resource before updating.
Definition Inversion of control is the principle by which the high-level resource calls the low-level one for attributes or references.
As a nontechnical example, you use inversion of control when you call to schedule a doctor’s appointment instead of the doctor’s office automatically scheduling your appointment.
Let’s apply inversion of control to implement the server’s dependency on the network. You create the network by using a network module. In the following listing, the network module outputs a network name and saves it in a file called terraform.tfstate. High-level resources, like the server, can parse the network name from this JSON file.
Listing 4.1 Network module outputs in JSON file
❶ Creating the network with Terraform generates a JSON file with a list of outputs. Terraform uses this file to track the resources it creates.
❷ The network module outputs the subnet name as a string.
❸ The remainder of the JSON file has been omitted for clarity.
Using inversion of control, the server calls the network’s terraform.tfstate file in listing 4.2 and reads the subnet name. Since the module expresses outputs in the JSON file, your server module needs to parse for the value of the subnet name (hello-world-subnet).
Listing 4.2 Applying inversion of control to create a server on a network
import json class NetworkModuleOutput: ❶ def __init__(self): with open('network/terraform.tfstate', 'r') as network_state: network_attributes = json.load(network_state) self.name = network_attributes['outputs']['name']['value'] ❷ class ServerFactoryModule: ❸ def __init__(self, name, zone='us-central1-a'): ❹ self._name = name self._network = NetworkModuleOutput() ❺ self._zone = zone self.resources = self._build() ❻ def _build(self): ❻ return { 'resource': [{ 'google_compute_instance': [{ ❹ self._name: [{ 'allow_stopping_for_update': True, 'boot_disk': [{ 'initialize_params': [{ 'image': 'ubuntu-1804-lts' }] }], 'machine_type': 'f1-micro', 'name': self._name, ❹ 'zone': self._zone, ❹ 'network_interface': [{ 'subnetwork': self._network.name ❼ }] }] }] }] } if __name__ == "__main__": ❽ server = ServerFactoryModule(name='hello-world') ❽ with open('main.tf.json', 'w') as outfile: ❽ json.dump(server.resources, outfile, sort_keys=True, indent=4)
❶ Creates an object that captures the schema of the network module’s output. This makes it easier for the server to retrieve the subnet name.
❷ The object for the network output parses the value of the subnet name from the JSON object.
❸ Creates a module for the server, which uses the factory pattern
❹ Creates the Google compute instance using a Terraform resource with a name and zone
❺ The server module calls the network output object, which contains the subnet name parsed from the network module’s JSON file.
❻ Uses the module to create the JSON configuration for the server using the subnetwork name
❼ The server module references the network output’s name and passes it to the “subnetwork” field.
❽ Writes the Python dictionary to a JSON file to be executed by Terraform later
Implementing inversion of control eliminates a direct reference to the subnet in your server module. You can also control and limit the information the network returns for high-level resources to use. More important, you improve my composability because you can create other servers and high-level resources on the subnet name offered by the network module.
What if other high-level resources need other low-level attributes? For example, you might create a queue that needs the subnet IP address range. To solve this problem, you evolve the network module to output the subnet IP address range. The queue can reference the outputs for the address it needs.
Inversion of control improves evolvability as high-level resources require different attributes. You can evolve low-level resources without rewriting the infrastructure as code for high-level resources. However, you need a way to protect the high-level resources from any attribute updates or renaming on the low-level resources.
4.2.2 Dependency inversion
While inversion of control enables the evolution of high-level modules, it does not protect them from changes to low-level modules. Let’s imagine you change the network name to its ID. The next time you deploy changes to your server module, it breaks! The server module does not recognize the network ID.
To protect your server module from changes to the network outputs, you add a layer of abstraction between the network output and server. In figure 4.4, the server accesses the network’s attributes through an API or a stored configuration instead of the network output. All of these interfaces serve as abstractions to retrieve network metadata.
Figure 4.4 Dependency inversion returns an abstraction of the low-level resource metadata to the resource that depends on it.
You can use dependency inversion to isolate changes to low-level modules and mitigate disruption to their dependencies. Dependency inversion dictates that high-level and low-level resources should have dependencies expressed through abstractions.
Definition Dependency inversion is the principle of expressing dependencies between high-level and low-level modules or resources through abstractions.
The abstraction layer behaves as a translator that communicates the required attributes. It serves as a buffer for changes to the low-level module away from the high-level one. In general, you can choose from three types of abstraction:
Some abstractions, such as attribute interpolation or module outputs, depend on your tool. Abstraction by infrastructure state will depend on your tool or infrastructure API. Figure 4.5 shows the abstractions by attribute interpolation, module output, or infrastructure state to pass network metadata to the server.
Figure 4.5 Depending on the tool and dependencies, abstractions for dependency inversion can use attribute interpolation, module outputs, or infrastructure state.
Let’s examine how to implement the three types of abstraction by building modules for the network and server in listing 4.3. I’ll start with attribute interpolation. Attribute interpolation handles attribute passing between resources or tasks within a module or configuration. Using Python, a subnet interpolates the name of the network by accessing the name attribute assigned to the network object.
Listing 4.3 Using attribute interpolation to get the network name
import json class Network: ❶ def __init__(self, name="hello-network"): ❶ self.name = name self.resource = self._build() ❷ def _build(self): ❷ return { 'google_compute_network': [ ❶ { f'{self.name}': [ { 'name': self.name ❶ } ] } ] } class Subnet: ❸ def __init__(self, network, region='us-central1'): ❹ self.network = network ❹ self.name = region ❸ self.subnet_cidr = '10.0.0.0/28' self.region = region self.resource = self._build() def _build(self): return { 'google_compute_subnetwork': [ ❸ { f'{self.name}': [ { 'name': self.name, ❸ 'ip_cidr_range': self.subnet_cidr, 'region': self.region, 'network': self.network.name ❺ } ] } ] } if __name__ == "__main__": network = Network() ❻ subnet = Subnet(network) ❼ resources = { ❽ "resource": [ ❽ network.resource, ❻ subnet.resource ❼ ] ❽ } ❽ with open(f'main.tf.json', 'w') as outfile: ❾ json.dump(resources, outfile, sort_keys=True, indent=4) ❾
❶ Creates the Google network using a Terraform resource named “hello-network”
❷ Uses the module to create the JSON configuration for the network
❸ Creates the Google subnetwork using a Terraform resource named after the region, us-central1
❹ Passes the entire network object to the subnet. The subnet calls the network object for the attributes it needs.
❺ Interpolates the network name by retrieving it from the object
❻ Uses the module to create the JSON configuration for the network
❼ Uses the module to create the JSON configuration for the subnet and passes the network object to the subnet
❽ Merges the network and subnet JSON objects into a Terraform-compatible JSON structure
❾ Writes the Python dictionary to a JSON file to be executed by Terraform later
Attribute interpolation works between modules or resources in a configuration. However, interpolation works for only specific tools and not necessarily across tools. When you have more resources and modules in composition, you cannot use interpolation.
One alternative to attribute interpolation uses explicit module outputs to pass resource attributes between modules. You can customize outputs to any schema or parameters you need. For example, you can group the subnet and network into one module and export its attributes for the server to use. Let’s refactor the subnet and network and add the server, as in the following listing.
Listing 4.4 Setting the subnet name as the output for a module
import json ❶ class NetworkModule: ❷ def __init__(self, region='us-central1'): self._region = region self._network = Network() ❷ self._subnet = Subnet(self._network) ❷ self.resource = self._build() ❸ def _build(self): ❸ return [ ❸ self._network.resource, ❸ self._subnet.resource ❸ ] ❸ class Output: ❹ def __init__(self, subnet_name): ❹ self.subnet_name = subnet_name ❹ def output(self): ❺ return self.Output(self._subnet.name) ❺ class ServerModule: ❻ def __init__(self, name, network, ❼ zone='us-central1-a'): self._name = name self._subnet_name = network.subnet_name ❽ self._zone = zone self.resource = self._build() ❾ def _build(self): ❾ return [{ 'google_compute_instance': [{ self._name: [{ 'allow_stopping_for_update': True, 'boot_disk': [{ 'initialize_params': [{ 'image': 'ubuntu-1804-lts' }] }], 'machine_type': 'e2-micro', 'name': self._name, 'zone': self._zone, 'network_interface': [{ 'subnetwork': self._subnet_name }] }] }] }] if __name__ == "__main__": network = NetworkModule() ❿ server = ServerModule("hello-world", ❻ network.output()) ❼ resources = { ⓫ "resource": network.resource + server.resource ⓫ } ⓫ with open(f'main.tf.json', 'w') as outfile: ⓬ json.dump(resources, outfile, sort_keys=True, indent=4) ⓬
❶ Network and subnet objects omitted for clarity
❷ Refactors network and subnet creation into a module. This follows the composite pattern. The module creates the Google network and subnet using Terraform resources.
❸ Uses the module to create the JSON configuration for the network and subnet
❹ Creates a nested class for the network module output. The nested class exports the name of the subnet for high-level attributes to use.
❺ Creates an output function for the network module to retrieve and export all network outputs
❻ This module creates the Google compute instance (server) using a Terraform resource.
❼ Passes the network outputs as an input variable for the server module. The server will choose the attributes it needs.
❽ Using the network output object, gets the subnet name and sets it to the server’s subnet name attribute
❾ Uses the module to create the JSON configuration for the server
❿ Refactors network and subnet creation into a module. This follows the composite pattern. The module creates the Google network and subnet using Terraform resources.
⓫ Merges the network and server JSON objects into a Terraform-compatible JSON structure
⓬ Writes the Python dictionary to a JSON file to be executed by Terraform later
Module outputs help expose specific parameters for high-level resources. The approach copies and repeats the values. However, module outputs can get complicated! You’ll often forget which outputs you exposed and their names. Contract testing in chapter 6 might help you enforce required module outputs.
Rather than use outputs, you can use infrastructure state as a state file or infrastructure provider’s API metadata. Many tools keep a copy of the infrastructure state, which I call tool state, to detect drift between actual resource state and configuration and track which resources it manages.
Definition Tool state is a representation of infrastructure state stored by an IaC tool. It tracks the configuration of resources managed by the tool.
Tools often store their state in a file. You already encountered an example of using tool state in listing 4.2. You parsed the name of the network from a file called terraform.tfstate, which is the tool state for Terraform. However, not all tools offer a state file. As a result, you may have difficulty parsing low-level resource attributes across tools.
If you have multiple tools and providers in your system, you have two main options. First, consider using a configuration manager as a standard interface to pass metadata. A configuration manager, like a key-value store, manages a set of fields and their values.
The configuration manager helps you create your own abstraction layer for tool state. For example, some network automation scripts might read IP address values stored in a key-value store. However, you have to maintain the configuration manager and make sure your IaC can access it.
As a second option, consider using an infrastructure provider’s API. Infrastructure APIs do not often change; they provide detailed information and account for out-of-band changes that a state file may not include. You can use client libraries to access information from infrastructure APIs.
You will encounter a few downsides to using the infrastructure API. Unfortunately, your IaC needs network access. You won’t know the value of the attribute until you run the IaC because the code must make a request to the API. If the infrastructure API experiences an outage, your IaC may not resolve attributes for low-level resources.
When you add an abstraction with dependency inversion, you protect high-level resources from changing attributes on lower-level resources. While you can’t prevent all failures or disruptions, you minimize the blast radius of potential failures due to updated low-level resources. Think of it as a contract: if both high- and low-level resources agree on the attributes they need, they can evolve independently of one another.
4.2.3 Applying dependency injection
What happens when you combine inversion of control and dependency inversion? Figure 4.6 shows how you can combine both principles to decouple the server and network example. The server calls the network for attributes and parses the metadata using the infrastructure API or state. If you make changes to the network name, it updates the metadata. The server retrieves the updated metadata and adjusts its configuration separately.
Figure 4.6 Dependency injection combines inversion of control and dependency inversion to loosen infrastructure dependencies and isolate low-level and high-level resources.
Harnessing the power of both principles helps promote evolution and composability because the abstraction layer behaves as a buffer between each building block of your system. You use dependency injection to combine inversion of control and dependency inversion. Inversion of control isolates changes to the high-level modules or resources, while dependency inversion isolates changes to the low-level resources.
Definition Dependency injection combines the principles of inversion of control and dependency inversion. High-level modules or resources call for attributes from low-level ones through an abstraction.
Let’s implement dependency injection for the server and network example with Apache Libcloud, a library for the GCP API, as shown in listing 4.5. You use Libcloud to search for the network. The server calls the GCP API for the subnet name, parses the GCP API metadata, and assigns itself the fifth IP address in the network’s range.
Listing 4.5 Using dependency injection to create a server on a network
import credentials import ipaddress import json from libcloud.compute.types import Provider ❶ from libcloud.compute.providers import get_driver ❶ def get_network(name): ❷ ComputeEngine = get_driver(Provider.GCE) ❸ driver = ComputeEngine( ❹ credentials.GOOGLE_SERVICE_ACCOUNT, ❹ credentials.GOOGLE_SERVICE_ACCOUNT_FILE, ❹ project=credentials.GOOGLE_PROJECT, ❹ datacenter=credentials.GOOGLE_REGION) ❹ return driver.ex_get_subnetwork( ❺ name, credentials.GOOGLE_REGION) ❺ class ServerFactoryModule: ❻ def __init__(self, name, network, zone='us-central1-a'): self._name = name gcp_network_object = get_network(network) ❷ self._network = gcp_network_object.name ❼ self._network_ip = self._allocate_fifth_ip_address_in_range( ❽ gcp_network_object.cidr) ❽ self._zone = zone self.resources = self._build() def _allocate_fifth_ip_address_in_range(self, ip_range): ❽ ip = ipaddress.IPv4Network(ip_range) ❽ return format(ip[-2]) ❽ def _build(self): ❾ return { 'resource': [{ 'google_compute_instance': [{ ❻ self._name: [{ 'allow_stopping_for_update': True, 'boot_disk': [{ 'initialize_params': [{ 'image': 'ubuntu-1804-lts' }] }], 'machine_type': 'f1-micro', 'name': self._name, 'zone': self._zone, 'network_interface': [{ 'subnetwork': self._network, ❿ 'network_ip': self._network_ip ⓫ }] }] }] }] } if __name__ == "__main__": server = ServerFactoryModule(name='hello-world', network='default') ⓬ with open('main.tf.json', 'w') as outfile: ⓭ json.dump(server.resources, outfile, sort_keys=True, indent=4) ⓭
❶ Imports the Libcloud library, which allows you to access the GCP API. You must import the provider object and Google driver.
❷ This function retrieves the network information using the Libcloud library. The network and subnet were created separately. Their code has been omitted for clarity.
❸ Imports the Google Compute Engine driver for Libcloud
❹ Passes the GCP service account credentials you want Libcloud to use for accessing the GCP API
❺ Uses the Libcloud driver to get the subnet information by its name
❻ This module creates the Google compute instance (server) using a Terraform resource
❼ Parses the subnet name from the GCP network object returned by Libcloud and uses it to create the server
❽ Parses the CIDR block from the GCP network object returned by Libcloud and uses it to calculate the fifth IP address on the network. The server uses the result as its network IP address.
❾ Uses the module to create the JSON configuration for the server
❿ Parses the subnet name from the GCP network object returned by Libcloud and uses it to create the server
⓫ Parses the CIDR block from the GCP network object returned by Libcloud and uses it to calculate the fifth IP address on the network. The server uses the result as its network IP address.
⓬ This module creates the Google compute instance (server) using a Terraform resource.
⓭ Writes the Python dictionary to a JSON file to be executed by Terraform later
Using the infrastructure API as an abstraction layer, you account for the evolution of the network independent of the server. For example, what happens when you change the IP address range for the network? You deploy the update to the network before you run the IaC for the server. The server calls the infrastructure API for network attributes and recognizes a new IP address range. Then it recalculates the fifth IP address.
Figure 4.7 shows the responsiveness of the server to the change because of dependency injection. When you change the IP address range for the network, your server gets the updated address range and reallocates the IP address if needed.
Figure 4.7 Dependency injection allows me to change the low-level module (the network) and automatically propagate the change to the high-level module (the server).
Thanks to dependency inversion, you can evolve low-level resources separately from dependencies. Inversion of control helps high-level resources respond to changes in low-level resources. Combining the two as dependency injection ensures the composability of the system, as you can add more high-level resources on the low-level ones. Decoupling due to dependency injection helps you minimize the blast radius of failed changes across modules in your system.
In general, you should apply dependency injection as an essential principle for infrastructure dependency management. If you apply dependency injection when you write your infrastructure configuration, you sufficiently decouple dependencies so that you can change them independently without affecting other infrastructure. As your module grows, you can continue to refactor to more specific patterns and further decouple infrastructure based on the type of resources and modules.
4.3 Facade
Applying the principle of dependency injection generates similar patterns for expressing dependencies. The patterns align with structural design patterns in software development. In the pursuit of decoupling dependencies, I often find myself repeating the same three patterns in my IaC.
Imagine you want to create a storage bucket to store static files. You can control who accesses the files with an access control API in GCP. Figure 4.8 creates the bucket and sets the outputs to include the bucket’s name. The access control rules for the bucket can use the outputs to get the bucket’s name.
Figure 4.8 The facade simplifies attributes to the name of the storage bucket for use by the access control module.
The pattern of using outputs and an abstraction layer seems very familiar. In fact, you encountered it in the chapter’s first half. You’ve been unknowingly using the facade pattern to pass multiple attributes between modules!
The facade pattern uses module outputs as the abstraction for dependency injection. It behaves like a mirror, reflecting the attributes to other modules and resources.
Definition The facade pattern outputs attributes from resources in a module for dependency injection.
A facade reflects the attributes and nothing more. The pattern does decouple dependencies between high- and low-level resources and conforms to the principle of dependency injection. The high-level resource still calls the low-level resource for information, and outputs serve as the abstraction.
The following listing implements the facade pattern in code by building an output method. Your bucket module returns the bucket object and name in its output method. Your access module uses the output method to retrieve the bucket object and access its name.
Listing 4.6 Outputting the bucket name as a facade for access control rules
import json import re class StorageBucketFacade: ❶ def __init__(self, name): ❶ self.name = name ❶ class StorageBucketModule: ❷ def __init__(self, name, location='US'): ❸ self.name = f'{name}-storage-bucket' self.location = location self.resources = self._build() def _build(self): return { 'resource': [ { 'google_storage_bucket': [{ ❸ self.name: [{ 'name': self.name, 'location': self.location, 'force_destroy': True ❹ }] }] } ] } def outputs(self): ❺ return StorageBucketFacade(self.name) class StorageBucketAccessModule: ❻ def __init__(self, bucket, user, role): ❼ if not self._validate_user(user): ❽ print("Please enter valid user or group ID") exit() if not self._validate_role(role): ❾ print("Please enter valid role") exit() self.bucket = bucket ❼ self.user = user self.role = role self.resources = self._build() def _validate_role(self, role): ❾ valid_roles = ['READER', 'OWNER', 'WRITER'] if role in valid_roles: return True return False def _validate_user(self, user): ❽ valid_users_group = ['allUsers', 'allAuthenticatedUsers'] if user in valid_users_group: return True regex = r'^[a-z0-9]+[._]?[a-z0-9]+[@]w+[.]w{2,3}$' if(re.search(regex, user)): return True return False def _change_case(self): return re.sub('[^0-9a-zA-Z]+', '_', self.user) def _build(self): return { 'resource': [{ 'google_storage_bucket_access_control': [{ self._change_case(): [{ 'bucket': self.bucket.name, ❿ 'role': self.role, 'entity': self.user }] }] }] } if __name__ == "__main__": bucket = StorageBucketModule('hello-world') with open('bucket.tf.json', 'w') as outfile: json.dump(bucket.resources, outfile, sort_keys=True, indent=4) server = StorageBucketAccessModule( bucket.outputs(), 'allAuthenticatedUsers', 'READER') with open('bucket_access.tf.json', 'w') as outfile: json.dump(server.resources, outfile, sort_keys=True, indent=4)
❶ Using the facade pattern, outputs the bucket name as part of the storage output object. This implements dependency inversion to abstract away unnecessary bucket attributes.
❷ Creates a low-level module for the GCP storage bucket, which uses the factory pattern to generate a bucket
❸ Creates the Google storage bucket using a Terraform resource based on the name and location
❹ Sets an attribute on the Google storage bucket to destroy it when you delete Terraform resources
❺ Creates an output method for the module that returns a list of attributes for the storage bucket
❻ Creates a high-level module to add access control rules to the storage bucket
❼ Passes the bucket’s output facade to the high-level module
❽ Validates that the users passed to the module match valid user group types for all users or all authenticated users
❾ Validates that the roles passed to the module match valid roles in GCP
❿ Creates Google storage bucket access control rules using a Terraform resource
Why output the entire bucket object and not just the name? Remember that you want to build an abstraction layer to conform to the principle of dependency inversion. If you create a new module that depends on the bucket location, you can update the bucket object’s facade to output the name and location. The update does not affect the access module.
You can implement a facade with low effort and still get the benefits from decoupling of dependencies. One such benefit includes the flexibility to make isolated, self-contained updates in one module without affecting others. Adding new high-level dependencies does not require much effort.
The facade pattern also makes it easier to debug problems. It mirrors the outputs without adding logic for parsing, making it simple to trace problems to the source and fix the system. You’ll learn more about reverting failed changes in chapter 11.
As a general practice, you’ll start a facade with one or two fields. Always keep this to the minimum number of fields you’ll need for high-level resources. Review and prune the fields when you don’t need them every few weeks.
The facade pattern works for simpler dependencies, such as a few high-level modules to one low-level one. However, when you add many high-level modules and the depth of your dependencies increases, you will have difficulty maintaining the facade pattern for the low-level modules. When you need to change a field name in the output, you must change every module that references it. Changing every module reference does not scale when you have hundreds of resources that depend on one low-level module.
4.4 Adapter
The facade mirrors the values as outputs for one infrastructure module to high-level modules in the previous section. It works well for simple dependency relationships but falls apart with more complex modules. More complex modules usually involve one-to-many dependencies or span multiple infrastructure providers.
Let’s say you have an identity module that passes a list of users and roles for configuring infrastructure. The identity module needs to work across multiple platforms. In figure 4.9, you set up the module to output a JSON-formatted object that maps permissions such as read, write, or admin with the corresponding usernames. Teams must map these usernames and their generic permissions to GCP-specific terms. GCP’s access management uses viewer, editor, and owner, which transform to read, write, and admin.
Figure 4.9 The adapter pattern transforms attributes to a different interface that high-level modules can consume.
How do you map a generic set of roles to the specific infrastructure provider roles? The mapping needs to ensure that you can reproduce and evolve the module over multiple infrastructure providers. You want to extend the module in the future to add users to equivalent roles across platforms.
As a solution, the adapter pattern transforms metadata from the low-level resource so that any high-level resource can use it. An adapter behaves like a travel plug. You can change the plug depending on the country’s outlet and still use your electronic devices.
Definition The adapter pattern transforms and outputs metadata from the low-level resource or module so any high-level resource or module can use it.
To start, you create a dictionary that maps generic role names to users. In listing 4.7, you want to assign a read-only role to the audit team and two users. These generic roles and usernames do not match any of the GCP permissions and roles.
Listing 4.7 Creating a static object that maps generic roles to usernames
class Infrastructure: def __init__(self): self.resources = { 'read': [ ❶ 'audit-team', ❶ 'user-01', ❶ 'user-02' ❶ ], 'write': [ ❷ 'infrastructure-team', ❷ 'user-03', ❷ 'automation-01' ❷ ], 'admin': [ ❸ 'manager-team' ❸ ] ❸ }
❶ Assigns the audit-team, user-01, and user-02 to a read-only role. The mapping describes that the user can only read information on any infrastructure provider.
❷ Assigns the infrastructure-team, user-02, and automation-01 to a write role. The mapping describes that the user can update information on any infrastructure provider.
❸ Assigns the manager team to the administrator role. The mapping describes that the user can manage any infrastructure provider.
However, you cannot do anything with the static object in role mappings. GCP does not understand the usernames or roles! Implement the adapter pattern to map generic permissions to the infrastructure-specific permissions.
The following listing builds an identity adapter specific to GCP, which maps generic permissions like read to GCP-specific terms like roles/viewer. GCP can use the map to add users, service accounts, and groups to the correct roles.
Listing 4.8 Using the adapter pattern to transform generic permissions
import json import access class GCPIdentityAdapter: ❶ EMAIL_DOMAIN = 'example.com' ❷ def __init__(self, metadata): gcp_roles = { ❸ 'read': 'roles/viewer', ❸ 'write': 'roles/editor', ❸ 'admin': 'roles/owner' ❸ } self.gcp_users = [] for permission, users in metadata.items(): ❹ for user in users: ❹ self.gcp_users.append( ❹ (user, self._get_gcp_identity(user), ❺ gcp_roles.get(permission))) ❹ def _get_gcp_identity(self, user): ❺ if 'team' in user: ❻ return f'group:{user}@{self.EMAIL_DOMAIN}' ❻ elif 'automation' in user: ❼ return f'serviceAccount:{user}@{self.EMAIL_DOMAIN}' ❼ else: ❽ return f'user:{user}@{self.EMAIL_DOMAIN}' ❽ def outputs(self): ❾ return self.gcp_users ❾ class GCPProjectUsers: ❿ def __init__(self, project, users): self._project = project self._users = users self.resources = self._build() ⓫ def _build(self): ⓫ resources = [] for (user, member, role) in self._users: ⓬ resources.append({ 'google_project_iam_member': [{ ⓭ user: [{ ⓭ 'role': role, ⓭ 'member': member, ⓭ 'project': self._project ⓭ }] ⓭ }] ⓭ }) return { 'resource': resources } if __name__ == "__main__": users = GCPIdentityAdapter(access.Infrastructure().resources).outputs() ⓮ with open('main.tf.json', 'w') as outfile: ⓯ json.dump( ⓯ GCPProjectUsers( ⓯ 'infrastructure-as-code-book', ⓯ users).resources, outfile, sort_keys=True, indent=4) ⓯
❶ Creates an adapter to map generic role types to Google role types
❷ Sets the email domain as a constant, which you’ll append to each user
❸ Creates a dictionary to map generic roles to GCP-specific permissions and roles
❹ For each permission and user, builds a tuple with the user, GCP identity, and role
❺ Transforms the usernames to GCP-specific member terminology, which uses user type and email address
❻ If the username has “team,” the GCP identity needs to be prefixed with “group” and suffixed with the email domain.
❼ If the username has “automation,” the GCP identity needs to be prefixed with “serviceAccount” and suffixed with the email domain.
❽ For all other users, the GCP identity needs to be prefixed with “user” and suffixed with the email domain.
❾ Outputs the list of tuples containing the users, GCP identities, and roles
❿ Creates a module for the GCP project users, which uses the factory pattern to attach users to GCP roles for a given project
⓫ Uses the module to create the JSON configuration for the project’s users and roles
⓬ Creates a dictionary to map generic roles to GCP-specific permissions and roles
⓭ Creates a list of Google project IAM members using a Terraform resource. The list retrieves the GCP identity, role, and project to attach a username to read, write, or administrator permissions in GCP.
⓮ Creates an adapter to map generic role types to Google role types
⓯ Writes the Python dictionary to a JSON file to be executed by Terraform later
You could extend your identity adapter to map the generic dictionary of access requirements to another infrastructure provider, like AWS or Azure. In general, an adapter translates the provider-specific or prototype module-specific language into generic terms. This pattern works best for modules with different infrastructure providers or dependencies. I also use the adapter pattern to create a consistent interface for infrastructure providers with poorly defined resource parameters.
For a more complex example, imagine configuring a virtual private network (VPN) connection between two clouds. Instead of passing network information from each provider through a facade, you use an adapter, as in figure 4.10. Your network modules for each provider output a network object with more general fields, such as name and IP address. This use case benefits from an adapter because it reconciles the semantics of two different languages (e.g., a GCP Cloud VPN gateway and AWS customer gateway).
Why use an adapter to promote composability and evolvability? The pattern heavily relies on dependency inversion to abstract any transformation of attributes between resources. An adapter behaves as a contract between modules. As long as both modules agree on the contract outlined by the adapter, you can continue to change high-level and low-level modules somewhat independently of each other.
However, the adapter pattern works only if you maintain the contract between modules. Recall that you built an adapter to transform permissions and usernames to GCP. What happens if your teammate accidentally updates the mapping for read-only roles to roles/reader, which doesn’t exist? Figure 4.11 demonstrates that if you don’t use the right role specific to GCP, your IaC fails.
In the example, you broke the contract between the generic and GCP roles! The broken contract causes your IaC to fail. Make sure you maintain and update the correct mappings in your adapter to minimize failure.
Furthermore, troubleshooting becomes more difficult with an adapter. The pattern obfuscates the resources depending on a specific adapter attribute. You need to investigate whether an error results from the wrong field output from the source module, an incorrect attribute in the adapter, or the wrong field consumed by the dependent module. Module versioning and testing in chapters 5 and 6, respectively, can alleviate the challenges and troubleshooting of an adapter.
4.5 Mediator
The adapter and facade patterns isolate changes and make it easy to manage one dependency. However, IaC often includes complex resource dependencies. To detangle the web of dependencies, you can build opinionated automation that structures when and how IaC should create resources.
Imagine you want to add a firewall rule to allow SSH to the server’s IP address in our canonical server and network example. However, you can create the firewall rule only if the server exists. Similarly, you can create the server only if the network exists. You need automation to capture the complexity of the relationships among firewall, server, and network.
Let’s try to capture the logic of creating the network, the server, and the firewall. Automation can help mediate which resources to create first. Figure 4.12 diagrams the workflow for the automation. If the resource is a server, IaC creates the network and then the server. If the resource is a firewall rule, IaC creates the network first, the server second, and the firewall rule third.
The IaC implements dependency injection to abstract and control network, server, and firewall dependencies. It relies on the principle of idempotency to run continuously and achieve the same end state (network, server, and firewall), no matter the existing resources. Composability also helps establish the building blocks of infrastructure resources and dependencies.
This mediator pattern works like air-traffic control at an airport. It controls and manages inbound and outbound flights. A mediator’s sole purpose is to organize the dependencies among these resources and to create or delete objects as needed.
Definition The mediator pattern organizes dependencies among infrastructure resources and includes logic to create or delete objects based on their dependencies.
Let’s implement the mediator pattern for the network, server, and firewall. Implementing a mediator in Python requires a few if-else statements to check each resource type and build its low-level dependencies. In listing 4.9, the firewall depends on creating the server and the network first.
Listing 4.9 Using the mediator pattern to organize server and dependencies
import json from server import ServerFactoryModule ❶ from firewall import FirewallFactoryModule ❶ from network import NetworkFactoryModule ❶ class Mediator: ❷ def __init__(self, resource, **attributes): self.resources = self._create(resource, **attributes) def _create(self, resource, **attributes): ❸ if isinstance(resource, FirewallFactoryModule): ❹ server = ServerFactoryModule(resource._name) ❹ resources = self._create(server) ❹ firewall = FirewallFactoryModule( ❺ resource._name, depends_on=resources[1].outputs()) ❺ resources.append(firewall) ❺ elif isinstance(resource, ServerFactoryModule): ❻ network = NetworkFactoryModule(resource._name) ❻ resources = self._create(network) ❻ server = ServerFactoryModule( ❼ resource._name, depends_on=network.outputs()) ❼ resources.append(server) ❼ else: ❽ resources = [resource] ❽ return resources def build(self): ❾ metadata = [] ❾ for resource in self.resources: ❾ metadata += resource.build() ❾ return {'resource': metadata} ❾ if __name__ == "__main__": name = 'hello-world' resource = FirewallFactoryModule(name) ❿ mediator = Mediator(resource) ❿ with open('main.tf.json', 'w') as outfile: ⓫ json.dump(mediator.build(), outfile, sort_keys=True, indent=4) ⓫
❶ Imports the factory modules for the network, server, and firewall
❷ Creates a mediator to decide how and in which order to automate changes to resources
❸ When you call the mediator to create a resource like a network, server, or firewall, you allow the mediator to decide all the resources to configure.
❹ If you want to create a firewall rule as a resource, the mediator will recursively call itself to create the server first.
❺ After the mediator creates the server configuration, it builds the firewall rule configuration.
❻ If you want to create a server as a resource, the mediator will recursively call itself to create the network first.
❼ After the mediator creates the network configuration, it builds the server configuration.
❽ If you pass any other resource to the mediator, such as the network, it will build its default configuration.
❾ Uses the module to create a list of resources from the mediator and render the JSON configuration
❿ Passes the mediator a firewall resource. The mediator will create the network, server, and then the firewall configuration.
⓫ Writes the Python dictionary to a JSON file to be executed by Terraform later
If you have a new resource, such as a load balancer, you can expand the mediator to build it after the server or firewall. The mediator pattern works best with modules that have many levels of dependencies and multiple system components.
However, you might find the mediator challenging to implement. The mediator pattern must follow idempotency. You need to run multiple times and achieve the same target state. You have to write and test all of the logic in a mediator. If you do not test your mediator, you may accidentally break a resource. Writing your own mediator takes lots of code!
Fortunately, you do not have to implement your own mediator often. Most IaC tools behave as mediators to resolve complex dependencies and decide how to create resources. The majority of provisioning tools have built-in mediators to identify dependencies and order of operations. For example, the container orchestration of Kubernetes uses a mediator to orchestrate changes to the resources in the cluster. Ansible uses a mediator to determine which automation steps to compose and run from various configuration modules.
Note Some IaC tools implement the mediator pattern by using graph theory to map dependencies between resources. The resources serve as nodes. Links pass attributes to dependent resources. If you want to create resources without a tool, you can manually diagram dependencies in your system. Diagrams can help organize your automation and code. They also identify which modules you can decouple. The exercise of graphing dependencies might help you implement a mediator.
I implement the mediator pattern only when I cannot find it in a tool or need something between tools. For example, I sometimes write a mediator to control creating a Kubernetes cluster in one tool before another tool deploys services on the Kubernetes cluster. A mediator reconciles automation between these two tools, such as checking cluster health before deploying services with the second tool.
4.6 Choosing a pattern
The facade, adapter, and mediator all use dependency injection to decouple changes between high-level and low-level modules. You can apply any of the patterns, and they will express dependencies between modules and isolate changes within them. As your system grows, you may need to change these patterns depending on the structure of your module.
Your choice of pattern depends on the number of dependencies you have on a low-level module or resource. The facade pattern works for one low-level module to a few high-level ones. Consider an adapter if you have a low-level module with many high-level module dependencies. When you have many dependencies among modules, you may need a mediator to control resource automation. Figure 4.13 outlines the decision tree for identifying which dependency pattern to use.
Figure 4.13 Choosing your abstraction depends on the relationship of the dependency, whether it is intra-module, one to one, or one to many.
All of the patterns promote idempotency, composability, and evolvability through dependency injection. However, why would you start with a facade and then consider the adapter or mediator? As your system grows, you will need to optimize your dependency management pattern to reduce the operational burden of changes.
Figure 4.14 shows the relationship between troubleshooting and implementation effort and scalability and isolation for facade, mediator, and adapter patterns. For example, a facade has the benefit of minimal effort for implementation and troubleshooting but does not scale or isolate changes with more resources. Adapters and mediators offer improved scalability and isolation at the cost of troubleshooting and implementation effort.
Figure 4.14 Some patterns may have a low cost of troubleshooting and implementation but cannot isolate changes to modules and scale.
Lower your initial effort by choosing a tool with a mediator implementation. Then use the tool’s built-in facade implementation to manage dependencies between modules or resources. When you find it difficult to manage a facade because you have multiple systems depending on each other, you can start examining an adapter or mediator.
An adapter takes more effort to implement but provides the best foundation for expanding and growing your infrastructure system. You can always add new infrastructure providers and systems without worrying about changing low-level modules. However, you cannot expect to use the adapter for every module because it takes time to implement and troubleshoot.
A tool with a mediator chooses which components get updated and when. An existing tool lowers your overall implementation effort but introduces some concerns during troubleshooting. You need to know your tool’s behavior to troubleshoot failed changes for dependencies. Depending on how you use the tool, a tool with a mediator allows you to scale but may not fully isolate changes to modules.
Summary
-
Apply infrastructure dependency patterns such as facade, adapter, and mediator to decouple modules and resources, and you can make changes to modules in isolation.
-
Inversion of control states that the high-level resource calls the low-level one for attributes.
-
The dependency inversion principle states that the high-level resource should use an abstraction of low-level resource metadata.
-
Dependency injection combines the principles of inversion of control and dependency inversion.
-
If you do not recognize an applicable pattern, you can use dependency injection for a high-level resource to call a low-level resource and parse its object structure for the values it needs.
-
Use the facade pattern to reference a simplified interface for attributes.
-
Use the adapter pattern to transform metadata from one resource for another to use. This pattern works best with resources across different infrastructure providers or prototype modules.
-
The mediator pattern organizes the dependencies between these resources and creates or deletes objects as needed. Most tools serve as a mediator between resources.