13 Managing tools

This chapter covers

• Assessing the use of open source infrastructure modules and tools
• Applying techniques to update or migrate IaC tools
• Implementing module patterns for event-driven IaC

You’ve learned how to write infrastructure as code, update it with your team by using delivery pipelines and testing, and manage its security and cost within your organization. As you evolve your infrastructure system, you adapt these patterns and practices and adjust them to fit new workflows and use cases. Similarly, tools change but should not disrupt the patterns and practices to scale, collaborate, and operate your infrastructure.

Updating your tool can require several actions. You could upgrade to a new version, replace it with a new tool, or handle more dynamic use cases for IaC. This chapter discusses common patterns and practices to handle updates to IaC tools.

The patterns apply to any tools that cover provisioning, configuration management, and image building use cases. You’ll find they also apply to software development, although I adapted them in an opinionated way to infrastructure. Use these patterns and practices to mitigate the blast radius of an update, scale the new tool across teams, and continue evolving your system to support business requirements.

Note This chapter does not include any code listings. Adding an example means introducing yet another tool. I describe the patterns and approaches at a higher level. You can apply the techniques to any tool that supports DSLs or programming languages.

You’ve read the many chapters in this book and practiced IaC in different industries and companies. You’ve built a reputation for setting up and scaling IaC practices. One day, a social media company offers you a role in its platform team.

The company has already established its IaC practice for a few years now. The staff needs your help to maintain and keep the tools they use for IaC updated. You accept and get a backlog of projects to start on the very first day.

13.1 Using open source tools and modules

The accessibility of version control and public repositories makes it simpler to search for an existing tool or infrastructure module instead of writing your own. You can go onto GitHub or any other service to find the automation and tooling to serve your needs. However, you need to make sure you do your due diligence before introducing new tooling to any organization.

For example, let’s imagine the team members who maintain the social media’s feed functionality approach you. They searched online and found an infrastructure module to create a database. They want to use it. They hope to speed up their development process and avoid waiting for another team to review their database configuration. Why reinvent the wheel, after all?

You offer to help review the module for security and best practices. Before you introduce the module to the other teams at the social media company, you use figure 13.1 to assess the database module for its functionality, security, and life cycle before officially adopting it.

Each time the open source maintainers release a new database module, you reassess the module. You can apply this decision workflow to safely and securely adopt external IaC modules and tools. You want to make sure you can use the tool or module and avoid insecure configuration in your infrastructure system that would allow a bad actor to exploit your system.

13.1.1 Functionality

You may find a module or tool promising to start. It allows you to configure the attributes you need very flexibly. However, recall from chapters 2 and 3 that modules should include some opinionated defaults. Without them, a module or tool with too much flexibility may lead to one-off configurations that eventually break your system.

You encourage the feed team to verify the default values in the database module. The team assesses the module, as shown in figure 13.2. The module uses very opinionated defaults, pins the version of the database, and tests compatibility thoroughly.

The documentation notes that the module pins the database version to thoroughly test the configuration and its compatibility with specific versions. The feed team members confirm they use the database version and approve the module’s default value for it.

Next, they assess the input variables for the module. The database module allows them to set the attributes they need, such as database name. It also lets them set the tagging and network. The feed team members confirm they do not need to set more than those variables.

Since the module does not offer every attribute as an input variable and offers opinionated defaults that fit the team, you approve the module based on functionality. In general, review the module’s documentation and commit history. If the module tests for version compatibility and a set of opinionated defaults that suit your functionality, you can move forward with a security assessment.

If you find the module does not offer some specific default values or input variables you need to change, you can find another module, write your own, or proceed to use the module with its limitations in mind. Similarly, you can apply the decision flow to a tool and find it lacking. A single tool cannot do everything! You must balance its flexibility in functionality with its predictability in making infrastructure changes.

13.1.2 Security

While you might first assess the functionality of a module or tool, you should next evaluate its security. Security tends to serve as the make-or-break criteria for whether you should use an open source tool or module. Without carefully assessing an open source module or tool for secure configuration or code, you may inadvertently open a door for a bad actor to compromise the system.

Before the feed team members can use their database module, they need you to check the module for any security concerns. You check whether the database module exposes or outputs sensitive information, sends information to third-party endpoints, and passes existing security and compliance tests, as shown in figure 13.3.

In the example, the database module does not output the password or any sensitive configuration or send information to a third-party endpoint. The module also passes your security and compliance tests for databases you wrote in chapter 8. Why should you verify all three checks before adopting a module?

A module may accidentally expose or output sensitive information. For example, a configuration could accidentally output a password during a dry run. If it does, make sure you have a way to mitigate or mask the password and rotate it.

Similarly, a module should not write or send information to an unauthorized third party. A bad actor might add a minor configuration that sends your network information to an HTTP endpoint. Review each resource and check that they do not send anything to a third party.

Finally, run existing security and compliance tests you wrote for the infrastructure resources. You want secure and compliant resources. Otherwise, you need to go back and update the module to suit your requirements.

Deploy the module in your IaC in an isolated testing environment. Then run the security and compliance tests against the module. Isolating the module in its own environment ensures that you do not introduce noncompliant configurations to running environments.

Keep in mind that not all security and compliance tests will pass. Those that fail need minor refactoring to work with the module. After all the tests pass, you can approve the module as secure for use.

Tools follow a similar decision workflow as you assess their security. However, security and compliance testing for IaC tools may include static code analysis and additional reviews from your organization. For tools that output configuration or other information during dry runs, you will want to apply the remediation steps from chapter 8.

13.1.3 Life cycle

You examined the functionality and security of the module, but the compliance team members raise a very good question. They ask about who maintains the public module. Your organization will need to take private (or maybe public) ownership of the module or tool if its maintainers no longer update it.

You examine the documentation to understand the life cycle of the module and who maintains it, as shown in figure 13.4. If the database module has company sponsorship and an appropriate license, you can likely use it.

You examine the maintainers of the database module. The maintainers come from a well-known technology company and have many contributors, which means the project fosters an active community. They update and release a new version of the module every few months. Each contribution must pass a testing suite to verify that changes do not break the module.

Next, you retrieve information about the database module’s open source license. You do not know how the open source license affects your company, so you contact your legal team. The legal team reviews the license before the feed team can use it.

The module includes an MIT license. As a permissive open source license, the MIT license means if you fork (maintain a copy) or modify the database module, you must include a copy of the license and the original copyright notice. If the maintainers deprecate the module or tool, the permissive license type allows you to modify and update the module yourself.

Definition A permissive open source license allows you to fork or modify code as long as you include a copy of the license and original copyright notice.

Your legal team approves the license because the module poses minimal risk to the overall infrastructure configuration. The company can edit the module if needed but does not have to release it publicly. Perhaps you can even contribute to the open source module yourself, pending legal approval.

A module or tool can also include an open source license in the copyleft category. The copyleft category of licenses contains a clause that you must release the codebase with your modifications.

Definition A copyleft open source license allows you to fork or modify code as long as you release the codebase with your modifications.

The copyleft category of licenses often includes more restrictions on modifying and distributing the tool or module. Your company’s legal team will assess whether the company can use open source IaC with more restrictive licenses.

Note For more information on open source licenses, review the licenses and standards outlined by the Open Source Initiative (https://opensource.org/licenses).

The feed team (overjoyed) can use the database module. You recommend that the team pins the module version by mirroring it to an internal artifact repository. Mirroring the module ensures that teams can use the approved module only in the internal repository. If the public endpoint for the module goes down, the team always has a copy in the internal repository. Each time the maintainers release a new module version, you must check the changes and approve the latest version.

Consider contributing back to open source if your organization allows it. Forking an open source module or tool and maintaining it yourself while keeping it updated with the public release has an operational overhead. You can dedicate many hours reconciling changes between the open source version and your version. Creating a process to contribute changes directly to the public release helps reduce the overhead of maintaining unique changes that may break your infrastructure.

13.2 Upgrading tools

When you run your IaC practice for a few years, you inevitably reach a point where you need to upgrade your tool or the plugins it uses. A wider gap in your tool version to the latest version can make it much harder for you to update your infrastructure with minimal disruption!

We learned about this challenge for module versioning in chapter 5. In this section, I’ll cover some of the considerations and patterns you can use when you need to upgrade your tool.

Note You will not find magic tools to migrate everything perfectly for you. Upgrading a tool will always have challenging obstacles. Unique patterns in your IaC (such as inline scripting) may break the system during the upgrade! As a result, try to limit the complexity of your IaC logic when possible.

Imagine you audit the company’s IaC tools (provisioning, configuration management, and image building). Most of the company IaC uses tool version 1.7. However, the latest tool version is at 4.0. Your first major project involves updating the IaC to use the tool version 4.0.

13.2.1 Pre-upgrade checklist

Before you start upgrading a tool or plugin, you need to check a few essential practices that will help minimize potential disruption to your infrastructure. Your checklist should include a few steps to decouple, stabilize, and reconcile IaC.

Figure 13.5 shows this checklist. You decouple all dependencies, check versioning, pin all versions, and deploy your IaC to reduce drift.

If the tool upgrade adds or removes fields, you need to pass the correct information expected by each resource subset. Dependency injection offers a layer of abstraction for configuration attributes between infrastructure resources (refer to chapter 4). It protects each subset from changes to the other.

You check that you added module versioning to all IaC modules (chapter 5). Similarly, you make sure any IaC or repositories using the module pin to a specific version.

For example, one of the teams at the social media company always uses the latest version of the module. You add module version 2.3.1 to the team’s repository. When you release module version 3.0.0 with the tool upgrade, the team’s IaC will not upgrade with breaking changes. You could update a module and push a breaking change to everyone using it without pinning the module version!

You also verify that each team pins the current version of the tool and its plugin versions. While the plugins may not have forward compatibility, you want to preserve the current version and avoid adding a new configuration in a different version. Finally, you push all your pinned module, tool, and plugin versions to each IaC module and configuration. You ensure that the version pinning does not introduce new changes.

After you complete your pre-upgrade checklist, you must plan your tool upgrade path. Figure 13.6 shows that upgrading from 1.0 to 4.0 of the tool introduces some breaking changes! You decide you can upgrade to 3.0, which provides backward compatibility with 1.7. Then you can upgrade from 3.0 to 4.0, which should reduce any other breaking changes.

Do not upgrade to the latest version immediately. Instead, examine a list of breaking changes in your tool and assess whether you can accommodate those changes. I avoid upgrading more than two versions (or subversions in case of beta releases) at one time. Most tools have breaking changes in either behavior or syntax.

Consider testing the upgrade in a testing environment before rolling it out to production. Based on your system and its testing environment, you can identify if a tool upgrade may disrupt infrastructure. Upgrading will never go as smoothly as you expect, and a testing environment helps identify significant issues before you upgrade production.

13.2.2 Backward compatibility

Many IaC tools offer some kind of backward compatibility for changes. They typically support old and new features for a version or two before deprecating the old feature. Even if a tool supports backward compatibility, make sure to port and refactor to new features as soon as you can.

Your example upgrades the tool from version 1.7 to 3.0. Fortunately, version 3.0 does support backward compatibility with 1.7. It offers new features but no breaking changes that would affect your IaC. Just in case, you take a careful approach to the upgrade.

You start with the feed team members since they agree to your help during the upgrade. The feed team deploys all changes and makes sure not to add new changes to the IaC. Then, you examine the configuration for the best way to upgrade the tool without disrupting the social media feed.

In figure 13.7, you apply the refactoring techniques from chapter 10 to upgrading the tool. You start with high-level resources because other resources do not depend on them, deploy changes, test the system, and upgrade the lower-level resources.

You notify the feed team that you will upgrade DNS and load balancer infrastructure resources first. Other resources do not depend on them. Updating them first allows you to test whether your upgrade patterns work. You can upgrade the tool version for the resource by changing the version and running the IaC. You check the dry run and tests in your delivery pipeline to ensure that you did not disrupt anything.

The high-level resources like DNS and load balancer update without any disruption. Next, you move to lower-level resources. You start to update the servers with the rolling update pattern from chapter 10. Rather than upgrade all servers simultaneously, you start the upgrade on one and quickly run into a problem.

The production server configuration has an override script that breaks when you upgrade the tool. Fortunately, you affected only one server, since you used a rolling update. After all, you want to keep the social media feed running and available.

In figure 13.8, you apply the roll-forward techniques from chapter 11 to fix the server that no longer works. You implement a manual fix and debug the old server for the problem. Once you fix the problem in the testing environment, you push out a change to the override script and proceed with your rolling update of servers.

The servers pass the test. You move to the lowest-level resource, the network. Just in case, you deploy a second version of the network using a blue-green deployment pattern from chapter 9. After deploying everything onto the new network, you run all the end-to-end tests and check that the system still works. You completed the tool upgrade!

Why revisit patterns like refactoring, blue-green deployment, and rolling upgrades? You want to minimize the blast radius of a potential failure. Many of these patterns seem repetitive, but they offer a structured, less risky approach to upgrading your system. Changing your tool does change your infrastructure, so you can apply very similar techniques with the same result.

In general, use rolling upgrades for IaC upgrades to servers or other compute resources. Blue-green deployment helps tool upgrades for high-risk, low-level infrastructure resources. You can usually update high-level resources in place.

However, you can’t prevent every failure. In that case, use the roll-forward practices and patterns from chapter 11 if the system breaks. High-level resources can revert the configuration in place, while lower-level resources benefit from creating new resources with previous changes.

13.2.3 Breaking changes in upgrades

Every once in a while, you’ll find your infrastructure tool or plugin releases a new version that includes breaking changes. New versions with breaking changes often happen with early versions of a tool, such as when the tool handles new or edge use cases. If you need to do a tool upgrade with breaking changes or features, use the techniques for making changes in chapter 9.

For the social media company, you upgraded the feed team’s tool from version 1.7 to 3.0. However, upgrading from version 3.0 to 4.0 involves some breaking changes! Version 4.0 contains backend schema changes that may affect your resources. How can you update your infrastructure to version 4.0 without impacting the system?

Remember, in chapter 4, I mentioned the existence of the tool state. Tools keep a copy of the infrastructure state to detect drift between actual resource state and configuration and to track which resources it manages. Tool state differs from the actual infrastructure state. When you update a tool, you want to break out the old tool state from the new one.

In our example, you want to isolate the old tool state with version 3.0 from a new tool state with version 4.0. Separating the tool state minimizes the blast radius of a potential failure by isolating the infrastructure resources a tool needs to change. Fewer resources means faster recovery and potentially less impact to other parts of the system.

In figure 13.9, each team in the social media company separates its tool state into a different location. Separate tool states ensure that changes to the feed team’s blue infrastructure does not affect the green infrastructure.

First, you pin tool and module versions across the feed team’s IaC. Next, you update infrastructure modules to tool version 4.0. You release a new version of the module, making a note of breaking changes.

Next, you copy the existing configuration into a new folder. Each new folder creates a new tool state. Find the network folder and create a new network using tool version 4.0. You should now have the original “blue” resources from tool version 3.0 and the new “green” resources from tool version 4.0.

You applied a blue-green deployment strategy to the tool state to create new resources with the new tool! Creating a new set of resources with a new version ensures that any breaking changes will not affect the existing infrastructure.

Definition Blue-green deployment for tool state is a pattern that creates a new subset of infrastructure resources with a new tool version. You gradually shift traffic from the old set of resources (blue) to the new set of resources (green). The pattern isolates breaking changes to the new set of resources for testing.

After creating the low-level resources, copy the high-level resources to a new folder. Update its dependencies to use the low-level resources from tool version 4.0. After all, you want all resources using the new tool version.

Figure 13.10 summarizes the strategy. Create the high-level resources, run your tests, and send traffic to the new resources.

This approach differs from the blue-green deployment in chapter 9. You create a new set of resources and an entirely new tool state. If you loosely coupled your dependencies, subsets of resources can have different tool versions without affecting the system’s functionality.

Recall that you can work on IaC with different repository structures (chapter 5) and branching models (chapter 7). Depending on your repository structure and branching model, you can isolate your tool state differently. You can always merge a new branch and alter its pipeline to deploy infrastructure or copy the separate folder into your main configuration for a repository.

Suppose your organization has an opinionated approach to delivery pipelines and allows production deployment only on the main branch. In that case, you can create a new repository for the tool upgrade and archive the old one.

You can apply an in-place upgrade if you dry-run and test the changes thoroughly. If it fails, you create an entirely new resource in a new tool state with the upgraded tool. When in doubt, refer to a tool’s upgrade documentation. As a general practice, I will try an in-place upgrade if the tool offers some kind of migration tool or script to ease the update.

13.3 Replacing tools

I attempted to keep this book as tool-agnostic as possible while offering concrete examples to demonstrate the patterns and practices. I recognize I will probably have to update this book and replace all the tools with the latest and greatest technology! When running IaC for a while, you inevitably change tools for improved functionality or vendor support. What patterns should we use when migrating to a new tool?

Many of the patterns and practices in this book should help protect your system from these changes. Scoping and modularizing your infrastructure with patterns from chapter 3 and decoupling your dependencies with patterns from chapter 4 allows your teams to use tools that fit their use case and replace them as needed. If you do not have these patterns in place, you will encounter some difficulties migrating to a new tool.

Imagine completing your tool upgrades across the IaC for the social media company. You think about taking a break when the network team members ask for your help. They want to move from a vendor’s DSL to an open source DSL. Their configuration will need an additional review from the social media feed team, which does not know anything about the vendor.

Your research cannot find a vendor or open source script to facilitate a straightforward migration. You wished you had something that could translate the vendor’s DSL to the open source one. Without it, you and the network team need to proceed with the migration carefully.

13.3.1 New tool supports import

You could not find automation to “translate” between tools. However, you can apply patterns in this book to migrate across tools. Some tools support an import capability to add new resources to a tool’s state. You can use the practices in chapter 2 to migrate existing resources to the new tool.

In figure 13.11, you upgrade the modules to use the new open source DSL. You also update its tests, which pass. To start, identify some low-level resources you can change. You create a separate folder, branch, or repository to isolate the new DSL from the vendor DSL. After writing the configuration for the new DSL, you import existing resources into the state of the new DSL.

Once again, you rewrite the tests to test the new syntax of the open source DSL. They pass, and you proceed to write configuration and import high-level resources. Finally, you delete the IaC.

Throughout each cycle of writing new IaC with the open source DSL, you want to check your dry run and rewrite your tests. The dry run shows if the defaults for the new tool do not match your existing state. You need to update the new IaC if they do not match and fix the drift.

13.3.2 No import capability

Some tools do not support import capabilities, and you need to create new resources for the new tool. Imagine if the network team asked you to help them convert from one vendor DSL to another. However, the new vendor DSL does not allow the import of existing resources into its state.

If your new tool does not support importing existing resources, you need to re-create resources with a blue-green deployment strategy for the tool state. In figure 13.12, you start by writing new IaC for the low-level resources and refactor tests for the new tool. As you repeat the process and finish migrating high-level resources, you cut over traffic and test the entire system.

The pattern of migrating tools remains consistent, import capability or none. However, migration without an import capability takes more effort because you re-create the system. Even if you had a magical script to migrate from one tool to another, you might consider applying some of these patterns and practices to avoid breaking critical infrastructure resources, like the network.

You will always replace or add tools to your infrastructure ecosystem. Your organization will choose the tools that fit its architectural objectives. Applying the techniques to modularize, isolate, and manage IaC will accommodate your IaC evolution. I always return to the practices and patterns to make infrastructure, IaC, module, tool, and organizational changes and mitigate their risk to critical systems.

Regardless of whether a tool has import capabilities, you need to rewrite the tests for each resource you refactor. Figure 13.13 shows that you must refactor the unit and contract tests for each module and subset of resources as you migrate. However, your end-to-end and integration tests may stay the same.

A new tool will affect unit and contract tests because the tool uses a different state and metadata format. Tests cannot parse the correct information from the new tool. Integration and end-to-end tests likely stay the same because they evaluate the functions of infrastructure and not the tool itself.

Refactoring the tests as you go along allows you to add more tests, remove redundant tests, or make updates to broader security or policy tests. You should update your manual, integration, and end-to-end tests with new input parameters because you have different resources. However, the tests themselves do not change too much because they test the system’s functionality, not infrastructure attributes.

13.4 Event-driven IaC

Most of the book covers the practices of collaborating on and writing IaC to reduce the impact of a potential failure of critical systems. Once you get familiar with the principles and practices, you can extend them to more dynamic use cases.

For example, a development team wants some very dynamic automation with its IaC. Whenever the development team members deploy a new instance of an application, they need to update a firewall rule to allow access from the instance to the database. Rather than push a new instance and then remember to update the firewall rule later, they want some automation to run an infrastructure module to configure the firewall rule after an application instance starts.

Figure 13.14 demonstrates the automation you implement. You deploy a new application with a new IP address. An automated script captures the new IP address and runs some IaC. The configuration updates the firewall with the new IP address. This automation repeats each time you deploy a new application with a new IP address.

You may consider running an infrastructure module automatically when the system changes. Event-driven IaC means running a minimally scoped infrastructure module to configure infrastructure in response to an event. You can use the automation to update another resource or fix the system based on the event.

Definition Event-driven IaC runs a minimally scoped infrastructure module to configure infrastructure in response to an event.

Updating the application equals an event. Some script, application, or automation detects the event and responds by running an infrastructure module! You can write your own or find an open source tool to identify and respond to the event. Some real-world automation you can use to detect and react to an event includes an operator for Kubernetes, a serverless function, or an application that consumes from an event queue.

The use case for event-driven IaC has become more prevalent with dynamic services and applications. If you do use event-driven IaC, keep a few practices in mind:

• Avoid infrastructure resources that take a long time to create or configure. You do not want to include an infrastructure resource that takes an hour to create.

• Do not add too many resources to an event-driven module. Otherwise, you will take a long time to create many instances.

• Balance the time it takes to run a change with the module versus the time interval between events.

• Combine the module practices from chapter 2 with the testing patterns from chapter 6 to verify that event-driven IaC runs quickly and correctly.

Some events happen with high frequency. You need infrastructure that deploys faster than the event’s frequency or an automation script to batch changes at a specific interval. You should choose the most minimal subset of infrastructure resources to deploy with event-driven IaC.

From static IaC that you deploy based on code commits to dynamic IaC run for an event, you apply the same patterns, practices, and principles to managing and collaborating. Your objectives, your team’s requirements, and your organization’s business will evolve and change over time—hopefully, your IaC practice grows with it. Keep in mind your testing strategy, the cost of infrastructure, and maintaining its security and compliance.

Summary

• Review the functionality, security, and life cycle of an open source tool or module before adopting it in your organization.

• An open source tool or module should have default values or behaviors that offer predictability and stability in changes. Otherwise, you will have to write a layer of code to add opinionated default attributes within your organization.

• Protect your infrastructure from supply-chain attacks by scanning an infrastructure tool or module for security, checking for third-party data collection, and running your security and compliance tests.

• The type and number of maintainers and the license associated with the tool or module also affect your organization’s use.

• Open source tools and modules can have two categories of licenses: permissive and copyleft.

• A permissive license lets you modify and update the module or tool as long as you include a copy of the license and its original copyright notice.

• A copyleft license allows you to modify and update the module or tool as long as you include a copy of the license, its original copyright notice, and release your copy open source.

• Before upgrading your IaC tool, decouple infrastructure dependencies and pin module, plugin, and tool versions.

• Start a tool update with backward compatibility by refactoring high-level resources mutably and proceeding to low-level resources.

• A blue-green deployment strategy for tool state means creating a new set of infrastructure resources with a state separate from the existing configuration.

• Tool state refers to a copy of the infrastructure state that an IaC tool uses to detect drift or resources under its management.

• Start a tool update with breaking changes by applying a blue-green deployment strategy to the tool state, starting from low-level to high-level resources.

• When replacing one tool with another, use the new tool’s import capability to migrate existing resources to the new tool. Start with low-level resources and apply them to high-level resources. Then, remove the configuration for the old tool.

• If a new tool does not have an importing feature for existing infrastructure resources, you will need to apply a blue-green deployment strategy for the tool state.

• Event-driven IaC runs a minimal IaC module in response to a system event to automate infrastructure changes.

• Ensure that a module for event-driven IaC deploys quickly with as few resources as possible.