Terraform is an open source project maintained by HashiCorp, which gives an infrastructure as code (IaC) interface to cloud providers.
The idea behind IaC is that instead of managing cloud infrastructure via the console UI or explicitly calling create/delete/update APIs. The code describes the desired state of affairs.
The system (in this case, Terraform) is responsible for reconciling the actual cloud infrastructure with the desired state. This means that the infrastructure is managed as code; updates to the infrastructure are code reviewed, approved, and then merged.
Terraform is a powerful and sophisticated project. There are many resources to learn how to use it, such as the official tutorial on terraform.io or many video tutorials on YouTube.
Terraform’s native language, HCL, is a domain-specific language to define infrastructure configuration. In more sophisticated use-cases, it is easy to run to its limit.
For example, HCL does have a for-like construct for creating several similar objects that are different by a parameter. It lacks, however, a conditional statement.
This means that while it is possible to create several AWS S3 buckets, it is not possible to set or not set a time-to-live (TTL) on the objects in them based on a parameter. In this example, it is possible to set a long TTL on the objects.
These workarounds and accommodations stack up quickly and complicate the HCL sources with comments explaining why each was necessary. For cases like this, it is possible to programmatically generate Terraform configurations.
One way to generate Terraform configurations is using the native Cloud Development Kit (CDK), which supports several languages, among them Python. At the time of writing, the CDK and cdktf tools are still in beta.
A different approach is to take advantage of a lesser-known feature of Terraform. Terraform can take its input not just in HCL but also in JSON. Any programming language, including Python, can generate these JSON files.
Terraform is used to configure cloud environments, and learning to use it properly takes time and practice. The following examples do not focus on teaching Terraform but on how to automate Terraform configuration using Python. Because of this, the examples contain a Terraform usage that is much simpler than configuring cloud environments.
Terraform contains the local provider, which allows reading and writing local files. This is not a good use case for Terraform but a great source of examples for automating Terraform configuration.
The following examples create Terraform configurations that produce a directory with files containing greetings for someone. In a more realistic Terraform usage, the files are stand-ins for some cloud-based resource, like an AWS S3 bucket or a Google GKS cluster.
The examples also contain the name of the person being greeted. This is a stand-in for the things which differ between different environments. The same Terraform configuration is used to create the staging and production clusters.
15.1 JSON Syntax
The native Terraform HCL is more convenient for writing Terraform configuration by hand. Before learning how to generate JSON configurations automatically, it is useful to learn how to write them by hand, even if this is not convenient.
Usually, the provider configuration—often put in a main.tf file—does not need to be generated since it tends to be constant across environments.
The main things that need to be configured are Terraform resources and Terraform variables. Unlike in HCL, when using JSON, each resource or variable needs to be in its own file. While this makes it harder to write by hand, it makes it somewhat easier to generate.
The following example writes a greeting into a file. The person being greeted is configured with a variable.
15.2 Generating Terraform Configurations
In the previous example, there was only one greeting: hello. If you need several greetings, some of them working slightly differently, it makes sense to generate the Terraform .tf.json files from code.
The code can have the relevant parameters baked into it if it makes sense or accepts them from some configuration source if the same code might need to generate different configurations. It is useful, in general, to use the native Terraform parametrizing abilities to create different environments while using the generators to create different objects in the same environment that still share some commonalities. The person variable is kept as a Terraform-level variable and not generated from the code.
The first step in generating a local_file resource, equivalent to the earlier resource written by hand, is to generate the right shape for the data structure.
This function is fairly generic. It does not really understand greetings. It is focused on the shape of the Terraform resource, which is often a useful abstraction. The more sophisticated the generator is, the more it makes sense to refactor abstractions like this into a Terraform generator utility library.
At some point, the generator does need to have a specific code. In this example, farewell greetings add a see you later. Silly as it sounds, this is a stand-in for a typical use case. Often, the different objects generated might have some conditional parts. This tends to be the simplest case where a generator is useful.
Using yield allows returning an iterator over the objects. Often, these tend to be a better abstraction than something that needs to explicitly call the function once at a time since they can maintain a local state. In this case, the state is the index, making sure the files do not collide.
After generating, Terraform needs to be run. Running Terraform post-generation depends on how the rest of the flow needs to be done. If this is done in a persistent directory, terraform init does not need to be re-run every time. If something like Terraform Cloud is used, after generation, you can use the TF Cloud API to upload the .tf.json files (along with any manually maintained .tf files) instead of having TF Cloud track your repository.
This creates all greeting files under a sandbox, with the farewell greetings having an additional “see you later.”
15.3 Summary
cdktf may become a reliable, production-grade tool in the future. It is not reliable enough to be the basis of an infrastructure as code pipeline.
Text templating languages are often limited themselves and extensible with Python. Directly using Python removes one step of indirection.
Text templating language does not understand HCL and can result in invalid HCL. When writing JSON, the JSON output is always valid JSON. Higher-level abstractions in Python can guarantee that higher-level syntax will be correct.
This mechanism is fully compatible with anything in the Terraform ecosystem since this ecosystem is already set up for integration with Terraform JSON configuration. Whether you run Terraform manually from a console, automatically from a CI, use Terraform Cloud, or Terraform Enterprise, this is compatible with all workflows. It is also compatible with separating the planning and execution phases and using a pre-written plan.