Terramate and Terragrunt

In case you haven’t seen the international headlines: We published an introduction blog post for Terramate, an orchestrator and code generator for Terraform environments at scale. Please read it in our blog if you want to become a better human and awesome at Terraform.

Jokes aside, this article dives deeper into why we built Terramate when existing tools like Terragrunt are solving very similar problems. So let’s compare the features in more depth.

Design decisions

When designing Terramate, we thought of ways to be less intrusive in existing setups no matter what tooling was being used. We explicitly wanted to support a broad list of setups people have built out there without causing migration efforts.

Most of the existing setups we’ve seen in live production with our clients are still run on mono state repositories, which have long run times for creating Terraform plans and applies. Another observation is that many teams work in an environment in which Terraform isn’t fully automated. Instead, we see individual contributors or platform teams that run Terraform manually on their workstations as they do not trust automation due to the increased blast radius and risks of using a single state.

Some people use tools like Terragrunt or Atlantis that introduced splitting a single state into multiple smaller units, via isolated stacks.

Also, when using plain Terraform in different stacks we observed that teams are writing their own scripts to orchestrate the various states and tackle the complexity such a split brought.

Before creating Terramate, we did the same: We built a set of Bash scripts, Makefiles and git commands. For some time it worked well and this setup helped us detect changes (only one module-level deep) and to run Terraform commands only on changed stacks(a static set of very specific commands). That allowed us to create previews in pull requests and provided our customers with a full CI/CD experience when using stacks. Of course, this approach was limited since this automation was hard to maintain across multiple projects, not very flexible, and not easily distributable (resulting in duplication).

Based on this experience, we started replacing the above-mentioned prototype with a single Go tool just for taking care of orchestration.

Orchestration

After conducting research on the existing tooling we came to the conclusion that no tool that integrates well with Git, and detects changes based on pull requests or after a merge existed. Some tools come with change detection (e.g. Atlantis) but we needed to support specific features in environments not having infrastructure ready to run such tools. As well as being able to use such powers manually on the command line.

One of the core design decisions here was to be able to execute any commands in the identified stacks. This can be used to run Terraform workflows but also to run maintenance or tooling for static analysis of the code found in stacks; or just simply listing the contents of a set of stacks by running a very basic ls -la.

As most of our customers and also our team is using Git as a VCS, the choice of integrating with Git for filtering stacks based on detected changes was easy to take. Other VCS can be integrated over time.

Terramate ended up being a lean tool that is able to detect stacks, detect changes made within git branches and thus detect changed stacks within a specific git branch.

While testing the first versions of Terramate we came to the conclusion that orchestration is helpful, but it only solves some issues we face on a day-to-day basis when dealing with stacks.

Code Generation

Once we promoted a stack from a staging environment to production, we couldn’t simply copy the stack. We had to take care of the changes within the configuration and numerous other aspects. Configuring providers, credentials, the Terraform backend, naming of variables and other environmental differences.

Of course, we are aware of tools like Terragrunt and also used them with various customers, but deciding to use Terragrunt when coming from a pure Terraform setup comes with some migration effort and some constraints that are not easy to take for some setups. For example, we experience high maintenance efforts when upgrading Terraform versions in previous projects, were also migrating Terragrunt config was needed on top. To make a long story short, we faced setups where Terragrunt was not an option for our clients.

But we still wanted to keep the code DRY. So we decided to dig into requirements and decided to make Terramate not only orchestrate stacks but also configure them.

Several approaches on how to configure a Terraform stack exist:

• Using input variables in stacks. Those can be populated
  — via environment variables with TF_VAR_ prefix (as done by Terragrunt)
  — via command line option -var when calling Terraform
  — using .tfvars or .auto.tfvars files
  — or by specifying default values in the variable definition
• Using Terraform Locals to split between configuration and code.
• Not using any variables and hard coding values, which is of course not recommended but still seen.

Terragrunt does well in generating the needed configuration for provider, backend, and even more — but as the code generation is based on strings, it is easy to generate simple HCL files but generating HCL files with more complex data structures is an open issue as of today.

As Terragrunt translates its own config into Terraform calls by providing input values to modules using environment variables, it is hard to just enter a stack and run any Terraform command without using Terragrunt as it takes care of populating those environment variables for you. This is not a big issue as Terragrunt wraps all commands very well, but of course, it is not easy to revert lock-in that we and our customers would like to avoid.

Our first approach when thinking about code generation was to simply add what is needed to solve the biggest pains which are generating the backend configuration and providers set up. To do so we need to get the desired configuration from somewhere and we came up with global variables, simply called: Globals.

Globals as a Source of Data in Code Generation

Terraform is great when it comes to local variables. They come with lazy evaluations, they can be defined in any order, in any file, and used anywhere.

But as we wanted to share data between stacks on a more global level within the repository, we needed to think about additional features our global variables need.

Globals need to be configurable within the hierarchy of a project to make the most use of them and allow to set eg. a region, environment, or other options for a subset of stacks.

That’s why we made globals aware of the hierarchy and allow for inheritance and overwriting out of the box. In addition, we allow lazy evaluation within the complete hierarchy. So the value of a global variable used on the top level does not need to be known on this level as long as all stacks will be able to set a value at some point. All globals are evaluated in the stack context allowing us to create complex logic on higher levels and share them within stacks.

We will providea detailed blog post on Globals in Terramate soon.

Generating valid HCL Code

Whilst working on the initial code generation features in Terramate such as generating backend and provider configuration, we came to the conclusion that globals would be powerful for more than just configuring Terraform.

An idea we had early on was to make the shared data available to all Terraform code within a stack by generating files that contain locals based on globals.

This was great but not really satisfying as it added a layer of complexity that we could avoid by allowing us to generate any HCL.

This led us to the decision to make Terramate aware of HCL and added the possibility to generate valid HCL code, allowing the user to use functions as known from Terraform and to create any Terraform code based on globals and Terramate metadata.

Having generated valid HCL we removed all previously implemented special handling of Terraform in both backend and provider config. This made Terramate more independent from just being used with Terraform and actually being used with any Tool that uses HCL as a configuration language, e.g. Hashicorps Packer or actually Terragrunt configuration which is also HCL.

Generating any file types

On top of generating valid HCL, Terramate also supports generation of files of any type. By supporting functions and Terramate Globals in file generation, Terramate can generate valid YAML and JSON from HCL structures but also any other string-based content.

Differences to Terragrunt

In the previous section, we already mentioned some opinionated concerns we and our customers have when it comes to adopting Terragrunt.

The biggest differences to Terragrunt from our point of view in the current state of development are:

• Terramate is not a wrapper for Terraform but can execute any command
• Terramate is focusing on providing a great orchestration experience as it comes with a Git integration to detect changes made in branches.
• Terramate can generate more complex HCL and validates it when reading the configuration and write out well formatted and valid generated HCL.
• Terramate provides lazy evaluated, hierarchy aware global variables that can be used in HCL and generic file generation to share any data between stacks.
• Terramate integrates well with any tooling including Terragrunt and can generate Terragrunt configurations as of today (it still lacks change detection for Terragrunt, which we already added to our roadmap)
• Terramate was not designed to replace any existing tool but to enable great features such tools introduced in a non-intrusive way without major lock-ins and for a broader audience that want to keep using such tools — maybe in combination with Terramate in the future.

More features will be added over time so feel free to watch and star our Terramate GitHub repository.

To get started with Terramate please see the documentation.

Also, we launched a Terramate Discord Server for you to ask questions about Terramate, suggest new features, connect with other members and broaden your circle to learn from one another. If you are working with Terramate, we recommend you join!
👉 https://terramate.io/discord

This article originally appeared in Rethinking IaC on May 24, 2022