Create and orchestrate Terraform Stacks with Terramate

The introduction of Infrastructure as Code, or IaC, has transformed how we provision and deploy cloud infrastructure as part of a once-in-a-generation shift to the cloud. However, adopting and managing Infrastructure as Code comes with certain challenges.

Today, we are kicking off a series of blog posts explaining how to manage Terraform and OpenTofu with stacks using Terramate efficiently. In this intro post, we’ll discuss how to create and orchestrate stacks. In the rest of this series, we will cover the following topics:

1. Create and orchestrate Terraform stacks with Terramate (this article)
2. Clone and manage stacks (coming soon)
3. Best practices for designing, sizing and structuring stacks (coming soon)
4. Share data between stacks (coming soon)

About Terramate

Terramate is a next-generation Infrastructure as Code (IaC) Management Platform that empowers teams to build, deploy, manage, and observe cloud infrastructure with IaC tools such as Terraform, OpenTofu, Terragrunt, Kubernetes, and others. Terramate offers developers a free and open-source CLI, which can be optionally integrated with its fully managed Cloud service, Terramate Cloud, in an automated manner.

Terramate can be integrated into any existing architecture and with every third-party tooling in less than 5 minutes, with no prerequisites, lock-in, or modifying any existing Terraform configuration. For example, you can onboard Terramate to existing Terraform or Terragrunt projects with a single command.

At the same time, it seamlessly integrates with all your existing tools, such as GitHub or Slack, in a non-intrusive way.

If you want to learn more about Terramate please find an overview at https://terramate.io/docs/introduction.

What are Stacks, and why use them?

Stacks provide a structured approach to organizing Terraform and OpenTofu into manageable units. They allow for independent deployment and management, which reduces execution runtimes and minimizes the impact scope of changes commonly known as “blast radius”.

You can think about a stack as a combination of:

• Infrastructure code which declares a set of infrastructure resources and their configuration. using Terraform or OpenTof (.tf files).
• State that describes the status of the assets according to the latest deployment (e.g., Terraform state, can be stored and managed either locally or in a remote backend)
• Configuration to configure the infrastructure assets and stack behavior (e.g., Variables, Stack Configuration, etc.)

Stacks are a great way to bundle all the resources and configurations required to deploy a specific service that can easily be promoted across different environments.

This guide explains how to create, manage, and orchestrate stacks, focusing on actionable and easy-to-understand examples.

Prerequisites

Ensure you have Terraform and Terramate installed and configured for your project.

Creating Stacks

Let’s dive into creating stacks and exploring Terramate’s capabilities.

Initialize a New Repository:

git init -b main terramate-stacks
cd terramate-stacks

Create Example Stacks: To understand how stacks function, let’s create a few examples:

terramate create --name "Example Stack A" --description "This is an awesome first example stack" stacks/a
terramate create --name "Example Stack B" --description "This is an awesome second example stack" stacks/b
terramate create --name "Example Stack C" --description "This is an awesome third example stack" stacks/c

These commands set up three distinct stacks that we can use for experimentation and learning.

Explore stack information: To view all available stacks in your project, use:

terramate list

stacks/a
stacks/b
stacks/c

Detailed stack metadata: For a more in-depth look at stack metadata, run:

terramate debug show metadata

This command provides comprehensive information about available stacks and their metadata.

terramate debug show metadata

Available metadata:
project metadata:
        terramate.stacks.list=[/stacks/a /stacks/b /stacks/c]
stack "/stacks/a":
        terramate.stack.id="e4d53140-a2de-4616-a667-0b6376f6dfd7"
        terramate.stack.name="Example Stack A"
        terramate.stack.description="This is an awesome first example stack"
        terramate.stack.tags=[]
        terramate.stack.path.absolute="/stacks/a"
        terramate.stack.path.basename="a"
        terramate.stack.path.relative="stacks/a"
        terramate.stack.path.to_root="../.."
stack "/stacks/b":
        terramate.stack.id="917d2f76-2f62-49f1-ade6-345d08191bd6"
        terramate.stack.name="Example Stack B"
        terramate.stack.description="This is an awesome second example stack"
        terramate.stack.tags=[]
        terramate.stack.path.absolute="/stacks/b"
        terramate.stack.path.basename="b"
        terramate.stack.path.relative="stacks/b"
        terramate.stack.path.to_root="../.."
stack "/stacks/c":
        terramate.stack.id="3fd79337-b83a-46a3-8d29-62f091df6e14"
        terramate.stack.name="Example Stack C"
        terramate.stack.description="This is an awesome third example stack"
        terramate.stack.tags=[]
        terramate.stack.path.absolute="/stacks/c"
        terramate.stack.path.basename="c"
        terramate.stack.path.relative="stacks/c"
        terramate.stack.path.to_root="../.."

Working with Terraform in Stacks

In Terramate, stacks are simple initially just directories with a stack.tm.hcl file defining the metadata and configuration of a stack. However, to leverage stacks for managing infrastructure using Terraform or OpenTofu, we must create actual Terraform configurations into each stack.

Adding Terraform configurations

Let’s dive into adding Terraform configurations to our stacks:

echo 'resource "null_resource" "stack" {}' | tee stacks/a/main.tf stacks/b/main.tf stacks/c/main.tf > /dev/null

The above command creates a main.tf file in each stack, declaring a Terraform null resource.

Now, each of our example stacks (stacks/a, stacks/b, stacks/c) contains a Terraform configuration that allows us to explore how to use commands such as terraform init by using the Terramate orchestration.

Orchestration with Terramate

Each of the created stacks is considered to be an isolated Terraform environment, which means we need to run terraform init in each.

The following orchestrate the init command in each stack (stacks/a, stacks/b, stacks/c) sequentially:

terramate run terraform init

Output:

terramate: Entering stack in /stacks/a
terramate: Executing command "terraform init"

Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/null from the dependency lock file
- Installing hashicorp/null v3.2.2...
- Installed hashicorp/null v3.2.2 (signed by HashiCorp)
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
terramate: Entering stack in /stacks/b
terramate: Executing command "terraform init"
Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/null from the dependency lock file
- Installing hashicorp/null v3.2.2...
- Installed hashicorp/null v3.2.2 (signed by HashiCorp)
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
terramate: Entering stack in /stacks/c
terramate: Executing command "terraform init"
Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/null from the dependency lock file
- Installing hashicorp/null v3.2.2...
- Installed hashicorp/null v3.2.2 (signed by HashiCorp)
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Upon executing the command, Terramate orchestrates terraform init across all stacks, following the order of execution similar to your file system structure. For instance, running tree in your Terramate project reveals the orchestration order:

❯ tree
.
└── stacks
    ├── a
    │   ├── main.tf
    │   └── stack.tm.hcl
    ├── b
    │   ├── main.tf
    │   └── stack.tm.hcl
    └── c
        ├── main.tf
        └── stack.tm.hcl

5 directories, 6 files

Terramate executes terraform init first in stack a, then stack b, and finally stack c. While this default behavior is convenient, you might need to change the order. That's where stack orchestration configuration becomes useful.

Customizing the order of execution

To modify the default order of execution, we can explicitly configure the stack’s orchestration behavior using Terramate’s before-and-after configuration.

Let’s adjust stack a to be orchestrated after stack b. Open the stack configuration(stack.tm.hcl) of stack a and add the following:

stack {
  name        = "Example Stack A"
  description = "This is an awesome first example stack"
  id          = "e4d53140-a2de-4616-a667-0b6376f6dfd7"

after = [
    "../b"
  ]
}

After configuring the stack, when we use Terramate to run the pwd command in all stacks, you'll notice that stack b is executed before stack a.

terramate run pwd

terramate: Entering stack in /stacks/a
terramate: Executing command "pwd"
/terramate-quickstart/stacks/a
terramate: Entering stack in /stacks/b
terramate: Executing command "pwd"
/terramate-quickstart/stacks/b
terramate: Entering stack in /stacks/c
terramate: Executing command "pwd"
/terramate-quickstart/stacks/c

Terramate provides a range of configuration options for stack orchestration. For detailed information, refer to the documentation here.

By the way, you can use terramate list --run-order to understand the order of execution of your stacks without having to execute a command in the first place.

Nesting Stacks

Terramate Stacks can be nested, allowing you to map your infrastructure code as a tree, which leads to a natural organization of your infrastructure resources with Infrastructure as Code.

Let’s explore nested stacks by adding several of them to stack a :

terramate create --name "Example Nested Stack X" --description "This is an awesome first example nested stack" stacks/a/x
terramate create --name "Example Nested Stack Y" --description "This is an awesome first example nested stack" stacks/a/y
terramate create --name "Example Nested Stack Z" --description "This is an awesome first example nested stack" stacks/a/z

Executing the above command results in the following structure:

❯ tree
.
└── stacks
    ├── a
    │   ├── main.tf
    │   ├── stack.tm.hcl
    │   ├── x
    │   │   └── stack.tm.hcl
    │   ├── y
    │   │   └── stack.tm.hcl
    │   └── z
    │       └── stack.tm.hcl
    ├── b
    │   ├── main.tf
    │   └── stack.tm.hcl
    └── c
        ├── main.tf
        └── stack.tm.hcl

8 directories, 9 files

Let’s run terraform init as before to observe the execution order:

terramate run terraform init

Output:

terramate: Entering stack in /stacks/b
terramate: Executing command "terraform init"

Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/null from the dependency lock file
- Installing hashicorp/null v3.2.2...
- Installed hashicorp/null v3.2.2 (signed by HashiCorp)
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
terramate: Entering stack in /stacks/a
terramate: Executing command "terraform init"
Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/null from the dependency lock file
- Installing hashicorp/null v3.2.2...
- Installed hashicorp/null v3.2.2 (signed by HashiCorp)
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
terramate: Entering stack in /stacks/a/x
terramate: Executing command "terraform init"
Terraform initialized in an empty directory!
The directory has no Terraform configuration files. You may begin working
with Terraform immediately by creating Terraform configuration files.
terramate: Entering stack in /stacks/a/y
terramate: Executing command "terraform init"
Terraform initialized in an empty directory!
The directory has no Terraform configuration files. You may begin working
with Terraform immediately by creating Terraform configuration files.
terramate: Entering stack in /stacks/a/z
terramate: Executing command "terraform init"
Terraform initialized in an empty directory!
The directory has no Terraform configuration files. You may begin working
with Terraform immediately by creating Terraform configuration files.
terramate: Entering stack in /stacks/c
terramate: Executing command "terraform init"
Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/null from the dependency lock file
- Installing hashicorp/null v3.2.2...
- Installed hashicorp/null v3.2.2 (signed by HashiCorp)
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

When running terraform init, Terramate follows these steps:

1. Executes in stack b first, as configured explicitly before stack a.
2. Runs the init command in stack a first and then in its nested stacks, executing them sequentially.
3. Finally, it executes stack c after stack a and its nested stacks.

Change Detection and Execution

Always executing commands such as terraform init in all available stacks can lead to long execution run times that are unnecessary and should be avoided. This is why Terramate offers support for detecting stacks with changes and orchestrating the execution of commands exclusively in those stacks.

Let’s start by committing our current changes:

git add stacks
git commit -m "feat: add some stacks"

Next, create a new branch and add a new stack within this branch:

git checkout -b new

terramate create --name "Example Stack D" --description "This is an awesome fourth example stack" stacks/d

Commit the newly created stack in the new branch:

git add stacks/d
git commit -m "feat: add a new stack"

First, we can get an overview of all stacks containing changes using the list command:

terramate list --changed

stacks/d

Next, let’s run terraform init using the change detection feature:

❯ terramate run --changed terraform init

Output:

terramate: Entering stack in /stacks/d
terramate: Executing command "terraform init"
Terraform initialized in an empty directory!

The directory has no Terraform configuration files. You may begin working
with Terraform immediately by creating Terraform configuration files.

As observed, Terramate detects changes in stack d within the current branch and executes terraform init exclusively in this stack. Leveraging change detection significantly reduces execution times and minimizes the impact scope, as unchanged stacks are ignored unless explicitly configured. Terramate's change detection is dynamic and configurable for various scenarios, including detecting changes in used Terraform modules. For detailed information, refer to the documentation at Terramate Change Detection - Git Integration and Terramate Change Detection - Terraform Integration.

Parallel Execution

After covering stack creation, nesting, and orchestration basics with Terramate, let’s focus on another awesome feature: executing commands in stacks concurrently.

Picture a scenario with dozens or hundreds of stacks that need to must be orchestrated. Since Terramate orchestrates commands in stacks sequentially by default, this could potentially take a long time! Luckily, Terramate supports parallel execution with the parallel flag, which lets you set the maximum number of concurrent runs and boost command execution speed.

For instance, consider the following command:

terramate run --parallel=3 terraform apply -auto-approve

Here’s what this command does:

1. terramate run --parallel=3: Orchestrates the execution of commands in stacks in parallel, limiting the concurrent execution to 3.
2. terraform apply -auto-approve: Executes the terraform apply command with auto-approval across all stacks in parallel.

Parallel execution is great for orchestrating multiple stacks in parallel that don’t depend on each other and can reduce the execution run time and waiting times significantly.

Summary

In this article, we’ve covered the basics of managing stacks with Terramate. We’ve learned how to create stacks, nest them, and orchestrate commands sequentially or in parallel.

In the next part of this series, we’ll explore cloning and managing stacks, and delve into structuring and sizing them effectively.

Stay tuned for more insights into maximizing your Terramate experience!

This article originally appeared in Rethinking IaC on March, 22, 2024