Skip to main content

Bootstrap Pipelines in an Existing Repository

This guide provides comprehensive instructions for integrating Gruntwork Pipelines into an existing repository with Infrastructure as Code (IaC). This is designed for Gruntwork customers who want to add Pipelines to their current infrastructure repositories for streamlined CI/CD management.

To configure Gruntwork Pipelines in an existing repository, complete the following steps (which are explained in detail below):

  1. Plan your Pipelines setup by identifying all environments and cloud accounts/subscriptions you need to manage.
  2. Bootstrap core infrastructure in accounts/subscriptions that don't already have the required OIDC and state management resources.
  3. Configure SCM access using either the Gruntwork.io GitHub App or machine users.
  4. Create .gruntwork HCL configurations to tell Pipelines how to authenticate and organize your environments.
  5. Create .github/workflows/pipelines.yml to configure your GitHub Actions workflow.
  6. Commit and push your changes to activate Pipelines.

Prerequisites​

Before starting, ensure you have:

  • An active Gruntwork subscription with Pipelines access. Verify by checking the Gruntwork Developer Portal and confirming access to "pipelines" repositories in your GitHub team.
  • Cloud provider credentials with permissions to create OIDC providers and IAM roles in accounts where Pipelines will manage infrastructure.
  • Git installed locally for cloning and managing your repository.
  • Existing IaC repository with Terragrunt configurations you want to manage with Pipelines (if you are using OpenTofu/Terraform, and want to start using Terragrunt, read the Quickstart Guide).

Planning Your Pipelines Setup​

Before implementing Pipelines, it's crucial to plan your setup by identifying all the environments and cloud resources you need to manage.

Identify Your Environments​

Review your existing repository structure and identify:

  1. All environments you want to manage with Pipelines (e.g., dev, staging, prod)
  2. Cloud accounts/subscriptions associated with each environment
  3. Directory paths in your repository that contain Terragrunt units for each environment
  4. Existing OIDC resources that may already be provisioned in your accounts
Progress Checklist

Determine Required OIDC Roles​

For each AWS Account / Azure Subscription you want to manage, you might already have some or all of the following resources provisioned.

Required AWS Resources:

  • An OIDC provider for GitHub Actions
  • An IAM role for Pipelines to assume when running Terragrunt plan commands
  • An IAM role for Pipelines to assume when running Terragrunt apply commands
Progress Checklist

Configuring SCM Access​

Pipelines needs the ability to interact with Source Control Management (SCM) platforms to fetch resources (e.g. IaC code, reusable CI/CD code and the Pipelines binary itself).

There are two ways to configure SCM access for Pipelines:

  1. Using the Gruntwork.io GitHub App (recommended for most GitHub users).
  2. Using a machine user (recommended for GitHub users who cannot use the GitHub App).
Progress Checklist

Bootstrapping Cloud Infrastructure​

If your AWS accounts / Azure subscriptions don't already have all the required OIDC and state management resources, you'll need to bootstrap them. This section provides the infrastructure code needed to set up these resources.

tip

If you already have all the resources listed, you can skip this section.

If you have some of them provisioned, but not all, you can decide to either destroy the resources you already have provisioned and recreate them or import them into state. If you are not sure, please contact Gruntwork support.

Prepare Your Repository​

Clone your repository to your local machine using Git if you haven't already.

tip

If you don't have Git installed, you can install it by following the official guide for Git installation.

For example:

git clone git@github.com:acme/infrastructure-live.git
cd infrastructure-live
Progress Checklist

To bootstrap your repository, we'll use Boilerplate to scaffold it with the necessary IaC code to provision the infrastructure necessary for Pipelines to function.

The easiest way to install Boilerplate is to use mise to install it.

tip

If you don't have mise installed, you can install it by following the official guide for mise installation.

mise use -g boilerplate@latest
tip

If you'd rather install a specific version of Boilerplate, you can use the ls-remote command to list the available versions.

mise ls-remote boilerplate
Progress Checklist

If you don't already have Terragrunt and OpenTofu installed locally, you can install them using mise:

mise use -g terragrunt@latest opentofu@latest
Progress Checklist

Cloud-specific bootstrap instructions​

The resources you need provisioned in AWS to start managing resources with Pipelines are:

  1. An OpenID Connect (OIDC) provider
  2. An IAM role for Pipelines to assume when running Terragrunt plan commands
  3. An IAM role for Pipelines to assume when running Terragrunt apply commands

For every account you want Pipelines to manage infrastructure in.

Don't Panic!

This may seem like a lot to set up, but the content you need to add to your repository is minimal. The majority of the work will be pulled from a reusable catalog that you'll reference in your repository.

If you want to peruse the catalog that's used in the bootstrap process, you can take a look at the terragrunt-scale-catalog repository.

The process that we'll follow to get these resources ready for Pipelines is:

  1. Use Boilerplate to scaffold bootstrap configurations in your repository for each AWS account
  2. Use Terragrunt to provision these resources in your AWS accounts
  3. (Optionally) Bootstrap additional AWS accounts until all your AWS accounts are ready for Pipelines

Bootstrap Your Repository for AWS

First, confirm that you have a root.hcl file in the root of your repository that looks something like this:

root.hcl
locals {
  account_hcl       = read_terragrunt_config(find_in_parent_folders("account.hcl"))
  state_bucket_name = local.account_hcl.locals.state_bucket_name

  region_hcl = read_terragrunt_config(find_in_parent_folders("region.hcl"))
  aws_region = local.region_hcl.locals.aws_region
}

remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite"
  }
  config = {
    bucket       = local.state_bucket_name
    region       = local.aws_region
    key          = "${path_relative_to_include()}/tofu.tfstate"
    encrypt      = true
    use_lockfile = true
  }
}

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
provider "aws" {
  region = "${local.aws_region}"
}
EOF
}

If you don't have a root.hcl file, you might need to customize the bootstrapping process, as the Terragrunt scale catalog expects a root.hcl file in the root of the repository. Please contact Gruntwork support for assistance if you need help.

For each AWS account that needs bootstrapping, we'll use Boilerplate to scaffold the necessary content. Run this command from the root of your repository for each account:

boilerplate \
  --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/aws/github/account?ref=v1.1.0' \
  --output-folder .
tip

You'll need to run this boilerplate command once for each AWS account you want to manage with Pipelines. Boilerplate will prompt you for account-specific values each time.

tip

You can reply y to all the prompts to include dependencies, and accept defaults unless you want to customize something.

Alternatively, you could run Boilerplate non-interactively by passing the --non-interactive flag. You'll need to supply the relevant values for required variables in that case.

e.g.

boilerplate \
  --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/aws/github/account?ref=v1.1.0' \
  --output-folder . \
  --var 'AccountName=dev' \
  --var 'GitHubOrgName=acme' \
  --var 'GitHubRepoName=infrastructure-live' \
  --var 'AWSAccountID=123456789012' \
  --var 'AWSRegion=us-east-1' \
  --var 'StateBucketName=my-state-bucket' \
  --non-interactive

You can also choose to store these values in a YAML file and pass it to Boilerplate using the --var-file flag.

vars.yml
AccountName: dev
GitHubOrgName: acme
GitHubRepoName: infrastructure-live
AWSAccountID: 123456789012
AWSRegion: us-east-1
StateBucketName: my-state-bucket
boilerplate \
  --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/aws/github/account?ref=v1.1.0' \
  --output-folder . \
  --var-file vars.yml \
  --non-interactive
Progress Checklist

Provision AWS Bootstrap Resources

Once you've scaffolded out the accounts you want to bootstrap, you can use Terragrunt to provision the resources in each of these accounts.

tip

Make sure that you authenticate to each AWS account you are bootstrapping using AWS credentials for that account before you attempt to provision resources in it.

You can follow the documentation here to authenticate with the AWS provider. You are advised to choose an authentication method that doesn't require any hard-coded credentials, like assuming an IAM role.

For each account you want to bootstrap, you'll need to run the following commands:

First, make sure that everything is set up correctly by running a plan in the bootstrap directory in name-of-account/_global where name-of-account is the name of the AWS account you want to bootstrap.

name-of-account/_global/bootstrap
terragrunt run --all --non-interactive --provider-cache --backend-bootstrap plan
tip

We're using the --provider-cache flag here to ensure that we don't re-download the AWS provider on every run by leveraging the Terragrunt Provider Cache Server.

We're using the --backend-bootstrap flag here to tell Terragrunt to bootstrap the OpenTofu backend automatically for the account.

Next, apply the changes to your account.

name-of-account/_global/bootstrap
terragrunt run --all --non-interactive --provider-cache apply
Progress Checklist

Creating .gruntwork HCL Configurations​

Create HCL configurations in the .gruntwork directory in the root of your repository to tell Pipelines how you plan to organize your infrastructure, and how you plan to have Pipelines authenticate with your cloud provider(s).

The repository block​

The core configuration that you'll want to start with is the repository block. This block tells Pipelines which branch has the "live" infrastructure you want provisioned. When you merge IaC to this branch, Pipelines will be triggered to update your infrastructure accordingly.

.gruntwork/repository.hcl
repository {
  deploy_branch_name = "main"
}
Progress Checklist

The environment block​

Next, you'll want to define the environments you want to manage with Pipelines using the environment block.

For each environment, you'll want to define a filter block that tells Pipelines which units are part of that environment. You'll also want to define an authentication block that tells Pipelines how to authenticate with your cloud provider(s) for that environment.

.gruntwork/environment-production.hcl
environment "production" {
  filter {
    paths = ["prod/*"]
  }

  authentication {
    aws_oidc {
      account_id         = "123456789012"
      plan_iam_role_arn  = "arn:aws:iam::123456789012:role/pipelines-plan"
      apply_iam_role_arn = "arn:aws:iam::123456789012:role/pipelines-apply"
    }
  }
}
tip

Learn more about how Pipelines authenticates to AWS in the Authenticating to AWS page.

tip

Check out the aws block for more information on how to configure Pipelines to reuse common AWS configurations.

Progress Checklist

Creating .github/workflows/pipelines.yml​

Create a .github/workflows/pipelines.yml file in the root of your repository with the following content:

.github/workflows/pipelines.yml
name: Pipelines
run-name: "[GWP]: ${{ github.event.commits[0].message || github.event.pull_request.title || 'No commit message' }}"
on:
  push:
    branches:
      - main
    paths-ignore:
      - ".github/**"
  pull_request:
    types:
      - opened
      - synchronize
      - reopened
    paths-ignore:
      - ".github/**"

# Permissions to assume roles and create pull requests
permissions:
  id-token: write
  contents: write
  pull-requests: write

jobs:
  GruntworkPipelines:
    uses: gruntwork-io/pipelines-workflows/.github/workflows/pipelines.yml@v4
tip

You can read the Pipelines GitHub Actions Workflow to learn how this GitHub Actions workflow calls the Pipelines CLI to run your pipelines.

Progress Checklist

Commit and Push Your Changes​

Commit and push your changes to your repository.

note

You should include [skip ci] in your commit message here to prevent triggering the Pipelines workflow before everything is properly configured.

git add .
git commit -m "Add Pipelines configurations and GitHub Actions workflow [skip ci]"
git push
Progress Checklist

🚀 You've successfully added Gruntwork Pipelines to your existing repository!

Next Steps​

You have successfully completed the installation of Gruntwork Pipelines in an existing repository. Proceed to Deploying your first infrastructure change to begin deploying changes.

Troubleshooting Tips​

If you encounter issues during the setup process, here are some common troubleshooting steps:

Bootstrap Resources Failure​

If your bootstrap resource provisioning fails:

HCL Configuration Issues​

If your HCL configurations aren't working as expected:

GitHub Actions Workflow Issues​

If your GitHub Actions workflow isn't working as expected: