Skip to main content

Bootstrap Pipelines in a New GitLab Project

To configure Gruntwork Pipelines in a new GitLab project, complete the following steps (which are explained in detail below):

  1. (If using a self-hosted GitLab instance) Ensure OIDC configuration and JWKS are publicly accessible.
  2. Create an infrastructure-live project.
  3. Configure machine user tokens for GitLab access, or ensure that the appropriate machine user tokens are set up as project or organization secrets.
  4. Create .gruntwork HCL configurations to tell Pipelines how to authenticate in your environments.
  5. Create .gitlab-ci.yml to tell your GitLab CI/CD pipeline how to run your pipelines.
  6. Commit and push your changes to your project.

Ensure OIDC configuration and JWKS are publicly accessible​

This step only applies if you are using a self-hosted GitLab instance that is not accessible from the public internet. If you are using GitLab.com or a self-hosted instance that is publicly accessible, you can skip this step.

  1. Follow GitLab's instructions for hosting your OIDC configuration and JWKS in a public location (e.g. S3 Bucket). This is necessary for both Gruntwork and the AWS OIDC provider to access the GitLab OIDC configuration and JWKS when authenticating JWT's generated by your custom instance.
  2. Note the <ISSUER_URL> (stored as ci_id_tokens_issuer_url in your gitlab.rb file per GitLab's instructions) generated above for reuse in the next steps.
Progress Checklist

Creating the infrastructure-live project​

Creating an infrastructure-live project is fairly straightforward. First, create a new project using the official GitLab documentation for creating repositories. Name the project something like infrastructure-live and make it private (or internal).

Configuring SCM Access​

Pipelines needs the ability to interact with Source Control Management (SCM) platforms to fetch resources (e.g. IaC code, reusable CI/CD code and the Pipelines binary itself).

For GitLab, you'll need to configure SCM access using machine users with appropriate Personal Access Tokens (PATs).

Progress Checklist

Creating Cloud Resources for Pipelines​

To start using Pipelines, you'll need to ensure that requisite cloud resources are provisioned in your cloud provider(s) to start managing your infrastructure with Pipelines.

note

If you are using the Gruntwork Account Factory, this will be done automatically during onboarding and in the process of vending every new AWS account, so you don't need to worry about this.

Clone your infrastructure-live project repository to your local machine using Git.

tip

If you don't have Git installed, you can install it by following the official guide for Git installation.

For example:

git clone git@gitlab.com:acme/infrastructure-live.git
cd infrastructure-live
Progress Checklist

To bootstrap your infrastructure-live repository, we'll use Boilerplate to scaffold it with the necessary IaC code to provision the infrastructure necessary for Pipelines to function.

The easiest way to install Boilerplate is to use mise to install it.

tip

If you don't have mise installed, you can install it by following the official guide for mise installation.

mise use -g boilerplate@latest
tip

If you'd rather install a specific version of Boilerplate, you can use the ls-remote command to list the available versions.

mise ls-remote boilerplate
Progress Checklist

Cloud-specific bootstrap instructions​

The resources that you need provisioned in AWS to start managing resources with Pipelines are:

  1. An OpenID Connect (OIDC) provider
  2. An IAM role for Pipelines to assume when running Terragrunt plan commands
  3. An IAM role for Pipelines to assume when running Terragrunt apply commands

For every account you want Pipelines to manage infrastructure in.

Don't Panic!

This may seem like a lot to set up, but the content you need to add to your infrastructure-live repository is minimal. The majority of the work will be pulled from a reusable catalog that you'll reference in your infrastructure-live repository.

If you want to peruse the catalog that's used in the bootstrap process, you can take a look at the terragrunt-scale-catalog repository.

The process that we'll follow to get these resources ready for Pipelines is:

  1. Set up the Terragrunt configurations in your infrastructure-live repository for bootstrapping Pipelines in a single AWS account
  2. Use Terragrunt to provision these resources in your AWS account
  3. (Optionally) Bootstrap additional AWS accounts until all your AWS accounts are ready for Pipelines

Bootstrap your infrastructure-live repository

To bootstrap your infrastructure-live repository, we'll use Boilerplate to scaffold it with the necessary content for Pipelines to function.

boilerplate \
  --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/aws/gitlab/infrastructure-live?ref=v1.1.0' \
  --output-folder .
tip

You can just reply y to all the prompts to include dependencies, and accept defaults unless you want to customize something.

Alternatively, you could run Boilerplate non-interactively by passing the --non-interactive flag. You'll need to supply the relevant values for required variables in that case.

e.g.

boilerplate \
  --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/aws/gitlab/infrastructure-live?ref=v1.1.0' \
  --output-folder . \
  --var 'AccountName=dev' \
  --var 'GitLabGroupName=acme' \
  --var 'GitLabRepoName=infrastructure-live' \
  --var 'GitLabInstanceURL=https://gitlab.com' \
  --var 'AWSAccountID=123456789012' \
  --var 'AWSRegion=us-east-1' \
  --var 'StateBucketName=my-state-bucket' \
  --non-interactive

You can also choose to store these values in a YAML file and pass it to Boilerplate using the --var-file flag.

vars.yml
AccountName: dev
GitLabGroupName: acme
GitLabRepoName: infrastructure-live
GitLabInstanceURL: https://gitlab.com
AWSAccountID: 123456789012
AWSRegion: us-east-1
StateBucketName: my-state-bucket
boilerplate \
  --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/aws/gitlab/infrastructure-live?ref=v1.1.0' \
  --output-folder . \
  --var-file vars.yml \
  --non-interactive

If you're using a self-hosted GitLab instance, you'll want to make sure the issuer is set correctly when calling Boilerplate.

boilerplate \
  --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/aws/gitlab/infrastructure-live?ref=v1.1.0' \
  --output-folder . \
  --var 'AccountName=dev' \
  --var 'GitLabGroupName=acme' \
  --var 'GitLabRepoName=infrastructure-live' \
  --var 'GitLabInstanceURL=https://gitlab.com' \
  --var 'AWSAccountID=123456789012' \
  --var 'AWSRegion=us-east-1' \
  --var 'StateBucketName=my-state-bucket' \
  --var 'Issuer=<ISSUER_URL>' \
  --non-interactive
Progress Checklist

Next, install Terragrunt and OpenTofu locally (the .mise.toml file in the root of the repository after scaffolding should already be set to the versions you want for Terragrunt and OpenTofu):

mise install
Progress Checklist

Provisioning the resources

Once you've set up the Terragrunt configurations, you can use Terragrunt to provision the resources in your AWS account.

tip

Make sure that you're authenticated with AWS locally before proceeding.

You can follow the documentation here to authenticate with the AWS provider. You are advised to choose an authentication method that doesn't require any hard-coded credentials, like assuming an IAM role.

First, make sure that everything is set up correctly by running a plan in the bootstrap directory in name-of-account/_global where name-of-account is the name of the first AWS account you want to bootstrap.

name-of-account/_global/bootstrap
terragrunt run --all --non-interactive --provider-cache --backend-bootstrap plan
tip

We're using the --provider-cache flag here to ensure that we don't re-download the AWS provider on every run by leveraging the Terragrunt Provider Cache Server.

We're using the --backend-bootstrap flag here to tell Terragrunt to bootstrap the OpenTofu backend automatically for the account.

Progress Checklist

Next, apply the changes to your account.

name-of-account/_global/bootstrap
terragrunt run --all --non-interactive --provider-cache apply
Progress Checklist

Optional: Bootstrapping additional AWS accounts

If you have multiple AWS accounts, and you want to bootstrap them as well, you can do so by following a similar, but slightly condensed process.

For each additional account you want to bootstrap, you'll use Boilerplate in the root of your infrastructure-live repository to scaffold out the necessary content for just that account.

tip

If you are going to bootstrap more AWS accounts, you'll probably want to commit your existing changes before proceeding.

git add .
git commit -m "Add core Pipelines scaffolding [skip ci]"

The [skip ci] in the commit message is just in-case you push your changes up to your repository at this state, as you don't want to trigger Pipelines yet.

Just like before, you'll use Boilerplate to scaffold out the necessary content for just that account.

boilerplate \
  --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/aws/gitlab/infrastructure-live?ref=v1.1.0' \
  --output-folder .
tip

Again, you can just reply y to all the prompts to include dependencies, and accept defaults unless you want to customize something.

Alternatively, you could run Boilerplate non-interactively by passing the --non-interactive flag. You'll need to supply the relevant values for required variables in that case.

e.g.

boilerplate \
  --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/aws/gitlab/account?ref=v1.1.0' \
  --output-folder . \
  --var 'AccountName=prod' \
  --var 'AWSAccountID=987654321012' \
  --var 'AWSRegion=us-east-1' \
  --var 'StateBucketName=my-prod-state-bucket' \
  --var 'GitLabGroupName=acme' \
  --var 'GitLabRepoName=infrastructure-live' \
  --var 'GitLabInstanceURL=https://gitlab.com' \
  --non-interactive

If you prefer to store the values in a YAML file and pass it to Boilerplate using the --var-file flag, you can do so like this:

vars.yml
AccountName: prod
AWSAccountID: 987654321012
AWSRegion: us-east-1
StateBucketName: my-prod-state-bucket
GitLabGroupName: acme
GitLabRepoName: infrastructure-live
GitLabInstanceURL: https://gitlab.com
boilerplate \
  --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/aws/gitlab/account?ref=v1.1.0' \
  --output-folder . \
  --var-file vars.yml \
  --non-interactive
Progress Checklist

Once you've scaffolded out the additional accounts you want to bootstrap, you can use Terragrunt to provision the resources in each of these accounts.

tip

Make sure that you authenticate to each AWS account you are bootstrapping using AWS credentials for that account before you attempt to provision resources in it.

For each account you want to bootstrap, you'll need to run the following commands:

cd <account-name>/_global/bootstrap
terragrunt run --all --non-interactive --provider-cache --backend-bootstrap plan
terragrunt run --all --non-interactive --provider-cache apply
Progress Checklist

Commit and push your changes​

Commit and push your changes to your repository.

note

You should include [skip ci] in your commit message here to prevent triggering the Pipelines workflow.

git add .
git commit -m "Add Pipelines GitLab CI workflow [skip ci]"
git push
Progress Checklist

🚀 You've successfully added Gruntwork Pipelines to your new repository!

Next steps​

You have successfully completed the installation of Gruntwork Pipelines in a new repository. Proceed to Deploying your first infrastructure change to begin deploying changes.