Building a Monorepository of Terraform Modules on GitLab

A developer details building a monorepository of Terraform modules on GitLab to consolidate best practices like testing, versioning, and self-service infrastructure while eliminating friction from managing individual repositories for a team of five maintaining modules for an app with 16 million daily users. The approach includes scaffolding with Brewfile, issue/PR templates, Makefiles, Renovate configs, and pre-commit hooks for linting, security scanning with Checkov, and documentation generation.

“It is such a terribly long time since I last wrote to you — almost two years but I know you’ll excuse me because you understand how I am, stubborn and realistic; and I thought there was no sense to writing with all these AI companies unethically stealing all of our intellectual property.” -Richard Feynman Sorry for the silence. I have had my head down on new and exciting tech adventures each day, but I wanted to share something cool and hopefully useful that I have been prototyping. I have not seen many articles on this approach, so maybe it will be both innovative and practical. In a previous role, we were leveraging GitLab’s internal Terraform registry, which was handy. It made it easy to follow best practices like testing and versioning modules, and it helped us provide self-service infrastructure to other teams with best practices and security baked in. Overall, it was a great experience with a few pain points. Each Terraform module had its own repository, which is super cool, but that starts to introduce friction when you only have 5 people on a team maintaining infrastructure and tons of modules for an app that gets 16 million users each day. I digress. Individual repositories can still be powerful. For example, Cloud Posse has tons of Terraform module and component repositories across their GitHub org. Each has workflows, example code, extensive automation, and tests that even run in a test AWS account. When an approved maintainer comments on an open source contribution PR with /terratest, a runner uses Terratest to deploy real infra from the codebase and make sure the module actually works. While this is pretty nice, I thought I could consolidate a lot of the pros into a monorepo and start eliminating the pain points. This blog details how I structured and built automation to provide self-service Terraform modules to the org. Maybe I talk about scaffolding too much, but I truly believe it lays the foundation for a successful repository. There’s always been a soft spot in my heart for developer experience, so I usually add a Brewfile with the dependencies needed to work in the repo NOTE: In the future, I might look into nixpkgs for this use case. Declarative configs ftw . Anyway, I also include the basic housekeeping items like issue and PR/MR templates, Makefiles, Renovate configs, security contacts, and the classics: .gitattributes, .gitignore, and .editorconfig. Since these are Terraform modules, I also run Checkov across the codebase, so you might find a .checkov.yaml sprinkled next to a .copywrite.hcl. Other than that, I keep the pre-commit hooks pretty light. The main pieces cover general file hygiene, Terraform linting, documentation generation, security scanning with Checkov, and some secrets scanning. I also run Copywrite because it makes me feel special that I can add copyright headers to some code. See https://pre-commit.com for more information See https://pre-commit.com/hooks.html for more hooksrepos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v5.0.0 hooks: - id: end-of-file-fixer - id: check-merge-conflict - id: trailing-whitespace args: --markdown-linebreak-ext=md - id: check-shebang-scripts-are-executable YAML - id: check-yaml Cross platform - id: check-case-conflict checks for files that would conflict in case-insensitive filesystems. - id: mixed-line-ending replaces or checks mixed line ending. args: --fix=lf - repo: https://github.com/antonbabenko/pre-commit-terraform Ensure the PRE COMMIT TERRAFORM VERSION value is the same in .gitlab-ci.yml rev: v1.105.0 Get the latest from: https://github.com/antonbabenko/pre-commit-terraform/releases hooks: - id: terraform fmt - id: terraform docs args: "--args=--lockfile=false" - id: terraform tflint - id: terraform checkov args: - --args=--framework=terraform - --args=--soft-fail - --args=--config-file= GIT WORKING DIR /.checkov.yaml - --args=--skip-path= GIT WORKING DIR /.cache/pre-commit - repo: https://github.com/bridgecrewio/checkov.git rev: 3.2.524 hooks: - id: checkov secrets args: --directory, ., --framework, secrets, --config-file, .checkov.yaml, --skip-path, .cache/pre-commit pass filenames: false - repo: https://github.com/hashicorp/copywrite rev: v0.25.3 hooks: - id: add-headers NOTE:These pre-commit hooks are typically run again in CI via:pre-commit run --all-files --show-diff-on-failure to ensure standards are met The repository structure is pretty straightforward. Each module contains the core Terraform logic, a tests directory with native terraform test coverage, and a CHANGELOG. For my use case, I've been using Changie https://changie.dev/ for automated changelog generation, but this is more of an add-on than a requirement. It just makes it easier to keep track of changes as the repository grows over time. ├── Brewfile├── Makefile├── Other stuff...├── modules│ ├── aws│ │ ├── account│ │ │ ├── CHANGELOG.md│ │ │ ├── main.tf│ │ │ ├── outputs.tf│ │ │ ├── README.md│ │ │ ├── tests│ │ │ │ └── basic.tftest.hcl│ │ │ ├── variables.tf└── └── │ └── versions.tf The GitLab CI pipelines consist of detection, validation, and dispatch stages. The goal is to keep the feedback loop tight while still enforcing the basics: pre-commit hooks pass in CI, formatting and code quality standards are followed, and security checks catch obvious issues before anything gets merged. We also leverage secrets scanning to make sure sensitive information does not get committed to version control. After those checks pass, Terraform tests run for the changed modules, and once the changes land on main, the module gets published with a version tag. NOTE:I prefer defining versions at the top level of my GitLab CI pipelines. It makes upgrades and version bumps easier, and it keeps versions in one place instead of scattered throughout the workflow files. The root pipeline is mostly orchestration. It defines the stages, pins the container versions used by the jobs, limits pipeline execution to merge requests from the same project and pushes to the default branch, and includes the smaller workflow files that do the actual work. I like this pattern because the main .gitlab-ci.yml stays readable and the implementation details can live under .gitlab/ci/. Root GitLab CI pipeline definition for the Terraform modules repository.stages: - detect - validate - dispatchdefault: timeout: 30 minutes interruptible: truevariables: TF VERSION: "1.15.5" TF IMAGE: "hashicorp/terraform:$TF VERSION" ALPINE GIT VERSION: "2.52.0" CURL IMAGE VERSION: "8.21.0" CURL IMAGE: "curlimages/curl:$CURL IMAGE VERSION" PRE COMMIT TERRAFORM VERSION: "1.105.0"workflow: name: '$CI PIPELINE SCHEDULE DESCRIPTION' rules: - if: '$CI PIPELINE SOURCE == "merge request event" && $CI MERGE REQUEST SOURCE PROJECT ID == $CI PROJECT ID' - if: '$CI PIPELINE SOURCE == "push" && $CI COMMIT BRANCH == $CI DEFAULT BRANCH'include: - local: .gitlab/ci/detect/ .gitlab-ci.yml - local: .gitlab/ci/validate/ .gitlab-ci.yml - local: .gitlab/ci/dispatch/ .gitlab-ci.yml The change detection job builds the list of Terraform modules that were touched in the current pipeline. For merge requests, it compares the branch against the target branch. For default branch pushes, it compares the current commit against the previous commit and falls back to the first commit if GitLab does not provide a usable base SHA. From there, it walks changed files under modules/, finds the owning module by looking for .changie.yaml, and writes a dotenv artifact that later jobs can consume. The important piece is that this turns a monorepo into a smaller work queue. Instead of running every test and publishing logic against every module, the pipeline only focuses on modules that actually changed. Detects changed Terraform modules for merge requests and main branch commits.terraform:changes: stage: detect image: name: "alpine/git:$ALPINE GIT VERSION" entrypoint: "" variables: GIT DEPTH: "0" TERRAFORM CHANGE MATRIX DIR: "terraform-change-matrix" script: - | set -eu mkdir -p "$TERRAFORM CHANGE MATRIX DIR" if "$CI PIPELINE SOURCE" = "merge request event" ; then git fetch origin "$CI MERGE REQUEST TARGET BRANCH NAME" BASE SHA="$ git merge-base HEAD "origin/$CI MERGE REQUEST TARGET BRANCH NAME" " else BASE SHA="${CI COMMIT BEFORE SHA:-}" if -z "$BASE SHA" || "$BASE SHA" = "0000000000000000000000000000000000000000" ; then BASE SHA="$ git rev-list --max-parents=0 HEAD " fi fi changed files="$ git diff --name-only "$BASE SHA" "$CI COMMIT SHA" -- modules/ || true " module dirs="$ for changed file in $changed files; do module dir="${changed file%/ }" while "$module dir" = "." && "$module dir" = "modules" ; do if -f "$module dir/.changie.yaml" ; then printf '%s\n' "$module dir" break fi module dir="${module dir%/ }" done done | sort -u " module test dirs="$ for module dir in $module dirs; do if -d "$module dir/tests" && find "$module dir/tests" -name ' .tftest.hcl' -print -quit | grep -q .; then printf '%s\n' "$module dir" fi done " { printf 'TF CHANGED MODULE DIRS=%s\n' "$ printf '%s' "$module dirs" | tr '\n' ' ' " printf 'TF CHANGED MODULE TEST DIRS=%s\n' "$ printf '%s' "$module test dirs" | tr '\n' ' ' " printf 'TF CHANGED MODULE TEST COUNT=%s\n' "$ printf '%s\n' "$module test dirs" | sed '/^$/d' | wc -l | tr -d ' ' " } "$TERRAFORM CHANGE MATRIX DIR/matrix.env" cat "$TERRAFORM CHANGE MATRIX DIR/matrix.env" artifacts: when: always expire in: 1 week reports: dotenv: terraform-change-matrix/matrix.env paths: - terraform-change-matrix/ rules: - if: '$CI PIPELINE SOURCE == "merge request event"' changes: - modules/ / - if: '$CI PIPELINE SOURCE == "push" && $CI COMMIT BRANCH == $CI DEFAULT BRANCH' changes: - modules/ / The test job consumes the dotenv artifact from change detection and only runs terraform test for changed modules that actually contain .tftest.hcl files. If no changed modules have tests, the job exits cleanly. This keeps CI from wasting time on empty test directories while still making native Terraform tests part of the normal merge request workflow. Runs terraform test for changed Terraform modules that include test coverage.terraform-test: stage: validate image: name: "$TF IMAGE" entrypoint: "" needs: - job: terraform:changes artifacts: true script: - | set -eu if "${TF CHANGED MODULE TEST COUNT:-0}" = "0" ; then echo "No changed Terraform modules with tests detected." exit 0 fi for module dir in $TF CHANGED MODULE TEST DIRS; do echo "Running terraform test in $module dir" cd "$module dir" terraform init -input=false terraform test done rules: - if: '$CI PIPELINE SOURCE == "merge request event"' changes: - modules/ / - if: '$CI PIPELINE SOURCE == "push" && $CI COMMIT BRANCH == $CI DEFAULT BRANCH' changes: - modules/ / Publishing only runs on pushes to the default branch. It uses the changed module list from the detection job, reads the latest semantic version from each module’s CHANGELOG.md, creates a module-specific Git tag, packages the module directory, and uploads it to GitLab's Terraform module registry. The tag format keeps each release tied back to its module path, which matters when multiple modules are released from the same repository. After the module is published, it shows up in GitLab’s Terraform module registry like any other module package. terraform-module:publish: stage: dispatch image: "$CURL IMAGE" needs: - job: terraform:changes artifacts: true script: - | set -eu if -z "${GITLAB RELEASE TOKEN:-}" ; then echo "GITLAB RELEASE TOKEN is required to create module tags." exit 1 fi if -z "${TF CHANGED MODULE DIRS:-}" ; then echo "No changed Terraform modules detected." exit 0 fi for MODULE DIR in $TF CHANGED MODULE DIRS; do MODULE VERSION="$ sed -nE 's/^ 0-9 +\. 0-9 +\. 0-9 + . /\1/p' "$MODULE DIR/CHANGELOG.md" | head -n 1 " if -z "$MODULE VERSION" ; then echo "Could not determine module version from $MODULE DIR/CHANGELOG.md" exit 1 fi MODULE TAG="${MODULE DIR}/v${MODULE VERSION}" MODULE SYSTEM="$ echo "$MODULE DIR" | cut -d/ -f2 " MODULE NAME="$ echo "$MODULE DIR" | cut -d/ -f3- | tr '/' '-' " echo "Creating $MODULE TAG" curl --fail-with-body --location --request POST \ --header "PRIVATE-TOKEN: ${GITLAB RELEASE TOKEN}" \ --form "tag name=${MODULE TAG}" \ --form "ref=${CI COMMIT SHA}" \ "${CI API V4 URL}/projects/${CI PROJECT ID}/repository/tags" tar -czf "/tmp/${MODULE NAME}-${MODULE SYSTEM}-${MODULE VERSION}.tgz" \ -C "$MODULE DIR" \ --exclude=.git \ . PACKAGE URL="${CI API V4 URL}/projects/${CI PROJECT ID}/packages/terraform/modules/${MODULE NAME}/${MODULE SYSTEM}/${MODULE VERSION}/file" echo "Publishing $MODULE NAME/$MODULE SYSTEM/$MODULE VERSION" curl --fail-with-body --location \ --header "JOB-TOKEN: ${CI JOB TOKEN}" \ --upload-file "/tmp/${MODULE NAME}-${MODULE SYSTEM}-${MODULE VERSION}.tgz" \ "$PACKAGE URL" done rules: - if: '$CI PIPELINE SOURCE == "push" && $CI COMMIT BRANCH == $CI DEFAULT BRANCH' changes: - modules/ / There are still a few rough edges, but so far, this solution has been what I need for a monorepo of Terraform modules that can be consumed by internal registry users and other teams. It’s still a work in progress, but having a consolidated repository with versioning, testing, changelogs, and linting has been the right fit. If you liked or hated this blog, feel free to check out my GitHub https://github.com/RoseSecurity Building a Monorepository of Terraform Modules on GitLab https://blog.devgenius.io/building-a-monorepository-of-terraform-modules-on-gitlab-bf86e40158e7 was originally published in Dev Genius https://blog.devgenius.io on Medium, where people are continuing the conversation by highlighting and responding to this story.