[Databricks on AWS #6] How We Structure the Terraform: Terragrunt, YAML-Driven Modules, and Atlantis GitOps

A developer detailed how their team structured Terraform for a multi-workspace Databricks AI platform on AWS using Terragrunt, YAML-driven modules, and Atlantis GitOps. The layout separates reusable modules from per-environment trees, with humans editing only YAML files while Terragrunt injects environment prefixes and generates provider overrides for each workspace. The approach avoids copy-paste sprawl and enforces naming discipline across dev and prod environments.

📚 Series: Databricks on AWS Part 6 - Building a Databricks AI Platform on AWS - RBAC with Function-Role Groups - Compute Governance: Pools, Policies, Clusters - The BOOTSTRAP TIMEOUT Mystery - Fixing It with AWS PrivateLink How We Structure the Terraform← you are hereFive parts of whatwe built — the workspaces, the RBAC, the compute, the network mystery, the PrivateLink fix. This one is thehow: the repo layout and Terragrunt patterns that hold it all together, plus the two footguns that will bite you on day one. Over five parts we stood up an AI platform on Databricks + AWS. Along the way I kept saying "we apply this," "the module for that," "the provider override" — and mostly waved my hands at the plumbing. This finale is the plumbing: how the Terraform is actually organized, why humans never touch a .tf file, and the two ordering gotchas that turn a clean plan into a red one. If you're standing up a multi-workspace Databricks estate and want a layout that scales past the first workspace without copy-paste sprawl, this is the shape that worked for us. Two ideas do all the work: reusable modules and per-environment trees . infra/ ├── modules/ reusable Terraform modules the "what" ├── <project / │ └── databricks/ │ ├── dev/ per-env tree the "where" │ └── prd/ ├── common vars.yaml shared config: hosts, account IDs, CIDRs └── provider.tmpl provider template modules/ is a flat library of single-purpose Terraform modules — one per resource concept: workspace, unity-catalog, catalog, schema, grants, group, group-ar, instance-pool, cluster-policy, compute, sql-warehouse, entitlements, external-location, workspace-assignment, acls, iamrole, iampolicy, s3, kms, vpc, service-principal, audit-log-delivery, workspace-conf None of these know about dev or prd . They take inputs and create resources. The environment tree decides which modules run, with what values , in what order . A new workspace is a new folder under dev/ , not a new module. Here's the rule we enforce: a person editing this repo edits YAML. They don't write Terraform, and mostly don't write Terragrunt either. The flow is three hops: human-authored .yaml → terragrunt.hcl: yamldecode + env-prefix → Terraform module inputs Each leaf directory has a small .yaml the desired state, in human terms and a terragrunt.hcl that reads it. The terragrunt.hcl does yamldecode on the YAML, then rewrites the keys with an environment prefix before handing them to the module. Concretely, in the dev tree an instance-pool key like ip cpu small becomes dev ip cpu small on the way in. Why the prefix? Because the same YAML shape lives in dev/ and prd/ , but the resource names, pool IDs, and group names must be globally distinct. The env prefix is injected once, in one place, by Terragrunt — so no human ever hand-types dev or prd and nobody fat-fingers a cross-environment collision. You change a number in YAML; the naming discipline is automatic. This is the single highest-leverage decision in the whole layout. The reviewer of a merge request reads a YAML diff — "pool max went 8 → 16" — not a wall of HCL. Databricks has a wrinkle that AWS doesn't: every workspace is its own API host. A databricks cluster in workspace A and one in workspace B are created against different host URLs, even though it's the same account, same provider, same code. We solve this with generated provider overrides. Each workspace layer knows its workspace key from a small workspace.hcl , and common vars.yaml maps that key to the workspace host. Terragrunt looks up the host and generates a provider override.tf on the fly: locals { ws host = local.common vars.databricks .projects.<project .workspace hosts local.env local.ws vars.locals.workspace key } generate "workspace provider override" { path = "provider override.tf" contents = <<-EOF provider "databricks" { host = "${local.ws host}" } EOF } The module code is workspace-agnostic. The host is injected at plan time based on which folder you're in. Add a workspace, add its host to common vars.yaml , drop a workspace.hcl with the key — every layer underneath now targets the right host with zero code changes. You'll notice the code uses the Databricks provider two different ways, and it's not an accident: | provider | scope | used for | |---|---|---| databricks.mws account-level | the Databricks account | creating workspaces, account-level groups, metastore assignment | databricks workspace-level | a single workspace host | clusters, pools, policies, catalogs, grants, ACLs | Account-level resources the workspace itself , account groups, wiring a metastore don't belong to any one workspace — they're configured against the account API, which is why they use the mws alias. Everything inside a workspace uses the workspace-scoped provider with the host we just injected. Mixing these up is a classic early mistake: try to create a cluster with the mws provider and you'll get authentication errors that make no sense until you realize you're pointed at the account, not the workspace. Layers depend on each other's outputs. cluster-policy needs the pool IDs that instance-pool produced; compute needs the policy IDs. Terragrunt expresses this with a dependency block: dependency "instance pool" { config path = "../instance-pool" mock outputs = { instance pool ids = { "dev ip cpu small" = "mock-pool-id-1" } } mock outputs allowed terraform commands = "validate", "plan" } The mock outputs let you plan a layer before its dependency has ever been applied — Terragrunt feeds fake IDs so the plan can render. Handy. But look closely at that allow-list: "validate", "plan" . No "init". That omission is the gotcha, and it ties straight back to the timeout saga from Parts 3 and 4. On a brand-new tree, the very first plan runs an init first — and because init isn't in the mock allow-list, Terragrunt refuses to mock the dependency and instead tries to read real outputs from a state that doesn't exist yet. Result: Error: detected no outputs from dependency ".../instance-pool" Your clean-looking layer fails to plan not because it's wrong, but because the thing upstream of it was never applied. Some layers in the repo workspace-assignment , unity-catalog deliberately include "init" in the allow-list to dodge this; the compute layers don't. The practical consequence is the same either way, and it's the whole reason for the next section. Because of the dependency chain above, you apply in order. Not "usually." Always, the first time through: Workspace line: workspace → instance-pool → cluster-policy → compute → workspace-assignment → acls Unity Catalog: unity-catalog → external-location → catalogs → schemas → groups → grants The rule underneath both lines is the same: apply a layer only after everything it depends on is already applied. Mocks let you plan out of order, but the first real apply has to walk the chain, because a mock pool ID doesn't create a cluster and a mock catalog doesn't hold a grant. Two failure modes we hit repeatedly, both just ordering: cluster-policy / compute plan fails with "detected no outputs" → the pool/policy above it wasn't applied. Apply upstream first. workspace-assignment plans to assignments = {} → the groups layer wasn't applied, so there's nothing to assign. Apply groups first.Neither is a bug. Both are the dependency graph telling you that you skipped a step. We don't run terragrunt apply from laptops. Every change is a merge request; Atlantis runs atlantis plan on the MR and atlantis apply once it's approved. The plan output lands in the MR as a comment, so the diff and its consequences are reviewable in one place. Account credentials live server-side on the Atlantis host, not in anyone's shell — which also means local plan is optional and honestly, usually skipped since you get the real plan from the MR anyway. Put together, the daily loop is small: edit a YAML value, open an MR, read the Atlantis plan, apply. The modules, the env-prefixing, the provider injection, the dependency graph — all of that is machinery you set up once and then mostly forget. That's the whole platform: If there's one throughline, it's that the boring parts — naming discipline, apply order, who-writes-what — are exactly the parts that decide whether a platform stays sane at the tenth workspace. Get the structure right and the interesting problems like a packet dying in a firewall are the only ones left to solve. Thanks for reading all six. If you're just arriving, Part 1 is where the estate gets built from nothing.