# [Databricks on AWS #6] How We Structure the Terraform: Terragrunt, YAML-Driven Modules, and Atlantis GitOps > Source: > Published: 2026-07-09 01:08:38+00:00 📚 Series: Databricks on AWS (Part 6) - Building a Databricks AI Platform on AWS - RBAC with Function-Role Groups - Compute Governance: Pools, Policies, Clusters - The BOOTSTRAP_TIMEOUT Mystery - Fixing It with AWS PrivateLink How We Structure the Terraform← you are hereFive parts of whatwe built — the workspaces, the RBAC, the compute, the network mystery, the PrivateLink fix. This one is thehow: the repo layout and Terragrunt patterns that hold it all together, plus the two footguns that will bite you on day one. Over five parts we stood up an AI platform on Databricks + AWS. Along the way I kept saying "we apply this," "the module for that," "the provider override" — and mostly waved my hands at the plumbing. This finale is the plumbing: how the Terraform is actually organized, why humans never touch a `.tf` file, and the two ordering gotchas that turn a clean plan into a red one. If you're standing up a multi-workspace Databricks estate and want a layout that scales past the first workspace without copy-paste sprawl, this is the shape that worked for us. Two ideas do all the work: **reusable modules** and **per-environment trees**. ``` infra/ ├── _modules/ # reusable Terraform modules (the "what") ├── / │ └── databricks/ │ ├── dev/ # per-env tree (the "where") │ └── prd/ ├── common_vars.yaml # shared config: hosts, account IDs, CIDRs └── provider.tmpl # provider template ``` `_modules/` is a flat library of single-purpose Terraform modules — one per resource concept: ``` workspace, unity-catalog, catalog, schema, grants, group, group-ar, instance-pool, cluster-policy, compute, sql-warehouse, entitlements, external-location, workspace-assignment, acls, iamrole, iampolicy, s3, kms, vpc, service-principal, audit-log-delivery, workspace-conf ``` None of these know about `dev` or `prd` . They take inputs and create resources. The environment tree decides *which* modules run, *with what values*, in *what order*. A new workspace is a new folder under `dev/` , not a new module. Here's the rule we enforce: **a person editing this repo edits YAML.** They don't write Terraform, and mostly don't write Terragrunt either. The flow is three hops: ``` human-authored *.yaml → terragrunt.hcl: yamldecode() + env-prefix → Terraform module inputs ``` Each leaf directory has a small `*.yaml` (the desired state, in human terms) and a `terragrunt.hcl` that reads it. The `terragrunt.hcl` does `yamldecode()` on the YAML, then rewrites the keys with an environment prefix before handing them to the module. Concretely, in the `dev` tree an instance-pool key like `ip_cpu_small` becomes `dev_ip_cpu_small` on the way in. Why the prefix? Because the *same* YAML shape lives in `dev/` and `prd/` , but the resource names, pool IDs, and group names must be globally distinct. The env prefix is injected once, in one place, by Terragrunt — so no human ever hand-types `dev_` or `prd_` and nobody fat-fingers a cross-environment collision. You change a number in YAML; the naming discipline is automatic. This is the single highest-leverage decision in the whole layout. The reviewer of a merge request reads a YAML diff — "pool max went 8 → 16" — not a wall of HCL. Databricks has a wrinkle that AWS doesn't: **every workspace is its own API host.** A `databricks_cluster` in workspace A and one in workspace B are created against different `host` URLs, even though it's the same account, same provider, same code. We solve this with generated provider overrides. Each workspace layer knows its `workspace_key` (from a small `workspace.hcl` ), and `common_vars.yaml` maps that key to the workspace host. Terragrunt looks up the host and **generates** a `provider_override.tf` on the fly: ``` locals { ws_host = local.common_vars.databricks .projects..workspace_hosts[local.env][local.ws_vars.locals.workspace_key] } generate "workspace_provider_override" { path = "provider_override.tf" contents = <<-EOF provider "databricks" { host = "${local.ws_host}" } EOF } ``` The module code is workspace-agnostic. The *host* is injected at plan time based on which folder you're in. Add a workspace, add its host to `common_vars.yaml` , drop a `workspace.hcl` with the key — every layer underneath now targets the right host with zero code changes. You'll notice the code uses the Databricks provider two different ways, and it's not an accident: | provider | scope | used for | |---|---|---| `databricks.mws` (account-level) | the Databricks account | creating workspaces, account-level groups, metastore assignment | `databricks` (workspace-level) | a single workspace host | clusters, pools, policies, catalogs, grants, ACLs | Account-level resources (the workspace *itself*, account groups, wiring a metastore) don't belong to any one workspace — they're configured against the account API, which is why they use the `mws` alias. Everything *inside* a workspace uses the workspace-scoped provider with the host we just injected. Mixing these up is a classic early mistake: try to create a cluster with the `mws` provider and you'll get authentication errors that make no sense until you realize you're pointed at the account, not the workspace. Layers depend on each other's outputs. `cluster-policy` needs the pool IDs that `instance-pool` produced; `compute` needs the policy IDs. Terragrunt expresses this with a `dependency` block: ``` dependency "instance_pool" { config_path = "../instance-pool" mock_outputs = { instance_pool_ids = { "dev_ip_cpu_small" = "mock-pool-id-1" } } mock_outputs_allowed_terraform_commands = ["validate", "plan"] } ``` The `mock_outputs` let you `plan` a layer before its dependency has ever been applied — Terragrunt feeds fake IDs so the plan can render. Handy. But look closely at that allow-list: `["validate", "plan"]` . **No "init".** That omission is the gotcha, and it ties straight back to the timeout saga from Parts 3 and 4. On a brand-new tree, the very first `plan` runs an `init` first — and because `init` isn't in the mock allow-list, Terragrunt refuses to mock the dependency and instead tries to read *real* outputs from a state that doesn't exist yet. Result: ``` Error: detected no outputs from dependency ".../instance-pool" ``` Your clean-looking layer fails to plan not because it's wrong, but because the thing *upstream* of it was never applied. Some layers in the repo (`workspace-assignment` , `unity-catalog` ) deliberately include `"init"` in the allow-list to dodge this; the compute layers don't. The practical consequence is the same either way, and it's the whole reason for the next section. Because of the dependency chain above, **you apply in order.** Not "usually." Always, the first time through: ``` Workspace line: workspace → instance-pool → cluster-policy → compute → workspace-assignment → acls Unity Catalog: unity-catalog → external-location → catalogs → schemas → groups → grants ``` The rule underneath both lines is the same: **apply a layer only after everything it depends on is already applied.** Mocks let you *plan* out of order, but the *first real apply* has to walk the chain, because a mock pool ID doesn't create a cluster and a mock catalog doesn't hold a grant. Two failure modes we hit repeatedly, both just ordering: `cluster-policy` / `compute` plan fails with "detected no outputs" → the pool/policy above it wasn't applied. Apply upstream first.`workspace-assignment` plans to `assignments = {}` → the `groups` layer wasn't applied, so there's nothing to assign. Apply `groups` first.Neither is a bug. Both are the dependency graph telling you that you skipped a step. We don't run `terragrunt apply` from laptops. Every change is a merge request; Atlantis runs `atlantis plan` on the MR and `atlantis apply` once it's approved. The plan output lands in the MR as a comment, so the diff *and* its consequences are reviewable in one place. Account credentials live server-side on the Atlantis host, not in anyone's shell — which also means local `plan` is optional (and honestly, usually skipped) since you get the real plan from the MR anyway. Put together, the daily loop is small: edit a YAML value, open an MR, read the Atlantis plan, apply. The modules, the env-prefixing, the provider injection, the dependency graph — all of that is machinery you set up once and then mostly forget. That's the whole platform: If there's one throughline, it's that the boring parts — naming discipline, apply order, who-writes-what — are exactly the parts that decide whether a platform stays sane at the tenth workspace. Get the structure right and the interesting problems (like a packet dying in a firewall) are the only ones left to solve. Thanks for reading all six. If you're just arriving, Part 1 is where the estate gets built from nothing.