{"slug": "databricks-on-aws-6-how-we-structure-the-terraform-terragrunt-yaml-driven-and", "title": "[Databricks on AWS #6] How We Structure the Terraform: Terragrunt, YAML-Driven Modules, and Atlantis GitOps", "summary": "A developer detailed how their team structured Terraform for a multi-workspace Databricks AI platform on AWS using Terragrunt, YAML-driven modules, and Atlantis GitOps. The layout separates reusable modules from per-environment trees, with humans editing only YAML files while Terragrunt injects environment prefixes and generates provider overrides for each workspace. The approach avoids copy-paste sprawl and enforces naming discipline across dev and prod environments.", "body_md": "📚 Series: Databricks on AWS (Part 6)\n\n- Building a Databricks AI Platform on AWS\n- RBAC with Function-Role Groups\n- Compute Governance: Pools, Policies, Clusters\n- The BOOTSTRAP_TIMEOUT Mystery\n- Fixing It with AWS PrivateLink\nHow We Structure the Terraform← you are hereFive parts of\n\nwhatwe built — the workspaces, the RBAC, the compute, the network mystery, the PrivateLink fix. This one is thehow: the repo layout and Terragrunt patterns that hold it all together, plus the two footguns that will bite you on day one.\n\nOver five parts we stood up an AI platform on Databricks + AWS. Along the way I kept saying \"we apply this,\" \"the module for that,\" \"the provider override\" — and mostly waved my hands at the plumbing. This finale is the plumbing: how the Terraform is actually organized, why humans never touch a `.tf`\n\nfile, and the two ordering gotchas that turn a clean plan into a red one.\n\nIf you're standing up a multi-workspace Databricks estate and want a layout that scales past the first workspace without copy-paste sprawl, this is the shape that worked for us.\n\nTwo ideas do all the work: **reusable modules** and **per-environment trees**.\n\n```\ninfra/\n├── _modules/               # reusable Terraform modules (the \"what\")\n├── <project>/\n│   └── databricks/\n│       ├── dev/            # per-env tree (the \"where\")\n│       └── prd/\n├── common_vars.yaml        # shared config: hosts, account IDs, CIDRs\n└── provider.tmpl           # provider template\n```\n\n`_modules/`\n\nis a flat library of single-purpose Terraform modules — one per resource concept:\n\n```\nworkspace, unity-catalog, catalog, schema, grants, group, group-ar,\ninstance-pool, cluster-policy, compute, sql-warehouse, entitlements,\nexternal-location, workspace-assignment, acls, iamrole, iampolicy,\ns3, kms, vpc, service-principal, audit-log-delivery, workspace-conf\n```\n\nNone of these know about `dev`\n\nor `prd`\n\n. They take inputs and create resources. The environment tree decides *which* modules run, *with what values*, in *what order*. A new workspace is a new folder under `dev/`\n\n, not a new module.\n\nHere's the rule we enforce: **a person editing this repo edits YAML.** They don't write Terraform, and mostly don't write Terragrunt either.\n\nThe flow is three hops:\n\n```\nhuman-authored *.yaml\n  → terragrunt.hcl: yamldecode() + env-prefix\n  → Terraform module inputs\n```\n\nEach leaf directory has a small `*.yaml`\n\n(the desired state, in human terms) and a `terragrunt.hcl`\n\nthat reads it. The `terragrunt.hcl`\n\ndoes `yamldecode()`\n\non the YAML, then rewrites the keys with an environment prefix before handing them to the module. Concretely, in the `dev`\n\ntree an instance-pool key like `ip_cpu_small`\n\nbecomes `dev_ip_cpu_small`\n\non the way in.\n\nWhy the prefix? Because the *same* YAML shape lives in `dev/`\n\nand `prd/`\n\n, but the resource names, pool IDs, and group names must be globally distinct. The env prefix is injected once, in one place, by Terragrunt — so no human ever hand-types `dev_`\n\nor `prd_`\n\nand nobody fat-fingers a cross-environment collision. You change a number in YAML; the naming discipline is automatic.\n\nThis is the single highest-leverage decision in the whole layout. The reviewer of a merge request reads a YAML diff — \"pool max went 8 → 16\" — not a wall of HCL.\n\nDatabricks has a wrinkle that AWS doesn't: **every workspace is its own API host.** A `databricks_cluster`\n\nin workspace A and one in workspace B are created against different `host`\n\nURLs, even though it's the same account, same provider, same code.\n\nWe solve this with generated provider overrides. Each workspace layer knows its `workspace_key`\n\n(from a small `workspace.hcl`\n\n), and `common_vars.yaml`\n\nmaps that key to the workspace host. Terragrunt looks up the host and **generates** a `provider_override.tf`\n\non the fly:\n\n```\nlocals {\n  ws_host = local.common_vars.databricks\n    .projects.<project>.workspace_hosts[local.env][local.ws_vars.locals.workspace_key]\n}\n\ngenerate \"workspace_provider_override\" {\n  path      = \"provider_override.tf\"\n  contents  = <<-EOF\n    provider \"databricks\" {\n      host = \"${local.ws_host}\"\n    }\n  EOF\n}\n```\n\nThe module code is workspace-agnostic. The *host* is injected at plan time based on which folder you're in. Add a workspace, add its host to `common_vars.yaml`\n\n, drop a `workspace.hcl`\n\nwith the key — every layer underneath now targets the right host with zero code changes.\n\nYou'll notice the code uses the Databricks provider two different ways, and it's not an accident:\n\n| provider | scope | used for |\n|---|---|---|\n`databricks.mws` (account-level) |\nthe Databricks account\n|\ncreating workspaces, account-level groups, metastore assignment |\n`databricks` (workspace-level) |\na single workspace host |\nclusters, pools, policies, catalogs, grants, ACLs |\n\nAccount-level resources (the workspace *itself*, account groups, wiring a metastore) don't belong to any one workspace — they're configured against the account API, which is why they use the `mws`\n\nalias. Everything *inside* a workspace uses the workspace-scoped provider with the host we just injected. Mixing these up is a classic early mistake: try to create a cluster with the `mws`\n\nprovider and you'll get authentication errors that make no sense until you realize you're pointed at the account, not the workspace.\n\nLayers depend on each other's outputs. `cluster-policy`\n\nneeds the pool IDs that `instance-pool`\n\nproduced; `compute`\n\nneeds the policy IDs. Terragrunt expresses this with a `dependency`\n\nblock:\n\n```\ndependency \"instance_pool\" {\n  config_path = \"../instance-pool\"\n  mock_outputs = {\n    instance_pool_ids = { \"dev_ip_cpu_small\" = \"mock-pool-id-1\" }\n  }\n  mock_outputs_allowed_terraform_commands = [\"validate\", \"plan\"]\n}\n```\n\nThe `mock_outputs`\n\nlet you `plan`\n\na layer before its dependency has ever been applied — Terragrunt feeds fake IDs so the plan can render. Handy. But look closely at that allow-list: `[\"validate\", \"plan\"]`\n\n. **No \"init\".**\n\nThat omission is the gotcha, and it ties straight back to the timeout saga from Parts 3 and 4. On a brand-new tree, the very first `plan`\n\nruns an `init`\n\nfirst — and because `init`\n\nisn't in the mock allow-list, Terragrunt refuses to mock the dependency and instead tries to read *real* outputs from a state that doesn't exist yet. Result:\n\n```\nError: detected no outputs from dependency \".../instance-pool\"\n```\n\nYour clean-looking layer fails to plan not because it's wrong, but because the thing *upstream* of it was never applied. Some layers in the repo (`workspace-assignment`\n\n, `unity-catalog`\n\n) deliberately include `\"init\"`\n\nin the allow-list to dodge this; the compute layers don't. The practical consequence is the same either way, and it's the whole reason for the next section.\n\nBecause of the dependency chain above, **you apply in order.** Not \"usually.\" Always, the first time through:\n\n```\nWorkspace line:  workspace → instance-pool → cluster-policy → compute\n                 → workspace-assignment → acls\n\nUnity Catalog:   unity-catalog → external-location → catalogs\n                 → schemas → groups → grants\n```\n\nThe rule underneath both lines is the same: **apply a layer only after everything it depends on is already applied.** Mocks let you *plan* out of order, but the *first real apply* has to walk the chain, because a mock pool ID doesn't create a cluster and a mock catalog doesn't hold a grant.\n\nTwo failure modes we hit repeatedly, both just ordering:\n\n`cluster-policy`\n\n/ `compute`\n\nplan fails with \"detected no outputs\" → the pool/policy above it wasn't applied. Apply upstream first.`workspace-assignment`\n\nplans to `assignments = {}`\n\n→ the `groups`\n\nlayer wasn't applied, so there's nothing to assign. Apply `groups`\n\nfirst.Neither is a bug. Both are the dependency graph telling you that you skipped a step.\n\nWe don't run `terragrunt apply`\n\nfrom laptops. Every change is a merge request; Atlantis runs `atlantis plan`\n\non the MR and `atlantis apply`\n\nonce it's approved. The plan output lands in the MR as a comment, so the diff *and* its consequences are reviewable in one place. Account credentials live server-side on the Atlantis host, not in anyone's shell — which also means local `plan`\n\nis optional (and honestly, usually skipped) since you get the real plan from the MR anyway.\n\nPut together, the daily loop is small: edit a YAML value, open an MR, read the Atlantis plan, apply. The modules, the env-prefixing, the provider injection, the dependency graph — all of that is machinery you set up once and then mostly forget.\n\nThat's the whole platform:\n\nIf there's one throughline, it's that the boring parts — naming discipline, apply order, who-writes-what — are exactly the parts that decide whether a platform stays sane at the tenth workspace. Get the structure right and the interesting problems (like a packet dying in a firewall) are the only ones left to solve.\n\nThanks for reading all six. If you're just arriving, Part 1 is where the estate gets built from nothing.", "url": "https://wpnews.pro/news/databricks-on-aws-6-how-we-structure-the-terraform-terragrunt-yaml-driven-and", "canonical_source": "https://dev.to/javaking1129/databricks-on-aws-6-how-we-structure-the-terraform-terragrunt-yaml-driven-modules-and-4959", "published_at": "2026-07-09 01:08:38+00:00", "updated_at": "2026-07-09 01:41:47.066297+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools", "mlops"], "entities": ["Databricks", "AWS", "Terraform", "Terragrunt", "Atlantis"], "alternates": {"html": "https://wpnews.pro/news/databricks-on-aws-6-how-we-structure-the-terraform-terragrunt-yaml-driven-and", "markdown": "https://wpnews.pro/news/databricks-on-aws-6-how-we-structure-the-terraform-terragrunt-yaml-driven-and.md", "text": "https://wpnews.pro/news/databricks-on-aws-6-how-we-structure-the-terraform-terragrunt-yaml-driven-and.txt", "jsonld": "https://wpnews.pro/news/databricks-on-aws-6-how-we-structure-the-terraform-terragrunt-yaml-driven-and.jsonld"}}