📚 Series: Databricks on AWS (Part 6)
- Building a Databricks AI Platform on AWS
- RBAC with Function-Role Groups
- Compute Governance: Pools, Policies, Clusters
- The BOOTSTRAP_TIMEOUT Mystery
- Fixing It with AWS PrivateLink How We Structure the Terraform← you are hereFive parts of
whatwe built — the workspaces, the RBAC, the compute, the network mystery, the PrivateLink fix. This one is thehow: the repo layout and Terragrunt patterns that hold it all together, plus the two footguns that will bite you on day one.
Over five parts we stood up an AI platform on Databricks + AWS. Along the way I kept saying "we apply this," "the module for that," "the provider override" — and mostly waved my hands at the plumbing. This finale is the plumbing: how the Terraform is actually organized, why humans never touch a .tf
file, and the two ordering gotchas that turn a clean plan into a red one.
If you're standing up a multi-workspace Databricks estate and want a layout that scales past the first workspace without copy-paste sprawl, this is the shape that worked for us.
Two ideas do all the work: reusable modules and per-environment trees.
infra/
├── _modules/ # reusable Terraform modules (the "what")
├── <project>/
│ └── databricks/
│ ├── dev/ # per-env tree (the "where")
│ └── prd/
├── common_vars.yaml # shared config: hosts, account IDs, CIDRs
└── provider.tmpl # provider template
_modules/
is a flat library of single-purpose Terraform modules — one per resource concept:
workspace, unity-catalog, catalog, schema, grants, group, group-ar,
instance-pool, cluster-policy, compute, sql-warehouse, entitlements,
external-location, workspace-assignment, acls, iamrole, iampolicy,
s3, kms, vpc, service-principal, audit-log-delivery, workspace-conf
None of these know about dev
or prd
. They take inputs and create resources. The environment tree decides which modules run, with what values, in what order. A new workspace is a new folder under dev/
, not a new module.
Here's the rule we enforce: a person editing this repo edits YAML. They don't write Terraform, and mostly don't write Terragrunt either.
The flow is three hops:
human-authored *.yaml
→ terragrunt.hcl: yamldecode() + env-prefix
→ Terraform module inputs
Each leaf directory has a small *.yaml
(the desired state, in human terms) and a terragrunt.hcl
that reads it. The terragrunt.hcl
does yamldecode()
on the YAML, then rewrites the keys with an environment prefix before handing them to the module. Concretely, in the dev
tree an instance-pool key like ip_cpu_small
becomes dev_ip_cpu_small
on the way in.
Why the prefix? Because the same YAML shape lives in dev/
and prd/
, but the resource names, pool IDs, and group names must be globally distinct. The env prefix is injected once, in one place, by Terragrunt — so no human ever hand-types dev_
or prd_
and nobody fat-fingers a cross-environment collision. You change a number in YAML; the naming discipline is automatic.
This is the single highest-leverage decision in the whole layout. The reviewer of a merge request reads a YAML diff — "pool max went 8 → 16" — not a wall of HCL.
Databricks has a wrinkle that AWS doesn't: every workspace is its own API host. A databricks_cluster
in workspace A and one in workspace B are created against different host
URLs, even though it's the same account, same provider, same code.
We solve this with generated provider overrides. Each workspace layer knows its workspace_key
(from a small workspace.hcl
), and common_vars.yaml
maps that key to the workspace host. Terragrunt looks up the host and generates a provider_override.tf
on the fly:
locals {
ws_host = local.common_vars.databricks
.projects.<project>.workspace_hosts[local.env][local.ws_vars.locals.workspace_key]
}
generate "workspace_provider_override" {
path = "provider_override.tf"
contents = <<-EOF
provider "databricks" {
host = "${local.ws_host}"
}
EOF
}
The module code is workspace-agnostic. The host is injected at plan time based on which folder you're in. Add a workspace, add its host to common_vars.yaml
, drop a workspace.hcl
with the key — every layer underneath now targets the right host with zero code changes.
You'll notice the code uses the Databricks provider two different ways, and it's not an accident:
| provider | scope | used for |
|---|---|---|
databricks.mws (account-level) |
||
| the Databricks account | ||
| creating workspaces, account-level groups, metastore assignment | ||
databricks (workspace-level) |
||
| a single workspace host | ||
| clusters, pools, policies, catalogs, grants, ACLs |
Account-level resources (the workspace itself, account groups, wiring a metastore) don't belong to any one workspace — they're configured against the account API, which is why they use the mws
alias. Everything inside a workspace uses the workspace-scoped provider with the host we just injected. Mixing these up is a classic early mistake: try to create a cluster with the mws
provider and you'll get authentication errors that make no sense until you realize you're pointed at the account, not the workspace.
Layers depend on each other's outputs. cluster-policy
needs the pool IDs that instance-pool
produced; compute
needs the policy IDs. Terragrunt expresses this with a dependency
block:
dependency "instance_pool" {
config_path = "../instance-pool"
mock_outputs = {
instance_pool_ids = { "dev_ip_cpu_small" = "mock-pool-id-1" }
}
mock_outputs_allowed_terraform_commands = ["validate", "plan"]
}
The mock_outputs
let you plan
a layer before its dependency has ever been applied — Terragrunt feeds fake IDs so the plan can render. Handy. But look closely at that allow-list: ["validate", "plan"]
. No "init".
That omission is the gotcha, and it ties straight back to the timeout saga from Parts 3 and 4. On a brand-new tree, the very first plan
runs an init
first — and because init
isn't in the mock allow-list, Terragrunt refuses to mock the dependency and instead tries to read real outputs from a state that doesn't exist yet. Result:
Error: detected no outputs from dependency ".../instance-pool"
Your clean-looking layer fails to plan not because it's wrong, but because the thing upstream of it was never applied. Some layers in the repo (workspace-assignment
, unity-catalog
) deliberately include "init"
in the allow-list to dodge this; the compute layers don't. The practical consequence is the same either way, and it's the whole reason for the next section.
Because of the dependency chain above, you apply in order. Not "usually." Always, the first time through:
Workspace line: workspace → instance-pool → cluster-policy → compute
→ workspace-assignment → acls
Unity Catalog: unity-catalog → external-location → catalogs
→ schemas → groups → grants
The rule underneath both lines is the same: apply a layer only after everything it depends on is already applied. Mocks let you plan out of order, but the first real apply has to walk the chain, because a mock pool ID doesn't create a cluster and a mock catalog doesn't hold a grant.
Two failure modes we hit repeatedly, both just ordering:
cluster-policy
/ compute
plan fails with "detected no outputs" → the pool/policy above it wasn't applied. Apply upstream first.workspace-assignment
plans to assignments = {}
→ the groups
layer wasn't applied, so there's nothing to assign. Apply groups
first.Neither is a bug. Both are the dependency graph telling you that you skipped a step.
We don't run terragrunt apply
from laptops. Every change is a merge request; Atlantis runs atlantis plan
on the MR and atlantis apply
once it's approved. The plan output lands in the MR as a comment, so the diff and its consequences are reviewable in one place. Account credentials live server-side on the Atlantis host, not in anyone's shell — which also means local plan
is optional (and honestly, usually skipped) since you get the real plan from the MR anyway.
Put together, the daily loop is small: edit a YAML value, open an MR, read the Atlantis plan, apply. The modules, the env-prefixing, the provider injection, the dependency graph — all of that is machinery you set up once and then mostly forget.
That's the whole platform:
If there's one throughline, it's that the boring parts — naming discipline, apply order, who-writes-what — are exactly the parts that decide whether a platform stays sane at the tenth workspace. Get the structure right and the interesting problems (like a packet dying in a firewall) are the only ones left to solve.
Thanks for reading all six. If you're just arriving, Part 1 is where the estate gets built from nothing.