Databricks Lakebase: Give Your Agent a Branch, Not Your Production Database

Databricks Lakebase, a Postgres-compatible database with decoupled storage on lake storage, enables sub-second branching and ephemeral instances, making it suitable for agentic data projects that need isolated, production-like environments. The system uses Paxos-based safekeepers and page servers to achieve low-latency reads and writes, with branches costing about 500 milliseconds to create and idle branches scaling to zero. Databricks reports 12 million database launches per day, indicating that ephemeral instances are the intended usage pattern.

There's an awkward moment in every agentic data project. The agent works. It writes decent SQL, it reasons about schema, it proposes a migration that looks right. And then somebody asks the question nobody wants to answer: what happens when it's wrong against production? The usual answers are all bad. Point it at production with read-only credentials, and you've capped the damage but also capped the agent — it can diagnose and never fix. Point it at a seeded staging database, and you get an agent that's confidently correct about data that stopped resembling production four months ago. Spin up a container per run, and you're maintaining fixture pipelines forever while paying for compute that sits idle between runs. What you actually want is a full copy of production that appears in under a second, costs almost nothing while it exists, and vanishes without a trace when the agent is done. Which is to say: you want git checkout -b for your database. Databricks Lakebase does this, and the mechanism is interesting enough to be worth understanding rather than just enabling. Lakebase is Postgres — genuinely Postgres, not a wire-compatible reimplementation. What Databricks changed is the part underneath. Standard Postgres is a monolith with compute and storage tightly coupled. Databricks decoupled the two and moved storage onto lake storage, which creates an immediate problem: object storage has high latency and no transactional consistency, and Postgres assumes neither. Two components patch the gap. Safekeepers , built on the Paxos consensus algorithm, handle low-latency durable writes. Page servers handle low-latency reads, materializing pages on demand from the lake. Once storage is decoupled and page-addressed, branching becomes almost free. It's copy-on-write: the data stays in one place on the lake, and Lakebase tracks only the deltas between branches. Writing to a branch records the change independently without touching the others. The numbers that matter for agent work: a branch takes roughly 500 milliseconds , a new instance comes up in under 500 milliseconds, rollback to an earlier snapshot is equally fast, and idle branches scale to zero so you pay only cheap lake storage while they sit around. Databricks reports 12 million database launches per day in production, which is less a benchmark than an indication that ephemeral instances are the intended usage pattern rather than an edge case. That last property is what makes this viable for agents specifically. As Ali Ghodsi put it on the summit keynote stage, agents "don't want to wait 10 minutes on a database to come up." An agent that has to wait ten minutes for an environment will simply be given production instead, because the alternative makes the loop unusable. Sub-second provisioning is what makes the safe option also the convenient one. | Approach | Provisioning time | Data realism | Idle cost | Blast radius of a bad write | Ongoing maintenance | |---|---|---|---|---|---| | Read-only prod credentials | None | Perfect | None | Zero — but agent can't act | None | | Full prod write access | None | Perfect | None | Catastrophic | None | | Seeded staging database | Minutes to hours | Degrades continuously | Full instance cost | Contained, but shared between runs | High — fixture pipelines, refresh jobs | | Container per run with dump restore | Minutes, scales with data size | Good at restore time | None between runs | Contained per run | Medium — dump/restore tooling | | Lakebase branch per run | ~500 ms | Identical to parent at branch time | Near zero storage deltas only | Contained per run, discardable | Low | The column that usually decides it is provisioning time, because it governs whether developers actually use the isolated path or route around it under deadline pressure. Here's the pattern end to end. The Databricks-specific surface is deliberately isolated into one thin layer; everything else is ordinary Postgres, because that's what it is. Step one — get an instance. Lakebase is now included in Databricks Free Edition alongside Genie Code, serverless GPUs, Agent Bricks and LakeFlow Designer, so this costs nothing to follow along with. lakebase control.py — the only Databricks-specific layer in this design. Endpoint shapes for branch operations are still settling; check current docs before shipping. Everything downstream of get dsn is plain Postgres. import os import time import requests WORKSPACE = os.environ "DATABRICKS HOST" .rstrip "/" TOKEN = os.environ "DATABRICKS TOKEN" HEADERS = {"Authorization": f"Bearer {TOKEN}"} def create branch parent: str, name: str - dict: """Branch an existing Lakebase instance. Copy-on-write; returns in ~500ms.""" started = time.perf counter resp = requests.post f"{WORKSPACE}/api/2.0/database/instances/{parent}/branches", headers=HEADERS, json={"name": name}, timeout=30, resp.raise for status elapsed ms = time.perf counter - started 1000 print f"branch {name r} ready in {elapsed ms:.0f} ms" return resp.json def delete branch parent: str, name: str - None: requests.delete f"{WORKSPACE}/api/2.0/database/instances/{parent}/branches/{name}", headers=HEADERS, timeout=30, .raise for status def get dsn instance: str, branch: str - str: """Standard Postgres connection string for a branch.""" resp = requests.get f"{WORKSPACE}/api/2.0/database/instances/{instance}/branches/{branch}", headers=HEADERS, timeout=30, resp.raise for status host = resp.json "read write dns" return f"postgresql://{os.environ 'LAKEBASE USER' }:{TOKEN}@{host}:5432/databricks postgres?sslmode=require" Step two — wrap it so the agent physically cannot see production. This is the important part, and it's boring on purpose. The agent never receives a DSN; it receives a session bound to a branch. python agent sandbox.py import contextlib import psycopg from lakebase control import create branch, delete branch, get dsn PROD INSTANCE = "orders-prod" @contextlib.contextmanager def branched session run id: str, keep on failure: bool = True : """ Yield a Postgres connection on a throwaway branch of production. On success the branch is discarded. On failure it is retained so you can attach a psql session and see exactly what the agent did — which is the single most useful debugging affordance in this whole design. """ branch = f"agent-{run id}" create branch PROD INSTANCE, branch conn = None try: conn = psycopg.connect get dsn PROD INSTANCE, branch , autocommit=False yield conn conn.commit delete branch PROD INSTANCE, branch except Exception: if conn: conn.rollback if not keep on failure: delete branch PROD INSTANCE, branch else: print f"retained branch {branch r} for inspection" raise finally: if conn: conn.close Step three — verify before you believe anything. An agent reporting success is not evidence. Snapshot the invariants you care about before and after, and diff them. python verify.py from dataclasses import dataclass INVARIANTS = { "order count": "SELECT count FROM orders", "open order total": "SELECT coalesce sum total cents , 0 FROM orders WHERE status = 'open'", "orphaned items": "SELECT count FROM order items i " "LEFT JOIN orders o ON o.id = i.order id WHERE o.id IS NULL", "negative totals": "SELECT count FROM orders WHERE total cents < 0", "duplicate skus": "SELECT count FROM " " SELECT sku FROM products GROUP BY sku HAVING count 1" " d", } @dataclass class Snapshot: values: dict @classmethod def take cls, conn - "Snapshot": out = {} with conn.cursor as cur: for name, sql in INVARIANTS.items : cur.execute sql out name = cur.fetchone 0 return cls out def diff self, other: "Snapshot" - dict: return { k: self.values k , other.values k for k in self.values if self.values k = other.values k } def assert no corruption before: Snapshot, after: Snapshot - None: """Hard invariants must not move regardless of what the task was.""" for guard in "orphaned items", "negative totals", "duplicate skus" : if after.values guard = before.values guard : raise AssertionError f"invariant {guard} violated: " f"{before.values guard } - {after.values guard }" Step four — the run itself. python run agent task.py import uuid from agent sandbox import branched session from verify import Snapshot, assert no corruption def run task: str, agent - dict: run id = uuid.uuid4 .hex :8 with branched session run id as conn: before = Snapshot.take conn The agent gets a live connection to real-shaped data and full write access — to a branch that exists only for this run. agent.execute task, conn after = Snapshot.take conn assert no corruption before, after return { "run id": run id, "changed": after.diff before , } This isn't a pattern I invented for the article — it's the shape Databricks uses internally for Genie ZeroOps, its autonomous pipeline-remediation agent. ZeroOps investigates a failure, drafts a fix, then creates shallow clones of production data using the same branching mechanism as Lakebase, deploys the proposed fix to the clone, verifies row counts, and presents the result as a pull request. Nothing reaches production without explicit human approval. The verification-on-a-clone step is what makes autonomous remediation defensible rather than reckless, and it's available to you as a primitive. Worth being direct about, because the isolation is narrower than it feels. Side effects escape the branch. Your database is isolated. The payment API your agent calls is not. Neither is the email it sends, the Slack message it posts, or the Kafka topic it publishes to. A branch protects state you own in Postgres and nothing else — every external integration needs its own sandboxing, and this is where teams get burned after assuming the branch covered them. Branch sprawl is real. Sub-second creation plus retain-on-failure produces a lot of branches. Enforce TTLs from day one, tag branches with the run that created them, and put a reaper on a schedule. Copy-on-write storage is cheap, not free, and a branch that diverges heavily from its parent stops being cheap. Fresh at branch time, stale immediately after. A long-running agent task is reasoning about a snapshot. If it takes twenty minutes and the underlying question was time-sensitive, the conclusion may be wrong by the time it lands — not because the isolation failed but because you asked a real-time question of a point-in-time copy. Invariant checks only catch what you thought to write down. The assert no corruption function above is a floor, not a ceiling. It catches structural damage. It won't catch an agent that correctly updated the wrong customer's record, because that's semantically valid and structurally clean. Human review of the diff remains load-bearing. The reason this matters beyond Databricks is that it closes a gap that has quietly constrained every agentic data project: agents were either safe or useful, and picking both required infrastructure most teams weren't going to build. Application code got this right decades ago. Nobody tests a refactor by editing production source — you branch, you break things freely, you merge what works. Databases never got that affordance because copying one was expensive enough that the ergonomics never arrived. Decoupling compute from storage is what finally made the copy cheap, and cheap copies are what make experimentation safe. Give the agent a branch. Let it break things. Read the diff. Lakebase branch APIs are evolving; the control-plane calls above are structurally accurate but verify endpoint and field names against current Databricks documentation before running in anger. Everything downstream of the DSN is standard Postgres and will behave exactly as you expect.