{"slug": "lakebase-branching-for-ci-cd-lessons-from-100-pr-databases", "title": "Lakebase branching for CI/CD: lessons from 100 PR databases", "summary": "Snorkel AI cut per-pull-request database provisioning from about 20 minutes to under a minute by replacing Aurora restores with copy-on-write branches of Databricks' serverless Postgres database, Lakebase, according to a Databricks blog post by Yasamin Tari and Kartik Mathur. Snorkel replicates its production Aurora PostgreSQL cluster into Lakebase's main branch via AWS DMS, keeping it within about 2 minutes of production, then creates an isolated writable branch for each of its roughly 100 PR databases when a pull request opens and deletes it on merge or close, with branch TTL as a backstop. The setup runs tests against real data, including 10 to 20 MB JSONB values carried over in full LOB mode, over private networking with Okta SSO for interactive developers and separate credentials for unattended CI pipelines.", "body_md": "Copy-on-write branching makes isolated copies of production data fast and cheap for CI/CD testing and agentic use cases.\n\nby  [Yasamin Tari](https://www.databricks.com/blog/author/yasamin-tari)  and  [Kartik Mathur](https://www.databricks.com/blog/author/kartik-mathur) \n\nSnorkel AI builds data development tools for AI teams, used by expert data annotators and frontier model providers. In this specific use case, Snorkel ships code through a CI pipeline built on Kubernetes. Every open pull request, plus ad-hoc commits, gets its own ephemeral Kubernetes namespace (a \"PR namespace\"). Developers deploy their branch there and run the full test suite before merging.\n\nThese tests are most useful when they run against real data. Snorkel's production database is an Amazon Aurora PostgreSQL cluster. Giving each PR namespace its own copy of that database was the hard part. Restoring a copy took about 20 minutes per environment, which is too slow when developers open PRs all day and the merge queue needs a fresh environment for each run. The alternatives (shared staging databases, trimmed fixtures, or synthetic data) bring back the problems production-like testing is supposed to solve: tests interfering with each other, stale schemas, and bugs that only show up with real data shapes.\n\nSnorkel would consider Lakebase a success for CI/CD if it met these criteria:\n\nLakebase is Databricks' serverless Postgres database. Its storage and compute are separate, and it supports branching: you can create a new, isolated, writable branch of a database from any point in time, the same way you create a Git branch.\n\nBranches are copy-on-write. Creating a branch is a metadata operation, so no data is copied when the branch is created. A new branch shares storage pages with its parent, and only the pages that the branch changes are written as new storage. For CI, this means:\n\n**1. Keep a fresh copy of production on the main branch.** Lakebase is standard Postgres, so it works as a target for AWS DMS with no custom tooling. Snorkel points a DMS task at its production Aurora database and replicates it into the Lakebase project's main branch: a full load first, then ongoing change data capture that keeps main within about 2 minutes of production. Full LOB mode carries over the 10 to 20 MB JSONB values. The team already runs DMS, so there was no new replication pipeline to build or maintain. The main branch is the \"golden copy\" every PR branch starts from.\n\n**2. Create a branch when a PR opens.** When a developer opens a PR, the CI pipeline calls the Lakebase API to create a branch from main, sets an expiration time on it, and passes the branch's connection details to the PR namespace. The branch is ready in under a minute.\n\n**3. Run tests against real data.** The application in the PR namespace connects to its branch over standard Postgres. Tests can run migrations, write data, and change schemas without affecting production, main, or any other PR.\n\n**4. Delete the branch when the PR closes or test passes, with TTL as a backstop.** When the PR is merged or closed, the pipeline deletes the branch. If that cleanup step doesn't run, the branch expires on its own when its TTL runs out.\n\nTwo security requirements ruled out other options, so they deserve their own section.\n\n**Private connectivity.** Production data moves from Aurora to Lakebase, and from Lakebase to the CI cluster, over private networking instead of the public internet.\n\n**Identity.** Snorkel has two kinds of database users: developers who connect interactively to debug a PR environment, and CI pipelines that run with no human involved. Okta SSO is built around interactive browser login, so we split the design by user type:\n\nTwo Lakebase features make this pattern work at scale: copy-on-write branching and branch expiration (TTL).\n\n**Copy-on-write makes branches instant.** A new branch doesn't copy any data. It points to its parent's existing storage pages, and only the pages the branch modifies are written as new storage. Creating a branch from a ~1 TB production copy takes about as long as creating one from a 1 GB database: seconds, not the ~20 minutes a full restore took. Each PR namespace gets its own isolated, writable copy of production right away.\n\n**Copy-on-write makes branches cheap.** Every branch shares most of its data with main, so storage grows with the changes each branch makes, not with the size of the database. For Snorkel, that's about 5 to 10 GB per PR branch on top of the shared base. Compute is per branch and scales to zero when idle, so only the branches that are actively running tests use compute.\n\n**Branch expiration keeps it cheap over time.** Copy-on-write saves the most when branches are short-lived. A branch keeps the version of data that existed when it was created. As DMS keeps updating main with production changes, a branch that stays around keeps holding those older pages, so the storage it uses grows over time. Setting a TTL on every CI branch means each branch is removed automatically once its PR is done. This keeps storage close to the size of the production base plus the changes from active PRs.\n\nPR namespaces aren't the only workloads that need a disposable copy of production. AI agents that write SQL, run migrations, or change data also need a safe place to try things, and copy-on-write branching gives them one.\n\n**One branch per agent task.** Each agent run gets its own branch of main. The agent can read real data, write to it, and even change the schema without touching production or any other agent's work. Creating a branch takes seconds no matter how large the database is, so it's practical to create one for every task.\n\n**Many agents in parallel.** Storage only grows with the pages each branch changes, and each branch's compute scales to zero when it's idle. Running dozens of agents side by side costs about as much as the changes they make, not dozens of copies of the database.\n\n**Review the changes, or throw them away.** Teams can check what an agent changed on its branch before applying anything to production. If the result isn't right, they delete the branch, and there's nothing to roll back.\n\n**TTLs clean up after agents.** Agents can create a lot of short-lived branches. An expiration on each branch removes the abandoned ones automatically, the same way TTLs clean up CI branches.\n\n**Scoped identity for agents.** Like CI pipelines, agents can run as service principals with short-lived OAuth tokens, so every query an agent runs can be traced in the audit logs.\n\nReady to give every pull request its own production-like database? [Get started with Lakebase](https://docs.databricks.com/aws/en/oltp/) and read the branching documentation to set up branch expiration in your CI pipeline.\n\nSubscribe to our blog and get the latest posts delivered to your inbox.", "url": "https://wpnews.pro/news/lakebase-branching-for-ci-cd-lessons-from-100-pr-databases", "canonical_source": "https://www.databricks.com/blog/lakebase-branching-cicd-lessons-100-pr-databases", "published_at": "2026-10-31 21:25:00+00:00", "updated_at": "2026-10-31 21:51:51.845752+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "developer-tools", "ai-tools"], "entities": ["Snorkel AI", "Databricks", "Lakebase", "Amazon Aurora", "AWS DMS", "Kubernetes", "Okta", "Yasamin Tari"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/lakebase-branching-for-ci-cd-lessons-from-100-pr-databases", "markdown": "https://wpnews.pro/news/lakebase-branching-for-ci-cd-lessons-from-100-pr-databases.md", "text": "https://wpnews.pro/news/lakebase-branching-for-ci-cd-lessons-from-100-pr-databases.txt", "jsonld": "https://wpnews.pro/news/lakebase-branching-for-ci-cd-lessons-from-100-pr-databases.jsonld"}}