Learn a phased migration approach to adopting Databricks Lakehouse
by Takero Ibuki Many organizations support a range of data and analytics workloads on BigQuery today, but are exploring ways to bring BI, ETL, and multi-model AI workloads into a single governance and operating model. An open Lakehouse with a unified governance layer can help simplify operations, streamline compliance across data and AI, provide predictable performance as needs grow, and create a foundation for new AI-driven use cases.
A successful migration requires more than copying tables. It demands a phased strategy: moving data into open formats on services like Google Cloud Storage, transforming logic thoughtfully, and validating results with automated tooling. This guide outlines a migration framework across three pillars – Process, Technology, and People – for navigating that transition with minimal disruption and measurable business impact.
The Process pillar defines how you migrate. Success depends on choosing the right entry point and managing the transition period effectively.
The migration journey runs in sequence: assess the estate, choose an entry point, migrate, validate in dual operation, then decommission. Assessment comes before everything else: you cannot pick the right entry point without knowing what you run today. Profile the BigQuery estate first — datasets, query history and slot consumption — to find which workloads drive cost, which dashboards are actually used, and which tables are never queried at all. Lakebridge, the open-source Databricks Labs migration toolkit, includes a BigQuery profiler that automates this discovery, so migration waves are planned around what is actually used rather than what merely exists.
Whichever entry point you choose, migrate in waves, not in one big bang. A big-bang cutover concentrates all the risk in a single moment — and if anything breaks, so does trust in the whole program. Waves keep the blast radius small: rank workloads on two axes — value to the organization (how visible it is to leadership, how directly it touches revenue, how urgent the compliance deadline behind it is, how many people rely on it day to day) and migration complexity — and start where value is high and complexity is low. Each wave then delivers a visible business win, reconciles before the next starts, and leaves the team's playbook faster for the one after it. Reserve big bang for the rare small, low-risk estate where running two platforms costs more than it protects.
Most organizations navigate a Dual Operation phase using Lakehouse Federation to "shadow" workloads for validation. The key to ROI is setting clear "success criteria" (e.g., 99.9% parity) to trigger the immediate decommissioning of legacy pipelines, eliminating the cost of running two platforms.
Dual operation works in both directions, and the right bridge depends on your entry point. BI-first teams use Lakehouse Federation so Databricks can read BigQuery while dashboards move first. ETL-first teams flip it: move ingestion and transformation to Databricks, land curated tables in open formats, and let BigQuery keep serving existing dashboards and applications from that same single copy — no dual-write pipelines, no export jobs, no second copy to reconcile.
Migrating in waves keeps this dual-operation window short and narrow: only the workloads in the current wave carry the cost of running on two platforms at once, so the bill for dual operation stays proportional to what's actually in flight, not the whole estate. Treat BigQuery-as-serving-layer as a transitional state rather than a destination: the external tables are read-only on BigQuery’s side and carry limitations such as manual schema refresh after schema changes, so plan the cutover of the serving layer onto Databricks Lakehouse as the closing step of the journey.
Unity Catalog transforms governance from a hurdle into a strategic engine. It offers a seamless 3-tier mapping (Project → Catalog, Dataset → Schema, Table → Table) that replicates BigQuery permissions while adding automatic, end-to-end lineage and treating AI models as first-class citizens.
The mapping goes deeper than the object hierarchy. Unity Catalog gives you the same fine-grained protection natively: row filters restrict access row by row, and column masks and tags handle column-level security — so protection travels with the data instead of being rebuilt from scratch. And one sequencing rule from the field: migrate permissions before data, so each wave's cutover changes where a table lives, never who can see it.
The technical pillar focuses on moving data into open formats on services like Google Cloud Storage, migrating logic to Databricks SQL and Spark, and validating results.
Migrating from BigQuery to Databricks involves three workstreams: Data Migration, Logic Migration, and Validation.
Choose the path by volume and freshness. Bulk history moves fastest through BigQuery’s export to Parquet on Google Cloud Storage, usually the economical route at scale, and the export doubles as a frozen point-in-time copy that simplifies validation. Continuously updated tables are read through the Storage API connector or redirected at the source, and small, frequently changing datasets can stay queryable through federation until their wave arrives. Everything lands in open formats, ready for the medallion layers.
Never hand-convert an estate. Three tiers cover it: rule-based transpilation (e.g. Lakebridge) for the bulk of routine SQL, LLM-assisted conversion for dialect quirks, and engineers reserved for the genuinely complex tail. Orchestration follows the same pattern: scheduled queries and Composer DAGs map to Lakeflow Jobs.
Budget for validation as seriously as for the migration itself — in the field it often consumes comparable effort. Run it at three levels: completeness (row counts), consistency (schemas and types), and accuracy (aggregate reconciliation plus row-level hashing), automated with Lakebridge reconcile, which supports BigQuery as a native source. One field lesson: small built-in function differences between the two SQL dialects can break hash comparisons — investigate mismatches before assuming data loss. Parity here is what triggers decommissioning in the Process journey.
The open foundation pays off before the migration even finishes. Because Delta Lake and Apache Iceberg are open formats, the tables you land on Google Cloud Storage are readable by more than Databricks: BigQuery reads Delta through BigLake external tables and Iceberg through Iceberg external tables, and catalog federation between Unity Catalog and BigQuery (currently in preview) lets both platforms govern and query the same tables without copying them. Storage is decoupled from the engine — one copy of data, many engines. That interoperability is not a side benefit; it is the capability behind the low-risk transition patterns described in the Process section.
The third pillar provides the ultimate ROI of a migration: a more capable, unified workforce.
Ending the handoff culture. In legacy stacks, the gap between SQL Analysts and Data Scientists creates silos. In Databricks, shared Notebooks allow the entire "squad" to live in the same workspace, reducing communication overhead.
Building the new skillset. Genie acts as the bridge for SQL-heavy teams. Analysts can use natural language to generate Python or Spark code, turning traditional analysts into versatile data practitioners without a steep learning curve. Pair that day-to-day AI assist with structured training, Databricks Academy courses and role-based certifications, so the new skillset sticks organization-wide instead of depending on a few self-taught early adopters.
Engineering rigor. Teams move from "writing queries" to "building data products" by adopting software engineering best practices like Unity Catalog, Git integration and CI/CD.
Six patterns repeat across successful BigQuery migrations based on our migration project experience:
Migrating to Databricks isn't one leap — it's a sequence of decisions you can actually plan for: which entry point fits your team, how to sequence waves so each one banks a win before the next starts, and which bridge keeps BigQuery and Databricks running side by side until the last dashboard cuts over. Get that sequencing right, and Process, Technology, and People reinforce each other: data migration, logic transformation, and validation retire the estate on one side, while analysts querying in natural language with Genie and engineers shipping governed data products build on it on the other.
The result is an open lakehouse, a migration your organization can trust wave over wave, and a team that's ready to build on what it lands on.
Ready to plan your migration? Explore the Databricks migration hub, assess your estate with the open-source Lakebridge toolkit, or contact your Databricks account team for a migration assessment.
Subscribe to our blog and get the latest posts delivered to your inbox.