# MiniLake Is a Local Databricks API Emulator with SQL, Delta Lake and Spark

> Source: <https://dmux.github.io/minilake/>
> Published: 2026-10-27 14:12:59+00:00

### Unity Catalog, for real

Catalogs, schemas, tables and volumes with full CRUD. Every catalog is its own DuckDB database file, so `catalog.schema.table` addressing is native — not a regex rewrite.

          MiniLake is a local **Databricks API emulator** with **real SQL**,
          **real Delta Lake** and **real Spark job execution**. Point
          `databricks-sdk`, the Databricks CLI, Terraform, Asset Bundles or an LLM agent at
          `http://localhost:8000` — no account, no API key, no cloud bill.
        

`pip install minilake && minilake --port 8000`
PyPI
`docker run -p 8000:8000 ghcr.io/dmux/minilake:latest`
GHCR
Then open `http://localhost:8000/ui/` for the built-in SQL workspace. The image downloads nothing at runtime — it works air-gapped.

Fake endpoints let broken code pass. MiniLake executes your SQL on DuckDB, writes real Delta files, and runs real `spark-submit` jobs in containers — so a green test locally means something.

Statement execution on real DuckDB: `INLINE` and `EXTERNAL_LINKS` dispositions, `JSON_ARRAY` / `ARROW_STREAM` / `CSV` formats, and a result manifest that carries true column types.

EXTERNAL Delta tables are genuine Parquet plus a `_delta_log/`. `INSERT`, `UPDATE` and `DELETE` are routed through a real Spark job; reads go through DuckDB's `delta_scan()`.

`spark_python_task` runs `spark-submit` in a sibling container and returns actual stdout. Real DAG scheduling honours `depends_on` and `run_if`, with independent branches in parallel.

`spark.table("cat.sch.tbl")` resolves against MiniLake through the official `UCSingleCatalog` connector — the same protocol Spark speaks to a real workspace.

An Athena-style editor at `/ui`: Monaco with catalog-aware completion, a virtualized result grid, charts, CSV/JSON export, saved queries, query history — and an embedded JupyterLab.

Point the `databricks` provider at localhost and `terraform apply` creates real catalogs. Verified on every test run: the real provider is driven through `apply`, `plan` and `destroy`, and a second plan has to come back empty. `databricks bundle deploy` and `bundle run` work end to end — jobs execute for real.

67 tools, 8 resources and 4 prompts at `/mcp` — including composites that collapse a five-call sequence into one, and a SQL-dialect resource that stops agents writing Spark SQL.

One `POST /_minilake/reset` gives every test a clean workspace without restarting the container. `/health`, `/ready` and `/services` round out the control plane.

File-backed notebook and file storage with chunked uploads. Secrets are real CRUD whose values are only ever resolvable inside a job's environment — exactly like Databricks.

Real execution
DuckDB's `delta` extension and the Delta / Unity Catalog jars are baked in at build time. The image downloads nothing at runtime — provable with `docker run --network none`.

Failures come back as `{"error_code", "message"}`, so the SDK raises its normal typed exceptions. Anything out of scope returns a loud `501 NOT_IMPLEMENTED` instead of a convincing lie.

MiniLake speaks the Databricks REST API on the wire, so every client you already use keeps working — the same calls, the same response shapes, the same typed errors.

`STRING`, `LONG`, `DECIMAL(10,2)`, `ARRAY<T>`, `MAP` and `STRUCT` map to DuckDB and come back as the SDK's `ColumnInfo`.` MINILAKE_PERSIST=1` snapshots state on shutdown and restores it on start.

``` python
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.catalog import ColumnInfo, TableType

# The only line that differs from production
w = WorkspaceClient(host="http://localhost:8000", token="dev")

w.catalogs.create(name="sales")
w.schemas.create(name="store", catalog_name="sales")
w.tables.create(
    name="orders", catalog_name="sales", schema_name="store",
    table_type=TableType.MANAGED,
    data_source_format="DELTA", storage_location="/data/sales/store/orders",
    columns=[
        ColumnInfo(name="id", type_text="BIGINT"),
        ColumnInfo(name="customer", type_text="STRING"),
        ColumnInfo(name="amount", type_text="DECIMAL(10,2)"),
    ],
)

wh = w.warehouses.create(name="dev", cluster_size="Small")
w.statement_execution.execute_statement(
    warehouse_id=wh.id,
    statement="INSERT INTO sales.store.orders VALUES (1, 'Ana', 459.90)",
)

result = w.statement_execution.execute_statement(
    warehouse_id=wh.id,
    statement="SELECT customer, amount FROM sales.store.orders",
)
print(result.result.data_array)   # [['Ana', '459.90']] — really stored
python
from pyspark.sql import SparkSession

UC = "http://localhost:8000"

spark = (
    SparkSession.builder
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
    # Unity Catalog connector — three-part names resolve against MiniLake
    .config("spark.sql.catalog.spark_catalog", "io.unitycatalog.spark.UCSingleCatalog")
    .config("spark.sql.catalog.spark_catalog.uri", UC)
    .config("spark.sql.catalog.sales", "io.unitycatalog.spark.UCSingleCatalog")
    .config("spark.sql.catalog.sales.uri", UC)
    .getOrCreate()
)

# Real Delta files: parquet + _delta_log/, written by real Spark
spark.createDataFrame([(1, "a"), (2, "b")], ["id", "name"]) \
     .write.format("delta").mode("overwrite").save("/data/delta/sales/store/events")

spark.sql("INSERT INTO sales.store.events VALUES (3, 'c')")
print(spark.table("sales.store.events").count())   # 3

# ...and the same bytes are readable through MiniLake's SQL API:
#   SELECT e.name, o.customer
#   FROM sales.store.events e JOIN sales.store.orders o ON o.id = e.id
provider "databricks" {
  host  = "http://localhost:8000"
  token = "dev"
}

resource "databricks_catalog" "sandbox" {
  name    = "sandbox"
  comment = "Local sandbox catalog"
}

resource "databricks_schema" "things" {
  catalog_name = databricks_catalog.sandbox.name
  name         = "things"
}

resource "databricks_sql_endpoint" "compute" {
  name         = "local-compute"
  cluster_size = "Small"
}

# terraform apply  -> creates real catalogs and real DuckDB databases
# terraform destroy -> removes them
# Resources backed by APIs MiniLake does not emulate fail loudly with 501,
# instead of silently pretending to succeed.
bundle:
  name: my_pipeline

targets:
  dev:
    mode: development
    workspace:
      host: http://localhost:8000

resources:
  jobs:
    daily_load:
      name: daily_load
      tasks:
        - task_key: main
          spark_python_task:
            python_file: ../src/load.py

# $ databricks bundle deploy -t dev
# $ databricks bundle run daily_load -t dev
#
# The CLI syncs files through the Workspace API, the provider creates the job,
# and `bundle run` executes it on real Spark in a sibling container —
# returning the job's actual output.
# Is it up, and which services answered?
curl http://localhost:8000/_minilake/health

# Create a catalog through the Databricks API surface
curl -X POST http://localhost:8000/api/2.1/unity-catalog/catalogs \
  -H 'Content-Type: application/json' \
  -d '{"name": "sales"}'

# Run SQL and get real rows back
curl -X POST http://localhost:8000/api/2.0/sql/statements \
  -H 'Content-Type: application/json' \
  -d '{"warehouse_id": "dev", "statement": "SELECT 42 AS answer"}'

# Give the next test a clean workspace — no restart needed
curl -X POST http://localhost:8000/_minilake/reset
```

A FastAPI application serves the Databricks REST surface, the web UI and the MCP endpoint together. Behind it sit DuckDB, real Delta files on disk, and Docker for Spark.

A Databricks workspace is an awkward thing for an agent to drive: every useful action is four calls deep, half the SQL it writes is the wrong dialect, and failures are silent. MiniLake's MCP server closes those gaps — and it costs nothing when the agent gets it wrong.

Plus 8 resources — including a SQL-dialect guide that stops agents writing Spark SQL at DuckDB — and 4 prompts that front-load the rules. Tools run through MiniLake's own ASGI stack, so an agent's error is exactly the error an SDK client would have seen.

```
# One line to register it with Claude Code
claude mcp add --transport http minilake http://localhost:8000/mcp

# Then just ask:
#   "Use minilake: create a sales catalog with a seeded
#    orders table, and tell me the total."

# The agent does this — three calls, no glue code:
#   setup_fixture({"catalog": "sales", "schema": "store"})
#   seed_table({"full_name": "sales.store.orders", ...})
#   run_sql({"statement": "SELECT SUM(amount) FROM sales.store.orders"})

# Enable it on the server side:
MINILAKE_MCP=1 minilake --port 8000
```

Every row below is either really executed or clearly labelled. Anything out of scope answers `501 NOT_IMPLEMENTED` rather than faking a success your production code will not get.

| Service | Status | What that means | 
|---|---|---|
| Unity Catalog | Real | Catalogs, schemas, tables, volumes, functions — each catalog is its own DuckDB database, with native three-part addressing | 
| UC grants | Unenforced CRUD | With real inheritance — a grant on a catalog is effective on its schemas and tables — but no privilege is ever checked | 
| UC functions | Real | A SQL function becomes a real DuckDB macro, so `SELECT cat.sch.fn(x)` actually computes — not a registry entry | 
| SQL statements | Real | Executed on DuckDB; all three formats and both dispositions, with a typed result manifest | 
| EXTERNAL Delta tables | Real | Genuine Delta files; writes go through a generated Spark job, reads through `delta_scan()` | 
| Jobs | Real | `spark-submit` in sibling containers (or a subprocess fallback), real DAG scheduling, real logs, one-shot`runs/submit` | 
| SQL warehouses | Real | Full CRUD and lifecycle; statements run regardless of state, as they do locally | 
| Query history & saved queries | Real | Server-backed, failures included — which is usually why you opened history | 
| Alerts | Real | The condition is evaluated against real query results, so an alert fires on your data — never on a stored constant | 
| Workspace, DBFS & Files | Real | File-backed storage with chunked upload; raw-bytes sync is what makes `bundle deploy` work | 
| Secrets | Real | Real CRUD; values are resolvable only inside job environment variables, never through the API | 
| Web UI & notebooks | Real | Athena-style SQL workspace at `/ui` , with JupyterLab proxied at`/jupyter` | 
| MCP server | Real | Opt-in with `MINILAKE_MCP=1` : 67 tools, 8 resources, 4 prompts on the same port | 
| Persistence | Real | `MINILAKE_PERSIST=1` snapshots state as JSON on shutdown and restores it on startup | 
| Clusters | State machine | CRUD plus timed lifecycle transitions — deliberately no Spark compute; that lives in Jobs | 
| Policies & instance pools | Unenforced CRUD | They resolve and validate, so a bundle naming one deploys — but there is no compute here for a policy to constrain | 
| Permissions | Allow-all CRUD | Real create/read/update, but nothing is ever enforced — this is a single-user tool | 
| Workspace admin | Unenforced CRUD | Git credentials, IP access lists, init scripts, notification destinations, instance profiles, workspace conf — what a `terraform apply` touches on its way past | 
| Identity & SCIM | Unenforced CRUD | Users, groups and service principals, with SCIM filtering and `PATCH` — records, not credentials; any token is still accepted | 
| Tokens | Unenforced CRUD | The value comes back exactly once, as it does for real — and then authenticates nothing | 
| Legacy SQL ( `preview/sql` ) | Real | Queries, alerts and dashboards on the deprecated routes — adapters over the modern stores, so both surfaces see one object | 
| DLT, Model Registry, Vector Search, Lakeview, Repos | Not implemented | Returns `501 NOT_IMPLEMENTED` — loudly, so you never mistake it for working | 

These are deliberate design decisions, not a backlog. MiniLake targets one developer running it locally — not a shared, multi-tenant server.

Any token is accepted and there is exactly one user. Never expose it beyond localhost.

The Permissions API is real CRUD that always allows everything — a passing test here says nothing about grants in a real workspace.

Statements run on DuckDB's dialect. The MCP server ships a dialect resource for exactly this reason; the docs carry the translation table.

The cluster lifecycle is a state machine. Real Spark happens through Jobs, in sibling containers.

DuckDB is single-writer, so concurrent load serializes on a lock. Fine for dev and CI; not a deployment target.

They are DuckDB tables with no files. Use EXTERNAL Delta when Spark has to see the data — that trade-off is what keeps SQL fast.

Being explicit about the edges is the point: an emulator that quietly fakes what it cannot do is worse than no emulator at all.

Free and MIT-licensed. No account, no API key, no cloud bill — and nothing to uninstall from your cloud account when you're done.