Unity Catalog, for real
Catalogs, schemas, tables and volumes with full CRUD. Every catalog is its own DuckDB database file, so catalog.schema.table addressing is native — not a regex rewrite.
MiniLake is a local **Databricks API emulator** with **real SQL**,
**real Delta Lake** and **real Spark job execution**. Point
`databricks-sdk`, the Databricks CLI, Terraform, Asset Bundles or an LLM agent at
`http://localhost:8000` — no account, no API key, no cloud bill.
pip install minilake && minilake --port 8000
PyPI
docker run -p 8000:8000 ghcr.io/dmux/minilake:latest
GHCR
Then open http://localhost:8000/ui/ for the built-in SQL workspace. The image downloads nothing at runtime — it works air-gapped.
Fake endpoints let broken code pass. MiniLake executes your SQL on DuckDB, writes real Delta files, and runs real spark-submit jobs in containers — so a green test locally means something.
Statement execution on real DuckDB: INLINE and EXTERNAL_LINKS dispositions, JSON_ARRAY / ARROW_STREAM / CSV formats, and a result manifest that carries true column types.
EXTERNAL Delta tables are genuine Parquet plus a _delta_log/. INSERT, UPDATE and DELETE are routed through a real Spark job; reads go through DuckDB's delta_scan().
spark_python_task runs spark-submit in a sibling container and returns actual stdout. Real DAG scheduling honours depends_on and run_if, with independent branches in parallel.
spark.table("cat.sch.tbl") resolves against MiniLake through the official UCSingleCatalog connector — the same protocol Spark speaks to a real workspace.
An Athena-style editor at /ui: Monaco with catalog-aware completion, a virtualized result grid, charts, CSV/JSON export, saved queries, query history — and an embedded JupyterLab.
Point the databricks provider at localhost and terraform apply creates real catalogs. Verified on every test run: the real provider is driven through apply, plan and destroy, and a second plan has to come back empty. databricks bundle deploy and bundle run work end to end — jobs execute for real.
67 tools, 8 resources and 4 prompts at /mcp — including composites that collapse a five-call sequence into one, and a SQL-dialect resource that stops agents writing Spark SQL.
One POST /_minilake/reset gives every test a clean workspace without restarting the container. /health, /ready and /services round out the control plane.
File-backed notebook and file storage with chunked uploads. Secrets are real CRUD whose values are only ever resolvable inside a job's environment — exactly like Databricks.
Real execution
DuckDB's delta extension and the Delta / Unity Catalog jars are baked in at build time. The image downloads nothing at runtime — provable with docker run --network none.
Failures come back as {"error_code", "message"}, so the SDK raises its normal typed exceptions. Anything out of scope returns a loud 501 NOT_IMPLEMENTED instead of a convincing lie.
MiniLake speaks the Databricks REST API on the wire, so every client you already use keeps working — the same calls, the same response shapes, the same typed errors.
STRING, LONG, DECIMAL(10,2), ARRAY<T>, MAP and STRUCT map to DuckDB and come back as the SDK's ColumnInfo. MINILAKE_PERSIST=1 snapshots state on shutdown and restores it on start.
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.catalog import ColumnInfo, TableType
w = WorkspaceClient(host="http://localhost:8000", token="dev")
w.catalogs.create(name="sales")
w.schemas.create(name="store", catalog_name="sales")
w.tables.create(
name="orders", catalog_name="sales", schema_name="store",
table_type=TableType.MANAGED,
data_source_format="DELTA", storage_location="/data/sales/store/orders",
columns=[
ColumnInfo(name="id", type_text="BIGINT"),
ColumnInfo(name="customer", type_text="STRING"),
ColumnInfo(name="amount", type_text="DECIMAL(10,2)"),
],
)
wh = w.warehouses.create(name="dev", cluster_size="Small")
w.statement_execution.execute_statement(
warehouse_id=wh.id,
statement="INSERT INTO sales.store.orders VALUES (1, 'Ana', 459.90)",
)
result = w.statement_execution.execute_statement(
warehouse_id=wh.id,
statement="SELECT customer, amount FROM sales.store.orders",
)
print(result.result.data_array) # [['Ana', '459.90']] — really stored
python
from pyspark.sql import SparkSession
UC = "http://localhost:8000"
spark = (
SparkSession.builder
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
.config("spark.sql.catalog.spark_catalog", "io.unitycatalog.spark.UCSingleCatalog")
.config("spark.sql.catalog.spark_catalog.uri", UC)
.config("spark.sql.catalog.sales", "io.unitycatalog.spark.UCSingleCatalog")
.config("spark.sql.catalog.sales.uri", UC)
.getOrCreate()
)
spark.createDataFrame([(1, "a"), (2, "b")], ["id", "name"]) \
.write.format("delta").mode("overwrite").save("/data/delta/sales/store/events")
spark.sql("INSERT INTO sales.store.events VALUES (3, 'c')")
print(spark.table("sales.store.events").count()) # 3
provider "databricks" {
host = "http://localhost:8000"
token = "dev"
}
resource "databricks_catalog" "sandbox" {
name = "sandbox"
comment = "Local sandbox catalog"
}
resource "databricks_schema" "things" {
catalog_name = databricks_catalog.sandbox.name
name = "things"
}
resource "databricks_sql_endpoint" "compute" {
name = "local-compute"
cluster_size = "Small"
}
bundle:
name: my_pipeline
targets:
dev:
mode: development
workspace:
host: http://localhost:8000
resources:
jobs:
daily_load:
name: daily_load
tasks:
- task_key: main
spark_python_task:
python_file: ../src/load.py
#
curl http://localhost:8000/_minilake/health
curl -X POST http://localhost:8000/api/2.1/unity-catalog/catalogs \
-H 'Content-Type: application/json' \
-d '{"name": "sales"}'
curl -X POST http://localhost:8000/api/2.0/sql/statements \
-H 'Content-Type: application/json' \
-d '{"warehouse_id": "dev", "statement": "SELECT 42 AS answer"}'
curl -X POST http://localhost:8000/_minilake/reset
A FastAPI application serves the Databricks REST surface, the web UI and the MCP endpoint together. Behind it sit DuckDB, real Delta files on disk, and Docker for Spark.
A Databricks workspace is an awkward thing for an agent to drive: every useful action is four calls deep, half the SQL it writes is the wrong dialect, and failures are silent. MiniLake's MCP server closes those gaps — and it costs nothing when the agent gets it wrong.
Plus 8 resources — including a SQL-dialect guide that stops agents writing Spark SQL at DuckDB — and 4 prompts that front-load the rules. Tools run through MiniLake's own ASGI stack, so an agent's error is exactly the error an SDK client would have seen.
claude mcp add --transport http minilake http://localhost:8000/mcp
MINILAKE_MCP=1 minilake --port 8000
Every row below is either really executed or clearly labelled. Anything out of scope answers 501 NOT_IMPLEMENTED rather than faking a success your production code will not get.
| Service | Status | What that means |
|---|---|---|
| Unity Catalog | Real | Catalogs, schemas, tables, volumes, functions — each catalog is its own DuckDB database, with native three-part addressing |
| UC grants | Unenforced CRUD | With real inheritance — a grant on a catalog is effective on its schemas and tables — but no privilege is ever checked |
| UC functions | Real | A SQL function becomes a real DuckDB macro, so SELECT cat.sch.fn(x) actually computes — not a registry entry |
| SQL statements | Real | Executed on DuckDB; all three formats and both dispositions, with a typed result manifest |
| EXTERNAL Delta tables | Real | Genuine Delta files; writes go through a generated Spark job, reads through delta_scan() |
| Jobs | Real | spark-submit in sibling containers (or a subprocess fallback), real DAG scheduling, real logs, one-shotruns/submit |
| SQL warehouses | Real | Full CRUD and lifecycle; statements run regardless of state, as they do locally |
| Query history & saved queries | Real | Server-backed, failures included — which is usually why you opened history |
| Alerts | Real | The condition is evaluated against real query results, so an alert fires on your data — never on a stored constant |
| Workspace, DBFS & Files | Real | File-backed storage with chunked upload; raw-bytes sync is what makes bundle deploy work |
| Secrets | Real | Real CRUD; values are resolvable only inside job environment variables, never through the API |
| Web UI & notebooks | Real | Athena-style SQL workspace at /ui , with JupyterLab proxied at/jupyter |
| MCP server | Real | Opt-in with MINILAKE_MCP=1 : 67 tools, 8 resources, 4 prompts on the same port |
| Persistence | Real | MINILAKE_PERSIST=1 snapshots state as JSON on shutdown and restores it on startup |
| Clusters | State machine | CRUD plus timed lifecycle transitions — deliberately no Spark compute; that lives in Jobs |
| Policies & instance pools | Unenforced CRUD | They resolve and validate, so a bundle naming one deploys — but there is no compute here for a policy to constrain |
| Permissions | Allow-all CRUD | Real create/read/update, but nothing is ever enforced — this is a single-user tool |
| Workspace admin | Unenforced CRUD | Git credentials, IP access lists, init scripts, notification destinations, instance profiles, workspace conf — what a terraform apply touches on its way past |
| Identity & SCIM | Unenforced CRUD | Users, groups and service principals, with SCIM filtering and PATCH — records, not credentials; any token is still accepted |
| Tokens | Unenforced CRUD | The value comes back exactly once, as it does for real — and then authenticates nothing |
Legacy SQL ( preview/sql ) |
Real | Queries, alerts and dashboards on the deprecated routes — adapters over the modern stores, so both surfaces see one object |
| DLT, Model Registry, Vector Search, Lakeview, Repos | Not implemented | Returns 501 NOT_IMPLEMENTED — loudly, so you never mistake it for working |
These are deliberate design decisions, not a backlog. MiniLake targets one developer running it locally — not a shared, multi-tenant server.
Any token is accepted and there is exactly one user. Never expose it beyond localhost.
The Permissions API is real CRUD that always allows everything — a passing test here says nothing about grants in a real workspace.
Statements run on DuckDB's dialect. The MCP server ships a dialect resource for exactly this reason; the docs carry the translation table.
The cluster lifecycle is a state machine. Real Spark happens through Jobs, in sibling containers.
DuckDB is single-writer, so concurrent load serializes on a lock. Fine for dev and CI; not a deployment target.
They are DuckDB tables with no files. Use EXTERNAL Delta when Spark has to see the data — that trade-off is what keeps SQL fast.
Being explicit about the edges is the point: an emulator that quietly fakes what it cannot do is worse than no emulator at all.
Free and MIT-licensed. No account, no API key, no cloud bill — and nothing to uninstall from your cloud account when you're done.