{"slug": "minilake-is-a-local-databricks-api-emulator-with-sql-delta-lake-and-spark", "title": "MiniLake Is a Local Databricks API Emulator with SQL, Delta Lake and Spark", "summary": "MiniLake, an open-source project from dmux, has released a local Databricks API emulator that executes SQL on DuckDB, writes real Delta Lake files and runs real spark-submit jobs in containers, installable via `pip install minilake` or a GHCR Docker image and reachable at http://localhost:8000. MiniLake speaks the Databricks REST API on the wire so databricks-sdk, the Databricks CLI, Terraform and Asset Bundles work against localhost with no account or API key, and it exposes 67 tools, 8 resources and 4 prompts at /mcp for LLM agents. The project says out-of-scope calls return a 501 NOT_IMPLEMENTED error rather than a simulated success, and the image downloads nothing at runtime, so it runs air-gapped.", "body_md": "### Unity Catalog, for real\n\nCatalogs, schemas, tables and volumes with full CRUD. Every catalog is its own DuckDB database file, so `catalog.schema.table` addressing is native — not a regex rewrite.\n\n          MiniLake is a local **Databricks API emulator** with **real SQL**,\n          **real Delta Lake** and **real Spark job execution**. Point\n          `databricks-sdk`, the Databricks CLI, Terraform, Asset Bundles or an LLM agent at\n          `http://localhost:8000` — no account, no API key, no cloud bill.\n        \n\n`pip install minilake && minilake --port 8000`\nPyPI\n`docker run -p 8000:8000 ghcr.io/dmux/minilake:latest`\nGHCR\nThen open `http://localhost:8000/ui/` for the built-in SQL workspace. The image downloads nothing at runtime — it works air-gapped.\n\nFake endpoints let broken code pass. MiniLake executes your SQL on DuckDB, writes real Delta files, and runs real `spark-submit` jobs in containers — so a green test locally means something.\n\nStatement execution on real DuckDB: `INLINE` and `EXTERNAL_LINKS` dispositions, `JSON_ARRAY` / `ARROW_STREAM` / `CSV` formats, and a result manifest that carries true column types.\n\nEXTERNAL Delta tables are genuine Parquet plus a `_delta_log/`. `INSERT`, `UPDATE` and `DELETE` are routed through a real Spark job; reads go through DuckDB's `delta_scan()`.\n\n`spark_python_task` runs `spark-submit` in a sibling container and returns actual stdout. Real DAG scheduling honours `depends_on` and `run_if`, with independent branches in parallel.\n\n`spark.table(\"cat.sch.tbl\")` resolves against MiniLake through the official `UCSingleCatalog` connector — the same protocol Spark speaks to a real workspace.\n\nAn Athena-style editor at `/ui`: Monaco with catalog-aware completion, a virtualized result grid, charts, CSV/JSON export, saved queries, query history — and an embedded JupyterLab.\n\nPoint the `databricks` provider at localhost and `terraform apply` creates real catalogs. Verified on every test run: the real provider is driven through `apply`, `plan` and `destroy`, and a second plan has to come back empty. `databricks bundle deploy` and `bundle run` work end to end — jobs execute for real.\n\n67 tools, 8 resources and 4 prompts at `/mcp` — including composites that collapse a five-call sequence into one, and a SQL-dialect resource that stops agents writing Spark SQL.\n\nOne `POST /_minilake/reset` gives every test a clean workspace without restarting the container. `/health`, `/ready` and `/services` round out the control plane.\n\nFile-backed notebook and file storage with chunked uploads. Secrets are real CRUD whose values are only ever resolvable inside a job's environment — exactly like Databricks.\n\nReal execution\nDuckDB's `delta` extension and the Delta / Unity Catalog jars are baked in at build time. The image downloads nothing at runtime — provable with `docker run --network none`.\n\nFailures come back as `{\"error_code\", \"message\"}`, so the SDK raises its normal typed exceptions. Anything out of scope returns a loud `501 NOT_IMPLEMENTED` instead of a convincing lie.\n\nMiniLake speaks the Databricks REST API on the wire, so every client you already use keeps working — the same calls, the same response shapes, the same typed errors.\n\n`STRING`, `LONG`, `DECIMAL(10,2)`, `ARRAY<T>`, `MAP` and `STRUCT` map to DuckDB and come back as the SDK's `ColumnInfo`.` MINILAKE_PERSIST=1` snapshots state on shutdown and restores it on start.\n\n``` python\nfrom databricks.sdk import WorkspaceClient\nfrom databricks.sdk.service.catalog import ColumnInfo, TableType\n\n# The only line that differs from production\nw = WorkspaceClient(host=\"http://localhost:8000\", token=\"dev\")\n\nw.catalogs.create(name=\"sales\")\nw.schemas.create(name=\"store\", catalog_name=\"sales\")\nw.tables.create(\n    name=\"orders\", catalog_name=\"sales\", schema_name=\"store\",\n    table_type=TableType.MANAGED,\n    data_source_format=\"DELTA\", storage_location=\"/data/sales/store/orders\",\n    columns=[\n        ColumnInfo(name=\"id\", type_text=\"BIGINT\"),\n        ColumnInfo(name=\"customer\", type_text=\"STRING\"),\n        ColumnInfo(name=\"amount\", type_text=\"DECIMAL(10,2)\"),\n    ],\n)\n\nwh = w.warehouses.create(name=\"dev\", cluster_size=\"Small\")\nw.statement_execution.execute_statement(\n    warehouse_id=wh.id,\n    statement=\"INSERT INTO sales.store.orders VALUES (1, 'Ana', 459.90)\",\n)\n\nresult = w.statement_execution.execute_statement(\n    warehouse_id=wh.id,\n    statement=\"SELECT customer, amount FROM sales.store.orders\",\n)\nprint(result.result.data_array)   # [['Ana', '459.90']] — really stored\npython\nfrom pyspark.sql import SparkSession\n\nUC = \"http://localhost:8000\"\n\nspark = (\n    SparkSession.builder\n    .config(\"spark.sql.extensions\", \"io.delta.sql.DeltaSparkSessionExtension\")\n    # Unity Catalog connector — three-part names resolve against MiniLake\n    .config(\"spark.sql.catalog.spark_catalog\", \"io.unitycatalog.spark.UCSingleCatalog\")\n    .config(\"spark.sql.catalog.spark_catalog.uri\", UC)\n    .config(\"spark.sql.catalog.sales\", \"io.unitycatalog.spark.UCSingleCatalog\")\n    .config(\"spark.sql.catalog.sales.uri\", UC)\n    .getOrCreate()\n)\n\n# Real Delta files: parquet + _delta_log/, written by real Spark\nspark.createDataFrame([(1, \"a\"), (2, \"b\")], [\"id\", \"name\"]) \\\n     .write.format(\"delta\").mode(\"overwrite\").save(\"/data/delta/sales/store/events\")\n\nspark.sql(\"INSERT INTO sales.store.events VALUES (3, 'c')\")\nprint(spark.table(\"sales.store.events\").count())   # 3\n\n# ...and the same bytes are readable through MiniLake's SQL API:\n#   SELECT e.name, o.customer\n#   FROM sales.store.events e JOIN sales.store.orders o ON o.id = e.id\nprovider \"databricks\" {\n  host  = \"http://localhost:8000\"\n  token = \"dev\"\n}\n\nresource \"databricks_catalog\" \"sandbox\" {\n  name    = \"sandbox\"\n  comment = \"Local sandbox catalog\"\n}\n\nresource \"databricks_schema\" \"things\" {\n  catalog_name = databricks_catalog.sandbox.name\n  name         = \"things\"\n}\n\nresource \"databricks_sql_endpoint\" \"compute\" {\n  name         = \"local-compute\"\n  cluster_size = \"Small\"\n}\n\n# terraform apply  -> creates real catalogs and real DuckDB databases\n# terraform destroy -> removes them\n# Resources backed by APIs MiniLake does not emulate fail loudly with 501,\n# instead of silently pretending to succeed.\nbundle:\n  name: my_pipeline\n\ntargets:\n  dev:\n    mode: development\n    workspace:\n      host: http://localhost:8000\n\nresources:\n  jobs:\n    daily_load:\n      name: daily_load\n      tasks:\n        - task_key: main\n          spark_python_task:\n            python_file: ../src/load.py\n\n# $ databricks bundle deploy -t dev\n# $ databricks bundle run daily_load -t dev\n#\n# The CLI syncs files through the Workspace API, the provider creates the job,\n# and `bundle run` executes it on real Spark in a sibling container —\n# returning the job's actual output.\n# Is it up, and which services answered?\ncurl http://localhost:8000/_minilake/health\n\n# Create a catalog through the Databricks API surface\ncurl -X POST http://localhost:8000/api/2.1/unity-catalog/catalogs \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"name\": \"sales\"}'\n\n# Run SQL and get real rows back\ncurl -X POST http://localhost:8000/api/2.0/sql/statements \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"warehouse_id\": \"dev\", \"statement\": \"SELECT 42 AS answer\"}'\n\n# Give the next test a clean workspace — no restart needed\ncurl -X POST http://localhost:8000/_minilake/reset\n```\n\nA FastAPI application serves the Databricks REST surface, the web UI and the MCP endpoint together. Behind it sit DuckDB, real Delta files on disk, and Docker for Spark.\n\nA Databricks workspace is an awkward thing for an agent to drive: every useful action is four calls deep, half the SQL it writes is the wrong dialect, and failures are silent. MiniLake's MCP server closes those gaps — and it costs nothing when the agent gets it wrong.\n\nPlus 8 resources — including a SQL-dialect guide that stops agents writing Spark SQL at DuckDB — and 4 prompts that front-load the rules. Tools run through MiniLake's own ASGI stack, so an agent's error is exactly the error an SDK client would have seen.\n\n```\n# One line to register it with Claude Code\nclaude mcp add --transport http minilake http://localhost:8000/mcp\n\n# Then just ask:\n#   \"Use minilake: create a sales catalog with a seeded\n#    orders table, and tell me the total.\"\n\n# The agent does this — three calls, no glue code:\n#   setup_fixture({\"catalog\": \"sales\", \"schema\": \"store\"})\n#   seed_table({\"full_name\": \"sales.store.orders\", ...})\n#   run_sql({\"statement\": \"SELECT SUM(amount) FROM sales.store.orders\"})\n\n# Enable it on the server side:\nMINILAKE_MCP=1 minilake --port 8000\n```\n\nEvery row below is either really executed or clearly labelled. Anything out of scope answers `501 NOT_IMPLEMENTED` rather than faking a success your production code will not get.\n\n| Service | Status | What that means | \n|---|---|---|\n| Unity Catalog | Real | Catalogs, schemas, tables, volumes, functions — each catalog is its own DuckDB database, with native three-part addressing | \n| UC grants | Unenforced CRUD | With real inheritance — a grant on a catalog is effective on its schemas and tables — but no privilege is ever checked | \n| UC functions | Real | A SQL function becomes a real DuckDB macro, so `SELECT cat.sch.fn(x)` actually computes — not a registry entry | \n| SQL statements | Real | Executed on DuckDB; all three formats and both dispositions, with a typed result manifest | \n| EXTERNAL Delta tables | Real | Genuine Delta files; writes go through a generated Spark job, reads through `delta_scan()` | \n| Jobs | Real | `spark-submit` in sibling containers (or a subprocess fallback), real DAG scheduling, real logs, one-shot`runs/submit` | \n| SQL warehouses | Real | Full CRUD and lifecycle; statements run regardless of state, as they do locally | \n| Query history & saved queries | Real | Server-backed, failures included — which is usually why you opened history | \n| Alerts | Real | The condition is evaluated against real query results, so an alert fires on your data — never on a stored constant | \n| Workspace, DBFS & Files | Real | File-backed storage with chunked upload; raw-bytes sync is what makes `bundle deploy` work | \n| Secrets | Real | Real CRUD; values are resolvable only inside job environment variables, never through the API | \n| Web UI & notebooks | Real | Athena-style SQL workspace at `/ui` , with JupyterLab proxied at`/jupyter` | \n| MCP server | Real | Opt-in with `MINILAKE_MCP=1` : 67 tools, 8 resources, 4 prompts on the same port | \n| Persistence | Real | `MINILAKE_PERSIST=1` snapshots state as JSON on shutdown and restores it on startup | \n| Clusters | State machine | CRUD plus timed lifecycle transitions — deliberately no Spark compute; that lives in Jobs | \n| Policies & instance pools | Unenforced CRUD | They resolve and validate, so a bundle naming one deploys — but there is no compute here for a policy to constrain | \n| Permissions | Allow-all CRUD | Real create/read/update, but nothing is ever enforced — this is a single-user tool | \n| Workspace admin | Unenforced CRUD | Git credentials, IP access lists, init scripts, notification destinations, instance profiles, workspace conf — what a `terraform apply` touches on its way past | \n| Identity & SCIM | Unenforced CRUD | Users, groups and service principals, with SCIM filtering and `PATCH` — records, not credentials; any token is still accepted | \n| Tokens | Unenforced CRUD | The value comes back exactly once, as it does for real — and then authenticates nothing | \n| Legacy SQL ( `preview/sql` ) | Real | Queries, alerts and dashboards on the deprecated routes — adapters over the modern stores, so both surfaces see one object | \n| DLT, Model Registry, Vector Search, Lakeview, Repos | Not implemented | Returns `501 NOT_IMPLEMENTED` — loudly, so you never mistake it for working | \n\nThese are deliberate design decisions, not a backlog. MiniLake targets one developer running it locally — not a shared, multi-tenant server.\n\nAny token is accepted and there is exactly one user. Never expose it beyond localhost.\n\nThe Permissions API is real CRUD that always allows everything — a passing test here says nothing about grants in a real workspace.\n\nStatements run on DuckDB's dialect. The MCP server ships a dialect resource for exactly this reason; the docs carry the translation table.\n\nThe cluster lifecycle is a state machine. Real Spark happens through Jobs, in sibling containers.\n\nDuckDB is single-writer, so concurrent load serializes on a lock. Fine for dev and CI; not a deployment target.\n\nThey are DuckDB tables with no files. Use EXTERNAL Delta when Spark has to see the data — that trade-off is what keeps SQL fast.\n\nBeing explicit about the edges is the point: an emulator that quietly fakes what it cannot do is worse than no emulator at all.\n\nFree and MIT-licensed. No account, no API key, no cloud bill — and nothing to uninstall from your cloud account when you're done.", "url": "https://wpnews.pro/news/minilake-is-a-local-databricks-api-emulator-with-sql-delta-lake-and-spark", "canonical_source": "https://dmux.github.io/minilake/", "published_at": "2026-10-27 14:12:59+00:00", "updated_at": "2026-10-29 09:18:25.813064+00:00", "lang": "en", "topics": ["ai-tools", "ai-agents", "developer-tools", "ai-infrastructure", "agent-protocols"], "entities": ["MiniLake", "Databricks", "DuckDB", "Delta Lake", "Apache Spark", "dmux", "Terraform", "JupyterLab"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/minilake-is-a-local-databricks-api-emulator-with-sql-delta-lake-and-spark", "markdown": "https://wpnews.pro/news/minilake-is-a-local-databricks-api-emulator-with-sql-delta-lake-and-spark.md", "text": "https://wpnews.pro/news/minilake-is-a-local-databricks-api-emulator-with-sql-delta-lake-and-spark.txt", "jsonld": "https://wpnews.pro/news/minilake-is-a-local-databricks-api-emulator-with-sql-delta-lake-and-spark.jsonld"}}