{"slug": "synnodb-synthesizing-database-engines-for-your-workloads", "title": "SynnoDB – Synthesizing Database engines for your workloads", "summary": "SynnoDB, a drop-in replacement for DuckDB that transparently accelerates SQL with auto-generated bespoke C++ engines, is now available on PyPI. The project packages the Bespoke-OLAP research into a production-ready router that falls back to DuckDB for everything else and cross-checks correctness. SynnoDB uses an LLM agent to synthesize workload-specific query engines and supports stages like storage planning, base implementation, optimization, multi-threading, and correctness checking.", "body_md": "**A drop-in replacement for DuckDB that transparently accelerates your SQL with auto-generated\nbespoke C++ engines** - falling back to DuckDB for everything else, cross-checked for correctness.\n\n[🌐 Website](https://synnodb.com) ·\n[📄 Paper](https://arxiv.org/pdf/2603.02001) ·\n[📦 PyPI](https://pypi.org/project/synnodb/) ·\n[📓 Demo Notebook](/SynnoDB/SynnoDB/blob/main/tutorials/gen_tpch_demo.ipynb)\n\n*Required Notice: Copyright 2026 SynnoDB*\n\nSynnoDB grew out of the research project [ Bespoke-OLAP](https://github.com/DataManagementLab/BespokeOLAP)\n(\n\n[paper](https://arxiv.org/pdf/2603.02001)): an LLM agent that synthesizes workload-specific, one-size-fits-one C++ query engines. SynnoDB packages that idea as a production-ready DuckDB drop-in.\n\nInstall from ** PyPI**:\n\n```\npip install synnodb              # the demo DuckDB drop-in router\npip install \"synnodb[factory]\"   # + the Bespoke-Agent factory that generates engines\n```\n\nNew here? [ tutorials/gen_tpch_demo.ipynb](/SynnoDB/SynnoDB/blob/main/tutorials/gen_tpch_demo.ipynb) runs the whole loop end\nto end - generate TPC-H data, build an engine, and drop it in against DuckDB. See\n\n[Installation](#installation)for the system libraries the generated engines compile against.\n\n`SYNNO_DATA_DIR`\n\nmust point at the data root (parquet, caches, logs); set it in\nthe environment or `.env`\n\n.\n\n``` python\nfrom synnodb import SynnoDB\n\ndb = SynnoDB.in_memory(workload=\"tpch\", model=\"anthropic/claude-sonnet-4-6\")\n\nplan = db.createStoragePlan(queries=\"1\")     # -> StoragePlan\nprint(plan.text)                             # the storage_plan.txt document\nprint(plan.path, plan.run_id)                # on disk + wandb provenance\n\nimpl = db.createBaseImpl(storage_plan=plan.text)  # pass the plan content (W&B-free)\nprint(impl.files[\"db_loader.cpp\"])           # -> BaseImplementation (generated C++)\n\nopt = db.runOptimLoop(base_impl=impl)        # -> OptimizedImplementation\n```\n\nEach stage returns a domain object (`StoragePlan`\n\n, `BaseImplementation`\n\n,\n`OptimizedImplementation`\n\n, `MultiThreadedImplementation`\n\n, `CorrectnessReport`\n\n)\nthat carries the produced artifact and chains into the next stage.\n`SynnoDB(...)`\n\ntakes enums or strings (`db_storage=\"ssd\"`\n\n), alternative\nconstructors (`in_memory`\n\n/`on_ssd`\n\n/`for_tpch`\n\n/`for_ceb`\n\n/`from_env`\n\n), and\n`with_(...)`\n\nfor per-call overrides.\n\nEvery stage is a method on `SynnoDB`\n\n; there are no per-stage scripts. Each call\nruns one stage to completion and returns an artifact that chains into the next.\nStages chain in-process (pass the artifact) or across runs via the W&B run id\n(`*_wandb_id=`\n\n, requires `wandb_entity`\n\n/`wandb_project`\n\non the producing run):\n\n``` python\nfrom synnodb import SynnoDB\n\ndb = SynnoDB.on_ssd(\n    workload=\"tpch\", queries=\"1-22\", model=\"anthropic/claude-sonnet-4-6\",\n    notify=True, wandb_entity=\"my-entity\",   # presence of entity/project enables W&B\n)\n\nplan = db.createStoragePlan()                          # -> StoragePlan\nimpl = db.createBaseImpl(storage_plan_wandb_id=\"8xn0t04p\")   # or storage_plan=plan\nopt  = db.runOptimLoop(base_impl_wandb_id=\"q45vm9fz\")        # or base_impl=impl\nmt   = db.addMultiThreading(optimized_wandb_id=\"0br4bjqb\")   # or optimized=opt\nrep  = db.checkSfCorrectness(source_wandb_id=\"0br4bjqb\", target_sf=50)\n```\n\nThe run output dir defaults to a local `./output`\n\n; set `workspace=`\n\n(or\n`SYNNO_WORKSPACE`\n\n). Any `RunConfig`\n\nsetting the typed config does not model can be\nforced through the `extra_config={...}`\n\nescape hatch.\n\nThe built-in stages are ordinary `ConversationPlan`\n\ns; you can assemble and run\nyour own conversation from the same primitives via `run_synthesis`\n\n, the single\nentry point every stage goes through:\n\n``` python\nfrom synnodb import (\n    AssertCorrect, Benchmark, Compact, ConversationPlan, ConvContext,\n    PerQueryLoop, PrepareFeatures, PromptStage, SynnoDB,\n)\n\ndb = SynnoDB.in_memory(workload=\"tpch\", queries=\"1,4,6\")\n\ndef my_stages(ctx: ConvContext):\n    return [\n        AssertCorrect(),\n        PromptStage(\n            descriptor=\"inspect hot loops\",\n            get_prompt=lambda _exec_settings, _rt: (\n                f\"Profile {ctx.filenames.query_impl_path} and summarize the hot loops.\"),\n            measure_performance_after_stage=False,\n            auto_revert_on_regression=False,\n        ),\n        Compact(),\n        PerQueryLoop(lambda qid, ctx: [\n            PromptStage(\n                descriptor=f\"tune {qid}\",\n                # runtime and tracing data arrive exactly as in the built-in stages\n                get_prompt_with_tracing=lambda _exec_settings, rt, trace: (\n                    f\"Query {qid} currently runs in {rt:.0f} ms.\\n\"\n                    f\"Trace:\\n{trace}\\nOptimize it.\"),\n                max_turns=125,\n                # defaults: measure after stage, auto-revert on regression\n            ),\n        ]),\n        Benchmark(),\n    ]\n\nplan = ConversationPlan(\n    name=\"myTuningPass\",                    # run identity: naming, logging, caching\n    prepare=PrepareFeatures(tracing=True),  # what the workspace must provide\n    stages=my_stages,\n)\nresult = db.run_synthesis(plan, start=base_impl)  # start: artifact | snapshot hash | None\n```\n\n`prepare`\n\nstates*what the workspace must have*(scaffold, tracing instrumentation, MT helpers, ...) as independent feature flags; the features actually applied are recorded in a git-tracked`.synnodb_prepare.json`\n\ninside every snapshot, so chained runs know what they start from.`stages`\n\nreceives a`ConvContext`\n\n(queries, workspace filenames, run tool, lazy helpers like`ctx.reference_plans(source=\"umbra\")`\n\n) and returns a flat list of stage items.`PerQueryLoop`\n\nruns one conversation branch per query, feeding each stage the freshly measured runtime and trace data.- The returned artifact carries the final snapshot hash and the prepare record,\nso it chains into\n`db.checkSfCorrectness(result, target_sf=100)`\n\nor further custom plans with no extra ceremony.\n\nSynnoDB is published on ** PyPI**, so a single\n\n`pip`\n\ncommand\npulls in every Python dependency - no need to manage them yourself:\n\n```\npip install synnodb              # the DuckDB drop-in router / runtime\npip install \"synnodb[factory]\"   # + the LLM factory that generates engines\n```\n\nThat is everything needed to `import synnodb`\n\n. Two things live outside the wheel:\n\nThe generated engines are compiled C++ against Apache Arrow / Parquet, so a toolchain and the dev\nheaders must be present. [ cloc](https://github.com/AlDanial/cloc) is optional - the factory uses\nit to report generated-code size. On Debian/Ubuntu:\n\n```\nsudo apt install -y build-essential cloc                # C++ compiler (+ optional cloc)\n\n# Apache Arrow + Parquet development libraries\nwget https://packages.apache.org/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb\nsudo apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb\nsudo apt update\nsudo apt install -y libarrow-dev libparquet-dev parquet-tools\n```\n\n(Linux x86-64, Python 3.13+.)\n\nCreate a `.env`\n\nin your working directory with the model credentials (and optional run tracking):\n\n```\nANTHROPIC_API_KEY=...            # for the default anthropic/... models\n# OPENROUTER_API_KEY=...         # for openrouter/... models\n# LLM_API_BASE=http://host:PORT/v1   # a self-hosted, OpenAI-compatible endpoint\n# WANDB_ENTITY=...  WANDB_PROJECT=... # optional Weights & Biases run tracking\n```\n\nPoint `SYNNO_DATA_DIR`\n\nat the data root that holds the parquet, caches, and published engines.\nThe CLI and API require it - export it, put it in `.env`\n\n, or pass `data_dir=...`\n\nto `SynnoDB(...)`\n\n.\nThe [demo notebook](/SynnoDB/SynnoDB/blob/main/tutorials/gen_tpch_demo.ipynb) is self-contained: it defaults to a\nproject-local `.synno_data/`\n\nwhen the variable is unset and generates its own TPC-H parquet, so\nthere is nothing else to configure or download to run it.\n\nBuilding from a source checkout (for contributors) uses\n[ uv](https://github.com/astral-sh/uv) instead of\n\n`pip`\n\n- it manages the virtualenv and the\noptional-dependency extras:\n\n```\ncurl -LsSf https://astral.sh/uv/install.sh | sh    # install uv\ngit clone https://github.com/JWehrstein/SynnoDB.git\ncd SynnoDB\nuv sync --extra factory --extra dev                # editable install: factory + test deps\n```\n\nThe extras map to `pyproject.toml`\n\n: `factory`\n\n(the engine-generation stack plus the standalone\ndashboard's `wandb`\n\n), `dev`\n\n(pytest), `notebook`\n\n(Jupyter kernel + nbformat), and `benchmark`\n\n(the ClickHouse comparison). Install only what you need, e.g. `uv sync`\n\nfor the runtime alone or\n`uv sync --extra factory`\n\nto generate engines. Still install the [system libraries](#1-system-libraries)\nabove. Run the test suite with `.venv/bin/python -m pytest`\n\n.\n\n```\nwatch -n1 -d ./misc/get_db_procs.sh\n```\n\nTo share snapshots across machines, set up a bare git repository and start a git daemon:\n\n```\ngit init --bare synno_cache.git\ntouch synno_cache.git/git-daemon-export-ok\n\ngit daemon \\\n    --base-path=./ \\\n    --export-all \\\n    --enable=receive-pack \\\n    --reuseaddr \\\n    --verbose\n```\n\nThe cache URL is `git://<hostname>/synno_cache.git`\n\n. Pass it via the `.env`\n\nfile, or leave it unset to use only the local snapshot cache (with `--disable_repo_sync`\n\n).\n\nDelete snapshot:\n\n```\ngit -C /home/jwehrstein/bespoke_olap/output --git-dir=/home/jwehrstein/bespoke_olap/output/.git update-ref -d\n      refs/snapshots/snapshot-<hash>\n```\n\n", "url": "https://wpnews.pro/news/synnodb-synthesizing-database-engines-for-your-workloads", "canonical_source": "https://github.com/SynnoDB/SynnoDB", "published_at": "2026-07-22 01:14:15+00:00", "updated_at": "2026-07-22 01:22:06.995618+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-products", "ai-tools"], "entities": ["SynnoDB", "DuckDB", "Bespoke-OLAP", "PyPI", "Anthropic", "Claude Sonnet 4-6"], "alternates": {"html": "https://wpnews.pro/news/synnodb-synthesizing-database-engines-for-your-workloads", "markdown": "https://wpnews.pro/news/synnodb-synthesizing-database-engines-for-your-workloads.md", "text": "https://wpnews.pro/news/synnodb-synthesizing-database-engines-for-your-workloads.txt", "jsonld": "https://wpnews.pro/news/synnodb-synthesizing-database-engines-for-your-workloads.jsonld"}}