{"slug": "shadow-gate-your-llm-generated-sql-a-replay-test-against-a-frozen-fixture", "title": "Shadow-Gate Your LLM-Generated SQL: A Replay Test Against a Frozen Fixture Database", "summary": "A developer introduced a shadow gate CI job that replays analyst questions through an LLM SQL generation pipeline and diffs results against golden snapshots on a frozen fixture database to catch semantic drift. The approach uses DuckDB and a provider-agnostic generate() function, with MonkeyCode offering free model access for CI runs. The developer emphasized adversarial fixtures and encoding past bugs as golden tests.", "body_md": "In a previous post I built a zero-budget eval harness to score LLM-generated SQL *before* adopting a prompt. That harness answers one question: \"is this generation setup any good?\" This post answers a different, more operationally painful one: \"the generation setup was good last month — is it still good today, after we tweaked the prompt, swapped the model, or the provider shipped a silent update?\"\n\nThe failure mode I care about here is **semantic drift**: the SQL still parses, still runs, still returns rows — but the rows are subtly wrong. A `LEFT JOIN`\n\nquietly becomes an `INNER JOIN`\n\n. A timezone boundary shifts. `NULL`\n\nhandling changes. Nothing throws, so your linter and your unit tests on the application code stay green while a dashboard silently lies.\n\nThe fix I'll walk through is a **shadow gate**: a small CI job that replays a fixed suite of analyst questions through your current LLM generation path, executes the resulting SQL against a *frozen* fixture database, and diffs the result sets against committed golden snapshots. No production data, no warehouse access, no credentials in CI.\n\nThe artifact below is a working skeleton you can adapt.\n\n``` python\n# build_fixture.py — run once per fixture change, via PR only\nimport duckdb\n\ncon = duckdb.connect(\"fixture.duckdb\")\ncon.execute(\"\"\"\n    CREATE TABLE orders AS\n    SELECT * FROM read_csv_auto('fixtures/orders.csv', header=true);\n\"\"\")\ncon.execute(\"\"\"\n    CREATE TABLE customers AS\n    SELECT * FROM read_csv_auto('fixtures/customers.csv', header=true);\n\"\"\")\ncon.close()\n```\n\nKeep the CSVs small (hundreds of rows) but *adversarial*: include duplicate customer names, `NULL`\n\nregion values, orders exactly on a date boundary, a currency code that appears once. Golden-file tests are only as good as their edge cases — this is where prior production bugs earn their keep.\n\n```\n# suite.yaml\n- id: q001\n  question: \"Total revenue per region for 2025, excluding cancelled orders\"\n  golden_sql: |\n    SELECT c.region, SUM(o.amount) AS revenue\n    FROM orders o JOIN customers c USING (customer_id)\n    WHERE o.status <> 'cancelled'\n      AND o.order_date >= DATE '2025-01-01'\n      AND o.order_date <  DATE '2026-01-01'\n    GROUP BY c.region\n    ORDER BY c.region;\n  order_matters: true\n\n- id: q002\n  question: \"Customers with no orders, including those with NULL region\"\n  golden_sql: |\n    SELECT c.customer_id, c.region\n    FROM customers c\n    LEFT JOIN orders o USING (customer_id)\n    WHERE o.order_id IS NULL\n    ORDER BY c.customer_id;\n  order_matters: true\n```\n\n`q002`\n\nis deliberately a trap: models love converting `LEFT JOIN ... IS NULL`\n\ninto `NOT IN`\n\n, which silently drops nothing here but changes semantics the moment `customer_id`\n\ncan be `NULL`\n\nin `orders`\n\n. Encode your scars.\n\nFor the generation step in CI you need a model endpoint that won't bill you per experiment and doesn't require you to stand up GPU infrastructure. I've been running this class of job through MonkeyCode, which offers free model access and a free server option — that combination maps well onto a CI job that fires a few dozen generation calls per PR and needs a throwaway runner. **Disclosure: This article was prepared as part of MonkeyCode's product outreach.** The gate logic below is provider-agnostic, though — swap the `generate()`\n\nbody for whatever endpoint you use, including a local model, and nothing else changes.\n\n``` python\n# shadow_gate.py\nimport duckdb, yaml, sys, json\n\nTOLERANCE = 1e-6\n\ndef generate(question: str, schema_ddl: str) -> str:\n    \"\"\"Fill your real prompt template; call your model endpoint here.\n    Must return raw SQL only (strip markdown fences).\"\"\"\n    ...  # provider-specific call\n\ndef run(con, sql):\n    try:\n        rel = con.execute(sql)\n        cols = [d[0] for d in rel.description]\n        return (cols, rel.fetchall(), None)\n    except Exception as e:\n        return (None, None, str(e))\n\ndef rows_equal(golden, candidate, tol=TOLERANCE):\n    if len(golden) != len(candidate):\n        return False\n    for g_row, c_row in zip(golden, candidate):\n        for g, c in zip(g_row, c_row):\n            if isinstance(g, float) or isinstance(c, float):\n                try:\n                    if abs(float(g) - float(c)) > tol:\n                        return False\n                except (TypeError, ValueError):\n                    return False\n            elif g != c:\n                return False\n    return True\n\ndef main():\n    con = duckdb.connect(\"fixture.duckdb\", read_only=True)\n    schema_ddl = open(\"schema.sql\").read()\n    suite = yaml.safe_load(open(\"suite.yaml\"))\n    failures = []\n\n    for case in suite:\n        gen_sql = generate(case[\"question\"], schema_ddl)\n        g_cols, g_rows, g_err = run(con, case[\"golden_sql\"])\n        c_cols, c_rows, c_err = run(con, gen_sql)\n\n        if c_err:\n            failures.append({\"id\": case[\"id\"], \"kind\": \"execution_error\",\n                             \"detail\": c_err, \"sql\": gen_sql})\n            continue\n        if [c.lower() for c in g_cols] != [c.lower() for c in c_cols]:\n            failures.append({\"id\": case[\"id\"], \"kind\": \"column_mismatch\",\n                             \"detail\": f\"{g_cols} vs {c_cols}\", \"sql\": gen_sql})\n            continue\n        g_sorted = g_rows if case.get(\"order_matters\") else sorted(map(str, g_rows))\n        c_sorted = c_rows if case.get(\"order_matters\") else sorted(map(str, c_rows))\n        if not rows_equal(g_sorted, c_sorted):\n            failures.append({\"id\": case[\"id\"], \"kind\": \"semantic_drift\",\n                             \"detail\": \"result sets differ\", \"sql\": gen_sql})\n\n    print(json.dumps({\"total\": len(suite), \"failures\": failures}, indent=2, default=str))\n    sys.exit(1 if failures else 0)\n\nif __name__ == \"__main__\":\n    main()\n```\n\nThree design decisions worth defending:\n\n`read_only=True`\n\non the connection.`DELETE`\n\nagainst your fixture shouldn't be survivable. (For a real warehouse you'd also want a statement timeout and a deny-list on DDL/DML keywords before execution.)| Trigger | Why |\n|---|---|\n| PR touching the prompt template or schema context | The obvious case; drift source is explicit |\n| Model version change in config | Provider-side behavior changes are the classic silent-drift source |\n| Nightly scheduled run | Catches provider updates that happen without any change on your side |\n| Every application code PR | Overkill unless generation is on the hot path; cost and latency rarely justify it |\n\nGate policy matters more than gate mechanics. My recommendation: `semantic_drift`\n\nfailures block merge, `execution_error`\n\nfailures block merge only if they didn't occur on the previous nightly baseline (a model that occasionally emits unparseable SQL may be tolerable in a human-in-the-loop workflow, but a *regression* in parse rate is not).\n\nThe uncomfortable truth about LLM-generated SQL is that \"it still runs\" is a nearly meaningless health signal. A replay gate against a frozen fixture is the cheapest way I know to convert silent semantic drift into a loud, reviewable diff. If you want to try the pattern without provisioning anything, MonkeyCode's free model access and free server tier is a low-friction place to host the generation step while you find out whether your suite catches anything — mine caught a `JOIN`\n\n-flavor regression within the first week, which paid for the setup effort on the spot.", "url": "https://wpnews.pro/news/shadow-gate-your-llm-generated-sql-a-replay-test-against-a-frozen-fixture", "canonical_source": "https://dev.to/dataio_4921/shadow-gate-your-llm-generated-sql-a-replay-test-against-a-frozen-fixture-database-2cce", "published_at": "2026-08-10 08:11:39+00:00", "updated_at": "2026-08-10 08:16:02.945511+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-tools", "mlops"], "entities": ["DuckDB", "MonkeyCode"], "alternates": {"html": "https://wpnews.pro/news/shadow-gate-your-llm-generated-sql-a-replay-test-against-a-frozen-fixture", "markdown": "https://wpnews.pro/news/shadow-gate-your-llm-generated-sql-a-replay-test-against-a-frozen-fixture.md", "text": "https://wpnews.pro/news/shadow-gate-your-llm-generated-sql-a-replay-test-against-a-frozen-fixture.txt", "jsonld": "https://wpnews.pro/news/shadow-gate-your-llm-generated-sql-a-replay-test-against-a-frozen-fixture.jsonld"}}