{"slug": "deterministic-grounding-checks-for-llm-agents-no-llm-in-the-hot-path", "title": "Deterministic grounding checks for LLM agents, no LLM in the hot path", "summary": "The open-source Python package ora-grounding, released under the MIT license and installable via `pip install ora-grounding`, adds deterministic post-response grounding checks plus cross-family adversarial review for LLM chat agents with zero dependencies and no LLM calls in the grounding check itself. The tool, extracted from a production AI-CTO assistant, uses regex and set-membership to flag fabricated file, symbol, line-number, and command claims — in its demo it caught an invented `redis_lock.py` while confirming the real `payments_client.py` — and pairs that with a reviewer LLM from a different model family to catch overconfident synthesis. The package requires Python 3.10+ and targets hallucination failure modes that prompting alone and sibling-model review do not fix.", "body_md": "// double-click race test v5 // QA regression test marker\n\nDeterministic post-response grounding checks + cross-family adversarial review for LLM chat agents — zero deps, bring your own LLM and your own database.\n\n[Quick start](#-30-second-demo) · [Why](#-why-this-exists) · [Docs](#-usage) · [Compare](#-vs-the-alternatives) · [Roadmap](#-roadmap)\n\nThis is an **anonymized real production case** — a chat agent claimed a\nfile existed that didn't. Here's `ora-grounding` catching it, deterministically, with zero LLM calls in the check itself:\n\n``` python\n>>> from ora_grounding.grounding import extract_claims, classify_claims\n>>>\n>>> reply = \"Fixed the retry logic in payments_client.py — added dedup via redis_lock.py\"\n>>>\n>>> canonical = {\n...     \"paths\": {\"src/payments_client.py\"},   # redis_lock.py does NOT exist\n...     \"basenames\": {\"payments_client.py\"},\n...     \"defs\": set(),\n... }\n>>>\n>>> classify_claims(extract_claims(reply), canonical=canonical)\n{'fabricated': ['redis_lock.py'], 'unverified': []}\n```\n\n**One real file. One invented file. Caught instantly.** That's the whole pitch — everything below is detail.\n\nLLMs hallucinate *confidently*. Two failure modes hurt users the most:\n\n| Failure mode | What it looks like | \n|---|---|\n| **Made-up specifics** | \"Fix at `services/auth.py:42` \" — the file doesn't exist. | \n| **Overconfident synthesis** | The model stitches together plausible claims nothing in its context supports. | \n\nPrompting alone doesn't fix this. **Sibling-model review doesn't fix it either** — GPT reviewing GPT shares blind spots. `ora-grounding` adds two deterministic defences that sit *outside* the model:\n\n- 🧮 **Cheap grounding check** — regex + set-membership,**no LLM in the hot path** . Catches file/symbol/line-number/command claims the retrieval context never supported.\n- 🥊 **Adversarial review** — a*different-family* reviewer LLM hostile-reads the draft, with a hard deterministic guard against the reviewer itself hallucinating flags.\n\nExtracted from a production AI-CTO assistant serving real users. Battle-tested against actual regressions — including the one above.\n\n```\npip install ora-grounding\n```\n\nZero dependencies. Python 3.10+.\n\n``` python\nfrom ora_grounding.grounding import extract_claims, classify_claims\n\n# 1. Your agent generates a reply\nreply = \"Fixed auth in backend/routers/auth.py line 42\"\n\n# 2. Build the canonical set from your retrieval context\ncanonical = {\n    \"paths\": {\"backend/routers/auth.py\"},\n    \"basenames\": {\"auth.py\"},\n    \"defs\": {\"verify_token\", \"login\"},\n}\n\n# 3. Check\nclaims = extract_claims(reply)\nresult = classify_claims(claims, canonical=canonical)\n\nif result[\"fabricated\"]:\n    print(f\"⚠️  Fabricated: {result['fabricated']}\")\npython\nfrom ora_grounding.review import adversarial_review\n\n# After the grounding check passes, run a cross-family review\nreview_result = adversarial_review(\n    draft_reply=reply,\n    retrieval_context=your_rag_chunks,\n    reviewer_llm=your_llm_client,  # Different family from the drafter\n)\n\nif review_result[\"flags\"]:\n    print(f\"🚩 Review flags: {review_result['flags']}\")\n```\n\n| Approach | Speed | Catches fabricated paths | Catches overconfident synthesis | Cross-family | \n|---|---|---|---|---|\n| **Prompting alone** | Fast | ❌ | ❌ | N/A | \n| **Sibling-model review** | Slow |  |  | ❌ | \n| **ora-grounding** | Fast (grounding) + Slow (review) | ✅ | ✅ | ✅ | \n\n- **Prompting alone** — \"Be accurate. Don't hallucinate.\" — doesn't work. The model doesn't*know* it's hallucinating.\n- **Sibling-model review** — GPT-4 reviewing GPT-4 shares blind spots. Same training data, same failure modes.\n- **ora-grounding** — Deterministic check (fast) + adversarial review (slow, opt-in) with a different-family reviewer.\n\n- Deterministic grounding check\n- Adversarial review with cross-family LLM\n- Structured output validation (Pydantic models)\n- Multi-turn conversation grounding\n- Benchmark suite (public dataset)\n\nMIT — see [LICENSE](https://github.com/polarisbuiltinc-wq/ora-grounding/blob/main/LICENSE).\n\nExtracted from [AUREM](https://aurem.com) — an AI-CTO assistant that reads your GitHub repo and ships code. Built by the AUREM team.\n\n**Questions?** Open an issue or reach out at [support@aurem.com](mailto:support@aurem.com).", "url": "https://wpnews.pro/news/deterministic-grounding-checks-for-llm-agents-no-llm-in-the-hot-path", "canonical_source": "https://github.com/polarisbuiltinc-wq/ora-grounding", "published_at": "2026-09-21 02:04:21+00:00", "updated_at": "2026-09-21 02:23:23.449120+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-tools", "large-language-models", "developer-tools"], "entities": ["ora-grounding", "GPT-4", "Python", "Pydantic", "redis_lock.py", "payments_client.py"], "alternates": {"html": "https://wpnews.pro/news/deterministic-grounding-checks-for-llm-agents-no-llm-in-the-hot-path", "markdown": "https://wpnews.pro/news/deterministic-grounding-checks-for-llm-agents-no-llm-in-the-hot-path.md", "text": "https://wpnews.pro/news/deterministic-grounding-checks-for-llm-agents-no-llm-in-the-hot-path.txt", "jsonld": "https://wpnews.pro/news/deterministic-grounding-checks-for-llm-agents-no-llm-in-the-hot-path.jsonld"}}