{"slug": "your-rag-copilot-can-t-count-stop-letting-it-try", "title": "Your RAG copilot can't count — stop letting it try", "summary": "A developer discovered that their RAG-based document-search copilot, built on LangGraph and Bedrock, gave incorrect counts for aggregation questions because the LLM counted the surfaced results rather than the true database total. The fix preserves the authoritative count in a separate state field so the model never computes aggregates. The developer argues that LLMs should never compute aggregates and that state fields with mutating meanings are a bug factory.", "body_md": "A user asked our document-search copilot a completely reasonable question: how many documents a specific person authored. The copilot answered with a number. Confidently. With a nice sentence around it.\n\nThe real answer, computed by the database, was **84**. The copilot said — because it did what every RAG system does when you ask it an aggregation question: it counted what it could see.\n\nHere's the claim I want to defend, because I think a lot of us are shipping this bug right now:\n\n💡 An LLM should never compute an aggregate. Not counts, not sums, not \"most recent\". If the question is arithmetic over the corpus, the answer comes from the system of record — the model's only job is to phrase it.\n\nThis is a multi-agent document-search copilot for a quality-management product, built on LangGraph and Bedrock. When a metadata question comes in (\"documents authored by X\"), a search agent asks the documents service for matching rows, reranks and permission-gates them, collapses multiple versions of the same document into one card, and surfaces at most 30 results.\n\nEvery one of those steps is correct for *search*. Every one of them destroys the *count*:\n\n| Stage | What it does | What \"how many\" means here |\n|---|---|---|\n| Documents service (SQL) | runs the filter against the database | true total: 84\n|\n| Retrieval | returns the top-k rows | capped |\n| View-permission gate | drops rows this user can't see | fewer |\n| Version collapse | 5 versions of one doc → 1 card | fewer still |\n| Surfaced set | at most 30 cards | ≤ 30, always |\n\nThe model sits at the bottom of that funnel. When it \"counts\", it counts the last row of the table. The user asked about the first row. Everything in between is retrieval hygiene that silently turned a database question into a context-window question.\n\nAnd this is the part that makes it worse than a crash: **the answer is plausible**. A count that's capped at 30 and deduped looks like a real number. Nobody squints at \"you have 27 documents\" the way they'd squint at a stack trace. In a quality-management context, someone might put that number in a report.\n\nThis is the bit that stung. The documents service *returns* the true total with every filtered query. The search node even stored it in agent state, in a field called `total_available_results`\n\n.\n\nThen, a few nodes downstream, the permission-gate node overwrote that same field with the surfaced count — deliberately, and for a good reason:\n\n```\nreturn {\n    \"sources\": diversified,\n    # Honest \"documents found\": distinct logical documents after the view gate and\n    # version-collapse, so the headline count matches the cards (5 versions → 4 docs),\n    # not the service's version-row total.\n    \"total_available_results\": len(diversified_all),\n    ...\n}\n```\n\nThat comment is right! If the UI says \"found 12\" and shows 12 cards, the numbers should agree. The bug wasn't either write in isolation — it was one state field carrying **two different meanings** at two different points in the pipeline. Upstream it meant \"matches in the database\"; downstream it meant \"cards on screen\". Whoever read it last got whichever meaning was freshest.\n\n💡 A state field whose meaning mutates mid-pipeline is a bug factory. If a value is authoritative, give it its own field and never let anything downstream touch it.\n\nSo that's exactly what I did — preserved the service's true total in its own state field, threaded it to the two places that answer the user, and left the surfaced count alone for the card list and pager.\n\nFirst, the state gets an untouchable field:\n\n```\nclass DocumentsSearchState(OrchestratorState, total=False):\n    ...\n    total_available_results: int   # surfaced: view-gated, version-collapsed, capped at 30\n\n    # The documents service's true match total for the filter, before the 30-cap and\n    # version-collapse. Authoritative for a \"how many\" count; 0 on a content-only\n    # search, which has no exact total.\n    total_matching_records: int\n```\n\nThen the outcome node — the one that turns search state into both the user-facing status line and the message the agent reasons over — uses the authoritative number in both:\n\n```\ntotal_matching = state.get(\"total_matching_records\") or 0\nheadline_total = total_matching if total_matching > total else total\n\nfound_msg = (\n    f\"Showing {len(sources)} of {headline_total} matching documents\"\n    if headline_total > len(sources)\n    else f\"Found {len(sources)} matching documents\"\n)\n...\ncount_note = (\n    f\" {total_matching} documents match in total \"\n    \"(authoritative count; use this for a count question).\"\n    if total_matching > len(sources)\n    else \"\"\n)\nreturn f\"Returned {len(sources)} document cards to the user.{count_note} ...\"\n```\n\nAnd the agent's prompt closes the loop with an explicit instruction:\n\n```\nFor a \"how many\" question, answer with the authoritative total the tool\nreports (\"N documents match in total\"), never the number of cards shown,\nwhich is capped.\n```\n\nNotice the belt-and-braces structure, because it matters: the number goes into the **agent's context** (so the model can phrase the answer around it) *and* into the **user-facing headline** (\"Showing 5 of 84 matching documents\"). Even on a turn where the model ignores the note and free-styles a count, the UI is showing the true total right next to it. The prompt asks; the headline doesn't have to.\n\nThree small tests pin the behavior: the count note appears when there are more matches than cards, disappears when everything is shown, and never appears for a content-only semantic search — because a similarity ranking has no crisp \"matches\" set, so pretending it has an exact total would be the same lie in a new costume.\n\nSame release, same root cause wearing a different hat. Users asked for \"documents created in the last 10 days\" and the copilot had to turn that into a date filter — without knowing what day it is. A model doesn't know today's date. It will happily guess one from its training data, which is exactly as wrong as counting its context window.\n\nThe fix is a tiny LangGraph middleware that injects the current date into the system message on **every model call**:\n\n``` php\ndef current_date_context(now: datetime | None = None) -> str:\n    day = (now or datetime.now(UTC)).date()\n    return (\n        f\"Today's date is {day.isoformat()} (UTC). Resolve any relative date \"\n        '(\"today\", \"yesterday\", \"the last 10 days\", \"since March\") '\n        \"to concrete ISO-8601 dates relative to it before building a date filter.\"\n    )\n\nclass CurrentDateMiddleware(AgentMiddleware):\n    async def awrap_model_call(self, request, handler):\n        base = request.system_message\n        line = current_date_context()\n        merged = line if base is None else f\"{base.content}\\n\\n{line}\"\n        return await handler(request.override(system_message=SystemMessage(content=merged)))\n```\n\nOne non-obvious detail: it's injected per call and never appended to the persisted conversation. Write \"today is 2026-07-30\" into the thread history and a conversation resumed next week carries a stale date the model will trust completely.\n\nBoth fixes are the same principle: **the model knows nothing you don't hand it, and it will answer anyway.** Counts, dates, totals — anything that's a fact about *your* system rather than about language has to arrive as context, computed by something that can't hallucinate.\n\nHonest limits of what shipped:\n\nAnd the design choice I keep turning over: I handed the model the number **inline, as prose in a tool result**. The agentic-purist alternative is a dedicated `count_documents`\n\ntool the model calls when it detects an aggregation question — cleaner separation, one more round trip, and a new failure mode where the model doesn't call it. I went with inline because the search already paid for the count; a tool call felt like latency for ideology.\n\nSo: where do you draw that line — hand the model the number, or hand it a tool to fetch the number? If your copilot answers \"how many\" questions today, I'd genuinely like to know which side you picked and whether it's held up.\n\nThanks for making it to the end 🙏 If your assistant has ever announced a total that made you go \"wait, that can't be right\", I'd love to swap notes on how you fixed it — come find me on [LinkedIn](https://www.linkedin.com/in/rodrigo-diego-67867185/).", "url": "https://wpnews.pro/news/your-rag-copilot-can-t-count-stop-letting-it-try", "canonical_source": "https://dev.to/rdiegoss/your-rag-copilot-cant-count-stop-letting-it-try-2ie3", "published_at": "2026-07-31 11:00:29+00:00", "updated_at": "2026-07-31 11:35:48.863137+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-products", "developer-tools"], "entities": ["LangGraph", "Bedrock"], "alternates": {"html": "https://wpnews.pro/news/your-rag-copilot-can-t-count-stop-letting-it-try", "markdown": "https://wpnews.pro/news/your-rag-copilot-can-t-count-stop-letting-it-try.md", "text": "https://wpnews.pro/news/your-rag-copilot-can-t-count-stop-letting-it-try.txt", "jsonld": "https://wpnews.pro/news/your-rag-copilot-can-t-count-stop-letting-it-try.jsonld"}}