cd /news/ai-tools/your-ai-cited-a-real-file-it-still-l… Β· home β€Ί topics β€Ί ai-tools β€Ί article
[ARTICLE Β· art-137279] src=dev.to β†— pub= topic=ai-tools verified=true sentiment=↑ positive

Your AI cited a real file. It still lied to you.

A developer built Enhanciar, a codebase Q&A tool that forces every generated answer to cite wiki-style page handles resolving to specific repo files and lines, then re-reads each cited source to grade claims as supported, partial, unsupported, or unverifiable. The system uses a deterministic tree-sitter AST index to adjudicate structural claims before any LLM call, reserving model-based verification for prose claims that must return a quoted span and line number. The stated goal is to catch the common failure mode where a model cites a real file but paraphrases something the file does not say.

by read6 min views2 publishedSep 22, 2026

TL;DR β€” A citation proves a document was retrieved. It does not prove the sentence next to it came from there. We built Enhanciar to (1) force every answer to cite, (2) resolve each citation to a repo file and line, and (3) re-read the cited source and grade each claim supported / partial / unsupported / unverifiable. The interesting part is not the LLM β€” it's the free, deterministic checker that runs before the LLM, and the rule that the verifier must never say "supported" just because it couldn't run.

Every "chat with your codebase" tool now prints little [1] [2] markers under its answers. Ask one of them "which service handles refunds for cancelled orders?" and you'll get a confident paragraph, a source link, and a warm feeling.

Then you click the link and the file doesn't mention refunds.

This is the failure mode we kept hitting when we tried to use these tools on real engineering teams: the model cites a real page and then paraphrases something the page does not say. Retrieval worked. The citation is "valid". The answer is still wrong, and the citation makes it more convincing, not less.

We wanted a tool an engineer could trust on their first day at a company β€” where "trust" means "I can click through and see the line that backs this sentence, or the tool tells me it couldn't find one."

Here's how we built it. It's three layers, and the order matters.

We don't index raw files for Q&A. At ingest time we build a wiki: one Markdown page per meaningful unit (a service, a route, a module, a decision), each with frontmatter that records exactly which repo files the page was built from. Something like:

---
slug: entities/cart-service
sources:
  - src/services/cart.ts
  - src/routes/cart.ts
---
Handles cart mutation and checkout hand-off. Exposes `POST /cart/items` ...

## Dependencies
- [[entities/pricing-service]]

Answers are generated over these pages, and the model is required to cite them with wiki-style handles:

Refunds for cancelled orders are handled by the payments worker
[[entities/payments-worker]], which is triggered from the order
state machine [[flows/order-cancellation]].

We parse those handles out of the finished answer with a boring regex (we tolerate a Β§ heading suffix and a trailing .md because models add them anyway, dedupe, and cap at 16). Every handle resolves to a page, and every page resolves to file(s) in the repo. So one hop from [[entities/payments-worker]] gets you to src/workers/payments.ts, and the AST index we build at ingest gets you to a line.

That's the "file:line" part. It's the easy part, and it's where most tools stop.

Before an answer ever cites a page, the page has to survive a per-claim verification pass. We break each generated page into individual claims and adjudicate each one, cheapest oracle first.

A surprising share of claims are structural:

generateMetadata"[[get-meta-details-service]]" GET /aboutUs" We already extracted all of that with tree-sitter at ingest time. So these claims are settled by lookup β€” no model call, no API key, no tokens. If the AST says generateMetadata is defined at app/layout.tsx:42, the claim is supported with evidence {file, line}. If the symbol doesn't exist anywhere, it's unsupported.

Anything tier 1 can judge, tier 1 judges. Tier 2 never sees it. That's the whole cost story.

Prose claims β€” "this service handles refunds for cancelled orders" β€” have no structural handle. A model has to re-read the cited source and say whether it supports the sentence. It must return a quoted span and the line it was found at, or it doesn't get to say supported.

Every claim ends up with exactly one of:

Verdict Meaning
supported evidence found for the specific assertion
partial something related found, but not what was asserted
unsupported the cited source was checked and does not say this
unverifiable nothing could check it β€” no citation, source gone, checker unavailable

unverifiable is load-bearing and is never a synonym for "fine". Early on, tier 1 had a bug where it reported every React hook in a repo as unsupported. The lesson stuck: if the oracle has no data, or the critic has no key, the claim stays unverifiable and the UI says what was not checked. Silently marking things supported because the checker broke would be the worst possible failure of the feature, so every "cannot judge" path returns unverifiable rather than a guess.

There was a real design fork here: drop unverified claims, or tag them? Tagging won. A wrong verifier that silently destroys real information is far worse than a wrong verifier that mislabels it β€” a tag is inspectable and reversible. So a page can carry a sentence flagged unsupported and you can see it, click through, and decide.

Layer 2 settles "is this page true about the code". It does not settle the failure I opened with: the model cites [[cart-service]] while paraphrasing something that page doesn't say.

So we run the same checker a second time, with one substitution: for a wiki page the source is the repo file; for an answer the source is the wiki pages it cites. Same verdict vocabulary, same evidence discipline (quoted span + page + line), same extractor, same free oracle, same paid critic. It is deliberately not a second verification system.

Four properties, in the order they matter:

unverifiable β‰  unsupported. supported because the checker broke. Ask through the MCP server (Claude Code, Cursor, Claude Desktop all work). Rendered here as plain text β€” the UI shows the same verdicts as badges:

> which service handles refunds for cancelled orders?

Refunds for cancelled orders are issued by the payments worker
[[entities/payments-worker]] βœ… src/workers/payments.ts:118
when the order state machine emits `order.cancelled`
[[flows/order-cancellation]] βœ… src/orders/machine.ts:64.
Partial refunds are not supported ⚠️ unverifiable β€” no cited page discusses partial refunds.

Two sentences are backed by a file and a line you can open. The third is honestly labelled: the model asserted it, nothing in the cited pages says it, so you don't get a green tick. That third line is the feature.

Tier 1 is free forever. Tier 2 and answer-verification spend the user's own key (BYOK), once per page at ingest and once per answer if the flag is on β€” it's off by default precisely because it costs something, and the token counts come back with the verdicts so you can see what you paid for.

If you're building anything RAG-shaped for engineers: the citation is the start of trust, not the end. Re-read the source. Grade the claim. Say "I couldn't check this" out loud.

We're building this as Enhanciar β€” a company brain for engineering teams that answers with receipts. It's waitlist-only right now; if you want to try it on your repo, the link is on the site. Happy to answer questions about the verifier design in the comments.

── more in #ai-tools 4 stories Β· sorted by recency
── more on @enhanciar 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain β€” perfect for shipping the agent you just read about.

$git push zahid main
β†’ Live at https://your-agent.zahid.host βœ“
Get free account β†’ Pricing
from €0/mo Β· no card required
LIVE [news/your-ai-cited-a-real…] indexed:0 read:6min 2026-09-22 Β· β€”