Hardening an AI coding agent: the failures, and the code that fixed them Univoco, a company building retrieval-augmented assistants, has detailed the process of hardening an AI coding agent that writes code for a proprietary document layout engine. The agent, which relies on private documentation and search tools rather than memorized public data, encountered a series of unexpected failures, including literal search behavior, repetitive searches, and invented APIs. The team implemented fixes such as filename matching, loop detection, query inflation, and self-auditing lookups to address these issues. At Univoco https://univoco.io we build retrieval-augmented assistants over a customer's own documentation. One of them is a coding agent that writes code for a proprietary document layout engine. The useful thing about that engine is that there is effectively nothing about it on the public internet. The model cannot have learned it. Everything the agent knows comes from a private documentation index and the tools we give it to search that index. That makes it an unusually clean laboratory. When memorisation is off the table, you get to watch the problem-solving process itself, and watch it fail. It failed a lot, and the failures were rarely the ones I expected. What follows is the list, in the order we hit them, with the code that fixed each one. | | Failure | Fix | |---|---|---| | 1 | Search tools are literal | filename matching, construct footprints | | 2 | It searches the same thing forever | five layers of loop detection | | 3 | One phrasing is a single point of failure | query inflation | | 4 | Goal words vs. index words | a deterministic phrasebook | | 5 | The retrieval tool invented APIs | self-auditing lookups | | 6 | Deciding an API is invented | the checker stack | | 7 | Findings get ignored | finish gates | | 8 | The wording of a correction is load-bearing | the envelope and the silence trailer | | 9 | A deletion that grew the file | take the generation out | The standard shape is fetch-then-answer: embed the question, pull the top-k chunks, hand them to the model. It works for a great many tasks, and it has one structural problem. The folder gets assembled before anyone has understood the problem, so you are deciding what might be useful before you know what is needed . Giving the agent search tools of its own fixes the ordering and immediately produces a different class of bug, because search tools are literal. The agent searched every file in the project for json , found nothing, and offered to create a sample data file. sampleData.json was sitting in the project. The word it needed was on the front of the folder, not inside it. for entry in project paths.iter project files ctx.project dir, glob=glob : A model hunting "the json"/"the xml" searches by TYPE, and content search can never find a file whose content doesn't mention its own format live failure: sampleData.json was invisible to pattern "json" and the agent offered to create a file that existed . Names are part of the searchable surface. if rx.search entry "path" : name hits.append f"{entry 'path' } filename match " Asked to remove a feature, the agent searched, found the one matching line, deleted it, and declared success. The five lines above it declared and configured the same object. So find in files now returns grep-style context plus an automatic variable footprint . A hit on layout.blockStart box also lists var box = new Block and every property line of that variable. Only declared variables expand, so globals like layout never blow the output up. The rule it encodes: the construct is all of its lines, not the one that matched. Neither fix is clever. Both exist because something shipped. There are two layers of tools, and the split explains several of the fixes that follow. The base tools ship with the platform and are deliberately generic. list files , read file , find in files , write file , edit file , apply edit , delete lines : every customer gets them, and not one of them knows anything about anybody's domain. The client tools are authored per customer and live in that customer's own directory. Here that means lookup docs , resolve topic , find construct , check symbols and dry run , none of which mean anything without this engine's documentation and this engine's object model. The mechanic that matters is that a client tool shadows a base tool of the same name . Two do. The client find in files adds something the generic one could not: when a literal search finds nothing, concept words like "block" or "the box" expand to the engine's real symbols, so the agent never has to translate a concept into code before it can search for it. And rag lookup , the knowledge lookup itself, is shadowed too, which is what makes the fix in section 5 possible. That split is the architecture. The generic layer is table stakes and everyone has it. The layer that knows what a "block" is called in this customer's world is the part that cannot be bought. These two need separating, because the gap between them drives a lot of what follows. | Tool | What it does | What it costs | |---|---|---| lookup docs | full-text search over the indexed documentation, returning the chunks that literally contain the term. No embeddings, no model, no generation. | milliseconds, and it cannot invent anything | rag lookup | semantic retrieval, then a model composes an answer out of whatever came back | ten to thirty seconds, and the composing step can invent | The second one is what most people mean by RAG: retrieve some relevant text, hand it to a model, let the model write the answer. It is the right tool for "how do I build a table", and the wrong tool for "does tableColumnRule exist", which is the distinction the agent is told to route on: TWO knowledge tools, and the choice matters. lookup docs is ~1000x cheaper. Naming a symbol you already know is a FACT CHECK, not a research question. That framing looks like a cost optimisation. It turns out to be a correctness one, for reasons section 5 gets to. My favourite failure of the whole project: The agent ran the same knowledge lookup twelve times in one turn, identical arguments each time, and the results themselves contained notes pointing out it had already asked. It made no difference. The variant version was worse, and no exact-match check can catch it, because the arguments differ every time. One table turn ran 12 steps and 170k input tokens , and three of its five knowledge lookups re-confirmed the same three API names. Those names were already in its context, listed in the prompt fragment it had been handed at the start of the turn. It was not searching. It was checking something it already knew, at ten to thirty seconds a time. The operator's note in the ledger puts it better than I can: once it knows how to do stuff it shouldn't need to keep trying to find it at the rag end A human has an internal sense of going in circles. The model does not, and telling it so in the tool output does nothing, because that text is just more context. Worse, on a provider without prompt caching the fixed ~11k of system prompt and tool schemas is re-sent every step, so cost is roughly fixed times steps plus transcript. Fewer steps is the only real lever, and a redundant lookup adds steps. The fix is five layers, and the layering is the point : each layer only catches what the one before it structurally cannot see. A counter in the tool loop, tripping at MAX IDENTICAL CALLS = 4 . Past that the call is not executed, but it still has to be answered, because every tool call in the transcript needs a result. Stuck-loop detector: the same call over and over is fixation, not progress. Answer it with a stub every tool call must be answered and head for the forced final answer instead of burning the rest of the budget. key = name, call.arguments or "" call counts key = call counts.get key, 0 + 1 if call counts key MAX IDENTICAL CALLS: stuck = True messages.append {"role": "tool", "tool call id": call.id, "content": f" not run - you have made this exact call " f"{call counts key } times this turn "} Reworded questions are distinct argument strings, so the identical-call counter never fires on them. Catching those needs a budget rather than a counter: | Constant | Value | Meaning | |---|---|---| MAX LOOKUPS PER TURN | 12 | live lookups across all wordings | MAX LIVE LOOKUPS | 2 | per identical query, then the turn cache answers | This one I like, because the obvious solution is wrong. You could embed every query and compare vectors. We do not, because that is a per-probe cost on the hot path for a problem that plain string handling solves. Matching is deliberately embedding-free: normalise lowercase, drop stopwords, light stemming with a double-consonant collapse so splitting to split , token-sort, then exact key hit or a difflib close match. Deterministic, stdlib, zero per-probe cost, and it catches exactly the reworded-not-rethought queries above. Genuinely different questions stay distinct. The similarity threshold is 0.8 , and the calibration comment is the reason I trust it: create table vs create box , score about The window between them is the sweet spot, and it was measured rather than guessed. Read-only calls get dispatched together, so three rewordings can all be in flight before any of them returns, and a dedupe that runs on completion never sees them. That needs a leader/follower collapse on the normalised query: the first runs, the rest wait and receive its answer. None of the above touches the failure this section opened with. A budget caps how much re-asking costs, but the agent re-confirming names it was handed at turn start should not be asking at all. Writes to domain code are normally refused unless a knowledge lookup has run in that turn, which is what forces grounding. That gate was also what forced the redundant lookups: the agent had the names, but the rule did not know that. So the prompt fragments now declare what they supply. Each construct fragment carries a provides list, generated rather than hand-written and scoped to one family, so the table fragment declares the table companions and not the rule ones. The task brief unions those into provided symbols , and the gate exempts a write whose new X constructors are all covered. A write reaching for a construct that was never delivered still grounds, and check symbols remains the backstop before anything lands. The general shape: a rule that forces good behaviour will also force pointless behaviour unless it can see what has already been satisfied. This was the most useful thing in the section. The budget once cut a model off mid-way through correctly verifying eleven symbols it was about to use. It was doing exactly the right thing and we throttled it. So budget is now charged per tool rather than per call: MAX LOOKUPS PER TURN caps LIVE lookups across all wordings; cached serves and cheap tools are free projects.budget tools , default rag lookup : a millisecond lookup docs fact-check per symbol is diligence we demand, not cost to ration. If your rate limit punishes the behaviour you are trying to encourage, the limit is wrong, not the behaviour. One last piece of absurdity, included because it is a genuine class of bug and not just a funny story. An earlier nudge injected into the tool results said "look up the interface's REAL members" . The model ran that sentence, verbatim, as a semantic search query. Twelve times. Nudges now name an action, and only ever suggest exact-name lookups, never a phrase, because anything phrase-shaped in an agent's context is a candidate query. One user phrasing is a single point of failure for recall. If the question happens to be worded unlike the corpus, retrieval returns nothing useful and the agent concludes the knowledge base is empty rather than that it asked badly. So before the lookups run, a forced tool call reformulates the question into several deliberately different queries. Not paraphrases: different angles . QUERIES TOOL = { "name": "search queries", "description": "Reformulate the question into optimised knowledge-base search queries using the right " "domain keywords / API terms. For a how-to / implementation / 'how do I ...' question, " "return 2-3 queries covering different angles e.g. the core API/object, and a usage " "example . Return a single query ONLY for a trivial factual lookup e.g. 'what is X' ." , ... } The user message that drives it is where the actual behaviour lives. Almost every clause is there to stop a specific degenerate output: Generate {max queries} search queries covering DIFFERENT angles so retrieval is thorough: 1. A focused how-to query for the specific task. 2. A BROAD keyword query - just the core nouns / concepts / terms e.g. 'user login session token' , to surface related material the specific query might miss. 3. If more are allowed, add a query aimed at a concrete example. Do NOT return only a verbatim copy of the question. Only a trivial factual lookup gets a single query. Examples apply the SAME pattern to the question's own domain : - 'set up user authentication' - 'how to set up user authentication', 'authentication login session token configuration' - 'export a report to CSV' - 'how to export a report to CSV', 'CSV export columns rows formatting' The system line is one sentence and does most of the work: You generate DISTINCT knowledge-base search queries using precise keywords / API terms. Never echo the question verbatim. Without it you get the question back with the word order shuffled, which is the reworded-not-rethought problem from section 2, except now we are paying to generate it. Each query carries a human label. The tool schema requires an about field alongside the query: "a short, natural human phrase for what this looks up, e.g. 'creating tables with borders'" . That is not for retrieval, it is for the activity line the user watches. Generating the explanation in the same call as the query is free; deriving it afterwards is another model call or a bad heuristic. The queries are not fanned out by code. They are injected back into the prompt as an instruction, so the agent still owns sequencing and can stop early. python def guided prompt question, query objs, adaptive=False : """Wrap a question with guidance to search each expanded query, then answer or dig deeper .""" qlist = "; ".join o "query" for o in query objs tail = ADAPTIVE TAIL if adaptive else " Then answer fully." return f"{question}\n\n Search the knowledge base for: {qlist}.{tail} " Expansion without a brake is just a licence to over-search , which is section 2 all over again. So ADAPTIVE TAIL ships with it: Then judge the results: if they already answer the question, STOP and answer now, do not run extra, redundant, or tangential searches. Only if a genuine gap remains or the results reference something specific you still need search again for exactly that. Aim for the fewest searches that fully cover the question. There is a ceiling of 6 regardless of configuration, and max queries: -1 means the model decides within that ceiling. One clarification, since the two get conflated: this is multi-query rewriting, not HyDE . No hypothetical answer document is generated. In a domain the model has never seen, a hypothetical document is a hallucination with extra steps, and embedding it just retrieves whatever is nearest to the invention. This is the failure that cost the most and looked the least like a bug. The agent phrases things by goal: "display the data in a table" , "put the XML in a table" . The corpus is indexed by the engine's own vocabulary: openStream , xmlDoc , selectNodes . To a human expert those are obviously the same topic. To a retriever they have almost nothing in common, so the lookups returned nothing, the agent concluded the capability did not exist, and it reached for browser JavaScript instead. Missing real content and inventing names is the expensive combination, because the invention then looks like a hallucination and you go and debug the wrong component. Three layers, in increasing order of force. resolve topic maps a goal to the real symbols, with no model involved at all: The agent phrases requests by goal; the docs are indexed by object-model vocabulary. That mismatch made it MISS real content and INVENT names, spiralling. This reads the terminology datagraph topics.json and hands back the real symbols, a note on the pattern, and the exact next lookup, so the agent searches truth instead of guessing.Zero LLM, deterministic. Call it BEFORE rag lookup when you don't already know the API names. The map is mined rather than hand-written: Matching is stopword-filtered token overlap ranked by hit count. No embeddings anywhere in it. Because a phrasebook that is confidently wrong is worse than none. The first version applied full force to any match at all: Confidence ladder live miss: 'footer with page numbers' shared ONE word with the layers topic and got the full-strength 'do NOT invent' route, walling the agent off from the real constructs . A weak match is a HINT, not a law: strict only when the route explains at least half the goal's significant words, or 2+ of them. coverage = top / max 1, len words goal if top < 2 and coverage < 0.5: return "WEAK MATCH - the goal only loosely fits a mapped topic; treat this as a " "hint, NOT the answer:\n\n" ... A guardrail with a false positive does not fail neutrally. It actively steers the agent away from the answer, and it does it with authority. Users phrase goals in their own nouns "display parts from an XML file grouped by category" and documentation indexes mechanisms "load an XML file, iterate elements into table rows, heading per group" . One cheap call normalises the retrieval query before the lookup stack sees it, layered outermost so that dedupe and the persistent cache both key on the normalised form and hit more often. The constraint on that rewriter is the interesting part: : Operations only, never API names: the rewriter half-knows jargon, and an invented : symbol in the query would poison retrieval with exactly the hallucination the : lookup exists to prevent. API discovery stays retrieval's job. It ships disabled, incidentally. The doctrine version in the prompt does most of the work and the mechanical version is held in reserve for when doctrine leaks. Where the phrasebook is applied matters more than the phrasebook. The terminology hint is prepended to every lookup result, whether or not the model asked for it, and whether or not retrieval returned anything, because a gap is exactly when the real names matter most. The agent kept writing invented API calls into its work. It would be caught, it would look the correct ones up, and then it would do it again. Seven rounds of this in one session. I assumed the model was inventing them. That is the famous failure mode and I never questioned it. Then I read what the lookup tool had actually returned, rather than the model's account of what it returned. The retrieval pipeline was generating the invented APIs. The style that answers conceptual questions composes an answer from retrieved examples, and that generation step had invented layout.tableCellEnd and Table.cellSpacing , neither of which exists, while asserting they came "directly from the example in the provided context" . That is not really a bug, it is what generation does. Ask an obliging model for a worked example and it will produce a worked example, and where the retrieved material runs thin it fills the gap with the most plausible name available. The word "lookup" was what fooled me: I had been treating an endpoint that ends in a generate step as though it returned facts, when the only thing separating it from the agent was that its inventions arrived with a citation attached. The model was faithfully copying bad information from the one source it had every reason to trust. Worse, the loop was closed: the check flagged the file, the agent re-ran the same lookup to find the right answer, and got the same poison back. The debugging rule that came out of it, because I lost days to not knowing it: read the actual tool results in the log, not the model's narration of them. A large proportion of what looked like model failures turned out to be a tool feeding it poison. The fix is a client-side shadow that wraps the built-in lookup and audits its output with the same symbol checker the write path uses: The examples style GENERATES its answers, and generation can invent APIs. A live run returned examples using APIs that do not exist, the agent wrote them, check symbols flagged the file, the agent re-ran the same lookup and got the same poison: a perfect loop. This shadow breaks it at the source: run the built-in lookup, then run the SAME symbol audit the write path uses over the returned example, and hand the agent the answer and the corrections in one result, so it learns the example is partly wrong BEFORE writing it, together with the interface's real members to build from instead. The lookup cache had pickled the poison. Of 137 cached entries, 126 carried warning text . Every fuzzy cache hit was re-serving a known-bad answer, forever. Flagged answers are now uncacheable, and the cache module has a rule list at the top that reads like scar tissue, because it is: Once the guardrails were in, we probed the same grounded prompt across three model tiers, all with identical retrieval and identical checks: | Model tier | Grounded probe | |---|---| | frontier | 5/5 | | mid | 2/5 | | cheap | 0/5 | The verdict in the notes: the guardrails make a weak model SAFE bounced , not GROUNDED. A gate stack stops bad output reaching the user. It does not turn a model that cannot follow grounding instructions into one that can. Budget for the model tier; do not expect the scaffolding to buy it back. This is the check everything else depends on, and the first version was wrong in an instructive way. It asked an LLM to list the symbols in the agent's code, then full-text searched each one. Every step leaned on the thing that was broken: an LLM summarising another LLM's hallucination, and a search whose only possible verdict was "not found", which tells an agent nothing about what to write instead, so it kept the invented call. That second clause is the part I underestimated. "Not found" is not actionable. An agent that receives it has no better move available than the one it already made, so it makes it again. The reframe: the documentation is complete and enumerable, so the question is decidable . Resolve what an object is, then ask whether the member exists on it. php layout.tableCellEnd - layout is Layout - no such member - INVENTED var cell = new TableCell ; cell.start - cell is TableCell - check start "Complete and enumerable" is carrying a lot of weight in that paragraph, so here is where the index actually comes from. It is not hand-written, and that is the only reason the verdict is trustworthy. A hand-maintained list of what exists is just one more thing that can drift from what exists. The member index falls out of the documentation's own structure. Every documented member is its own chunk, titled Interface-member , so parsing titles yields {interface: {members}} for the entire surface. Two patterns in the prose supply the rest of what the resolver needs: | Pattern in the docs | What it yields | |---|---| Interface-member chunk titles | every member of every interface | accessed using the global property "layout" | layout resolves to Layout | Parent interface: Block | inherited members | The middle row is what makes receiver typing possible at all. The set of documented globals, the thing that turns layout.x into a decidable question rather than a guess, is extracted from a sentence the documentation happens to write about itself. A second datagraph describes constructs rather than members: how each one is declared, and which calls open and close it. It is generated from the API definitions rather than curated, and the reason is the best argument in the codebase for never hand-writing ground truth: Hand-curating it is error-prone: the first cut invented a List / listStart construct that does not exist. We hallucinated an API while building the tool whose entire job is catching hallucinated APIs. The generator now derives the pairs mechanically: on a global-reachable interface, a