Key Takeaways From Trying to Do It All With One LLM Call

A developer building a LangChain-based survey-report agent found that Anthropic's Claude Haiku 5.5 generated 17 bad citations in total by truncating 36-character respondent UUIDs to their first 8 characters, a failure the Sonnet model never produced, and fixed it by having the model cite short per-run aliases (q1…qN, p1…pN) that code maps back to real IDs. The developer also moved all statistics out of the model, computing figures in code and having the model emit references like {{stat:q1.mean}} that are substituted after generation, after a model-reported average of 4.3 differed from the true 4.2. The conclusion is that a model should get as little to do as possible and everything that can be done by ordinary code should be.

For the past few months I've been building an agent app in my evenings, using LangChain in Python. You feed it a big batch of survey answers and it writes a report on what the respondents think. I started with game feedback in mind: players fill in a form, and the developer gets a readable summary instead of a spreadsheet. I was new to Python, to LangChain and to agents, so I built the version a newcomer builds. One LLM call received all the answers and was asked to deliver the whole report: the numbers, the themes, the quotes, the prose. It worked well enough to be convincing, and that was the problem. Almost everything I did afterwards was taking a job away from that one call. The short version of what I learned: a model should get as little to do as possible, and everything that can be done by ordinary code should be. Setup: written in November 2026, using LangChain Python and Anthropic's Claude Haiku 5.5 and Sonnet models, with evaluation runs tracked in LangSmith. The data is a game-feedback survey, but nothing here depends on games. The report needs figures, such as the average rating in the summary. In my first version the model calculated them itself, and I planned to build guardrails that would catch its mistakes. Then one repouuüîurrt said the average was 4.3 when it was 4.2. A gap that small passes every sanity check, which is exactly why it is dangerous. Building guardrails around a calculation the model shouldn't be making was backwards. Now the statistics are computed in code before the model sees anything. The model is given the results, or more precisely a reference to each one. It writes something like {{stat:q1.mean}} in its sentence, and code replaces the reference with the real value once the text comes back. The 4.2 in the finished report never went through the model. I don't want to oversell this. A model can still choose a reference that exists but is wrong, for instance the mean of another question, and the substitution will succeed without complaint. What I've changed is the shape of the error. Before, the model could produce any number that looked plausible. Now it can only choose from a fixed list, code rejects references that don't exist and any bare number the model writes on its own, and each figure in the report leads back to one spot in the code. Errors still happen, but they're far easier to find. My plan was to build on Sonnet. Costs during development made me switch to Haiku, a smaller and cheaper model, at least for a while. I assumed the only difference would be prose quality. Instead the cheaper model exposed problems I hadn't known I had. Every report began failing verification, with 17 bad citations in total, something Sonnet had never caused. The cause is in the next section. The general point is that a smaller model trips over weaknesses a stronger one quietly absorbs, and those weaknesses can reach production just as easily. Passing on a cheap model proves nothing about report quality, though. I measure quality separately with evals and use the cheaper model only as a way to test how robust the pipeline is. Each survey response has a UUID in the database, 36 characters long, for example 3f2b8c1e-9a4d-4e7b-b1c2-5d6f7a8b9c0d. I included those IDs in the prompt so the model could point to the respondents it was quoting. For a model, that string is random and meaningless, and it must be reproduced character for character. Haiku kept the first 8 characters and dropped the rest. The citation no longer matched any record, so verification rejected it. The model hadn't invented anything. It had simply copied a long random string imperfectly. One option was to make the verifier more forgiving. I changed what the model sees instead. Now it works with its own short aliases: q1…qN for questions and p1…pN for respondents, assigned from scratch on every run. Before citations are checked, code maps each alias back to the real ID. The verifier is exactly as strict as before, so an alias that doesn't exist, such as p99, or a truncated UUID still fails. This also made the prompts shorter and cheaper in tokens. One quirk: aliases sometimes slip into the prose, as in " p5 ". Code removes them from the report text and leaves the quotations untouched. When a model would have to copy a long, unreadable identifier, give it a short alias it can handle, and let code translate. You spend fewer tokens and get far fewer failures. After the model writes a report, five checks run on it, including citations, references to statistics and required sections. My first loop handled failures one at a time: the first failing check sent its error to the model, the model rewrote, the checks ran again, and the next failure went back the same way. One report could take four model calls. That is wasteful, because checks are plain code. They finish immediately and cost nothing, while each model call costs real money. So the loop now runs all the checks, collects everything that failed into one list and sends the model that full list for a single rewrite. Then the checks run once more. Repeating checks that already passed costs almost nothing next to a model call that wasn't needed. The pattern is close to what's called evaluator-optimizer: one part writes, another judges, and they loop. The difference is that my judge is code rather than a second model. Reporting every failure at once resembles a compiler printing its whole error list instead of stopping at the first. The loop is capped at two correction rounds by default, and a report that still fails after that is rejected instead of being returned unchecked. I've settled on two habits for prompts. The first is version control, with a stricter rule than usual. A prompt lives in its own file, and every version is a separate file. Once a version has been used, it is never edited. Changes become a new version. A test stores each released file's hash, so editing an old version by accident makes the test fail. The second is metadata. A prompt file can start with a header, called frontmatter, that describes what a good answer looks like: --- pipeline: chunkedrequires: min findings: 3 summary keywords: recommend --- The model never sees it, because the header is removed before the prompt goes out, and I make no claim that a model would follow it. The code uses it in two ways. Before the call, it reads the settings and applies them the pipeline value, for example, selects how the processing runs . After the call, it acts as a guardrail and checks that the report meets what the header asks for, such as at least three findings or a particular keyword in the summary. The upshot is that whoever writes the prompt also defines what counts as a good answer, in the same file, and a new prompt version brings its own definition along. My first eval gave poor scores. The cause was my test data, not the agent. I wrote an artificial form with a known right answer: 16 players, 6 praising one theme and 5 complaining about another. With that, I can compare the agent's output against the answer key player by player. The set is tiny and measures one thing, whether the agent picks the themes out of the comments. Rather than a single score there are four. Did the agent find the themes that are really there? Did it avoid making up themes that aren't? If a theme appeared in different chunks of text, did the merge step combine them? And did it avoid fusing two distinct themes into one? When a score drops, I know which stage to look at. To compare two runs I change exactly one variable, the prompt, the model or a setting, and keep the data and scoring fixed. Otherwise I can't attribute the difference. Runs also vary naturally, so a small gap counts as noise, not as a result. Back to the poor scores: I opened the actual outputs beside the key and saw that neutral comments were being scored as an invented theme, and that my planted themes blended several sub-topics, so the supposed correct answer was ambiguous. I corrected the key, giving neutral players an explicit ignore list, and the scores became meaningful. This eval is still too easy. The baseline scores 1.0 on all four measures, so it can't separate a strong run from a slightly weaker one. A theme also counts as found when half of its players are matched, so 4 of 5 earns full marks. And it tests only theme finding, leaving the report, the statistics and the guardrails uncovered, on forms that are artificial and tidy. It is a starting point. Writing evals gave me a mental hurdle. The principle is appealing: evals do for agent apps what unit tests do for an API, and without them it's hard to say anything solid about quality, consistency, factual accuracy or relevance. The difficulty is that an eval result isn't a clean pass or fail. It's fuzzy, so reading it takes more effort than reading a green or red test. You watch for trends: cost per run creeping up, responses getting slower, quality scores sliding, retries accumulating because the guardrails keep firing. That calls for a dedicated tool. I use LangSmith, and other options exist in the agent world, such as Mastra and CrewAI. My working split is simple. The repository holds the code and the definitions, and the external tool holds what I learn from running them. Building my own dashboards for agent runs isn't worth the time, and tying everything to one vendor isn't wise, so I keep as much as possible in code. In practice, the prompts, the test data with its answer key and the scoring functions all sit in the repo and import nothing from the vendor. The tool surfaces insights and people make the decisions. Where this left me Looking back at the 4.3 that should have been 4.2, my mistake was giving one call too many jobs and then trying to catch its errors afterwards. The model now handles what it does best: it reads what players wrote, sorts it into themes and describes them in clear language. Code handles the rest: counting, tracking IDs and citations, checking the output and storing the prompt versions. The biggest improvements came from no longer asking the model to do the second set of things. Can one LLM call do everything in an agent pipeline? It can attempt it, but the more jobs one call carries, the harder its mistakes are to find. Splitting out the deterministic jobs counting, ID handling, validation made my agent's errors visible and checkable. How do you stop an LLM from getting numbers wrong in a report? Compute the numbers in code and let the model write references such as {{stat:q1.mean}}. Code substitutes the real values, rejects references that do not exist and flags bare numbers the model typed itself. Why do LLMs fail to cite long UUIDs correctly? A UUID is a long, random string with no meaning, and the model has to reproduce it exactly. Smaller models shorten it. Give the model short aliases p1, p2 and map them back to real IDs in code. Is it worth testing an AI agent on a cheaper model? Yes, as a robustness test. A cheaper model surfaced citation failures that a stronger model hid. It does not tell you whether the report is good. Use evals for that. What is the evaluator-optimizer pattern? One component writes, another judges, and the loop repeats until the output passes. In my agent the judge is code, not a second model, so re-checking is almost free. Built with LangChain Python and LangSmith. Tags: LLM, AI Agents, LangChain, Prompt Engineering, Evals. Key Takeaways From Trying to Do It All With One LLM Call https://pub.towardsai.net/key-takeaways-from-trying-to-do-it-all-with-one-llm-call-ac6a3aa8f580 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.