{"slug": "four-ways-agents-fail-on-long-real-world-tasks", "title": "Four Ways Agents Fail on Long, Real-World Tasks", "summary": "A team deploying AI agents for private equity, private credit and investment banking deal work found that 18 of 56 agent sessions on a single project contained at least one recurring failure, and that coding agents have not shown the self-improvement a human analyst gains within days. The failures fall into four categories in the order they appear in long tasks: exploring the environment, grounding evidence, understanding the task and verifying the result. Examples include a parser that returned null for notices labeled \"Date\" instead of \"Effective Date,\" a PDF extractor that left a scanned annual review blank, and an agent that recorded nine rent-stabilized units for a single-apartment collateral.", "body_md": "Frontier models now know as much finance as a strong new hire. They can normalize EBITDA, read financial statements and write an extensive due diligence report. Drop one into a long-horizon task, though, and it makes a new analyst's first-month mistakes: it adds back a cost the fund doesn't allow, takes a number from the wrong part of the file, and copies the borrower's figures into the column meant to check them.\n\nA good analyst stops making those mistakes within days. In our testing and benchmarking so far, coding agents have not shown that kind of self-improvement.\n\nWe work with leading private equity and private credit funds and investment banks to deploy agents on deal work and portfolio operations, including underwriting extraction from commercial real estate loan files, monthly credit reconciliations and covenant monitoring.[^1] On one project alone, 18 of 56 agent sessions contained at least one recurring failure.\n\nAs we covered in [our last post on haircutting benchmark scores](https://lessuncertain.substack.com/p/how-to-read-and-haircut-ai-benchmark), real-world results are less encouraging than the headline metrics. The failures we saw most often fall into four kinds, in the order they show up in a long task: exploring the environment, grounding evidence, understanding the task and verifying the result.\n\n### Tool use errors or missing values?\n\nGiven tens or hundreds of files, a long-horizon agent sees that world only through tools: shell commands, file readers, search, MCP servers. Each observation is partial, and an empty one is ambiguous: the information may not exist, or the tool may have failed to surface it.\n\nFor example, in a monthly reconciliation task, the agent had to take each loan's effective date from its transaction notice. The task prompt stressed that \"the date should be the effective date of the notice, not a payment or wiring date\". After reading a few sample PDFs in full, the agent wrote a parser that looked for \"Effective Date\", so notices that labeled it simply \"Date\" came back null. A human analyst would have easily corrected this generalization during the task, but it happens often, even with the most capable models.\n\nOther tool errors lead to the same confusion. In a loan underwriting task, the PDF text extractor got a few hundred bytes out of a scanned annual review, and the agent left the review details blank because the file \"did not yield extractable text\", though the title is right on page one. In another run, the agent's loop only opened files ending in .pdf, so every field from the Excel rent roll and operating statement came back empty.\n\nEach of these has a straightforward fix. The hard part is seeing the miss, because a blank is also a legitimate answer: some notices have no effective date, and some loans have no default clause. Nothing in the immediate observation tells the agent whether to keep exploring with a different strategy or library, or to accept the blank. The agent needs a calibrated sense of when **absence of evidence is evidence of absence**.\n\n### Context failure: the right evidence for the wrong question\n\nA number is only right in its context: which entity, which period, which source. Agents often keep the citation and drop the context. In one underwriting task, the collateral was a single apartment, but the agent recorded nine rent-stabilized units for it, taken from a review of other units in the building. In another, it took the monthly payment from the promissory note instead of the payment history, and even wrote that figure into the answer key, so the eval graded against wrong labels.\n\nBoth outputs look well sourced: the number is right there on the cited page. A human analyst would first ask what the number describes. Grounding has to record the scope of each fact (entity, period, source), not just the page.\n\n### Task understanding: general knowledge versus task-specific requirements\n\nReal-world instructions usually come as procedural steps or a description of the deliverable. They rarely state the high-level goal, which the day-to-day business implies. This context is easy to write into memory files or a skills.md file, but in practice agents struggle to follow it consistently, especially where house rules depart from standard practice.\n\nFor example, in a workbook monitoring a borrower's financials and covenants, the job was to copy the figures the borrower reports, recompute key ones such as leverage and interest coverage, and compare the two. Despite the task prompt, the agent copied the reported leverage into the column meant to recompute it, and recalculated figures that should have been taken as reported.\n\nExplicit rules lost to general knowledge too. Under one lender's rules, a building's NOI was −$167,730. Following standard practice, the agent added back $236,400 of roof and boiler work that a later review had called one-time, and reported +$68,670, even though the schema prohibited capital-expenditure add-backs in writing. Clearer prompts alone has not fixed this.\n\n### Verification: the tests pass and the big picture goes unchecked\n\nCoding agents are good at writing checks that execute: schema validation, arithmetic identities, row counts, file hashes. For knowledge work, that isn't enough. Before calling an output final, a human analyst reviews it as a whole and spots inconsistencies, gaps or statistical outliers. A big-picture check like that would have flagged the loan whose spreadsheets were never opened, and the building reported as profitable while its own statements showed a loss.\n\nThe same gap shows up after feedback. When a user pointed out that a loan's NOI should be blank, the agent agreed, but the workbook it had already called final still held the old number. Corrections also didn't spread across parallel agents: the same wrong deduction showed up in several of ten workers, and each had to be fixed on its own. A team of analysts would share the lesson after the first fix.\n\n### Training agents to learn the task\n\nIn all these cases, the agents had the knowledge and the outputs looked finished, but they still missed the task. How could we make agents learn a task as fast as a good analyst does? We think post-training for knowledge work could reward these behaviors more heavily:\n\n- **Exploring before executing:** sampling and reading files to understand what is there, beyond basic file-system exploration, and establishing what success means for itself before executing.\n- **Holding task-level background context:** tracking which entity, period or source each fact belongs to throughout the task.\n- **Asking clarifying questions** when the purpose or an exception is unstated, instead of filling the gap with a prior by default.\n- **Contextual verification:** checking results against what they should look like, so that corrections made during the task persist into the final output.\n\nEnvironments that teach these need what real work has: heterogeneous files, true blanks mixed with unreadable ones, house rules that contradict standard practice, and a reviewer who corrects the agent partway through the episode. The fully automated, long-horizon agent for knowledge work has not yet arrived as coding agents in their current state of capabilities and harnesses, but we are optimistic in its opportunity for continual learning and self improvement.\n\n[^1]: Figures, names and identifying details in the examples have been changed.", "url": "https://wpnews.pro/news/four-ways-agents-fail-on-long-real-world-tasks", "canonical_source": "https://lessuncertain.substack.com/p/four-ways-agents-fail-on-long-real", "published_at": "2026-10-13 09:14:59+00:00", "updated_at": "2026-10-15 05:08:39.338619+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "large-language-models", "ai-products"], "entities": ["MCP"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/four-ways-agents-fail-on-long-real-world-tasks", "markdown": "https://wpnews.pro/news/four-ways-agents-fail-on-long-real-world-tasks.md", "text": "https://wpnews.pro/news/four-ways-agents-fail-on-long-real-world-tasks.txt", "jsonld": "https://wpnews.pro/news/four-ways-agents-fail-on-long-real-world-tasks.jsonld"}}