# Atlas-Finance: Evaluating AI Agents Inside a Bank > Source: > Published: 2026-09-15 23:37:11+00:00 ## **TL;DR** **The gap:** Existing finance benchmarks test agentic financial reasoning, data retrieval, and tool use through static, fully specified tasks. These one-off requests in clean contexts are not reflective of actual deployment. In reality, analysts must operate under ambiguity, gather key context from multiple (and sometimes conflicting) sources, and coordinate with leadership, colleagues across teams, and clients that may send relevant updates mid-task. **Our benchmark:** Introducing ATLAS-Finance–100 expert-level tasks inside 13 realistic financial firm environments with multi-party working group teams, full inboxes, drives, notebooks, and calendars. Agent-generated work outputs are graded criterion-by-criterion against expert-authored rubrics with 42-486 binary criteria per task ([code](https://github.com/Handshake-AI-Research/ATLAS-Finance); [data](https://huggingface.co/datasets/handshake-ai-research/ATLAS-Finance)). **Pushing the frontier:** We designed ATLAS-Finance to closely reflect realistic working conditions in a financial firm. Agents that reliably navigate ATLAS-Finance are more prepared to help professionals deliver the real-world performance their clients expect. **What we found:** Across 11 frontier models, the highest observed pass rate is 12.3% achieved by Claude Opus 5. Claude Fable 5.1 and GPT-6 Astra achieve 12.0% and 11.3%, respectively. The other eight models remain below 10%. Models consistently failed in three areas:  - Applying the wrong financial logic, methodology, timing, or perimeter - Omitting required scope in the final output - Failing to propagate correctly calculated values downstream in the analyses. The specific mix varies by model and workflow, but recurring, plausible-looking financial mistakes appear across all model families. **Why it matters:** Identifying where agents fail under realistic working conditions like ATLAS-Finance environments is a key step in improving them to deliver larger productivity gains in the financial sector. In an actual bank, associates that make critical errors like the ones indicated above force senior members to audit all of their work. Similarly, if an agent makes a mistake like this in the workplace, it negates all of the time savings and expertise acceleration potentially provided by that model. ## **1.** **What’s different about ATLAS-Finance** Recent benchmarks have moved from short-form question answering to end-to-end financial work. Handshake’s BankerToolBench (BTB), in particular, evaluated the financial models produced by AI agents completing self-contained junior investment banker workflows. Its results show that frontier agents can complete parts of these workflows but remain unreliable at the standard required for professional delegation. BTB also identifies a remaining gap: simplified, static environments exclude the scattered, evolving, and sometimes contradictory nature of information found in the real world, in addition to organizational-level complexities that must be navigated where this work happens. We address this gap with ATLAS-Finance by changing the unit of evaluation from a linear workflow to a workplace world. As a vastly richer RL environment, each world contains an advisory firm, specialized teams, updates from senior and junior professionals, clients and counterparties, active and background mandates, communication histories, shared files, calendars, permissions, and firm-specific conventions. The agent works through email, chat, document repositories, notes, calendars, and a local workstation. It must produce the requested financial artifacts and meet an explicit submission contract. Table 1. Comparison of ATLAS Finance against existing finance and professional-work benchmarks. BTB (BankerToolBench, Handshake AI), APEX (APEX-Agents, Mercor; investment-banking subset), BFB (BigFinanceBench, Rogo), ALE (Agents' Last Exam, UC Berkeley), FAB (Finance Agent Benchmark, Vals AI), FinB (FinanceBench, Patronus AI), FF (FrontierFinance, Samaya AI), and GDPval (OpenAI). Built by bankers up to VP-level from institutional banks and elite boutiques (such as Morgan Stanley, Bank of America, UBS, and Rothschild & Co), ATLAS-Finance reflects the real-world complexity that professionals navigate every day. The benchmark comprises 100 expert-level tasks in 13 environments, covering 9 industries and 20+ financial-analysis methodologies. These tasks typically take human finance experts 15-30 hours to complete manually. Table 2. Expert-level workflows that undergird the ATLAS Finance benchmark. To accurately complete tasks, agents must navigate organizational and relational complexity while collaborating with dozens of unique coworkers and clients. Before it populates a single spreadsheet cell, the agent must: - Assemble its workflow details from environment context - Understand its relationship to each counterparty and the role they play for the job at hand - Resolve controlled contradictions by finding governing sources or identifying the right decision-maker (e.g. “Who has the final say on the financing assumption?”). A subset of tasks includes a real-time update with corrections to prior assumptions that must be incorporated before delivery. Agent outputs are evaluated against 14,600 environment-grounded rubric criteria (146 per task on average and all expert-verified). ATLAS-Finance grades the work the way an industry professional would: every task carries 42-486 binary checks organized by schedule, with gating dependencies between them–if the debt schedule is not built with live formulas, everything downstream of it is recorded as unreachable, not merely wrong. A failing run produces a diagnosis in addition to a rubric score: which schedule broke, what the break cost downstream, and, read against the run's transcript, whether the model didn't know the finance, knew it and built it wrong, or built it right and delivered a workbook with static values instead of working formulas. Each rubric criterion comes with a weight indicating its relative importance according to experts. To grade any agent output against these rubrics, we use [Gandalf the Grader](https://joinhandshake.com/research/ai/gandalf-the-grader/): an agent-as-judge framework for verifying financial artifacts.  ## **2. Headline results** We evaluate 11 frontier models (see Figure 1), all run within the OpenCode agentic harness. Claude Opus 5 performs the best, yet still only manages **to pass 12.3% of tasks**. Claude Fable 5.1 and GPT-6 Astra are close behind, but the other eight models have significantly lower pass rates. **What counts as a pass:** A run passes only if it meets every rubric criterion deemed *critical* (those weighted 3 or higher) and triggers no penalties (negative criteria). Penalties catch unacceptable issues such as fabricated data or a missing audit trail. Figure 1. Pass@1 on ATLAS Finance. Error bars show 95% confidence intervals computed via bootstrap. ## **3.** **Example Task - LBO Rebuild Staffing at a FIG & media advisory** Below is an example task showcasing what agents must do in ATLAS-Finance environments. Here *Project Osprey* is a full LBO rebuild involving a specific tax treatment. It would take most human analysts 15+ hours to complete just the Excel modeling involved in this task, not counting the additional fact-finding work necessary to pull the relevant information into the analysis from the bank environment. All ATLAS-Finance tasks start with the agent’s project “staffing”–it is assigned to a new project at the firm exactly how thousands of industry professionals experience it daily. The agent receives a heads-up message with high-level context that points to a subsequent project details email. This begins the workflow chain of understanding, discovery, analysis, and delivery. Understanding this kickoff notice is the first in dozens of steps an agent must handle to complete this ATLAS Finance task. Step 1: Agent Staffing Kickoff *Look through your inbox and the Project Osprey attached source materials and rebuild the US sponsor LBO model your engagement lead assigned you (a sponsor’s 2019 LBO of a payments target, with the 338(h)(10) election). Your assignment email is titled “Project Osprey – rebuild the 2019 sponsor LBO (338(h)(10) election)”.*  The brief goes on to fix the delivery contract that the agent must satisfy–an exact filename, exact recipients, an exact subject line– and the working standard: the finished workbook must carry live in-cell formulas so hardcoding a computed number where a formula belongs is an error even when the number is right. ### **3.1.** **Running Opus 5 powered Agent on this task (73 execution steps, outcome: Fail, rubric score: 0.336)** Here are the steps that a Claude agent took when assigned this task. **Steps 1–19 – Discovery.** Searches the assignment subject (1) → reads the assignment thread and seven related threads (2–4, 14) → finds the desk’s OneNote: Modelling Approach, Output Standards, call notes, open items (5–6, 15–19) → lists and downloads every attachment (7–11) → dumps every source workbook programmatically to read real cell values (12–13). **Steps 20–25 – The shadow model.** Before touching a single spreadsheet cell, the agent writes the complete LBO in pure Python: sources and uses, debt sweep, tax computation, returns, and iterates it to convergence (20–23), then extends it with a second validation pass (24–25). Every later self-check will be measured against this blueprint. **Steps 26–51 – Build.** A single build script generates the whole workbook - nine tabs in the briefed order (26–27); first recalculation with the environment’s spreadsheet engine (28); targeted fixes to the NOL rows and a revenue formula (34–41). When convergence stalls, the agent reads the spreadsheet-recalculation utility’s own source code and reproduces its environment to debug the circularity (43–49). “Converged.” (52) **Steps 52–67 – Validate.** Sweeps every output tab and ties the numbers back to the shadow model (52–67). All thirteen integrity checks on the Control sheet read OK; the workbook carries 2,641 live formulas and zero error cells. **Steps 68–73 – Delivery.** Verifies the collection copy and the delivery-named copy are byte-identical by SHA-256 (68); sends to the exact recipients with the exact subject and attachment (69); reopens the sent message to confirm the attachment landed (70–72); closes: “Done. The model is built, recalculated, saved and delivered.” (73) Figure 2. Anatomy of one agent rollout: 73 steps of work, two wrong ideas, and where the rubric deducted points (out of 1460). ### **3.2** **What went wrong** During discovery, the agent surfaces a key email from the client’s tax lead. R. Calloway lays out each input for the tax-basis build and annotates each one with its operation: “*added to purchase price,*” “*subtracted in the basis calc,*” “*subtracted*.” One line carries no operation at all: *Existing Goodwill*. The client gave Existing Goodwill no specified treatment, because under the record’s method Existing Goodwill plays no role in the amortizable step-up. The agent’s main error is born at step 20, inside the agent’s own verification machinery. The Python shadow model–written to protect the build–contains one silent line that subtracts existing goodwill from the amortizable basis. Critically, the model failed to add a comment, consider alternative treatments given the task context, or flag the line for review. Six steps later, transcribing the inputs into the workbook, the agent rewrites the label itself: “*Existing Goodwill – carry-over tax basis*”–a treatment the governing email never assigned is recorded as if it were source data.  Figure 3. The LLM invents an incorrect treatment for the Existing Goodwill value shared in the client’s email. This treatment breaks the financial methodology for this scenario. From that moment, the run’s guardrails point the wrong way. All of the agent’s self-checks verified against this shadow model, so all thirteen integrity checks read green, the recalculated values matched the blueprint exactly, and the deduction shipped. The subtraction cut every year’s tax amortization by roughly a quarter and understated the election’s value. The delivery email then walked the client through the netted figure as if it were the record’s method. A second, unrelated idea in the same script drifted the revenue build: the agent authored its own cohort-penetration assumptions where the record prescribed specific ones. The two segments moved in opposite directions–one about five percent low, the other ten percent high–so the total nearly cancelled and no internal check flagged it. But exit-year EBITDA landed 5.9% low, and all downstream values it feeds: investment exit math, resulting GP and LP returns, etc. were incorrect. The rubric records dependencies the way a professional would: once the tax construction is wrong, everything downstream of it is untrustworthy, so the gate for the tax section of this rubric marks all 180 of its points unreachable rather than crediting arithmetic built on a bad number. The purchase-price bridge lost most of its points the same way–its lines were built on the netted basis and the revenue drift propagated through all 530 Returns points. Table 3. ATLAS Finance rubrics are calibrated to real-world performance expectations. Scores for the run above, by rubric section. If the agent’s Goodwill treatment ships to the client, it's a major embarrassment with real consequences. The client notices “something is off” (compared to their own math, or worse, another bank’s analysis on the same deal), re-derives the schedule, finds the invented deduction, and the bank’s entire work product is called into question. In highly competitive industries, teams have lost bakeoffs and millions in revenue over less. In our rollouts, every Claude Opus 5 run on this task makes the same deduction and lands on the same invented figure. Runs from Kimi K3 and GPT-5.6 Sol arrive at the same incorrect treatment independently. While experienced bankers would keep the 338(h)(10) tax election top of mind to avoid this mistake, AI agents routinely fail to do so during key calculations. ## **4. Failure mode analysis** Agent failures predominantly fell into six recurring categories. Our failure mode study analyzes 1,100 rollouts across different models. Each failing rubric criterion was then traced through the submitted workbook, governing inputs and dependency chain to the earliest independently repairable root cause. Table 4 reports the resulting breakdown of failure causes across the analyzed rollouts. Financial logic/methodology issues are the largest category of failure causes, accounting for 52.2% of the total failures across models. Finance integration issues were also responsible for another 1.4% of total failures, bringing finance-related causes to 53.7% of the total failures. Scope/completeness issues accounted for 26.6% of total failures, spreadsheet engineering issues for 10.8%, presentation/delivery for 8.8%, and source/data handling for only 0.1%. Table 4. Breakdown of model errors by failure category. These compositions should not be used for ranking models. Financial logic/methodology: The model builds the calculation but applies the wrong financial treatment, convention, timing, or perimeter. Finance integration/flow-through: A financial treatment is built locally but does not flow correctly through the model’s downstream outputs. Spreadsheet engineering: The financial reasoning may be sound, but the workbook fails because of incorrect formulas, references, signs, hardcodes, or model structure. Source/data handling: The model selects the wrong input, document, version, scenario, or correction—or cannot demonstrate the input’s provenance. Scope/completeness: The model omits or leaves unfinished a required part of the deliverable. Presentation/delivery: The model completes the work but does not present or deliver it as required. The breakdown of failure types varies materially across models. Compared to other models, a higher fraction of the failures of Fable 5.1 and Opus 5 lie in financial logic issues. Relative to the Claude models, more of GPT-6 Astra's and GPT-5.6 Sol's failures are due to scope/completeness issues. Gemini 3.1 Pro has the highest scope/completeness share, and the Gemini models and DeepSeek V4 Pro lose the largest fractions to spreadsheet engineering. Table 5 shows concrete examples of each failure mode. Each example highlights the first mistake made in the agent’s trajectory and the evidence used to distinguish the root cause of failure from its downstream effects. Table 5. Verified examples of model errors and their causes. Several examples show models failing to connect related financial calculations. Opus omitted a borrowing amount from its debt schedule, understating interest expense. Gemini 3.1 Pro failed to carry acquisition-related asset adjustments into the combined balance sheet. In both cases, information needed in one part of the financial model was missing from another part. Across the audited trajectories, visible completion was not a reliable indicator of financial correctness. Models often delivered live-formula, complete-looking workbooks that nevertheless applied the wrong financial treatment or failed to propagate a valid upstream component. ## **5. Next steps** The low pass rates on ATLAS-Finance leave substantial room for improvement, making it a useful benchmark for tracking progress toward financial agents that can reliably complete real banking work. Our forthcoming paper will present the full evaluation methodology, results, and detailed failure analysis. The code to run the benchmark (as a Harbor RL environment) and data are available here: