{"slug": "harvey-lab-aa-evaluating-ai-agents-on-real-world-legal-work", "title": "Harvey LAB-AA: evaluating AI agents on real-world legal work", "summary": "Harvey LAB-AA, a new legal agent benchmark from Artificial Analysis, evaluates AI models on 120 real-world legal tasks across 24 practice areas. Claude Fable 5 leads with a 14.2% all-pass rate, nearly double the next best models, but the best model still fails to fully complete 86% of professional legal deliverables, highlighting that frontier legal work remains largely unsolved.", "body_md": "[All articles](/articles)\n\n# Announcing Harvey LAB-AA: evaluating AI agents on real-world legal work\n\nHarvey LAB-AA (Legal Agent Benchmark) is our implementation of Harvey's new agentic legal benchmark, evaluating language models on real-world legal work across 24 practice areas.\n\nModels are tested on a private set of 120 legal tasks built by the team at Harvey, spanning practice areas from corporate M&A and capital markets to tax, litigation, and bankruptcy. Models work to create the legal outputs specified in each task, and each task is graded against a rubric of binary criteria. The primary metric we present is the all-pass rate: the share of tasks where all criteria in the rubric are satisfied, reflecting the high standard of real-world professional legal deliverables.\n\n## Score\n\n### Harvey LAB-AA: All-pass Rate\n\nClaude Fable 5 (max, with Opus 4.8 fallback) leads Harvey LAB-AA with a 14.2% all-pass rate, after falling back to Claude Opus 4.8 on only one task. This is almost double the next best models, Claude Opus 4.8 (max) and GLM-5.2 (max), which tie at 7.5%, followed by MiniMax-M3 at 6.7% and Claude Sonnet 5 at 5.0%.\n\nFrontier legal work is far from solved: most models pass a majority of individual rubric criteria but fully satisfy the requirements of very few tasks. The best model still leaves ~86% of professional legal deliverables incomplete, and 13 of the 28 models evaluated at launch fully pass zero tasks. Only four models score above 90% on criterion pass rate: Claude Fable 5 (93.6%), Claude Opus 4.8 (91.1%), GLM-5.2 (91.0%), and Claude Sonnet 5 (90.1%).\n\nFor up-to-date results see the [Harvey LAB-AA evaluation page](/evaluations/harvey-lab-aa). Charts show data as at 7 July 2026.\n\n## Cost\n\n### Harvey LAB-AA: Cost per Task\n\n## Token Usage\n\n### Harvey LAB-AA: Output Tokens per Task\n\n## Speed\n\n### Harvey LAB-AA: Time per Task\n\n## Turns\n\n### Harvey LAB-AA Benchmark Leaderboard: Average Turns per Task\n\n## Score vs. Release Date\n\n### Harvey LAB-AA: All-pass Rate vs. Release Date\n\n## Example Tasks & Submissions\n\nBrowse representative Harvey LAB tasks from the public task set, the reference files each model was given, and the deliverables it produced.\n\nInstructions\n\nReview the attached acquisition data room contracts and internal memo for change of control and assignment provisions, and prepare a comprehensive deal team report.\n\nOutput: `coc-analysis-report.docx`\n\nDeliverables\n\nExpected outputs the model must produce\n\n- coc-analysis-report.docxA comprehensive deal team report analyzing change of control and assignment provisions across the target’s material contracts.\n\nReference files\n\nProvided to the model\n\nModel submissions\n\nDeliverables produced by each model\n\n[Open](https://artificialanalysiscdn.com/harvey-lab-example-files/corporate-ma/analyze-change-of-control-provisions-across-targets-material-contracts/submissions/claude-fable-5/coc-analysis-report.docx)\n\n## How Harvey LAB-AA differs from Harvey's LAB\n\nHarvey LAB-AA is our independent reimplementation of Harvey's evaluation, and there are several key differences to the original version:\n\n- Models are run on our\n[Stirrup](https://github.com/ArtificialAnalysis/Stirrup)agent harness, enabling features such as context compaction rather than failure when reaching context limits, with simplified Artificial Analysis-authored agent and judge prompts - We do not include Harvey's custom tools and document-generation skill scripts (e.g. pptx, docx), instead providing a simple code execution tool to reflect raw model ability\n- Deliverables must match the exact filename specified, rather than fuzzy matching when models produce incorrect filenames\n- Grading uses a single Gemini 3.1 Pro judge, tested to be well-calibrated against a frontier panel\n\n## Harvey LAB-AA resources\n\n- The leaderboard and full results live on the\n[Harvey LAB-AA evaluation page](/evaluations/harvey-lab-aa), updated as new models are released - The\n[methodology page](/methodology/intelligence-benchmarking#harvey-lab-aa)documents the full implementation, including the agent and grading prompts [Harvey's original LAB announcement](https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark)introduces the benchmark and its design- A public set of representative tasks is available on\n[GitHub](https://github.com/harveyai/harvey-labs) - Harvey LAB-AA runs on\n[Stirrup](https://github.com/ArtificialAnalysis/Stirrup), our open-source agent framework\n\n#### Read the latest\n\n### How GPT-5.6 Sol, Terra, Luna compare on intelligence vs cost\n\nGPT-5.6 Sol and Luna are ahead of Terra at every point on the Intelligence vs Cost per Task chart. GPT-5.6 Luna stands out as a particularly cost efficient model\n\nJuly 13, 2026\n\n### Muse Spark 1.1: Meta gains 8 Intelligence Index points in three months\n\nMeta's Muse Spark 1.1 scores 51 on the Artificial Analysis Intelligence Index and is cost and token efficient compared to its peers\n\nJuly 10, 2026\n\n### GPT-5.6 benchmarks across Intelligence, Speed and Cost\n\nGPT-5.6 Sol comes close second to Claude Fable 5 in the Artificial Analysis Intelligence Index at one third of the cost, and leads the Artificial Analysis Coding Agent Index in OpenAI’s Codex harness\n\nJuly 9, 2026", "url": "https://wpnews.pro/news/harvey-lab-aa-evaluating-ai-agents-on-real-world-legal-work", "canonical_source": "https://artificialanalysis.ai/articles/harvey-lab-aa", "published_at": "2026-07-14 06:09:42+00:00", "updated_at": "2026-07-14 06:18:19.155595+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "large-language-models"], "entities": ["Harvey", "Artificial Analysis", "Claude Fable 5", "Claude Opus 4.8", "GLM-5.2", "MiniMax-M3", "Claude Sonnet 5", "Gemini 3.1 Pro"], "alternates": {"html": "https://wpnews.pro/news/harvey-lab-aa-evaluating-ai-agents-on-real-world-legal-work", "markdown": "https://wpnews.pro/news/harvey-lab-aa-evaluating-ai-agents-on-real-world-legal-work.md", "text": "https://wpnews.pro/news/harvey-lab-aa-evaluating-ai-agents-on-real-world-legal-work.txt", "jsonld": "https://wpnews.pro/news/harvey-lab-aa-evaluating-ai-agents-on-real-world-legal-work.jsonld"}}