{"slug": "new-access-to-justice-benchmark-finds-leading-ai-models-miss-critical-legal-252", "title": "New 'Access to Justice' Benchmark Finds Leading AI Models Miss Critical Legal Details Across 252 Scenarios", "summary": "Vals AI and NextLadder Ventures released the Access to Justice Benchmark on November 10, testing 23 AI models on 252 simulated legal-aid conversations, and no model reached the 80% threshold the organizers set for a reliable first point of contact, with Alibaba's Qwen 3.8 Max topping the field at 76.4%. Every model met 78–87% of criteria for general legal explanations but only 30–50% for identifying specific local resources, and 21 of 23 models fell back on national directories such as LawHelp.org, the ABA's Find Legal Help and 211 in at least 70% of final answers. Free-tier models averaged 69.7% versus 70.4% for premium models, a 0.7-point gap, with Anthropic's free-tier Claude Sonnet 5.5 scoring 1.2 points above premium Claude Opus 5.5 and only xAI's Grok 4.7 beating Grok 4.6 by about 2.3–2.4 points.", "body_md": "# New 'Access to Justice' Benchmark Finds Leading AI Models Miss Critical Legal Details Across 252 Scenarios\n\nVals AI and NextLadder Ventures released the Access to Justice Benchmark on November 10, evaluating 23 AI models on 252 simulated legal-aid conversations. No model reached the 80% threshold the organizers set for a reliable first point of contact. The top performer, Alibaba's Qwen 3.8 Max, scored 76.4% [\\[1\\]](https://www.prnewswire.com/news-releases/new-test-of-leading-ai-models-legal-aid-advice-finds-key-gaps-still-to-be-closed-across-issues-commonly-faced-by-low-income-americans-302935069.html)[\\[3\\]](https://benchlm.ai/benchmarks/access-to-justice).\n\nThe benchmark was built to test how well general-purpose models handle the civil legal problems low-income Americans actually face — evictions, child custody, debt collection, domestic violence filings — rather than the corporate legal work most legal AI benchmarks measure [\\[1\\]](https://www.prnewswire.com/news-releases/new-test-of-leading-ai-models-legal-aid-advice-finds-key-gaps-still-to-be-closed-across-issues-commonly-faced-by-low-income-americans-302935069.html)[\\[4\\]](https://www.unite.ai/vals-ai-legal-aid-benchmark-finds-models-fall-short-on-local-help). Six legal aid experts in California, Louisiana, Virginia, New York, Maryland and Texas adapted real queries from legal aid organizations into scenarios with background facts withheld unless the model asked [\\[4\\]](https://www.unite.ai/vals-ai-legal-aid-benchmark-finds-models-fall-short-on-local-help)[\\[7\\]](https://www.vals.ai/benchmarks/a2j).\n\n### Models explain law but miss local help\n\nEvery model met 78–87% of criteria for general legal explanations, but only 30–50% for identifying specific resources such as a local legal aid program, court self-help center or particular form [\\[1\\]](https://www.prnewswire.com/news-releases/new-test-of-leading-ai-models-legal-aid-advice-finds-key-gaps-still-to-be-closed-across-issues-commonly-faced-by-low-income-americans-302935069.html)[\\[7\\]](https://www.vals.ai/benchmarks/a2j). Twenty-one of 23 models fell back on national directories — LawHelp.org, the ABA's Find Legal Help, 211 — in at least 70% of their final answers instead of naming the relevant local office [\\[4\\]](https://www.unite.ai/vals-ai-legal-aid-benchmark-finds-models-fall-short-on-local-help)[\\[7\\]](https://www.vals.ai/benchmarks/a2j).\n\n\"Models read enough law to explain it, but rarely open the local self-help, legal aid or court pages that would let them name the specific program or office a person should go to next,\" Vals AI's report states [\\[7\\]](https://www.vals.ai/benchmarks/a2j). Statutes, court rules and case law made up 42% of pages models read; legal aid and nonprofit self-help sites accounted for 10% [\\[4\\]](https://www.unite.ai/vals-ai-legal-aid-benchmark-finds-models-fall-short-on-local-help)[\\[7\\]](https://www.vals.ai/benchmarks/a2j).\n\n### Free-tier models keep pace with premium\n\nFree-tier chatbot models averaged 69.7% and premium ones 70.4%, a gap of 0.7 percentage points [\\[1\\]](https://www.prnewswire.com/news-releases/new-test-of-leading-ai-models-legal-aid-advice-finds-key-gaps-still-to-be-closed-across-issues-commonly-faced-by-low-income-americans-302935069.html)[\\[7\\]](https://www.vals.ai/benchmarks/a2j). Anthropic's free-tier Claude Sonnet 5.5 scored 1.2 points higher than its premium Claude Opus 5.5. Only xAI showed a clear premium advantage: Grok 4.7 beat Grok 4.6 by about 2.3–2.4 points, larger than the margin of error [\\[1\\]](https://www.prnewswire.com/news-releases/new-test-of-leading-ai-models-legal-aid-advice-finds-key-gaps-still-to-be-closed-across-issues-commonly-faced-by-low-income-americans-302935069.html)[\\[7\\]](https://www.vals.ai/benchmarks/a2j). The tier assignments are Vals AI's estimates, not confirmed provider mappings, and API tests cannot fully capture consumer chatbot behavior [\\[4\\]](https://www.unite.ai/vals-ai-legal-aid-benchmark-finds-models-fall-short-on-local-help).\n\n### Wide variance across legal topics\n\nHousing and landlord-tenant questions scored highest across all models at 75.4%. Public benefits ranked lowest at 57.3%, though that category contained only five scenarios [\\[1\\]](https://www.prnewswire.com/news-releases/new-test-of-leading-ai-models-legal-aid-advice-finds-key-gaps-still-to-be-closed-across-issues-commonly-faced-by-low-income-americans-302935069.html)[\\[4\\]](https://www.unite.ai/vals-ai-legal-aid-benchmark-finds-models-fall-short-on-local-help). Family law, the largest topic with 110 scenarios, averaged 71.2% but varied sharply by subtopic: visitation (61.3%) and kinship caregiving with child protective services (63.0%) scored lowest, while guardianship, divorce finances, family-member adoptions and domestic violence fell between 66% and 68% [\\[1\\]](https://www.prnewswire.com/news-releases/new-test-of-leading-ai-models-legal-aid-advice-finds-key-gaps-still-to-be-closed-across-issues-commonly-faced-by-low-income-americans-302935069.html)[\\[4\\]](https://www.unite.ai/vals-ai-legal-aid-benchmark-finds-models-fall-short-on-local-help).\n\n### Two examples show the gaps\n\nIn a Louisiana small-claims scenario, a New Orleans parent asked about suing the other parent over a verbal repayment promise. Most models cited the correct court but incorrectly cited a 10-year prescriptive period; the correct period for money lent is three years under La. C.C. art. 3494, as the benchmark's methodology page clarifies [\\[7\\]](https://www.vals.ai/benchmarks/a2j). The press release and some coverage repeated the 10-year figure [\\[1\\]](https://www.prnewswire.com/news-releases/new-test-of-leading-ai-models-legal-aid-advice-finds-key-gaps-still-to-be-closed-across-issues-commonly-faced-by-low-income-americans-302935069.html)[\\[5\\]](https://agentlocker.ai/news/ai-chatbots-explain-the-law-well-but-often-fail-to-point-people-to-local-help). Only 8 of 23 models asked whether the debt involved real estate — which decides whether a verbal agreement can be enforced at all. The expert rubric evaluated whether models warned that the claim is lost once the 10-year period runs out (the figure most models cited); 4 of 23 did so. Only 3 directed the user to the court clerk's website for claim and fee-waiver forms [\\[1\\]](https://www.prnewswire.com/news-releases/new-test-of-leading-ai-models-legal-aid-advice-finds-key-gaps-still-to-be-closed-across-issues-commonly-faced-by-low-income-americans-302935069.html)[\\[5\\]](https://agentlocker.ai/news/ai-chatbots-explain-the-law-well-but-often-fail-to-point-people-to-local-help).\n\nIn a New York City constructive eviction scenario, a tenant who had moved out asked how to recover an unreturned deposit. Nearly every model cited the landlord's 14-day deadline and laid out a demand letter followed by small claims filing. Fourteen models were marked down for pushing that out-of-court step, which the expert rubric warned against. Only 4 referred the tenant to NYC's Right to Counsel program; most offered general hotlines like 311 or Legal Services NYC [\\[1\\]](https://www.prnewswire.com/news-releases/new-test-of-leading-ai-models-legal-aid-advice-finds-key-gaps-still-to-be-closed-across-issues-commonly-faced-by-low-income-americans-302935069.html)[\\[5\\]](https://agentlocker.ai/news/ai-chatbots-explain-the-law-well-but-often-fail-to-point-people-to-local-help).\n\n### Context: the justice gap\n\nThe benchmark is part of a NextLadder Ventures effort to accelerate AI tools for low-income Americans. The Legal Services Corporation's 2022 Justice Gap study found low-income Americans did not get any or enough legal help for 92% of their civil legal problems; 46% of those who didn't seek help cited cost [\\[1\\]](https://www.prnewswire.com/news-releases/new-test-of-leading-ai-models-legal-aid-advice-finds-key-gaps-still-to-be-closed-across-issues-commonly-faced-by-low-income-americans-302935069.html)[\\[8\\]](https://www.nextladder.com/news/new-test-of-leading-ai-models-legal-aid-advice-finds-key-gaps-still-to-be-closed-across-issues). NextLadder, backed by over $1 billion, aims to empower millions by 2040 [\\[1\\]](https://www.prnewswire.com/news-releases/new-test-of-leading-ai-models-legal-aid-advice-finds-key-gaps-still-to-be-closed-across-issues-commonly-faced-by-low-income-americans-302935069.html)[\\[8\\]](https://www.nextladder.com/news/new-test-of-leading-ai-models-legal-aid-advice-finds-key-gaps-still-to-be-closed-across-issues). Vals AI, a Series A company backed by a16z, provides independent benchmarking used by frontier labs [\\[1\\]](https://www.prnewswire.com/news-releases/new-test-of-leading-ai-models-legal-aid-advice-finds-key-gaps-still-to-be-closed-across-issues-commonly-faced-by-low-income-americans-302935069.html)[\\[9\\]](https://www.vals.ai/methodology).\n\n### What's missing\n\nThe full dataset contains 504 scenarios split evenly between a public test set and a private validation set; only the 252 public scenarios were used for published results [\\[4\\]](https://www.unite.ai/vals-ai-legal-aid-benchmark-finds-models-fall-short-on-local-help)[\\[7\\]](https://www.vals.ai/benchmarks/a2j). Models were run through their APIs, which approximates but does not reproduce the consumer chatbot experience [\\[3\\]](https://benchlm.ai/benchmarks/access-to-justice). Content-filter refusals counted as zero — Muse Spark 1.3 scored 41.5% on criminal and records questions because its provider blocked five scenarios on every retry [\\[7\\]](https://www.vals.ai/benchmarks/a2j). Vals AI and NextLadder say they are collaborating on a fuller report and a practitioner brief to help frontline workers understand remaining gaps [\\[1\\]](https://www.prnewswire.com/news-releases/new-test-of-leading-ai-models-legal-aid-advice-finds-key-gaps-still-to-be-closed-across-issues-commonly-faced-by-low-income-americans-302935069.html)[\\[4\\]](https://www.unite.ai/vals-ai-legal-aid-benchmark-finds-models-fall-short-on-local-help).\n\n## Sources\n\n1. [New Test of Leading AI Models' Legal Aid Advice Finds ...](https://www.prnewswire.com/news-releases/new-test-of-leading-ai-models-legal-aid-advice-finds-key-gaps-still-to-be-closed-across-issues-commonly-faced-by-low-income-americans-302935069.html) (prnewswire.com)\n2. [New Test of Leading AI Models' Legal Aid Advice Finds ...](https://www.morningstar.com/news/pr-newswire/20261110ph92963/new-test-of-leading-ai-models-legal-aid-advice-finds-key-gaps-still-to-be-closed-across-issues-commonly-faced-by-low-income-americans) (morningstar.com)\n3. [Access to Justice Leaderboard & Scores — November 2026](https://benchlm.ai/benchmarks/access-to-justice) (benchlm.ai)\n4. [Vals AI Legal Aid Benchmark Finds Models Fall Short on ...](https://www.unite.ai/vals-ai-legal-aid-benchmark-finds-models-fall-short-on-local-help) (unite.ai)\n5. [AI Chatbots Explain the Law Well but Often Fail to Point ...](https://agentlocker.ai/news/ai-chatbots-explain-the-law-well-but-often-fail-to-point-people-to-local-help) (agentlocker.ai)\n6. [Leading AI Models Still Missing Critical Details When Giving Financial Advice, New Research Finds](https://www.prnewswire.com/news-releases/leading-ai-models-still-missing-critical-details-when-giving-financial-advice-new-research-finds-302938452.html) (prnewswire.com)\n7. [Access to Justice Benchmark Leaderboard and Methodology | Vals AI](https://www.vals.ai/benchmarks/a2j) (vals.ai)\n8. [New Test of Leading AI Models’ Legal Aid Advice Finds Key Gaps Still to be Closed Across Issues Commonly Faced by Low-Income Americans — NextLadder](https://www.nextladder.com/news/new-test-of-leading-ai-models-legal-aid-advice-finds-key-gaps-still-to-be-closed-across-issues) (nextladder.com)\n9. [AI Evaluation Methodology | Vals AI](https://www.vals.ai/methodology) (vals.ai)", "url": "https://wpnews.pro/news/new-access-to-justice-benchmark-finds-leading-ai-models-miss-critical-legal-252", "canonical_source": "https://epyhia.com/new-access-to-justice-benchmark-finds-leading-ai-models-miss-critical", "published_at": "2026-11-14 10:09:18+00:00", "updated_at": "2026-11-14 10:22:19.667819+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-safety"], "entities": ["Vals AI", "NextLadder Ventures", "Alibaba", "Qwen 3.8 Max", "Anthropic", "Claude Sonnet 5.5", "Claude Opus 5.5", "xAI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/new-access-to-justice-benchmark-finds-leading-ai-models-miss-critical-legal-252", "markdown": "https://wpnews.pro/news/new-access-to-justice-benchmark-finds-leading-ai-models-miss-critical-legal-252.md", "text": "https://wpnews.pro/news/new-access-to-justice-benchmark-finds-leading-ai-models-miss-critical-legal-252.txt", "jsonld": "https://wpnews.pro/news/new-access-to-justice-benchmark-finds-leading-ai-models-miss-critical-legal-252.jsonld"}}