{"slug": "sherpa-labs-launches-ai-benchmarks-built-around-finance-deals-and-production", "title": "Sherpa Labs launches AI benchmarks built around finance deals and production code", "summary": "Sherpa Labs on October 20th released finance and coding benchmarks for AI agents, and founder Justin Jiang reported a 3.6% overall pass rate across 3,750 runs on the finance test, which is drawn from private-equity deal work. Jiang said Sherpa Labs is building reinforcement-learning environments for finance, coding and quantitative research, and that the low scores expose gaps in long-horizon professional work where formulas, files and software changes must tie together. The company is selling infrastructure to train and measure agents on those workflows.", "body_md": "# Sherpa Labs launches AI benchmarks built around finance deals and production code\n\n**Founder Justin Jiang says the benchmarks expose gaps in long-horizon work; the finance test recorded a 3.6% overall pass rate across 3,750 runs.**\n\n        By [Ryan Merket](https://runtimewire.com/author/ryan-merket)\n        · Published \n\nPrimary source: [X](https://x.com/jostinjiang/status/2112616822092755385)\n\n## Why it matters\n\nSherpa Labs is selling the infrastructure to train and measure agents on professional workflows, where a plausible answer can still fail when formulas, files or software changes do not tie together. Its early scores illustrate the gap the company hopes labs will pay it to close.\n\nOn October 20th, [Justin Jiang (@jostinjiang)](https://x.com/jostinjiang) introduced Sherpa Labs' finance and coding benchmarks for AI agents, tests built around completing professional work rather than answering isolated questions. Jiang says the company is building [reinforcement-learning](https://runtimewire.com/models/huggingface/ydiffraction-reinforcement-learning-7897455e5c11c599) environments for finance, coding and quantitative research, with the first two benchmarks reporting low pass rates on tasks drawn from private-equity deal work and a production software repository.…\n\nWhen you purchase through links in our articles, [we may earn a small commission](https://runtimewire.com/affiliate-monetization-standards). This doesn’t affect our editorial independence.", "url": "https://wpnews.pro/news/sherpa-labs-launches-ai-benchmarks-built-around-finance-deals-and-production", "canonical_source": "https://runtimewire.com/article/sherpa-labs-finance-coding-ai-benchmarks", "published_at": "2026-10-21 01:27:38+00:00", "updated_at": "2026-10-21 04:48:40.314040+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "ai-startups", "large-language-models", "mlops"], "entities": ["Sherpa Labs", "Justin Jiang", "X"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/sherpa-labs-launches-ai-benchmarks-built-around-finance-deals-and-production", "markdown": "https://wpnews.pro/news/sherpa-labs-launches-ai-benchmarks-built-around-finance-deals-and-production.md", "text": "https://wpnews.pro/news/sherpa-labs-launches-ai-benchmarks-built-around-finance-deals-and-production.txt", "jsonld": "https://wpnews.pro/news/sherpa-labs-launches-ai-benchmarks-built-around-finance-deals-and-production.jsonld"}}