# Specific Labs launches Real-SWE, where 38.8% is enough for first place > Source: > Published: 2026-09-12 21:27:42+00:00 # Specific Labs launches Real-SWE, where 38.8% is enough for first place **Specific Labs co-founders Janak Sunil and Siddhant Paliwal built the benchmark from licensed production code, turning a recurring lab question into a data product.** By [RuntimeWire Staff](/author/runtimewire-staff) ยท Published Primary source: [Specific Labs](https://withspecific.com/benchmarks/real-swe) ## Why it matters Public coding scores can reward familiarity with visible repositories. Real-SWE tests unfamiliar production systems while giving Specific a route from evaluation failures to training-data customers. [Specific Labs](https://withspecific.com/?ref=runtimewire) co-founders [Janak Sunil (@janaksunil)](https://x.com/janaksunil?ref=runtimewire) and [Siddhant Paliwal (@siddhantpa1iwal)](https://x.com/siddhantpa1iwal?ref=runtimewire) have launched [Real-SWE](https://withspecific.com/benchmarks/real-swe?ref=runtimewire), a benchmark that puts coding agents to work on licensed, private production codebases. The assignments cover billing, taxes, customer migrations and infrastructure changes rather than neatly isolated bugs from public repositories. Specific's benchmark grows out of its work with enterprises and frontier labs to turn company workflows into datasets. [Y Combinator's company page](https://www.ycombinator.com/companies/specific-labs?ref=runtimewire) says Specific acquires and licenses operational data and codebases. Sunil, who studied econometrics and quantitative economics at UCLA, previously worked at Coinbase. Paliwal previously co-founded Third Chair, another YC-backed company, and worked at Intel. YC backed Specific in its Fall 2025 batch and lists the company as founded in 2025 with a two-person team. Real-SWE turns the benchmark's central question into a product aligned with the founders' broader business. Specific builds datasets from the decisions, exceptions and handoffs inside real companies, material that rarely appears in public training corpora. ### The leaderboard tops out below 40% [Specific's published leaderboard](https://withspecific.com/benchmarks/real-swe?ref=runtimewire) reports that Fable 5.1 running through Claude Code led Real-SWE with a 38.8% resolution rate. GPT-6 Astra with Codex CLI followed at 33.8%, while [Gemini 3.8 Flash](/models/google/gemini-3.8-flash) with Gemini CLI scored 31.2%. [GLM 5.3](/models/z-ai/glm-5.3) using Claude Code placed fourth at 28.8%. The remaining combinations finished below 25%. [Grok 4.6](/models/azure/grok-4.6) with Grok Build and Muse Spark 1.3 with Muse Code each scored 23.8%. Kimi K3 with Kimi Code reached 18.8%, and GPT-5.6 Sol with Codex CLI scored 16.2%. Specific defines resolution rate as pass@1 averaged across eight independent runs per task. The published analysis shows ten tasks, eight model-and-harness combinations and eight attempts for each pairing, producing 640 rollouts. Specific's published analysis says six of the ten analyzed tasks had aggregate resolution rates below 15%. The supplied release does not disclose the full task set or enough methodology for outside researchers to reproduce the result. The low resolution rates point to a practical risk for businesses deploying coding agents. A plausible patch can still apply the wrong tax rule, break an existing workflow or update code the running service never calls. ### Private code makes the test harder to game Real-SWE's central design choice is keeping the underlying software and solutions away from the public internet. Public coding benchmarks can expose repositories, issues and eventual patches that may enter model training data. A private codebase forces an agent to read the system in front of it and infer conventions from the available workspace. Specific says its sample codebases include an events product with more than 200,000 users, a consumer fintech platform that has processed more than 100,000 bank statements, and enterprise sales software. The businesses remain anonymous, so those descriptions and usage figures are Specific's claims. One sample assignment asks an agent to repair invoice taxation across businesses with different tax configurations. The agent must account for customer exemptions, buyer destinations, European VAT registrations and separate sandbox and production tax services. It must also record rates and totals on invoices and report settled sales to an outside authority. The environments can expose PostgreSQL, MySQL, MongoDB, Redis, Kubernetes, Docker, Slack, Intercom, Google Drive, ClickUp and other services. Specific uses each model's native coding harness, measuring a model-and-harness combination instead of holding the agent scaffold constant. That choice reflects how developers buy and use coding systems, though it limits what the ranking can prove about the underlying models. A score can capture differences in tool use, context handling and orchestration alongside model capability. The leaderboard should be read as a comparison of complete coding setups, not an isolated model contest. For eight repository-backed sample tasks, Specific reports a median instruction length of 1,742 characters and a median reference solution touching 11 files. Short prompts therefore leave substantial discovery work to the agent, including locating the correct services and reconstructing business rules from existing code. The privacy advantage comes with an auditability cost. The public release identifies ten analyzed assignments but does not specify the size of the full task set, the total number of licensed codebases or their distribution by language and company type. Outside researchers also cannot inspect the proprietary repositories and reference solutions. Buyers must weigh contamination resistance against a methodology they cannot fully reproduce. ### A benchmark that feeds Specific's core business Real-SWE arrives amid a push toward harder coding evaluations. [Scale AI's SWE-Bench Pro](https://scale.com/blog/swe-bench-pro?ref=runtimewire) contains 1,865 instances across 41 repositories, including 276 commercial tasks. [DeepSWE](https://deepswe.datacurve.ai/?ref=runtimewire) uses 113 original, longer-horizon tasks across 91 repositories. [CCBench](https://ccbench.org/?ref=runtimewire) tests agents on roughly 180 tasks derived from private user submissions involving smaller codebases. The scores across these benchmarks are not directly comparable. Their repositories, prompts, verification systems, time limits and agent scaffolds differ. Real-SWE's narrower proposition is that an evaluation should preserve the business context around work already performed inside an operating company. That proposition also gives Sunil and Paliwal a direct commercial path. On [Specific's YC launch page](https://www.ycombinator.com/companies/specific-labs?ref=runtimewire), the company invites AI labs to submit models for evaluation or request the full task set for training. It also asks businesses to contribute codebases, with anonymization and private task handling. Those offers tie the benchmark directly to Specific's work creating datasets from private company operations. Real-SWE therefore functions as both a benchmark and a customer-acquisition channel for Specific's dataset business. Every failed billing migration or missed permission rule makes the founders' case that frontier models still need access to the operational knowledge stored inside private organizations. The first leaderboard gives them plenty of evidence. The strongest tested setup resolved fewer than four tasks in ten, and six of the ten analyzed tasks had resolution rates below 15%. Sunil and Paliwal are betting that proprietary, company-specific data can help close that gap.