GPT-6.2 Astra tops Vals AI's new multi-turn coding benchmark, which checks whether agents can keep extending a working app through up to ten requests without breaking existing features.
- GPT-6.2 Astra leads Vibe Code Bench 1-100 at 55.16%, ahead of Gemini 4 Carbon's 49.73%.
- The benchmark tests whether agents can extend a working web app through up to ten dependent requests.
- If a new feature breaks an existing one, the task ends there and no later steps count.
- Astra had no failures where only old behavior broke, ran fastest, and did best on schema migrations.
- GPT-6.2 Sol scores 40.94% at $12.74 per task, under a third of Astra's $45.66.
- Astra is available via the API at $8/$40 per million tokens, plus ChatGPT, Codex, andCopilot .
Most coding benchmarks give a model one prompt and grade its first answer. Vals AI’s Vibe Code Bench 1-100 measures sequential maintenance: adding features to a working application while preserving existing behavior. OpenAI’s newly released GPT-6.2 Astra leads the VCB 1-100 results with 55.16%, followed by Gemini 4 Carbon at 49.73% and GPT-6.2 Sol at 40.94%.
The test begins with working software #
The original greenfield benchmark asks a model to build an application from scratch. Vibe Code Bench 1-100 starts with a complete web application whose tests already pass, then gives the model a sequence of feature requests. Later requests depend on earlier features, data models, permissions, and user journeys.
Each request arrives in plain English, similar to a product ticket. Across 50 scenarios, the benchmark contains 456 requests and 6,022 browser workflows. Regression checks account for 26.1% of those workflows, testing whether earlier behavior survives each change.
A controlled, tool-rich test bed
- Environment: Every model uses the same modified OpenHands V1 agent environment with terminal access.
- Services: Supabase supplies database and authentication infrastructure, Stripe test services handle payments, and MailHog captures test email.
- Time limit: Each scenario receives one 10-hour generation budget shared across all its iterations.
- Grading: A GPT-6 Luna browser agent tests the application after each change. Claude Sonnet 5.5 reviews every reported failure against recorded evidence.
- Workload: Agents processed about 46.3 million tokens per scenario on average and made a median of 136 file-editor calls, about twice the original benchmark’s total.
Vals divides the requests into five categories, led by cross-cutting changes that apply one rule across several parts of an application.
| Request mix | |
|---|---|
| Request type | Share |
| --- | --- |
| Cross-cutting rule | 35.31% |
| Remove or revert | 18.86% |
| Bug fix | 18.64% |
| Add feature | 17.98% |
| Schema migration | 9.21% |
Schema migrations change the data model while retaining existing records. Cross-cutting tasks can touch authorization, interfaces, database queries, and business logic in the same iteration, making consistency across the codebase part of the test.
One failed iteration ends the attempt #
Vals uses prefix scoring: a scenario receives credit only for consecutive successful iterations from its start. Each iteration must pass checks for the new request and existing behavior. The first failure stops the scenario, and the leaderboard score is the mean result across all 50 scenarios.
def task_score(iteration_results, planned):
passed = 0
for ok in iteration_results: # acceptance and regression checks
if not ok:
break
passed += 1
return passed / planned
A failure halfway through leaves credit for earlier iterations, while all later planned requests contribute zero. The scoring method therefore rewards models that can sustain correct changes across a long sequence.
Astra gains ground through clean migrations #
GPT-6.2 Astra completed every planned request in 18 of 50 scenarios, the highest total among the tested models. Gemini 4 Carbon completed 10. Vals identifies three factors behind Astra’s lead:
- No regression-only failures: Every Astra failure included at least one failed check for the new request. None occurred solely because an earlier feature broke. Carbon and Claude Opus 5.5 each recorded three regression-only failures.
- Faster runs: Astra averaged 2 hours 48 minutes per scenario and never exhausted the 10-hour budget.
- Stronger migrations: Astra passed 88% of schema-migration iterations, compared with 68% for Carbon.
Astra averaged $45.66 per scenario, costing less than Carbon and Opus 5.5 while exceeding Sol’s cost. These figures reflect the benchmark’s agent configuration and workloads rather than flat API charges.
| Selected leaderboard results | |||
|---|---|---|---|
| Model | Score | Cost per scenario | Time per scenario |
| --- | --- | --- | --- |
| GPT-6.2 Astra | 55.16% | $45.66 | 2 h 48 m |
| Gemini 4 Carbon | 49.73% | $57.92 | 4 h 07 m |
| GPT-6.2 Sol | 40.94% | $12.74 | 2 h 31 m |
| Claude Opus 5.5 | 30.95% | $115.10 | 6 h 46 m |
| MiMo V2.6 Flash | 24.50% | $0.44 | Not reported |
Sol scored 14.22 percentage points below Astra and cost about 72% less per scenario. MiMo V2.6 Flash delivered a lower score at $0.44 per scenario, giving high-volume agent deployments a much cheaper option.
Claude Opus 5.5 was the only entry whose run used fallback models. Twelve of its 50 scenarios triggered 16 fallback events, with Claude Opus 4.8 or Claude Opus 5 taking over a request. Restricting each affected scenario to work completed before its first fallback reduces the score from 30.95% to 23.55%.
New features cause most failures #
Seven of Astra’s 50 scenarios earned no credit because the model failed the first request. Across all tested models, failures were concentrated in checks for newly requested behavior.
| Failure types across all models | |
|---|---|
| Failed checks | Share of failures |
| --- | --- |
| New request only | 73.81% |
| Regression only | 5.29% |
| New request and regression | 20.90% |
The distribution identifies incomplete feature implementation as the main bottleneck. Regressions appeared in 26.19% of failures, including cases where both the new feature and old behavior failed.
Vals published examples that resemble routine production defects. One default booking flow displayed no available times until the user selected a custom service. Another implementation deleted a former member’s historical RSVP instead of preserving the record with identifying information removed.
The comparison has two gaps #
Anthropic’s Claude Fable 5.2 is absent from the leaderboard. Artificial Analysis places Fable ahead of Astra on its broader index, where Astra ranks second, so Vibe Code Bench does not compare the two directly.
Vals also keeps the benchmark proprietary, preventing developers and independent researchers from running the complete evaluation on their own models or agent configurations.
Where developers can use Astra #
OpenAI offers GPT-6.2 Astra and Sol in ChatGPT Work and Codex for Plus, Pro, Enterprise, and Edu users. API customers can call the models as gpt-6.2-astra and gpt-6.2-sol.
GPT-6.2 Astra costs $8 per million input tokens, $0.40 per million cached input tokens, and $40 per million output tokens. Both models are also available through GitHub Copilot for Copilot Pro+, Max, Business, and Enterprise customers.
Vals configured Astra with a 1 million-token context window, a 128,000-token maximum output, and maximum reasoning effort. Lower-compute settings may produce different scores, costs, and completion times.
Maintenance changes the model order #
Artificial Analysis’s Coding Agent Index, measured with the Codex harness at maximum compute, shows no movement between GPT-6 Astra and GPT-6.2 Astra. Vibe Code Bench produces a different ranking because it measures repeated modifications to an existing application, including migrations, reverts, and regression control.
OpenAI’s launch post emphasizes professional and scientific workloads. GitHub’s early testing also found that Astra completed agentic coding tasks with fewer tokens, consistent with its comparatively short benchmark runs.
Among the evaluated models, Astra produced the highest completion score for long sessions on an existing application. Sol offers a lower-cost option when a human can review each step and rerun failed work.
Astra completed every request in 18 of 50 scenarios and failed the first request in seven. Those results support a supervised workflow with acceptance checks for each feature, regression tests after every agent turn, and staged review for schema migrations and destructive operations.