BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution

Frontier models now exceed 99% Pass@1 on LiveCodeBench easy, saturating our benchmarks. BenchEvolver automatically transforms existing coding problems into substantially harder, verified variants — tasks that are hard even for the model that created them.

On LiveCodeBench, state-of-the-art models exceed 99% Pass@1 on the newest easy split and over 90% on average. Building new, sufficiently hard datasets by hand is slow and expensive — a bottleneck for continued progress.

Most benchmark generation methods are problem-centric: they start by writing a new task and hope it requires new reasoning. In practice, this often produces surface-level variants of existing problems, while still relying on increasingly strong models to solve and validate them. BenchEvolver flips the direction. We evolve solutions first, then derive tasks from them. Because the reasoning structure changes before the problem statement is written, the resulting benchmarks impose genuinely new algorithmic demands while retaining executable ground truth by construction.

Mutate the reference solution to force a dominant algorithmic lift, then derive the statement and tests around the evolved, executable solution.

Brute-force triangulation and statement-faithfulness checks ensure statement, solution, and tests define the same task — not a single LLM judge.

Difficulty is measured, not assigned: a candidate is accepted only if a panel of target models empirically fails more than on the seed.

The surface story stays familiar; the underlying computation jumps to a different regime. The same solution-centric principle works across two very different coding domains.

Count arrays whose adjacent differences match the original and whose entries satisfy per-index bounds.

Now adjacent XORs must match. The feasible sets are no longer contiguous — interval intersection fails.

Implement a classical fourth-order Runge–Kutta integrator for a driven damped pendulum, returning the full state-space trajectory.

A Proposer evolves solutions and writes tasks; an Evaluator validates and measures empirical difficulty; a Memory module feeds accepted lineages and past failures back into search — turning repeated sampling into adaptive evolution.

Mutates the parent solution into a structurally different one, then derives a natural statement, public examples, and tiered hidden tests — all anchored by executing the evolved reference.

Triangulates the reference, a brute-force solver, and a statement-only oracle to catch inconsistencies; runs bounded repair; then accepts only if the target panel empirically fails more.

Local memory tracks each seed's lineage and error patterns; global memory enforces diversity across seeds — a family that already succeeded must clear a higher difficulty bar.

Across two domains, four target models, and multiple evolvers, evolved problems consistently and substantially reduce Pass@1 relative to their seeds. Crucially, each evolver also drops on its own evolved tasks — this is self-challenging generation, not teacher-to-student distillation.

Six competitive-programming experts (Codeforces master / IOI / ICPC level) blindly reviewed 207 evolved problems across 72 seeds. Evolved tasks are rated more novel and far more difficult, span a much broader algorithmic surface — and are actually rated clearer than their seeds.

A 91-problem benchmark combining 64 human-vetted evolved tasks with 27 difficult original LCB-v6 problems. Every problem passes correctness, quality (≥3/5, Olympiad standard), and difficulty-range gates. Frontier Pass@1 spans 27.5%–62.6% — restoring clear discrimination among the strongest models.

Reasoning settings: GPT models use medium reasoning effort, DeepSeek uses high, and Gemini models use adaptive reasoning.

Averaged across all evaluated models, the Hard split drops from 87.0% → 45.7% Pass@1 — an absolute reduction of 41.3 points.

Using gpt-oss-20b as both evolver and target, we evolve problems it already solves into harder verified variants, then train on them with RL. Evolved tasks improve held-out coding performance beyond training on the original seeds alone — the same model exposes and then learns from its own weaknesses. Any fixed benchmark eventually saturates. BenchEvolver points to a different model of evaluation: a reproducible pipeline that periodically generates, validates, and calibrates new tasks against current frontier models — aligning evaluation with training, so the same verified tasks that reveal failures also become the environments that fix them.

Difficulty is measured by executable model failure — including the generator's own. Frontier models can expose and train on their own weaknesses, not just distill from a larger model.

Only the execution harness is domain-specific. The same mutate → write → verify → select loop works on stdin/stdout competitive programming and assertion-based scientific coding alike.

If you find our work useful, please consider citing: