Epoch AI benchmarked Fable 5.2, GPT-6.2 Astra, and four other frontier models on real workplace tasks. None met the employee bar.
- No model met Epoch's employee bar on Epoch Automation Reports , despite top ECI scores.
- Fable 5.2 averaged 82%, GPT-6.2 Astra 65%, even with tied ECI of 174.
- Fable 5.2 and Opus 5.5 hit contractor-level on AI data center analysis (97.5% and 87.5%).
- Opus 5.5 finished in 3 hours what takes a contractor 16, at lower cost.
- Every model reused existing chess variants instead of inventing truly out-of-distribution ones.
- Models preferred their own output over human gold standards in all but one case.
Epoch AI evaluated six frontier systems on open-ended research, writing, and analysis drawn from its employees’ daily work. No model completed any task to the standard Epoch applies to employee submissions, including Fable 5.2 and GPT-6.2 Astra, which share the top position on the Epoch Capabilities Index.
The updated benchmark page tests complete workflows instead of isolated skills. Models must find evidence, make defensible judgments, use tools, evaluate intermediate results, and deliver work suitable for publication. Epoch scored each submission with its internal rubrics; for the data center task, a score of 100% means the output could be published without edits.
The research report found substantial differences between models with similar aggregate rankings. Fable 5.2 averaged 82%, while GPT-6.2 Astra averaged 65%, a 17-point gap despite their roughly tied Epoch Capabilities Index scores. Opus 5.5 and Gemini 4 Argon also failed in distinct parts of the workflow.
A detailed playbook gets one pipeline close #
The strongest results came from a structured data center research task. Epoch asked models to add EcoDataCenter 1 to its public database by researching news and regulatory filings, ordering satellite imagery, identifying relevant buildings, and estimating their computing infrastructure.
Fable 5.2 scored 97.5%, and Opus 5.5 scored 87.5%. Epoch is now considering moving contractor time from initial data collection to reviewing model output for this pipeline.
The time and cost results varied by model. A contractor typically spends about 16 hours on one data center. Opus 5.5 finished in just under three hours at a lower cost, while Fable 5.2 took about 6.5 hours and cost more than a contractor but earned the higher score.
GPT-6.2 Astra’s submission showed how excessive caution can reduce completeness. Its sources indicated that the site contained multiple AI buildings, yet it documented one. The model also omitted compute and chip estimates that the available evidence and task guide supported because it sought a separate source for each figure.
Novel experiments remain a weak spot #
Research design tasks exposed failures in problem framing and validation. Epoch asked models to measure out-of-distribution generalization with chess variants, testing performance under rules that models were unlikely to have encountered during training. Every system selected existing variants, leaving open the possibility that the evaluated models had memorized them.
Gemini 4 Argon ran an experiment showing that the evaluated models already knew its chosen variants, then continued with the invalidated design. GPT-6.2 Astra changed piece labels without changing the rules, using “R” for a knight even though that symbol conventionally denotes a rook. Fable 5.2 proposed “Kingsgate,” which plays identically to an existing variant called “Midline Invasion.”
A steganography task revealed a related failure to iterate. Models had to design prompts that induced other models to hide messages inside SVG files, a text-based image format. Fable 5.2 submitted its first attempt even though most generated images displayed the supposedly hidden message directly on the canvas. Human researchers revised their approach until they produced functional encodings, including one that mapped star heights to letter positions.
Self-grading rewards the wrong signals #
Epoch also asked each model to compare its output with a human gold-standard submission using the organization’s internal rubric. The comparison concealed which submission came from the human. Every model selected its own work in every case except one, when Gemini 4 Argon preferred the human steganography submission.
The models’ explanations showed three recurring evaluation errors:
- They magnified minor flaws in the human submissions.
- They rewarded visible signs of rigor, such as a larger number of experiments, without checking whether those experiments answered the research question.
- They overlooked the value of narrower experiments with cleaner controls and more relevant evidence.
Gemini 4 Argon’s lone preference for human work followed a stronger evaluation process. It recognized that the human submission tested the central research question directly and criticized its own experiments as “superficially more formal.” The broader results indicate that uncalibrated model-as-judge systems can approve weak outputs when the judge shares the generator’s habits.
Production systems need workflow-level tests #
The results support several concrete design choices for teams deploying frontier models:
- Test the complete workflow. Aggregate benchmark rankings did not predict performance on research tasks that combined browsing, tool use, judgment, and final delivery.
- Encode inference rules. Prompts should specify when estimates are required, what evidence supports them, and which fields must be completed.
- Keep human approval at judgment points. Topic selection, experimental design, interpretation, and publication decisions produced the largest failures.
- Separate generation from acceptance. Use deterministic checks, calibrated independent review, or human evaluation instead of relying on a model to grade similar output.
- Measure quality, runtime, and cost together. Opus 5.5 was faster and cheaper than a contractor on the data center task, while Fable 5.2 earned a higher score at a higher cost.
Epoch plans to keep the benchmark current by refining elicitation methods and retiring tasks that may have entered model training data. For developers, the central result is practical: model selection requires tests built around the exact workflow, evidence standards, tools, and acceptance criteria used in production.