2 months back, I started the Prime Residency with Florian Brand as my peer or shall I say my verifier. My first task was to port the Agents’ Last Exam Linux CLI subset to verifiers. Then one day Florian asked me to look into the data and this quest slowly evolved into finding issues in the benchmark and the making of what we call ALE-Gold.
It was more challenging than I expected. I was new to this kind of work, so there was some skill issue on my part. Along the way, I also hit benchmark defects, inference provider failures, and gaps in the freshly released Verifiers v1 (that have been fixed now so you don't need to worry). This post is a work log of my porting work, what the full runs revealed about the benchmark, how that led to ALE-Gold, followed by an analysis of how the models performed.
A small odyssey from porting ALE to ALE-Gold, with a few unexpected detours.
Agents’ Last Exam is considered a frontier benchmark made by Berkeley RDI, Dawn Song’s group at UC Berkeley. They worked with more than 300 experts across 100+ institutions to make long horizon economically valuable tasks spanning over 55 domains. 147 tasks are public out of 1500+ task corpus.
My task was to port the Linux CLI subset, which consisted of 105 terminal-only tasks spanning 41 subdomains across 12 broad domain categories.
What’s in ALE Linux CLI?
105 tasks across 12 broad domain categories
Health & medicine
■■■■■■■■■■■■■■■■■■■■■■
22 · 21.0%
Computing & mathematics
■■■■■■■■■■■■■■■■■■■
19 · 18.1%
Life sciences
■■■■■■■■■■■■■■■■■■■
19 · 18.1%
Business & finance
■■■■■■■■■■■■■■
14 · 13.3%
Physical sciences
■■■■■■■■■■■
11 · 10.5%
Engineering
■■■■■■■■
8 · 7.6%
Education & information
■■■
3 · 2.9%
Transport & safety
■■■
3 · 2.9%
Legal
■■
2 · 1.9%
Psychology & neuroscience
■■
2 · 1.9%
Social sciences
■
1 · 1.0%
Other
■
1 · 1.0%
Share of the original 105-task CLI subset. Domains are grouped by task-name prefix throughout this post. Percentages are rounded.
One may wonder “what does it mean to port a benchmark to verifiers”. Verifiers is a library by Prime Intellect that enables you to run “rollouts” in an easy and elegant manner. I mention the word rollout multiple times in this post so it's worth doing a small detour to understand it.
What is a Rollout?
Rollout in context of evaluation and reinforcement learning is defined as a single run of a policy/model in an environment from start to finish, recording the states, actions, and rewards along the way.
For example when doing LLM-RL, you will be running multiple rollouts of a single task and these can have different scores/rewards. The policy parameters are updated using these rewards. If you use GRPO, differences in rewards within a group of rollouts provide a learning signal by showing which rollouts performed better relative to the others. Same task, different attempts. Each rollout has its own sandbox and harness, takes a different path, and receives a score. Timelines and scores are illustrative.
If you will see benchmarks like FrontierSWE or TerminalBench (harbor framework), they are usually divided into three parts - the task’s input in the form of files or docker image, a harness where the model will run like claude code and codex and the runtime (where the rollout happens). The runtime can be your local machine, docker instance or an AWS instance. Verifiers provides this exact abstraction for you to port the benchmark and then run the benchmark on Prime Intellect’s infra including support for several sandbox providers.
The three pieces: a taskset defines the work and scoring, a harness controls the agent’s interaction loop, and a runtime provides the execution environment.
What porting ALE involved and some small challenges I faced along the way
The porting work mainly involves porting the taskset/dataset to verifiers and then writing relevant adapter code between verifiers and original public benchmark code (i refer to as “upstream”) to support evaluation (as it’s defined by upstream). In short, port the input and hook up the scorer code. In addition to this, one may need to write some more compatibility related code depending on the benchmark. The “porting work” itself can vary depending on if the benchmark providers have provided sandbox images or not.
In ALE’s case, images were not provided so I had to make sandbox images for 105 tasks with the mentioned dependency specifications and input files. This was a code intensive and time consuming process and was only made possible due to coding agents.
One must minimise the rollout time
Once the images were made, I would run a rollout or two of the task to verify if the image was fine and the rollout happened successfully. Halfway through my first attempt of making images, I made a mistake. I was not baking the “input files” onto the images. Instead I was down them on runtime. This means the input files were being downloaded from the Huggingface archive during each rollout. Now most of the input files were small in size but for a few tasks, they ranged in GBs and HF started hitting resolver limits… This is when I realised my mistake.
From an evaluation point of view, it’s also a redundant operation and in general people make effort to minimize the rollout time.
From an RL post-training point of view, this is an even bigger issue because you run multiple rollouts of the same task repeatedly during training, so down the same input files for every rollout adds up quickly. RL researchers/engineers strive to reduce the rollout time (though mainly inference side) as it’s often the biggest bottleneck. In a synchronous RL training setup, you need to wait for all rollouts to finish before running the gradient steps so the slowest rollout becomes the bottleneck. You can read a bit about it here (as how Prime Intellect engineers solve for inference throughput).
Missing dependency specs
Another challenge I faced was the lack of sufficient system or library level dependency information for some tasks. This led me to run rollouts, figure out the missing dependencies and then fix the images.
I also adjusted the memory or disk requirements for eight tasks and configured their sandbox resources accordingly.
Full eval runs
Once the images were ready, I started doing full evaluation runs with different models and harnesses to verify my implementation. This is when you start realising how important a reliable infrastructure is and how fast the money burns when you use models via API.
A full eval run with a model like GPT-5.6 Sol could take 6–12 hours, including retries. Some individual tasks ran for hours, and sandbox or inference provider failures meant retrying affected rollouts, making the feedback loop quite slow.
Occasionally, I ran into more specific issues. During my Kimi runs, failures in the ACP integration caused many rollouts to end midway through a task.
Running these full evaluations also surfaced infrastructure issues, which I reported to the Prime team. Verifiers v1 had just been released, and these reports helped the team identify and address issues in the new release. I also dabbled a bit in the sandbox infra code to see if I could use a “separate sandbox judging runtime” for my benchmark. Some ALE tasks produced output artifacts larger than 1 GB, but transferring artifacts from the rollout sandbox to the judge sandbox was limited to 32 MB at the time. I found out the bottleneck was the sandbox SDK’s lack of support for chunked file streaming.
These runs also helped uncover issues in the benchmark itself. Florian also encouraged me to "look at the data". A closer look revealed more issues. Note: look at the data at your own risk. You never know what crimes you will uncover.
The next section covers the issues found in the ALE Linux CLI subset, the fixes, and the nine tasks excluded because their defects remained unresolved. The resulting subset, ALE-Gold, contains 96 of the original 105 tasks, spanning 40 subdomains.
Look at the data, they said
Before we’re too hard on the creators, I should mention that frontier benchmarks are hard to build. These tasks span specialized domains, involve complicated dependencies, and need verifiers that can distinguish a wrong answer from a valid solution the author didn’t anticipate. Getting all of this right takes a lot of care, both when building the tasks and when validating them.
With that context, here are some of the issues I found while inspecting the tasks and rollout traces.
Underspecified task contracts and evaluator defects
I observed agents were failing 5-6 tasks despite producing correct solutions because the task prompt / input didn’t specify the expected output format correctly. To be more specific, essential schemas, enums, or semantics were absent or ambiguous in the visible prompt.
"frontier benchmark" >rollout has partial score >verifier is deterministic scorer >look inside >verifier expects output format with field names agent can't infer from task prompt even by hallucinating pic.twitter.com/qGpDddm1Mn
The verifier expected several JSON keys and CSV column names that were not mentioned in the input. The prompt also left the domain specific details unclear i.e O’Brien–Fleming implementation, event target, conditional-power futility rule, and follow-up interpretation were unclear.
Here the agent had to reconcile a Moodle course gradebook against a supplied grading policy and produce the final grades and exports. One policy decision was how to handle empty grade entries, but the prompt did not specify the accepted values for empty_grade_behavior.
Across four rollouts, GPT-5.6 Sol chose "count_as_zero", which was a reasonable description of the required behavior. All four produced otherwise correct deliverables but scored 0.95 because the evaluator expected the exact string "zero".
This could have been resolved by mentioning the expected enum in the output contract data. I made relevant changes to the input data to fix this issue and a subsequent rollout scored 1.0.
I was able to fix some of the tasks by minimally modifying the input on the images while others had to be kept out.
Answer Leakage and network issues
ALE does not restrict network as such. It’s a generally online bench and one of the authors mentioned “we keep it online so agent can sometimes fix its dependencies”.
Some tasks require the agent to look up things online or query APIs and these often came with URLs mentioned in the allow-lists. A few tasks enforced network restrictions by mentioning in the prompt.
“The VM has no internet access during solve time.”
The metadata for network policy (which tasks run online or offline) was not public at the time. Upstream later published a list of 5 tasks that must run offline.
I did 2-3 runs with GPT-5.6 Sol (Medium) without network isolation and found a few tasks that looked up solutions online. These tasks are made to run with network isolation now.
My personal favourite examples of agents looking up stuff on the internet were:
computing_math/go_game_reconstruction_1
The goal was to reconstruct a Go game from the evidence staged in the sandbox. Instead, the agent realised that the target may be an identifiable public professional game.
The prompt mentioned “Do not use a Go engine or a web browser.” so the agent proceeded to use curl and download an archive of more than 90,000 professional SGF games, identified the exact historical game, and recovered its 168 moves.
No Go engine or web browser? The agent used curl to find the game in a public archive instead.
health_medicine/wsi_tumor_localization_1
The task asks for the center point of a tumor in a whole-slide pathology image. The agent started by doing the actual image analysis, zooming into suspicious tissue patches.
The input had retained its public filename, tumor_001.tif, which probably motivated the agent to look up online, so the agent searched for the corresponding annotation, tumor_001.xml. It tried GitHub, grep.app, Hugging Face, Google, Bing, DuckDuckGo, the original CAMELYON16 challenge website, and finally Kaggle.
Broken inputs and missing dependencies
In one task, the ordering of the input datasets gave away answers worth 70% of the rubric, allowing the agent to infer them from dataset numbers.
Another task depended on a software package that was no longer downloadable. It had to be excluded until a refreshed task artifact could provide the required dependencies.
Missing or corrupt reference data
For some tasks, the reference data was missing or corrupt. This manifested as bad ground truth data, references being questionable or in 1-2 cases as scientifically questionable. Some of these were fixed upstream (see “Working with upstream” below). The unresolved ones had to be excluded from the subset.
I reported these issues to the ALE authors on Discord and they were fast to respond. They fixed the reference data for a few tasks in the upstream code and published the list of offline-only tasks (PR 1, PR 2).
The mitigations mentioned above in the benchmark issues section led us to the formation of ALE-Gold. “Gold” just means tasks I believe are not bad and measure capability well. Nine tasks had to be excluded, leaving us with 96 tasks.
One detail worth mentioning is that I looked at the inputs and traces of problematic tasks and decided on a case-by-case basis what to include. In some cases, the tasks were badly designed or had some issue, but were still included because they seemed to do fine across multiple rollouts.
Bad tasks
9 excluded tasks · Scroll horizontally to explore →
■ Business & finance01
BPMN governance
Contract mismatch
The scorer rejects the documented schema and requires undisclosed coordination tasks. Either mismatch can force a zero.
■ Computing & mathematics02
Ranking feature recovery
Scorer crash
Scoring crashes after task completion: the evaluator treats a string payload as a mapping.
■ Computing & mathematics03
Causal structure inference
Answer leakage
Dataset order reveals scenario families and answers worth 70% of the rubric, bypassing much of the intended inference.
■ Engineering04
Chisel–Verilog alignment
Reference mismatch
The hidden source-location reference contradicts the supplied comparison evidence, putting 80% of credit at risk.
■ Engineering05
Humanoid policy evaluation
Missing dependency
A required MuJoCo development wheel is no longer downloadable. The archived payload lacks the replacement wheelhouse.
■ Engineering06
OpenROAD chip signoff
Scorer crash
A shell PID-parsing bug crashes scoring before the verifier starts, returning zero even for completed submissions.
■ Health & medicine07
Healthcare bias reproduction
Unenforced constraint
The output-only scorer accepts a direct race bonus prohibited by the prompt; a controlled probe scored 0.861.
■ Health & medicine08
Prostate radiotherapy planning
Scoring timeout
The matRad scorer repeatedly exceeds its declared one-hour scoring timeout, preventing reliable evaluation.
■ Health & medicine09
Variant annotation pipeline
Questionable reference
Hidden truth appears to mishandle deletion normalization and allele-specific ClinVar, rewarding reference-matching errors.
The nine excluded tasks and why I left them out. These issues refer to the task versions used in this evaluation.
ALE-Gold evaluation results
I ran full evaluation runs for 4 latest models at the time which also supported both text and vision capabilities. Opus 5 was passed on due to higher cost and GLM 5.3 because it didn't support vision.
These results are based on the 96-task ALE-Gold subset, with one reported attempt per model per task. The reasoning level and harness were chosen based on the best-performing settings for each model on the official ALE leaderboard. If a rollout didn’t end successfully as indicated by the harness, I retried it, with at least three retries for persistent failures. This included rollouts that stopped during tool call execution.
Aggregate score is the mean task score on a 0–1 scale, including partial credit, with missing scores counted as zero. Pass rate counts only recorded scores exactly equal to 1.0.
ALE-Gold results
Model / configurationAggregate scorePass rate
GPT-5.6 Sol
Medium reasoning · Codex
Aggregate score0.5689
Pass rate28.13%
GPT-5.6 Luna
High reasoning · Codex
Aggregate score0.5566
Pass rate29.17%
Kimi K3
Max reasoning · Kimi Code
Aggregate score0.4686
Pass rate26.04%
GLM 5.3 Flash
Max reasoning · Codex
Aggregate score0.4071
Pass rate19.79%
Kimi K3 and GLM 5.3 Flash had several rollouts ending mid-turn / partial tool call execution. For Kimi K3, it was ACP server issue from their side whereas for GLM 5.3 Flash, the issue seemed to be more due to model's skill issue.
Sol (med) vs Luna (high) Sol vs LunaSol mediumLuna high
Mean score56.89%55.66%
Perfect-pass rate28.13%29.17%
Completion tokens1.12M2.40M
Recorded cost$56.43$16.67
96 shared tasks. Cost is the model-call total recorded in the traces; runtime infrastructure is excluded.
Luna interestingly has a higher pass rate than Sol on ALE-Gold and this matches the score displayed on the official ALE website leaderboard.
Sol and Luna follow similar broad workflows and finish close together: Sol has the higher mean score (0.569 vs 0.557), while Luna has the slightly higher exact-1 pass rate (29.17% vs 28.13%). Across all 96 tasks, Sol scores higher on 23, Luna on 14, and 59 tie. Sol earns more partial credit outside the exact-pass boundary.
Luna having a higher pass rate than Sol also raises the question of how well the benchmark differentiates between models’ capabilities. This suggests there may be scope for improvement in the tasks.
Luna used 114% more completion tokens, including reasoning tokens, and made about 86% more model calls. Its recorded model-call cost was nevertheless about 70.5% lower: $16.67 versus $56.43.
The four tasks below are the ones Luna passed with an exact score of 1.0 while Sol fell short.
SSE trading-rule review
Review the naming rules for northbound programmatic-trading software.
Luna 1.000Sol 0.667
Sol interpreted permission to use English as a prohibition on a French software name. It turned a limited permission into a stronger restriction than the cited rule stated.
My interpretation is that Sol was overly strict in its instruction following.
Model-parallel checkpoint consolidation
Reassemble a model from its distributed checkpoint shards.
Luna 1.000Sol 0.000
Sol replaced embeddings supplied consistently across the checkpoint shards with a guessed initialization. Its own measurements showed that this change worsened the result, but it finalized anyway.
My interpretation is that Sol has a tendency to substitute requested work with an approximation.
Healthcare bias-audit replication
Reproduce a healthcare bias audit using the supplied software environment.
Luna 1.000Sol 0.000
Sol ignored the supplied Python runtime and built a different software stack. The task required numerical reproduction within a tight tolerance, making the runtime change the strongest supported explanation for divergence. The exact failing field was not retained.
Pseudotime differential expression
Identify genes whose expression changes along a cell-development trajectory.
Luna 1.000Sol 0.000
Sol applied an additional gene filter, reducing its result from the broader accepted construction of 2,157 genes to 1,049. That likely pushed recall below the scorer’s threshold.
The instructions were ambiguous about this filter, so some bad behaviour can be attributed to the task specification here.
Where each model performs better than others
Every model has tasks where it outscores all three peers. GLM 5.3 Flash, for example, has the lowest overall mean but is the only model to get full credit on CT geometry calibration. Each card below shows a task, its goal, and the four scores; use the arrows to see a second example for each model. These are selected score advantages, not proof of a general domain specialty.
Scores include partial credit. Each model’s arrows cycle independently. Keyboard: Tab to a model, then use the arrow keys.
Failure modes
Common failure modes
Share of runs that fell short, shown as the range across models. Labels overlap. Bars mark each range on a 0–100% scale.
Verification issueChecks format or self-consistency, but never tries to falsify the result
64–74%
Skill issueUses the wrong method, formula, model, mapping, or implementation
57–71%
Instruction issueSkips, changes, or misreads an explicit requirement
17–34%
Execution issueRepeats, explores, or waits until time runs out without finishing
18–33%
Output issueMisses a required filename, path, header, schema key, unit, or literal
3–13%
Harness issueProvider, tool, or runtime failures interrupt or block the run
6–21%
The models sometimes replaced a difficult workflow with a simpler method that did not meet the original requirements. Their checks could miss the difference because they repeated the same assumptions as the implementation. In other cases, the checks exposed a problem but the model submitted the result anyway. E.g., in checkpoint consolidation, Sol and GLM submitted despite failed numerical checks, and Sol retained a change that made its measured result worse.
There are multiple interpretations of this: skill issue, lack of capability, or model simply picked up a wrong trajectory.
Some runs spent all their time on intermediate work without producing the final deliverable. Repeated experiments, restarts, and monitoring left Sol and Luna without a final CSV in the disordered-protein ensemble scoring task. Other losses came from provider interruptions or the benchmark itself. In the data pipeline extract, transform, and load (ETL) warehouse-construction task, for example, Sol and Luna followed the column order required by the instructions, but the scorer expected the reverse order. These external problems need to be distinguished from model mistakes.
More model-specific failure patterns also stood out.
Sol has a tendency to attempt an approximate version of a task when it finds the work difficult. It can also follow instructions too rigidly, taking things literally and, in my view, showing a lack of common sense at times. Based on my observations, it also showed the most reward-hacking tendencies.
Luna persisted with weak approaches and sometimes ran out of time before producing the deliverable.
Kimi K3 substituted shortcuts for the requested workflow without adequately checking that they met the original requirements.
GLM 5.3 Flash often made implementation errors or used the wrong formulas. My interpretation is that these mistakes reflected the capability and knowledge limitations of a smaller model.
Performance by domain
Across the five domains with at least 10 tasks, the four-model average is highest in business and finance, followed by computing and mathematics, then physical sciences. Sol leads in business and finance, physical sciences, and life sciences; Luna leads in computing and mathematics and health and medicine. These are small observed advantages. Kimi leads education and information, which has only three tasks. GLM has individual task wins but no outright domain-mean lead. Social sciences and legal have higher four-model averages, but only one and two tasks respectively.
Performance by domain
Mean task score01
Domain
Sol
Luna
Kimi K3
GLM 5.3 Flash
Health & medicine 19 tasks
0.487
0.497
0.423
0.410
Life sciences 19 tasks
0.544
0.536
0.450
0.426
Computing & mathematics 17 tasks
0.626
0.633
0.612
0.357
Business & finance 13 tasks
0.695
0.685
0.510
0.477
Physical sciences 11 tasks
0.651
0.612
0.498
0.333
Engineering 5 tasks
0.206
0.207
0.207
0.207
Education & information 3 tasks
0.617
0.611
0.634
0.504
Transport & safety 3 tasks
0.611
0.467
0.534
0.541
Legal 2 tasks
0.834
0.694
0.665
0.667
Psychology & neuroscience 2 tasks
0.404
0.380
0.003
0.381
Other 1 task
0.000
0.000
0.000
0.000
Social sciences 1 task
1.000
1.000
0.000
1.000
Domains with scope for significant improvement
Hello benchmaxxers, this section is for you.
To see domains that are not saturated, let's look at the highest whole domain mean achieved by the models. A low score indicates model lacking capability in that domain. The panel includes domains with at least five tasks where the best model mean is below 0.55.
How much room is left?
Highest model mean · bars run from 0 to 1. Open a domain to explore its results.
Engineering0.2075 tasks · Best: Kimi K33 shared zeros · Explore
Life sciences0.54419 tasks · Best: Sol6 shared zeros · Explore
Sol0.544
Luna0.536
Kimi K30.450
GLM Flash0.426
Scored zero for all four models
Amber minimization workflow
Amber three-stage MM/GBSA workflow
Cell translocation analysis
Disordered-protein ensemble scoring
Marrow cell-type annotation
Yeast-colony detection
Among domains with at least five tasks, the models perform worst in engineering. All four score zero on low-thrust trajectory design, building model-predictive control, and urban traffic calibration. Their roughly 0.21 averages come mostly from one common full-score robot-description task, plus very little credit on power-feeder reliability.
Health and medicine and life sciences provide larger samples of unfinished work. No model reaches a 0.50 mean in health and medicine. All four score zero on limited-angle CT reconstruction, the longitudinal targeted-bootstrap study, radiomics survival prediction, and one clinical-form mapping task. In life sciences, the highest mean is 0.544, with shared zeros on tasks including both Amber workflows, cell translocation, marrow cell-type annotation, and yeast-colony detection.
Conclusion
Once you start looking into benchmarks, you will realise that they are not perfect. Often, they are not measuring what they claim to measure, are fully saturated, and/or are unable to differentiate between the capabilities of models. This post was my attempt to highlight some of these issues. E.g., Terminal Bench 2.1 is totally saturated and unable to differentiate now. We already have Terminal Bench 4, which is successfully able to do so.
That said, Agents’ Last Exam is not yet saturated. At the same time, it’s not the best at differentiating capabilities. One can expect many domains of ALE to be rapidly hillclimbed in the next 6 months. Recently, we saw how GPT-6 Astra got so much better at computer use.
If you read till here, thank you so much. I hope you got to learn something new. This exploration was a highly dense learning experience for me. Acknowledgements
Xeophon (Florian Brand), for being my peer, discussing ideas with me, and guiding me whenever I was stuck.
Snimu (Sebastian Muller), for coordinating the residency and providing the compute and guidance whenever I requested them.
Thanks to Prime Intellect for the Prime Residency program. Thanks also to the Prime Intellect team for building such great infrastructure. This work would not have been possible without their infrastructure.