{"slug": "porting-ale-linux-cli-i-looked-at-the-data-of-a-frontier-benchmark-and-found", "title": "Porting ALE Linux CLI: I Looked at the Data of a Frontier Benchmark and Found some Issues", "summary": "A Prime Residency researcher ported the 105-task Linux CLI subset of Agents' Last Exam, a frontier benchmark from Berkeley RDI spanning 41 subdomains across 12 domain categories, to Prime Intellect's Verifiers library, and the full runs surfaced benchmark defects that led to the creation of ALE-Gold. The porting work also hit inference provider failures and gaps in the freshly released Verifiers v1, which the researcher says have since been fixed. Agents' Last Exam was built by Berkeley RDI, Dawn Song's group at UC Berkeley, with more than 300 experts across 100+ institutions, and only 147 of its 1500+ tasks are public.", "body_md": "2 months back, I started the Prime Residency with Florian Brand as my peer or shall I say my verifier. My first task was to port the Agents’ Last Exam Linux CLI subset to verifiers. Then one day Florian asked me to look into the data and this quest slowly evolved into finding issues in the benchmark and the making of what we call ALE-Gold.\n\nIt was more challenging than I expected. I was new to this kind of work, so there was some skill issue on my part. Along the way, I also hit benchmark defects, inference provider failures, and gaps in the freshly released Verifiers v1 (that have been fixed now so you don't need to worry). This post is a work log of my porting work, what the full runs revealed about the benchmark, how that led to ALE-Gold, followed by an analysis of how the models performed.\n\nA small odyssey from porting ALE to ALE-Gold, with a few unexpected detours.\n\nAgents’ Last Exam is considered a frontier benchmark made by Berkeley RDI, Dawn Song’s group at UC Berkeley. They worked with more than 300 experts across 100+ institutions to make long horizon economically valuable tasks spanning over 55 domains. 147 tasks are public out of 1500+ task corpus.\n\nMy task was to port the Linux CLI subset, which consisted of 105 terminal-only tasks spanning 41 subdomains across 12 broad domain categories.\n\nWhat’s in ALE Linux CLI?\n\n105 tasks across 12 broad domain categories\n\nHealth & medicine\n\n■■■■■■■■■■■■■■■■■■■■■■\n\n22 · 21.0%\n\nComputing & mathematics\n\n■■■■■■■■■■■■■■■■■■■\n\n19 · 18.1%\n\nLife sciences\n\n■■■■■■■■■■■■■■■■■■■\n\n19 · 18.1%\n\nBusiness & finance\n\n■■■■■■■■■■■■■■\n\n14 · 13.3%\n\nPhysical sciences\n\n■■■■■■■■■■■\n\n11 · 10.5%\n\nEngineering\n\n■■■■■■■■\n\n8 · 7.6%\n\nEducation & information\n\n■■■\n\n3 · 2.9%\n\nTransport & safety\n\n■■■\n\n3 · 2.9%\n\nLegal\n\n■■\n\n2 · 1.9%\n\nPsychology & neuroscience\n\n■■\n\n2 · 1.9%\n\nSocial sciences\n\n■\n\n1 · 1.0%\n\nOther\n\n■\n\n1 · 1.0%\n\nShare of the original 105-task CLI subset. Domains are grouped by task-name prefix throughout this post. Percentages are rounded.\n\nOne may wonder “what does it mean to port a benchmark to verifiers”. Verifiers is a library by Prime Intellect that enables you to run “rollouts” in an easy and elegant manner. I mention the word rollout multiple times in this post so it's worth doing a small detour to understand it.\n\nWhat is a Rollout?\n\nRollout in context of evaluation and reinforcement learning is defined as a single run of a policy/model in an environment from start to finish, recording the states, actions, and rewards along the way.\n\nFor example when doing LLM-RL, you will be running multiple rollouts of a single task and these can have different scores/rewards. The policy parameters are updated using these rewards. If you use GRPO, differences in rewards within a group of rollouts provide a learning signal by showing which rollouts performed better relative to the others.\n\nSame task, different attempts. Each rollout has its own sandbox and harness, takes a different path, and receives a score. Timelines and scores are illustrative.\n\nIf you will see benchmarks like FrontierSWE or TerminalBench (harbor framework), they are usually divided into three parts - the task’s input in the form of files or docker image, a harness where the model will run like claude code and codex and the runtime (where the rollout happens). The runtime can be your local machine, docker instance or an AWS instance.\n\nVerifiers provides this exact abstraction for you to port the benchmark and then run the benchmark on Prime Intellect’s infra including support for several sandbox providers.\n\nThe three pieces: a taskset defines the work and scoring, a harness controls the agent’s interaction loop, and a runtime provides the execution environment.\n\nWhat porting ALE involved and some small challenges I faced along the way\n\nThe porting work mainly involves porting the taskset/dataset to verifiers and then writing relevant adapter code between verifiers and original public benchmark code (i refer to as “upstream”) to support evaluation (as it’s defined by upstream). In short, port the input and hook up the scorer code. In addition to this, one may need to write some more compatibility related code depending on the benchmark. The “porting work” itself can vary depending on if the benchmark providers have provided sandbox images or not.\n\nIn ALE’s case, images were not provided so I had to make sandbox images for 105 tasks with the mentioned dependency specifications and input files. This was a code intensive and time consuming process and was only made possible due to coding agents.\n\nOne must minimise the rollout time\n\nOnce the images were made, I would run a rollout or two of the task to verify if the image was fine and the rollout happened successfully. Halfway through my first attempt of making images, I made a mistake. I was not baking the “input files” onto the images. Instead I was downloading them on runtime. This means the input files were being downloaded from the Huggingface archive during each rollout. Now most of the input files were small in size but for a few tasks, they ranged in GBs and HF started hitting resolver limits… This is when I realised my mistake.\n\nFrom an evaluation point of view, it’s also a redundant operation and in general people make effort to minimize the rollout time.\n\nFrom an RL post-training point of view, this is an even bigger issue because you run multiple rollouts of the same task repeatedly during training, so downloading the same input files for every rollout adds up quickly. RL researchers/engineers strive to reduce the rollout time (though mainly inference side) as it’s often the biggest bottleneck. In a synchronous RL training setup, you need to wait for all rollouts to finish before running the gradient steps so the slowest rollout becomes the bottleneck. You can read a bit about it here (as how Prime Intellect engineers solve for inference throughput).\n\nMissing dependency specs\n\nAnother challenge I faced was the lack of sufficient system or library level dependency information for some tasks. This led me to run rollouts, figure out the missing dependencies and then fix the images.\n\nI also adjusted the memory or disk requirements for eight tasks and configured their sandbox resources accordingly.\n\nFull eval runs\n\nOnce the images were ready, I started doing full evaluation runs with different models and harnesses to verify my implementation. This is when you start realising how important a reliable infrastructure is and how fast the money burns when you use models via API.\n\nA full eval run with a model like GPT-5.6 Sol could take 6–12 hours, including retries. Some individual tasks ran for hours, and sandbox or inference provider failures meant retrying affected rollouts, making the feedback loop quite slow.\n\nOccasionally, I ran into more specific issues. During my Kimi runs, failures in the ACP integration caused many rollouts to end midway through a task.\n\nRunning these full evaluations also surfaced infrastructure issues, which I reported to the Prime team. Verifiers v1 had just been released, and these reports helped the team identify and address issues in the new release. I also dabbled a bit in the sandbox infra code to see if I could use a “separate sandbox judging runtime” for my benchmark. Some ALE tasks produced output artifacts larger than 1 GB, but transferring artifacts from the rollout sandbox to the judge sandbox was limited to 32 MB at the time. I found out the bottleneck was the sandbox SDK’s lack of support for chunked file streaming.\n\nThese runs also helped uncover issues in the benchmark itself. Florian also encouraged me to \"look at the data\". A closer look revealed more issues. Note: look at the data at your own risk. You never know what crimes you will uncover.\n\nThe next section covers the issues found in the ALE Linux CLI subset, the fixes, and the nine tasks excluded because their defects remained unresolved. The resulting subset, ALE-Gold, contains 96 of the original 105 tasks, spanning 40 subdomains.\n\nLook at the data, they said\n\nBefore we’re too hard on the creators, I should mention that frontier benchmarks are hard to build. These tasks span specialized domains, involve complicated dependencies, and need verifiers that can distinguish a wrong answer from a valid solution the author didn’t anticipate. Getting all of this right takes a lot of care, both when building the tasks and when validating them.\n\nWith that context, here are some of the issues I found while inspecting the tasks and rollout traces.\n\nUnderspecified task contracts and evaluator defects\n\nI observed agents were failing 5-6 tasks despite producing correct solutions because the task prompt / input didn’t specify the expected output format correctly. To be more specific, essential schemas, enums, or semantics were absent or ambiguous in the visible prompt.\n\n>\"frontier benchmark\" >rollout has partial score >verifier is deterministic scorer >look inside >verifier expects output format with field names agent can't infer from task prompt even by hallucinating pic.twitter.com/qGpDddm1Mn\n\nThe verifier expected several JSON keys and CSV column names that were not mentioned in the input. The prompt also left the domain specific details unclear i.e O’Brien–Fleming implementation, event target, conditional-power futility rule, and follow-up interpretation were unclear.\n\nHere the agent had to reconcile a Moodle course gradebook against a supplied grading policy and produce the final grades and exports. One policy decision was how to handle empty grade entries, but the prompt did not specify the accepted values for empty_grade_behavior.\n\nAcross four rollouts, GPT-5.6 Sol chose \"count_as_zero\", which was a reasonable description of the required behavior. All four produced otherwise correct deliverables but scored 0.95 because the evaluator expected the exact string \"zero\".\n\nThis could have been resolved by mentioning the expected enum in the output contract data. I made relevant changes to the input data to fix this issue and a subsequent rollout scored 1.0.\n\nI was able to fix some of the tasks by minimally modifying the input on the images while others had to be kept out.\n\nAnswer Leakage and network issues\n\nALE does not restrict network as such. It’s a generally online bench and one of the authors mentioned “we keep it online so agent can sometimes fix its dependencies”.\n\nSome tasks require the agent to look up things online or query APIs and these often came with URLs mentioned in the allow-lists. A few tasks enforced network restrictions by mentioning in the prompt.\n\n“The VM has no internet access during solve time.”\n\nThe metadata for network policy (which tasks run online or offline) was not public at the time. Upstream later published a list of 5 tasks that must run offline.\n\nI did 2-3 runs with GPT-5.6 Sol (Medium) without network isolation and found a few tasks that looked up solutions online. These tasks are made to run with network isolation now.\n\nMy personal favourite examples of agents looking up stuff on the internet were:\n\ncomputing_math/go_game_reconstruction_1\n\nThe goal was to reconstruct a Go game from the evidence staged in the sandbox. Instead, the agent realised that the target may be an identifiable public professional game.\n\nThe prompt mentioned “Do not use a Go engine or a web browser.” so the agent proceeded to use curl and download an archive of more than 90,000 professional SGF games, identified the exact historical game, and recovered its 168 moves.\n\nNo Go engine or web browser? The agent used curl to find the game in a public archive instead.\n\nhealth_medicine/wsi_tumor_localization_1\n\nThe task asks for the center point of a tumor in a whole-slide pathology image. The agent started by doing the actual image analysis, zooming into suspicious tissue patches.\n\nThe input had retained its public filename, tumor_001.tif, which probably motivated the agent to look up online, so the agent searched for the corresponding annotation, tumor_001.xml. It tried GitHub, grep.app, Hugging Face, Google, Bing, DuckDuckGo, the original CAMELYON16 challenge website, and finally Kaggle.\n\nBroken inputs and missing dependencies\n\nIn one task, the ordering of the input datasets gave away answers worth 70% of the rubric, allowing the agent to infer them from dataset numbers.\n\nAnother task depended on a software package that was no longer downloadable. It had to be excluded until a refreshed task artifact could provide the required dependencies.\n\nMissing or corrupt reference data\n\nFor some tasks, the reference data was missing or corrupt. This manifested as bad ground truth data, references being questionable or in 1-2 cases as scientifically questionable.\n\nSome of these were fixed upstream (see “Working with upstream” below). The unresolved ones had to be excluded from the subset.\n\nI reported these issues to the ALE authors on Discord and they were fast to respond. They fixed the reference data for a few tasks in the upstream code and published the list of offline-only tasks (PR 1, PR 2).\n\nThe mitigations mentioned above in the benchmark issues section led us to the formation of ALE-Gold. “Gold” just means tasks I believe are not bad and measure capability well. Nine tasks had to be excluded, leaving us with 96 tasks.\n\nOne detail worth mentioning is that I looked at the inputs and traces of problematic tasks and decided on a case-by-case basis what to include. In some cases, the tasks were badly designed or had some issue, but were still included because they seemed to do fine across multiple rollouts.\n\nBad tasks\n\n9 excluded tasks · Scroll horizontally to explore →\n\n■ Business & finance01\n\nBPMN governance\n\nContract mismatch\n\nThe scorer rejects the documented schema and requires undisclosed coordination tasks. Either mismatch can force a zero.\n\n■ Computing & mathematics02\n\nRanking feature recovery\n\nScorer crash\n\nScoring crashes after task completion: the evaluator treats a string payload as a mapping.\n\n■ Computing & mathematics03\n\nCausal structure inference\n\nAnswer leakage\n\nDataset order reveals scenario families and answers worth 70% of the rubric, bypassing much of the intended inference.\n\n■ Engineering04\n\nChisel–Verilog alignment\n\nReference mismatch\n\nThe hidden source-location reference contradicts the supplied comparison evidence, putting 80% of credit at risk.\n\n■ Engineering05\n\nHumanoid policy evaluation\n\nMissing dependency\n\nA required MuJoCo development wheel is no longer downloadable. The archived payload lacks the replacement wheelhouse.\n\n■ Engineering06\n\nOpenROAD chip signoff\n\nScorer crash\n\nA shell PID-parsing bug crashes scoring before the verifier starts, returning zero even for completed submissions.\n\n■ Health & medicine07\n\nHealthcare bias reproduction\n\nUnenforced constraint\n\nThe output-only scorer accepts a direct race bonus prohibited by the prompt; a controlled probe scored 0.861.\n\n■ Health & medicine08\n\nProstate radiotherapy planning\n\nScoring timeout\n\nThe matRad scorer repeatedly exceeds its declared one-hour scoring timeout, preventing reliable evaluation.\n\n■ Health & medicine09\n\nVariant annotation pipeline\n\nQuestionable reference\n\nHidden truth appears to mishandle deletion normalization and allele-specific ClinVar, rewarding reference-matching errors.\n\nThe nine excluded tasks and why I left them out. These issues refer to the task versions used in this evaluation.\n\nALE-Gold evaluation results\n\nI ran full evaluation runs for 4 latest models at the time which also supported both text and vision capabilities. Opus 5 was passed on due to higher cost and GLM 5.3 because it didn't support vision.\n\nThese results are based on the 96-task ALE-Gold subset, with one reported attempt per model per task. The reasoning level and harness were chosen based on the best-performing settings for each model on the official ALE leaderboard. If a rollout didn’t end successfully as indicated by the harness, I retried it, with at least three retries for persistent failures. This included rollouts that stopped during tool call execution.\n\nAggregate score is the mean task score on a 0–1 scale, including partial credit, with missing scores counted as zero. Pass rate counts only recorded scores exactly equal to 1.0.\n\nALE-Gold results\n\nModel / configurationAggregate scorePass rate\n\nGPT-5.6 Sol\n\nMedium reasoning · Codex\n\nAggregate score0.5689\n\nPass rate28.13%\n\nGPT-5.6 Luna\n\nHigh reasoning · Codex\n\nAggregate score0.5566\n\nPass rate29.17%\n\nKimi K3\n\nMax reasoning · Kimi Code\n\nAggregate score0.4686\n\nPass rate26.04%\n\nGLM 5.3 Flash\n\nMax reasoning · Codex\n\nAggregate score0.4071\n\nPass rate19.79%\n\nKimi K3 and GLM 5.3 Flash had several rollouts ending mid-turn / partial tool call execution. For Kimi K3, it was ACP server issue from their side whereas for GLM 5.3 Flash, the issue seemed to be more due to model's skill issue.\n\nSol (med) vs Luna (high)\n\nSol vs LunaSol mediumLuna high\n\nMean score56.89%55.66%\n\nPerfect-pass rate28.13%29.17%\n\nCompletion tokens1.12M2.40M\n\nRecorded cost\\$56.43\\$16.67\n\n96 shared tasks. Cost is the model-call total recorded in the traces; runtime infrastructure is excluded.\n\nLuna interestingly has a higher pass rate than Sol on ALE-Gold and this matches the score displayed on the official ALE website leaderboard.\n\nSol and Luna follow similar broad workflows and finish close together: Sol has the higher mean score (0.569 vs 0.557), while Luna has the slightly higher exact-1 pass rate (29.17% vs 28.13%). Across all 96 tasks, Sol scores higher on 23, Luna on 14, and 59 tie. Sol earns more partial credit outside the exact-pass boundary.\n\nLuna having a higher pass rate than Sol also raises the question of how well the benchmark differentiates between models’ capabilities. This suggests there may be scope for improvement in the tasks.\n\nLuna used 114% more completion tokens, including reasoning tokens, and made about 86% more model calls. Its recorded model-call cost was nevertheless about 70.5% lower: $16.67 versus $56.43.\n\nThe four tasks below are the ones Luna passed with an exact score of 1.0 while Sol fell short.\n\nSSE trading-rule review\n\nReview the naming rules for northbound programmatic-trading software.\n\nLuna 1.000Sol 0.667\n\nSol interpreted permission to use English as a prohibition on a French software name. It turned a limited permission into a stronger restriction than the cited rule stated.\n\nMy interpretation is that Sol was overly strict in its instruction following.\n\nModel-parallel checkpoint consolidation\n\nReassemble a model from its distributed checkpoint shards.\n\nLuna 1.000Sol 0.000\n\nSol replaced embeddings supplied consistently across the checkpoint shards with a guessed initialization. Its own measurements showed that this change worsened the result, but it finalized anyway.\n\nMy interpretation is that Sol has a tendency to substitute requested work with an approximation.\n\nHealthcare bias-audit replication\n\nReproduce a healthcare bias audit using the supplied software environment.\n\nLuna 1.000Sol 0.000\n\nSol ignored the supplied Python runtime and built a different software stack. The task required numerical reproduction within a tight tolerance, making the runtime change the strongest supported explanation for divergence. The exact failing field was not retained.\n\nPseudotime differential expression\n\nIdentify genes whose expression changes along a cell-development trajectory.\n\nLuna 1.000Sol 0.000\n\nSol applied an additional gene filter, reducing its result from the broader accepted construction of 2,157 genes to 1,049. That likely pushed recall below the scorer’s threshold.\n\nThe instructions were ambiguous about this filter, so some bad behaviour can be attributed to the task specification here.\n\nWhere each model performs better than others\n\nEvery model has tasks where it outscores all three peers. GLM 5.3 Flash, for example, has the lowest overall mean but is the only model to get full credit on CT geometry calibration. Each card below shows a task, its goal, and the four scores; use the arrows to see a second example for each model. These are selected score advantages, not proof of a general domain specialty.\n\nScores include partial credit. Each model’s arrows cycle independently. Keyboard: Tab to a model, then use the arrow keys.\n\nFailure modes\n\nCommon failure modes\n\nShare of runs that fell short, shown as the range across models. Labels overlap. Bars mark each range on a 0–100% scale.\n\nVerification issueChecks format or self-consistency, but never tries to falsify the result\n\n64–74%\n\nSkill issueUses the wrong method, formula, model, mapping, or implementation\n\n57–71%\n\nInstruction issueSkips, changes, or misreads an explicit requirement\n\n17–34%\n\nExecution issueRepeats, explores, or waits until time runs out without finishing\n\n18–33%\n\nOutput issueMisses a required filename, path, header, schema key, unit, or literal\n\n3–13%\n\nHarness issueProvider, tool, or runtime failures interrupt or block the run\n\n6–21%\n\nThe models sometimes replaced a difficult workflow with a simpler method that did not meet the original requirements. Their checks could miss the difference because they repeated the same assumptions as the implementation. In other cases, the checks exposed a problem but the model submitted the result anyway. E.g., in checkpoint consolidation, Sol and GLM submitted despite failed numerical checks, and Sol retained a change that made its measured result worse.\n\nThere are multiple interpretations of this: skill issue, lack of capability, or model simply picked up a wrong trajectory.\n\nSome runs spent all their time on intermediate work without producing the final deliverable. Repeated experiments, restarts, and monitoring left Sol and Luna without a final CSV in the disordered-protein ensemble scoring task. Other losses came from provider interruptions or the benchmark itself. In the data pipeline extract, transform, and load (ETL) warehouse-construction task, for example, Sol and Luna followed the column order required by the instructions, but the scorer expected the reverse order. These external problems need to be distinguished from model mistakes.\n\nMore model-specific failure patterns also stood out.\n\nSol has a tendency to attempt an approximate version of a task when it finds the work difficult. It can also follow instructions too rigidly, taking things literally and, in my view, showing a lack of common sense at times. Based on my observations, it also showed the most reward-hacking tendencies.\n\nLuna persisted with weak approaches and sometimes ran out of time before producing the deliverable.\n\nKimi K3 substituted shortcuts for the requested workflow without adequately checking that they met the original requirements.\n\nGLM 5.3 Flash often made implementation errors or used the wrong formulas. My interpretation is that these mistakes reflected the capability and knowledge limitations of a smaller model.\n\nPerformance by domain\n\nAcross the five domains with at least 10 tasks, the four-model average is highest in business and finance, followed by computing and mathematics, then physical sciences. Sol leads in business and finance, physical sciences, and life sciences; Luna leads in computing and mathematics and health and medicine. These are small observed advantages. Kimi leads education and information, which has only three tasks. GLM has individual task wins but no outright domain-mean lead. Social sciences and legal have higher four-model averages, but only one and two tasks respectively.\n\nPerformance by domain\n\nMean task score01\n\nDomain\n\nSol\n\nLuna\n\nKimi K3\n\nGLM 5.3 Flash\n\nHealth & medicine 19 tasks\n\n0.487\n\n0.497\n\n0.423\n\n0.410\n\nLife sciences 19 tasks\n\n0.544\n\n0.536\n\n0.450\n\n0.426\n\nComputing & mathematics 17 tasks\n\n0.626\n\n0.633\n\n0.612\n\n0.357\n\nBusiness & finance 13 tasks\n\n0.695\n\n0.685\n\n0.510\n\n0.477\n\nPhysical sciences 11 tasks\n\n0.651\n\n0.612\n\n0.498\n\n0.333\n\nEngineering 5 tasks\n\n0.206\n\n0.207\n\n0.207\n\n0.207\n\nEducation & information 3 tasks\n\n0.617\n\n0.611\n\n0.634\n\n0.504\n\nTransport & safety 3 tasks\n\n0.611\n\n0.467\n\n0.534\n\n0.541\n\nLegal 2 tasks\n\n0.834\n\n0.694\n\n0.665\n\n0.667\n\nPsychology & neuroscience 2 tasks\n\n0.404\n\n0.380\n\n0.003\n\n0.381\n\nOther 1 task\n\n0.000\n\n0.000\n\n0.000\n\n0.000\n\nSocial sciences 1 task\n\n1.000\n\n1.000\n\n0.000\n\n1.000\n\nDomains with scope for significant improvement\n\nHello benchmaxxers, this section is for you.\n\nTo see domains that are not saturated, let's look at the highest whole domain mean achieved by the models. A low score indicates model lacking capability in that domain. The panel includes domains with at least five tasks where the best model mean is below 0.55.\n\nHow much room is left?\n\nHighest model mean · bars run from 0 to 1. Open a domain to explore its results.\n\nEngineering0.2075 tasks · Best: Kimi K33 shared zeros · Explore\n\nLife sciences0.54419 tasks · Best: Sol6 shared zeros · Explore\n\nSol0.544\n\nLuna0.536\n\nKimi K30.450\n\nGLM Flash0.426\n\nScored zero for all four models\n\nAmber minimization workflow\n\nAmber three-stage MM/GBSA workflow\n\nCell translocation analysis\n\nDisordered-protein ensemble scoring\n\nMarrow cell-type annotation\n\nYeast-colony detection\n\nAmong domains with at least five tasks, the models perform worst in engineering. All four score zero on low-thrust trajectory design, building model-predictive control, and urban traffic calibration. Their roughly 0.21 averages come mostly from one common full-score robot-description task, plus very little credit on power-feeder reliability.\n\nHealth and medicine and life sciences provide larger samples of unfinished work. No model reaches a 0.50 mean in health and medicine. All four score zero on limited-angle CT reconstruction, the longitudinal targeted-bootstrap study, radiomics survival prediction, and one clinical-form mapping task. In life sciences, the highest mean is 0.544, with shared zeros on tasks including both Amber workflows, cell translocation, marrow cell-type annotation, and yeast-colony detection.\n\nConclusion\n\nOnce you start looking into benchmarks, you will realise that they are not perfect. Often, they are not measuring what they claim to measure, are fully saturated, and/or are unable to differentiate between the capabilities of models. This post was my attempt to highlight some of these issues. E.g., Terminal Bench 2.1 is totally saturated and unable to differentiate now. We already have Terminal Bench 4, which is successfully able to do so.\n\nThat said, Agents’ Last Exam is not yet saturated. At the same time, it’s not the best at differentiating capabilities. One can expect many domains of ALE to be rapidly hillclimbed in the next 6 months. Recently, we saw how GPT-6 Astra got so much better at computer use.\n\nIf you read till here, thank you so much. I hope you got to learn something new. This exploration was a highly dense learning experience for me.\n\nAcknowledgements\n\nXeophon (Florian Brand), for being my peer, discussing ideas with me, and guiding me whenever I was stuck.\n\nSnimu (Sebastian Muller), for coordinating the residency and providing the compute and guidance whenever I requested them.\n\nThanks to Prime Intellect for the Prime Residency program. Thanks also to the Prime Intellect team for building such great infrastructure. This work would not have been possible without their infrastructure.", "url": "https://wpnews.pro/news/porting-ale-linux-cli-i-looked-at-the-data-of-a-frontier-benchmark-and-found", "canonical_source": "https://sankalp.bearblog.dev/porting-ale-linux-cli-i-looked-at-the-data-of-a-frontier-benchmark-and-found-some-issues/", "published_at": "2026-09-12 10:46:00+00:00", "updated_at": "2026-09-12 11:06:15.028795+00:00", "lang": "en", "topics": ["ai-research", "ai-safety", "ai-agents", "developer-tools"], "entities": ["Agents' Last Exam", "Berkeley RDI", "Dawn Song", "UC Berkeley", "Prime Intellect", "Verifiers", "ALE-Gold", "Florian Brand"], "alternates": {"html": "https://wpnews.pro/news/porting-ale-linux-cli-i-looked-at-the-data-of-a-frontier-benchmark-and-found", "markdown": "https://wpnews.pro/news/porting-ale-linux-cli-i-looked-at-the-data-of-a-frontier-benchmark-and-found.md", "text": "https://wpnews.pro/news/porting-ale-linux-cli-i-looked-at-the-data-of-a-frontier-benchmark-and-found.txt", "jsonld": "https://wpnews.pro/news/porting-ale-linux-cli-i-looked-at-the-data-of-a-frontier-benchmark-and-found.jsonld"}}