{"slug": "benchmarks-are-dead-for-us", "title": "Benchmarks Are Dead (For Us)", "summary": "Poetiq claims its Recursive Self-Improvement (RSI) Metasystem has autonomously set state-of-the-art results on six diverse benchmarks, including outperforming Muse Spark 1.1 within 48 hours of publication and surpassing Anthropic's Claude Fable 5 using a previous-generation model. The company argues that conventional benchmarks are inadequate for evaluating systems with true RSI capabilities.", "body_md": "Mainstream AI research is fixated on a capital-intensive strategy — intelligence via\nincremental weight updates, costing billions in compute. At Poetiq, we believe that\nintelligence does not need to live solely within model weights. Instead, we treat the LLM\nas a single component of a larger, self-improving reasoning architecture. This\nself-optimizing architecture leads to a Recursive Self-Improvement (RSI) loop that is\ndesigned to automatically construct entire harnesses (consisting of code, prompts, tools,\nand search strategies) for any benchmark it encounters.\n\nAt Poetiq, we're building a **Metasystem** capable of autonomously generating\nthe harnesses and auxiliary data required to solve any challenge. Setting state of the art\n(SOTA) results on newly introduced benchmarks is now a routine procedure for us. However,\nbenchmarks only serve as proxies for real world challenges; their diagnostic utility\nplummets once our system automatically masters them. Conventional, static, benchmark\nframeworks are inadequate for evaluating systems with true RSI capabilities.\n\nThe Benefits of our RSI Loop\n\n-\n**Autonomous SOTA Generation:** With *zero* human intervention, our\nMetasystem independently built harnesses that set SOTA milestones across six diverse\nbenchmarks — including long-context retrieval, agentic tool use, complex system\nplanning, and competitive mathematics. This includes outperforming a SOTA benchmark\nresult from Muse Spark 1.1 within 48 hours of publication.\n-\n**Disruptive Performance:** We frequently secure these SOTA outcomes\nwithout using the benchmark's leading model. In multiple instances, our system overtook\nexisting benchmarks held by Anthropic's Claude Fable 5 using a previous generation\nmodel.\n-\n**A Model-Agnostic, Compounding Moat:** By decoupling reasoning mechanics\nfrom underlying weights, our design is completely model-agnostic. As proprietary and\nopen-source models advance, our Metasystem utilizes them as tools, continuously\ncompounding their value.\n\nThe Path That Brought Us Here\n\nIn the past year, we've been deliberate about which benchmarks we attempt. We picked our\ninitial three to match critical categories of tasks for LLMs:\n\n-\n**Reasoning** — synthesizing provided information in inventive ways.\n[ARC-AGI](/posts/arcagi_verified/)\n([followup](/posts/update_arcagi_gpt52/)) is the premier\ntest of this.\n-\n**Retrieval** — testing the limits of the breadth of knowledge embedded in\na model's weights.\n[Humanity's Last Exam](/posts/raising_the_bar_hle_simpleqa/)\naudits this across a vast spectrum of disciplines.\n-\n**Coding** — the most pervasive commercial application of AI, melding\nreasoning and retrieval with specialized procedural logic.\n[LiveCodeBench Pro](/posts/recursive_self_improvement_coding/) tests coding ability while minimizing the risk of model memorization of answers.\n\nWe set SOTA on all three with no fine-tuning of weights and no privileged access. Across the\nboard, we see gains on all proprietary and open-source models. With each test, our\nMetasystem constructed and optimized increasingly large portions of its own harness,\nwidening its capabilities through RSI.\n\nThe loop is now good enough that it no longer needs us to do more than pick the benchmark.\nNo hand tuning or customization needed.\n\nThe Results: Brand New SOTA Benchmarks\n\nIn the following table, we demonstrate the efficacy of our automatic\n**Metasystem** on six highly complex benchmarks. Each benchmark was chosen to\ntarget new and diverse tasks of types that our system had not previously seen. The same\nMetasystem built every harness automatically. It designed the code, prompts, and\nhyperparameters. For half the benchmarks, SOTA was achieved even with one-generation older\nmodels than the current SOTA holder.\n\nEach harness, created by Poetiq's Metasystem, improved across all models (both proprietary\nand open-source). Below, we give our results and the models that our system used to reach\nit.\n\n[1](#footnote-1) *(Note: While we did not use Anthropic's Claude Fable 5 as our underlying model, we\nroutinely outperformed its capabilities.)*\n\n| Benchmark | Category | Underlying models used by Poetiq | Poetiq | Prev. SOTA | Prev. model |\n**ArXivMath** | Competition math | GPT-5.5 | **89.2** | 87.5* | Claude Fable 5 |\n**SciCode** | Scientific coding | Gemini 3.1 Pro | **61.5** | 60.2 | Claude Fable 5 |\n**Haladir — Challenge Set** | Long horizon planning | Gemini 3.1 Pro | **0.47** | 0.28 | Claude Fable 5 |\n**Haladir — Full Set** | Long horizon planning | Gemini 3.1 Pro | **0.69** | 0.33‡ | Claude Opus 4.8 |\n**MCP-Atlas** | Agentic tool use | Muse Spark 1.1 & Gemini 3.5 Flash | **89.8** | 88.1 | Muse Spark 1.1 |\n**MRCR v2** | Long-context | Gemini 3.1 Flash-Lite | **99.26** | 97.3 | GPT-5.4 |\n**Toolathlon** | Agentic tool use | Gemini 3.5-Flash | **59.26** | 56.5† | Gemini 3.5 Flash |\n\nScores are percentages except Haladir, whose native score is 0–1. * Model released after\nour results were reached. † Prior best on the standard, unmodified benchmark. Per\nAnthropic's own note, its reported Claude Fable 5 (61.7) and Opus 4.8 (59.9) run “~3 points\nabove a strictly upstream-equivalent harness” (\n\n[Fable 5 System Card, §8.17.8](https://www.anthropic.com/claude-fable-5-mythos-5-system-card)). ‡ Mean of ending budget ÷ initial budget over 200 environments.\n\n-\n**ArXivMath — leapfrogging on the same base model:** ArXivMath is a\ncontamination-resistant math benchmark: its problems are drawn monthly from fresh ArXiv\npapers published after the models' training cut-offs. It probes genuine mathematical\nreasoning — multi-step derivation and proof — rather than recall. It is one of the\ncleanest available measures of generalized reasoning.\n-\n*Result:* Poetiq took GPT-5.5 from 77.5% to 89.2%, and Gemini 3.1 Pro from\n51.7% to 68.3%. Our GPT-5.5 result also exceeds the previous SOTA of 87.5% posted by\nClaude Fable 5 (max) — a model released after our results were achieved.\n\n-\n**SciCode — clearing the SOTA with a smaller model:** SciCode asks models\nto implement research-grade scientific code from a natural-language problem statement.\nThese problems are derived from complex domains such as physics, chemistry, biology, and\nbeyond. The benchmark rewards the synthesis of domain knowledge with precise procedural\nlogic, not competitive-programming tricks. It measures whether AI can do the kind of\ncoding that moves science forward.\n-\n*Result:* We achieved 61.5% with Gemini 3.1 Pro, overtaking Claude Fable 5's\nrecent SOTA of 60.2%.\n\n-\n**Haladir: Operations Research — more than double the previous state of the art:** In the\n[Haladir](https://www.haladir.com/blog/production-simulation)\nbenchmark an agent is tasked with running a simulated factory: 200 scenarios across 10\nsynthetic domains — commercial baking, CNC machining, brewing, textiles manufacturing\nand more. In each scenario, the agent must build stations, hire workers, order\nmaterials, and ship product orders with operational constraints (limited budget, harsh\ndeadlines). It is scored on the ratio of ending budget to initial budget. Because winning\nmeans holding many interacting constraints at once (station health, ingredient spoilage,\nworker fatigue, deadlines) it is a model of real operational work and the difficulties of\nlong-horizon planning.\n-\n*Result:* On the full 200-environment set, Poetiq lifted the score from a\nprior best of 0.326 (Opus 4.8) to 0.688 with Gemini 3.1 Pro — more than double the\nprevious state of the art. The lift holds across nearly every domain.\n-\nAdditionally, Haladir released results on a harder 20-scenario subsample. On this\nsubsample, Haladir reported results for Claude Fable 5 at 0.28. Using our same\nexpert, we reached 0.470 — 1.7× that of Fable.\n\n-\n**MCP-Atlas — SOTA on tool use:** MCP-Atlas evaluates real-world tool use\nover the Model Context Protocol. Across simulated servers and real APIs, a model must\ndiscover the right tools, call them correctly, recover from errors, and synthesize the\nresults. It measures whether a model can act independently, not just answer. This is the\ncapability that separates an agent from a chatbot.\n-\n*Result:* We reached 89.8% on the 500-problem public set using a combination\nof Muse Spark 1.1 and Gemini 3.5 Flash, exceeding the prior best of 88.1%.\n\n-\n**MRCR v2 — reading two million tokens with a model that isn't on the leaderboard:** MRCR v2 is a “needle-in-a-haystack” benchmark: the model must find and correctly\ndisambiguate eight nearly-identical passages buried in roughly 2M tokens of context. It\nmeasures whether a model can genuinely use its whole context window. As context windows\nkeep growing, it's the test that shows whether those extra tokens in context are truly\nusable.\n-\n*Result:* We hit 99.26% – past the SOTA of 97.3% set by GPT-5.4 — using\nGemini 3.1 Flash-Lite, a small, inexpensive model whose base score doesn't even\ncrack the leaderboard (even Gemini 3.1 Pro manages only 84.9%).\n\n-\n**Toolathlon — SOTA on the real, unmodified set:** Toolathlon is a suite of\n108 real-world tool-use tasks spanning office productivity, e-commerce, data analysis\nand web research. These tasks are graded by execution-based checkers that verify the\nactual side effects each task should produce. With 600+ tools across dozens of\napplications and roughly twenty turns per task, it stresses long-horizon orchestration\nacross many apps. It is a demanding measure of whether an agent can reliably finish real\nmulti-application workflows.\n-\n*Result:* We reached 59.26% with Gemini 3.5 Flash — the best result on the\nbenchmark as of this publication, past Gemini 3.5 Flash's 56.5% (without Poetiq) and\nevery off-the-shelf frontier model (GPT-5.5: 55.6%, Sonnet 5: 54.3%, Gemini 3.1 Pro:\n48.8%).\n-\nThe only higher figures you may have seen — Claude Fable 5's 61.7% and Opus 4.8's\n59.9% — come from Anthropic's\n*modified Toolathlon* set along with Anthropic's\nown caveat that its reported numbers “consistently run ~3 points above a strictly\nupstream-equivalent harness”[2](#footnote-2).\n**On the clean, unmodified benchmark, we hold the top spot.**\n\nThe Future of Benchmarks\n\nEvery result above shares one property: **we didn't do the work.** We didn't\nread the benchmark, design a prompt, or hand-build a scaffold. The RSI loop ingested the\nbenchmark and produced a custom, task-specific harness on its own — across retrieval, math,\ncoding, and tool-use alike.\n\nAnd that's been our goal since we started Poetiq. The goal was not to create a harness to\nbeat a specific problem but to create a **Metasystem** that can automatically\ncreate harnesses, data, and procedures to solve *any* problem. We treat the LLM as a\nsingle component of a larger reasoning system (code, prompts, exploration/exploitation\nstrategies, and more) and what improves in each iteration is the system, not necessarily the\nmodel's weights.\n\nThis reframing is what lets our RSI loop run quickly and cheaply today while still\ncompounding its benefits. It also frees us from betting on any single model: we use models\nas interchangeable tools. The Poetiq Metasystem is a self-optimizing optimizer. Each\nproblem, task, and dataset it tackles helps it to optimize itself, building itself into a\nmore and more powerful optimizer.\n\nOf course, the statement “benchmarks are dead” is a bit tongue-in-cheek. An important\nquestion, in the light of RSI systems like ours which are already constantly self-improving,\nis how to find gaps in the current model's knowledge and enable further improvement. Perhaps\nunsurprisingly, we've turned to our own systems for the answer. Because the Poetiq Metasystem\nmaps the capabilities of the underlying models it uses, we can also task the system to find\nhighly challenging datasets along the model's jagged frontiers.\n\nBeyond just discovering such static datasets, we can dynamically generate benchmarks that\ncombine information and abilities that no single model has — a **living benchmark**\nthat *can't* be trained against. So, perhaps it is more appropriate to say: the static\nbenchmark is dead; long live the living benchmark.\n\nWhat's Next for Poetiq: Continuous RSI\n\nWe're turning the RSI loop on and leaving it on. Concretely, that means:\n\n-\n**Ever-improving systems fueled by solving real problems for enterprise partners:** By design, we created the Poetiq Metasystem to improve through the diversity of tasks\nthat it is given; the main source of our RSI improvements is tackling new problems. We\nwork with select partners whose datasets and challenges we haven't encountered — and as\neach is solved, the procedures and search strategies for that domain are automatically\nfolded into the RSI system, without retaining any of the data.\n-\n**Continuously-updated model capability maps:** Running a variety of models\nlets us map frontier and open-source models' capabilities — and in doing so, pinpoint\nexactly where each model is weak. Those weak points tell us precisely how to test and\nmeasure models as they are released. These dynamic assessments keep up with changing\ncapabilities without being susceptible to confounding training data contamination.\n-\n**Compounding transfer across domains:** Every harness the Metasystem\nbuilds becomes reusable raw material for the next. Search strategies discovered while\nsolving factory logistics resurface in agentic tool use; prompt structures from\ncompetition math accelerate scientific coding. The more diverse the problems, the faster\neach new domain falls.\n\nBeating benchmarks is how we launched our company. Solving commercial applications that give\nus the fuel to continuously improve our system is how we grow it.\n\n[Contact us](https://docs.google.com/forms/d/e/1FAIpQLScbaUMREJqBupnSwkpPaLyJFASZZbET7CjAqBq46mD9vnCGvg/viewform) for more information.\n\nThe Team Building RSI\n\nPoetiq is a rapidly growing technical team with backgrounds from Google DeepMind, Apple,\nMicrosoft, Amazon, and ByteDance, all focused on the goal of creating AI systems that\nrapidly improve themselves.\n\nAre you ready to experience intelligence beyond Fable for a fraction of the cost? We'd like\nto\n\n[hear from you](https://docs.google.com/forms/d/e/1FAIpQLScbaUMREJqBupnSwkpPaLyJFASZZbET7CjAqBq46mD9vnCGvg/viewform). And if you want to be a part of the team that's building RSI,\n\n[join us](/careers/).\n\nAfter SOTA was reached, we did not continue running to\nfurther improve SOTA.", "url": "https://wpnews.pro/news/benchmarks-are-dead-for-us", "canonical_source": "https://poetiq.ai/posts/benchmarks_are_dead/", "published_at": "2026-07-23 00:34:34+00:00", "updated_at": "2026-07-23 00:52:17.515397+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-agents", "ai-products"], "entities": ["Poetiq", "Muse Spark 1.1", "Anthropic", "Claude Fable 5", "ARC-AGI", "Humanity's Last Exam", "LiveCodeBench Pro"], "alternates": {"html": "https://wpnews.pro/news/benchmarks-are-dead-for-us", "markdown": "https://wpnews.pro/news/benchmarks-are-dead-for-us.md", "text": "https://wpnews.pro/news/benchmarks-are-dead-for-us.txt", "jsonld": "https://wpnews.pro/news/benchmarks-are-dead-for-us.jsonld"}}