Everyone should build their own AI benchmarks. Why?
Labs optimize their agents toward public benchmarks, so changes in benchmarks often overstate capability changes.
What you want to use AIs for is often particular to you, and has quirks that don’t fit cleanly into a generic benchmark.
Subjective benchmarks often give you a feel for the “texture” of quality that an agent produces, in a way that a benchmark that has to be reduced to a score/pass-fail cannot.
Today’s benchmarks are comically overweighted toward coding and logic reasoning, just because they’re verifiable.
Building your own benchmarks is fun, and adds a sense of excitment for new model releases.
Thus, every time I hit a problem an AI agent can’t solve, or does a very poor job on, I copy the prompt down in a little file on my computer in ~/benchmarks/[NAME].md, sometimes with attached context like the files needed for the task. Then, when new models come out I run a few of them to quickly get a sense for what the new models are capable of.
Steller
One such (subjective) benchmark, which I’m now satisfied enough with the results of to call “passed” is STELLAR.md.
Please create a detailed interactive environment in 3JS which features a single Stellar’s Jay, with the ability to have it move around (by both flying and hopping) in response to mouse clicks, and have it passively move around without human intervention, emulating the motion and passive activities of the animal. It should be in a simplified 3D scene that allows demonstration of the motion of the jay. Your goal is to capture in extreme visual fidelity the physical attributes of the animal, its modes of movement and patterns of behavior. Aim for realism, and avoid simplifications that would yield distracting visual artifacts. It should only use Javascript and HTML, and should work offline. You have access to run subagents through other models from other providers (via codex, claude, and agy CLIs) to check your work and provide visual critiques. You have 6h, and up to $100 in API credits to use (across all models). Any questions before you begin?
I then answer any questions the model asks.
A careful reader will realize in this prompt I type it “StellAr” (with an A), when it is in fact spelled “StellEr” (with an E) - something I learned while writing this blog post. I guess I’m not a great SPELLAR (lol).
Progression
I started asking this question in June of 2026, when people were first starting to talk about the huge improvements in 3d modeling.
Early results were bad, but solved an increasingly sophisticated set of problems around interactivity, motion, texture/pattern, and eventually, flight physics. Like any good project, they resulted in some interesting failures
Terra 6.1, Effort High, Released October 26, 2026.
Astra 6.2 and Fable 5.2
Within 5 months, in November of 2026, I deemed this benchmark “solved”. Both Astra 6.2 and Fable 5.2 can do a really good job on it while staying within the cost thresholds.
The result embedded below is a Fable 5.2 agent, coordinating with an Astra 6.2 agent for visual reviews. They communicated back and forth many of times in formulating this result (more than I’ve typically seen in other forms of cross model collaboration), with both making some modifications to the code at times. I give the final result to Fable 5.2 just because it coordinated it and wrote almost all of the code, and Astra 6.2’s performance solo was much less impressive. The final run took 4.5h and took ~$80 in API cost equivalent. I’ve made some visual tweaks to this (such as increasing the saturation of the feathers, and having another agent take a pass at improving CPU performance so it performs well on mobile for this article), but the core quality of the visual reasoning is from that one run.
Fable 5.2 high, Released November 9, 2026. Astra 6.2 high, released November 17, 2026, was used for reviews and similar support.
What is next?
Eventually, at some point, I want to play a video game that encodes ~everything we know about how birds behave, communicate, grow, and interact with their environments, into a game where you are a bird, moving through the world in VR, interacting with other birds and the habitat, avoiding predators, forraging, and eventually trying to raise a fledgling (or if you’re lucky, multiple) to adulthood.
This idea stems from a trio of inspirations:
the “American Truck Simulator” game (where hyperrealism makes something possibly boring into something fascinating)
the “Untitled Goose Game” (where simple click and point mechanics, interacting with a rich world is just delightful)
and the famous “What is it like to be a bat?” paper by philosopher Thomas Nagel (which points to the fundamental limitations of ever understanding what it is like to be another creature)
Imagine if all of the world’s information (videos, behavioral analysis, scientific research, etc.) about a species could be interpreted by an AI to build an experience that tries to emulate what it might be like to live as that species.
My hope is eventually to play a Steller’s Jay Simulator to try to understand what it might be like to be a bird - to huddle under a plant in a thunderstorm, to fly high above a forest, to forage close to the protective cover of bushes and fences, to communicate and dance and have little social groups. I think it could be fun, educational, and hard, and I can imagine it really broadening our minds to play it (not just a Steller’s Jay, but also a Minnow, a cricket, a tiger, an orca, a ladybug, etc).
This benchmark isn’t the endpoint for such a game, just a milestone along that road.
What is a benchmark, really?
I give all of my agents access to previous conversation histories through Pharos, and don’t make an exception for benchmarks. This is inherently “unfair” to earlier runs, because in the final run, the agent team (without my instruction) used this MCP to inspect prior attempts and avoid some pitfalls, and got some ideas from prior runs. This definitely biases toward later models, and blurs the line between “the model got better” and “the model learned from past mistakes”.
Now, if you’re thinking “that defeats the point of a benchmark”, I’d narrowly disagree.
Most people and companies treat benchmarks as a way to evaluate the question “what can this agent do with no help or support”. I’m more interested in asking “what at the frontier is newly possible”. Information about past efforts is typically going to be available to my agents (thanks to Pharos), so I want to evaluate and think about model capability on the frontier with this advantage is priced in. The purpose of my personal benchmarks is helping me understand what is newly possible, and that possibility is a function of history and tools, not only of model intelligence.
So the next time AI fails you at a task, rather than bemoaning it, squirrel it away for the future - so that you can dance around and better understand the frontier of the possible.