Physical AI lives or dies on whether the modeled physics is correct. A model of an aircraft, a separation column or a charged particle can compile and run cleanly while the physics it encodes is impossible. Agentic AI makes this failure mode worse, because agents steer by feedback from tests, and the tests are often written by the same agent. In domains like web development or compilers that loop works well enough, since correct behavior is contained and checkable. In engineering, the real question is “does this match the real world?”, and that is much harder to close: the agent can pass every test it wrote while those tests rest on simplifications that don’t hold in the regime where the model will actually be used.
There is also a trust problem with the numbers that already exist. Nobody takes the model providers’ self-reported benchmarks at face value, and for good reason: the provider publishing the score is the same party with every incentive to tune for it. Our incentives point the other way. JuliaHub ships Dyad with multiple agent backends across vendors, and we win when our users get the best possible Dyad experience, whichever lab’s model delivers it. If one model is better at physical modeling, it is in our interest to find that out and recommend it. So we investigated: which one actually is?
To measure which frontier model handles that work best, we hold everything else still. The Dyad AI agent harness - purpose-built for modeling & simulation workflows - is pinned across every trial, along with the problems, the reasoning effort (xhigh), the context window (1M) and the token budget (128k). The only variable is the model: OpenAI’s GPT 5.6 family (terra, sol, luna) against Anthropic’s claude-fable-5. Three trials per model on each of the four core problems, one long-horizon trial each on the fifth: 52 graded runs in all.
01 · The problems
How Physical AI Evaluation Works. #
We ground our exploration in a subset of our internal evals: five sealed problems drawn from the daily work of modeling & simulation. We order them by difficulty, defined in terms of the stages a correct solution must chain through, the careful details where a small slip is fatal, the easy shortcuts within reach, and the engineering work around the modeling itself - from parsing specs to repairing harnesses. We selected each for the same property: they admit solutions that compile and run green while being physically wrong. Grading therefore ignores the code and scores the full simulated trajectory against sealed ground truth. The hardest of the five is NASA’s HL-20 flight vehicle. An easier version of the problem is walked through in detail in this video; the study runs a harder, sealed variant.
02 · The scoreboard
What they scored. #
We run every model through the full suite - three trials on each of P1–P4, one on the long-horizon P5 - and grade every trial against sealed ground truth. Per-problem scores fold into a single number through a difficulty-weighted average, the hardest weighing the most. Cost and time carry no weighting: average cost per trial and average wall-clock per trial each get a tab:
The single number above has anatomy. Per problem, averaged across trials, the same three lenses:
03 · Work-style fingerprints
Each model has its own approach. #
We analyze the transcripts behind every trial. Each tool call is bucketed by what the agent was doing at that moment - orienting in the problem, deriving and testing physics, editing the model, compiling, or hunting for something to copy - then counted per trial and averaged per model. The result is a work-style fingerprint:
The bars say how the work divides; the transcripts show the temperament behind it. Four portraits, drawn from where each model was tested hardest:
Fable verifies by trying to fail. In one trial on the relativistic dynamics problem it swept solver tolerances across three decades, watching the mass-shell invariant u·u drift, before committing the run at 1e-10. It checked its fields against an independent reimplementation at 200 random points - then deliberately sign-flipped a field component to prove the check could fail (residual 2.0 wrong, 0.066 right). It refused to treat the time component of the initial four-velocity as a free input, deriving it from the mass-shell constraint instead. Twenty-eight calls per trial is not haste: it is a model that writes once because it has already tried to break what it is about to write. The bill for that conviction arrives later, in tokens.
Sol specifies more than the task asks. The habit cuts both ways. On the constitutive problem it built one of only two genuinely independent oracles in the cohort, closing a quadrature check of the constitutive identity to 3.7e-12 under the session title “Falsify the update against the independent bulk-modulus definition.” On the relativistic problem the same instinct manufactured its slip: the problem pins the z-component of the initial four-velocity at -0.62, and sol renormalized the state so the coordinate velocity came out at -0.62 instead, handing the particle 27% excess longitudinal momentum. Its own consistency check then blessed the misreading: mass-shell exactly one, velocity exactly as read, every downstream check self-consistent and wrong. Precision is not the same thing as fidelity to the spec.
Luna iterates where it should investigate. It carries the least configuration into a run and reads more of the problem than anyone, which serves it well on the easier problems. When the physics pushed back on the constrained-consistency problem, the same temperament turned into churn: 68 tool calls in a single trial, and a committed solution still 30% off. That trial was also the most-validated in the cohort - 22 checks, every one verifying its own equations against themselves - and the pattern repeats: on the constitutive problem, luna was the only model that never tested against anything independent of its committed equations. Validation without an external reference isn’t validation.
Terra economizes. It spends the smallest share of its clock on derivation and testing and the largest on editing - more trial and error against the file - and it broadly gets away with it: correct formulations, the cheapest and fastest trials in the study. Its slip on the steady-state linearization problem is the study’s narrowest miss, and its strangest: the physics was exact, but terra rewrote the required simulation horizon (stop = 5.0
became stop = 3.0
), and the comparison drifts to 1.34% past the truncated run. The cruel twist: terra was the only model to build a nonlinear reference to validate its reduction - and it gave the reference the same truncated horizon, blinding its own check. The temperament that saves the money is the same one that trims the margins.
04 · The frontier problem
The hardest problem, up close. #
P5 asks for a complete flight vehicle: NASA’s HL-20 lifting body on approach, with six-degree-of-freedom rigid-body dynamics, aerodynamics from roughly 170 wind-tunnel tables, a standard atmosphere, and the control-surface mixer described in two NASA technical memos. Eight physics scenarios grade the result, from a gravity-only sanity check to a ten-second closed-loop flight. No model solved it. This is where the frontier separates:
Terra shipped a flight vehicle without reading the flight manual. Its one attempt to extract the NASA memos failed at minute 3.5 and it never tried again - so it invented the control mixer, reverse-engineering gains from the scenario tables. It ran the full flight four times and never looked past the return code; its own log shows a control surface settling eleven degrees from the spec’s stated value, unremarked. Cheapest attempt in the cohort, worst full flight.
Sol read everything and verified the parts. It extracted both memos by minute three, built its aerodynamics as an interpreter over the supplied tables, and locked them against the 24 check cases NASA embedded in the data file - all 24 passing by minute 27 of the fastest run in the cohort. What it never verified was a trajectory: its dutch-roll run ends below sea level in a 48-degree dive, and its conclusion was that “the rudder’s behavior seems correct.”
Luna never got the airplane to fly. It fought one compile error twenty-three times across eighty minutes, then resolved it by rewiring the vehicle’s vertical aerodynamic force and pitch torque to zero and shipping scenarios it only ever ran for a tenth of a second. Its committed vehicle descends without ever rotating: the attitude quaternion never moves. Two and a quarter hours, 544 tool calls, last place.
Fable spent a third of its run reading before it wrote a line. Spec, tables, both memos - it rendered PDF pages as images to settle one ambiguous constant - then wrote sixteen files that compiled on the first try, ran all eight scenarios, and closed with a falsification battery against every oracle it could find. Where an external reference existed, it scored highest in the cohort; where none did, it fell with everyone else.