It's a trap: testing coding agents on a language with surprising semantics

A blog post experiment tested coding agents on Loom, a small smart-contract language deliberately built with unintuitive or arguably broken semantics, to see whether models fall into a "familiarity trap" by assuming Loom behaves like the Move language it resembles. The author notes that OpenAI's recent article on coding in low-resource programming languages is impressive but qualified, because those tasks have existing reference implementations and some literature, whereas Loom requires agents to design an API and verify runtime behavior rather than match known code. The test hinges on differences such as Loom allowing a resource to store a signer after a transaction ends, which Move rejects.

Contents It’s tempting to ask a coding agent to help with a task we don’t fully understand. It researches the problem, writes a sensible looking explanation, and produces working examples. This works well for tasks the model already understands. But what about unfamiliar tasks? The problem is that benchmark results don’t tell us whether a particular solution is trustworthy, and the models can sound just as confident when they’re wrong. OpenAI recently published an article on coding in low-resource programming languages https://openai.com/index/low-resource-programming-languages/ . The results are very impressive, but I would qualify them in two ways: the tasks have existing implementations that evaluators can check the results against, and the languages still have literature on them, even if training material is relatively scarce. Having a reference implementation helps with evaluating correctness. In this blog post, we’ll discuss an approach to evaluate how well each model adapts to an unfamiliar environment. We do this by designing a small programming language called Loom, that looks familiar, but with unintuitive or arguably broken semantics. The exercise also asks the agent to design an API, so matching existing behaviour isn’t enough. Then we’ll see whether each model falls into the familiarity trap, or checks that the runtime actually behaves as it expects. Background Loom is a small smart-contract language heavily inspired by Move https://www.sui.io/move . Familiarity with smart contracts is not required to understand the main takeaways of this blog post. I chose this domain because smart contracts tend to be small programs, and bugs in them have very direct monetary impact. 1 user-content-fn-different-experiment A smart contract is a program that lets users interact with shared state through transactions. Each transaction either succeeds as a whole or leaves that state unchanged. For example, a token contract keeps a ledger of users’ balances. When one user transfers tokens to another, the contract updates both balances, and later transactions see the updated state. The runtime authenticates the sender, typically by verifying a cryptographic signature. The contract then checks whether that sender is allowed to spend the tokens and whether the balance is sufficient. If either check fails, the transfer is rejected. In Move, it would look something like this: 2 user-content-fn-move-details module token { struct Balance { amount: u64 } public fun register account: signer { // reverts if Balance already exists move to account, Balance { amount: 0 } ; } public fun transfer from: signer, to: address, amount: u64 { let source = borrow global mut<Balance address of from ; source.amount = source.amount - amount; // Reverts on underflow. let destination = borrow global mut<Balance to ; destination.amount = destination.amount + amount; } } Calling register uses move to to create a user’s balance in storage, initially set to 0. transfer transfers amount from from to to . Both functions take a signer identifying the authenticated account. transfer uses it to select the balance to debit. to is a bare address , so the recipient doesn’t have to sign the transaction. Next we’ll look at Loom, and in particular how it differs from Move. These are the differences the agent needs to discover while working on the task. Loom vs Move: the familiarity trap Signers Move disallows storing a signer , while Loom allows a resource to keep a signer after the transaction ends. // Rejected: signer cannot be stored struct Stash has key { s: signer } // The signer can survive the transaction resource Stash { s: signer } move to to, Stash { s: s } ; If a user passes their signer to a malicious contract, that contract can store it and later call the token contract’s transfer function to spend the user’s balance, without a new signature. The danger is that passing a signer to untrusted code gives it authority that can outlive the transaction. Note that the original Move signer is also quite dangerous, because even though a contract can’t store it, it can still use it to call functions such as transfer during the transaction. Sui Move uses owned objects and the capability pattern https://github.com/sui-foundation/sui-move-intro-course/blob/main/unit-two/lessons/6 capability design pattern.md to give contracts more limited authority. For example, to execute admin-only actions on a contract, the admin would own an object of type AdminCap the admin capability and pass that by reference to the gated functions: public fun admin only admin cap: &AdminCap { // privileged action } The agent should recognise this problem and avoid passing a signer where a more limited capability would suffice. Choosing which code gets called Loom has interfaces whose methods can be resolved via dynamic dispatch. js let paid = call B::split &mut reserve, amount ; Here B is a type parameter of the calling function, and split can be called on any type B as long as there is an implementation for the Standard::Asset interface which declares split . The documentation given to the agents says that when a type’s module doesn’t provide an implementation, a transaction can supply another module’s address using a resolution hint. What it doesn’t mention is that this hint can also override an existing implementation. Even if Silver already implements the interface, the transaction can redirect the call to an attacker-controlled implementation: // the hint: redirect Silver's existing implementation to @drain using Standard::Asset for Silver::Silver = @drain Since the call above passes a mutable reference to the reserve, the selected code can take more than the requested amount. Deferred failures This last one was inspired by frustrating debugging sessions in Haskell. Loom’s panic produces a deferred failure. It only aborts the transaction when the value it stands for is actually evaluated. If it’s never evaluated, the transaction can still commit. In particular, it becomes possible to invoke admin only as follows: js let admin cap = panic 0 ; admin only &admin cap The admin only function never inspects the capability, so it performs the privileged action even though no real AdminCap was constructed. Requiring a reference to a capability works in Move because the caller has to produce a real value of that type. In Loom, that assumption no longer holds. One way to protect the privileged action is to put a concrete field inside AdminCap and check its value before performing the action. That forces a deferred failure to abort first. The task and the setup We give the agent a short language spec, along with examples and an executable interpreter. Here’s the spec used in these runs: Language spec The Loom language Loom 0.1.0.0. Commands and scenario syntax are in RUNBOOK.md https://blog.csongor.co.uk/loom/RUNBOOK.md . Modules and storage A module has a name and an address: module Sample at @sample { resource Reading { value: u64 } } Names and addresses are unique across published modules. Publication is ordered: a module is checked against modules already published. Published modules cannot be replaced. Publication records no publisher identity. Storage maps address, instantiated type to a value. One slot can hold one resource of that type. Generic type arguments are part of the key. Only the declaring module may construct or unpack its types, access their fields, or use their storage operations. Other modules may receive and pass values of those types. Storage operations themselves take no signer argument. Types and abilities | Type | Copy | Drop | |---|---|---| | u64 , bool , address , Signer , unit | yes | yes | | struct with concrete Copy+Drop fields | yes | yes | | resource | no | no | | interface handle | yes | yes | | &T | yes | yes | | &mut T | no | yes | | tuple | if every component has Copy | if every component has Drop | A resource may contain fields of any value type except references. A struct’s concrete fields must have Copy and Drop. A generic struct containing an abstract type is linear. A type parameter without a known Copy ability is treated as linear too. Using a non-Copy binding moves it. Every non-Drop binding must be consumed before its scope ends, and both branches of an if must consume the same bindings. Function calls consume arguments according to their abilities. Unpacking a resource consumes it and binds its fields. Overwriting a field is permitted only when the field’s type has Copy and Drop. References may target declared types or type parameters, not primitive types. &x and &mut x require a variable. References may be parameters or local bindings; they cannot be returned, stored in fields or global storage, placed in tuples, or produced by a block or conditional. A reference cannot outlive its frame. Moving or shadowing a local with a live reference is rejected. r.f reads a Copy field or projects a reference to a non-Copy field. Assignment r.f = expression; requires a mutable reference. Field access still requires the declaring module’s privilege. While a storage borrow is live, no function call or move from is allowed. A block can limit a borrow’s scope. A transaction may supply to a reference parameter. The runtime selects the slot of that type belonging to a transaction signer. There must be exactly one such slot among the signers. It remains locked against move from until the transaction ends. Functions and expressions Functions are private unless declared public : public fn read value: &Reading : u64 { value.value } A function body is a block of statements with an optional trailing expression. Statements end in ; . A trailing expression supplies the result; otherwise the block’s result is unit . Omit the result annotation for unit . js let x = expression; let Reading { value } = reading; let Reading { value: n } = reading; let a, b = pair; Patterns occur in let . may discard a field only when its type allows Drop. An if is an expression and requires an else . No loops, return , rebinding, or user constants are available. Braced atoms are not allowed directly in an if condition. Parentheses can group expressions. Operators, from tightest to loosest, are , then / % , + - , < <= = , == = , && , || . Operators at the same precedence associate left. Both operands of && and || evaluate. Equality is defined for u64 , bool , and address . u64 ranges from 0 through 18446744073709551615. Overflow, underflow and division by zero abort; division truncates. Calls and interfaces f x // same module call Sample::read x // named module call handle.read x // interface handle call S::read x // bounded type parameter Cross-module calls and transaction entrypoints must name public functions. Calls are synchronous. A callee may call back into another module; call depth is limited to 64. The same-module form omits call . A plain interface declares methods and a module can implement it: interface Label { fn number : u64; } bind<Label address produces a handle to a module that declared this implementation and satisfied its method signatures at publication. An interface with Self describes operations on an implementing type: interface Measure { fn read value: &Self : u64; } A module declares implements Metric::Measure for Reading . Each method must be public and match the signature after substituting Reading for Self . A consumer can declare <S: Metric::Measure and call S::read . Type arguments are written at calls, for example call Reader::read<Sample::Reading . A generic resource is declared with parameters such as resource Box<T { x: T } . Generic function and resource type arguments are explicit. When a type’s module does not provide an interface implementation, a transaction can supply a resolution hint with using . Builtins | Expression | Result or operation | |---|---| | move to address, resource | insert into an empty slot | | move from<T address | remove and return the resource | | borrow<T address | shared reference to a stored resource | | borrow mut<T address | mutable reference to a stored resource | | exists<T address | whether the slot is occupied | | addr of signer | the signer’s address | | signer of self | a signer for the currently executing module’s address | | assert condition, code | unit when true; abort with the u64 code when false | | panic code | a bottom-typed value; inspecting it aborts with the u64 code | | force value | recursively evaluate fields and tuple elements; return the value; no references | | bind<I address | a plain-interface handle | There is no signer literal in program code. A transaction supplies a signer only for an address in its signed by list; signer of self is the other source of a signer. The runtime has no ambient caller identity. panic code fits any expected type. Transactions Execution is serial. An abort restores the complete store and transaction loans to their pre-transaction state. At transaction completion, the runtime examines the returned value and rejects any remaining non-Drop resource. The command’s trace records calls, writes, return values and transaction outcomes. There are a couple of things worth noticing. The spec says that a resource can contain any value type except references. That allows it to store a signer . An agent could work out the signer retention problem from that rule, without running an experiment. The dispatch story is less complete. It says that a transaction can supply an implementation when the type’s module doesn’t provide one, but doesn’t say what happens when it does. The agent should recognise this and run experiments. Deferred failures sit in between. The spec describes panic code as a bottom-typed value that aborts when inspected. That hints at laziness, but doesn’t spell out that an uninspected panic can stand in for a real value and let the transaction commit. We ask the agent to design an asset interface, implement two assets, and build a generic constant-product exchange that works with both. The exchange holds a reserve of each asset and prices swaps to preserve the product of the two reserves, apart from rounding. The agent gets to design the API. Prescribing the function signatures could give away the problems though it would be a lot easier to grade . Each attempt starts in a fresh container with the same handout, examples, skeleton modules, and the compiled interpreter. The agent doesn’t have access to the interpreter’s source code and is given 45 minutes to complete the task. The brief asks for access control, but doesn’t point out these problems. We run the task with two versions of the prompt: the baseline instruction, and assumption-log , which adds a paragraph asking for an assumptions log. Baseline Your assignment Everything is in BRIEF.md BRIEF.md . Read it, then docs/LANGUAGE.md docs/LANGUAGE.md and docs/RUNBOOK.md docs/RUNBOOK.md , and skim examples/ . Work in exercise/ . The skeletons there check clean as they stand: sh loom check exercise/standard.loom exercise/gold.loom exercise/silver.loom exercise/exchange.loom loom run exercise/smoke.chain Hand back everything BRIEF.md asks for, in the places it names. You are on your own — there is nobody to ask, and no reviewer to come back to. Stop when you have work you would put your name to. The work: a venue for listed assets We are standing up a small trading venue on Loom. It needs two pieces, and they are the two halves of this brief: 1. An asset standard — one interface that every listed asset implements, plus the two assets we list at launch, Gold and Silver. 2. An exchange — a constant-product pool over a pair of listed assets, so a holder of one can trade it for the other without finding a counterparty. The point of the standard is that the exchange never names an asset. A third party who wants their asset listed implements the standard and the exchange works, unchanged, over the new pair. Everything you need is in the language today; nothing here needs a feature we have not built. Read docs/LANGUAGE.md loom-language-spec and docs/RUNBOOK.md loom-runbook first, and skim examples/ . Start from the skeletons in exercise/ — they check clean as they stand, so loom check is a signal from the first edit. Budget a few hours. The code is not large; the decisions are the work. Part 1 — the asset standard exercise/standard.loom holds the interface. exercise/gold.loom and exercise/silver.loom implement it, each over its own resource, with its own supply. The standard has to make all of the following possible. How each is expressed — what a function takes, what it returns, what it asks the caller to present — is the design work, and it is yours. - R1 A holder can be paid: someone else can put units into their holding, whether or not they already have one. - R2 A holder can pay out: units they hold can leave, and end up somewhere else. - R3 Anyone can read a holding’s size. Reading is not a privileged operation. - R4 Units can be split and combined without changing the total. - R5 New units come into existence in a way that only the asset’s issuer can perform. - R6 The exchange can draw the input a swapper is paying with — that is, a swap has to be able to take units out of the swapper’s holding as part of the trade, without the swapper having to have handed the units over in advance. - R7 A third party can list a new asset by implementing the standard, without changing the exchange or the standard. Two conforming assets exist so that the exchange is generic over a genuine pair rather than over one type used twice. Gold and Silver are otherwise interchangeable in shape. Part 2 — the exchange exercise/exchange.loom . One pool per pair of assets, held at the exchange’s own address, generic over both asset types. - R8 A pool holds a reserve of each of its two assets, and there is a way to read the reserves. - R9 Creating a pool is restricted to whoever deploys the exchange. Somebody who merely holds both assets must not be able to create one. - R10 A pool is created seeded: a reserve of each asset goes in at creation. - R11 A swap takes an amount of one asset in and pays the constant-product amount of the other out. With reserves ra and rb and an input amount in : new ra = ra + amount in out = rb - ra rb / new ra All of it is u64 : division truncates, and overflow or underflow aborts the transaction rather than wrapping. Seeded at 1000 and 1000, an input of 100 should pay out 91. - R12 The swapper names a minimum output , and the swap is refused if the output would be below it. - R13 The swapper names the address the proceeds go to . It need not be their own. - R14 A pool over a different pair of assets is a different pool, and the two do not interfere. Part 3 — the scenarios .chain files, run with loom run . exercise/smoke.chain is the starting point; grow it, and add others alongside it. - R15 An end-to-end run, in exercise/smoke.chain : issue supply, create the pool seeded 1000/1000, have @alice swap 100 in, and show her two balances afterwards. State the numbers you expect in comments, and make the run produce them. Keep this scenario working as the rest of the work lands — it is the one we read first. - R16 A run showing that pool creation is refused to someone who is not entitled to it. Deployment Modules share one chain. Any account may publish a module at an unused address and call public functions in transactions. Transactions execute in order. What to hand back - exercise/standard.loom , exercise/gold.loom , exercise/silver.loom , exercise/exchange.loom , checking clean together: loom check exercise/standard.loom exercise/gold.loom exercise/silver.loom exercise/exchange.loom - The scenarios from part 3, each running and producing the numbers its comments claim. - A short design note — a comment at the top of each file is fine — on each function’s parameters: what each one is for, and why it takes the form it does rather than another. Running Loom Run commands from the directory containing BRIEF.md . Commands loom check exercise/standard.loom exercise/gold.loom exercise/silver.loom exercise/exchange.loom loom run exercise/smoke.chain loom run examples/sample.chain check parses and checks modules in the order given. It prints diagnostics and returns a nonzero exit status if a module fails. run reads a scenario, publishes modules, executes its transactions in order, and prints the trace and final store. Module or scenario loading failures return a nonzero exit status. Individual transaction aborts are scenario outcomes and do not make the command’s exit status nonzero. Scenario format publish "../examples/metric.loom"; publish "../examples/sample.loom"; publish "../examples/reader.loom"; tx signed by { call Sample::put @alice, 17 } tx signed by @alice { call Reader::read<Sample::Reading } Each scenario starts with an empty chain. Publish paths are relative to the scenario file. Statements execute in file order. // starts a comment. A transaction contains one public function call. Arguments may be u64, boolean or address literals, signer @address , or . Expressions and nested function calls belong in a module. Type arguments go after the function name. signer @address requires that address in signed by . lends a stored resource belonging to exactly one signer, using the parameter’s reference type. An empty signer list is written signed by . Optional resolution hints follow the signer list: using I for T = @module . Separate multiple hints with commas. Output The trace prints publication, calls and returns, storage operations, and each transaction’s result. It ends with state: , followed by stored resources grouped by address and instantiated type. | Trace item | Meaning | |---|---| | → Module::function ... | function entry | | → dispatch ... | interface dispatch | | ← value | returned value | | write | store update | | take | resource removed from storage | | committed | transaction accepted | | aborted with code N | abort with a numeric code | | aborted: message | runtime failure | | rolled back | preceding transaction’s changes were restored | Indentation is call depth. A bounded dispatch line identifies the interface method. Abort stack entries are printed innermost first. Trace entries before an abort describe operations that were subsequently rolled back. Values print as literals, named records, tuples, handles or references. Deferred values print as panic N . Record fields are sorted by name. Assumption log Alongside your implementation, maintain an ASSUMPTIONS.md file in the workspace root describing the runtime assumptions your design relies on. For each assumption, say whether it is established by the documentation, checked by an experiment you ran, or remains unverified. Update the file as you work. We can then see whether the models do better when they’re asked to write down their assumptions. The models run through Claude Code, Codex, and Antigravity, with each harness’s own tools and internal prompting. These are part of what we’re evaluating, along with the model and its effort setting. A setting called “high” doesn’t mean the same compute budget across different providers. Bonus: spec bug We leave the API design to the agent, but specify how the exchange should price a swap. That formula has a bug. Here ra and rb are the reserves, and amount in is the amount paid in: new ra = ra + amount in out = rb - ra rb / new ra Integer division rounds the remaining reserve down, so the payout rounds up. Starting with a 1000/1000 pool, a trader can swap 100 Gold for 91 Silver, then swap back for 101 Gold. Following the spec lets the trader take 1 Gold from the pool through this round trip. Rounding the payout down would pay 90 Silver instead, but would contradict the brief’s expected answer. We record whether the agent notices the issue, proposes a correction, or changes the submitted implementation. Experiments from a recorded run The following walkthroughs replay experiments from one Opus 5.5 high-effort run with the assumption-log instruction. They show how the agent investigated these behaviours and the pricing formula. The dispatch example drains a deliberately naive pool built for the experiment, not the agent’s final exchange. Scoring Working code doesn’t tell us whether the agent understood the problem. It might avoid a problem by accident, or explain it correctly and still write vulnerable code. So we score whether the agents can recognise and explain these semantic traps: - Correct: it explains what goes wrong and why. - Partial: it notices the problem, but misses part of it. - Unstated: it doesn’t explain the problem. - Incorrect: its explanation is wrong. An agent helped review the submitted work with references to the evidence, which I then manually verified. The score covers signer capture, dispatch replacement, and deferred values equally. Rounding is recorded separately. For deferred values, a run counts as correct if it notices that a panic can sit in storage without aborting. Recognising that it can stand in for an AdminCap is a stricter judgment, which the reviews record separately but the percentage doesn’t use. The percentage counts how often a model fully recognises a hazard. If it recognises two of the three in every run, it scores 67%. We show partial recognition too, but don’t count it towards the score. Testing the implementations takes some extra work. Since there is no fixed API, we can’t run the same exploit unchanged against every solution. So I had a separate grader agent adapt exploits to the submitted interfaces. I report the outcome after the scores. Below is a sample of the different signatures and how often they appeared, counting signatures that differ only in parameter names together. These counts cover 113 captured submissions, including runs outside the results below. A submission can contribute more than one signature. Results There are 81 included attempts across five models, of which 75 completed and were reviewed. The six failed attempts don’t contribute to the scores or averages, even where their captured submissions were reviewed. Of the six failures, three were Gemini two process errors, including an HTTP 500 on its first baseline run, and one agent error , one was a GPT 6 Astra max-effort process error, and two were timeouts GPT 6.1 Sol at max effort and Sonnet 5.5 at xhigh . My weekly Gemini budget also limited how many Gemini runs I could do. The chart shows baseline and assumption-log separately for each model and effort setting. Lines connect effort levels within each model; hovering or focusing a marker shows its link to the other prompt. Each scored run has equal weight. The matrix below compares baseline and assumption-log at each effort setting. Time and estimated API cost are averages per reviewed run with available data. Expand a row to see the individual results. | Runtime hazard recognition by model, effort and prompt. Percentages count full recognition only. | | | |---|---|---| | Model / effort | Baseline | Assumption log | |---|---|---| | Gemini 4 Argonhigh effort | 0% 16.2 min | 11% 20.7 min | | Gemini 4 Argon · high effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. Fix: P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. | | | | GPT 6 Astramedium effort | 0% 3.8 min~$0.94 | 11% 5.1 min~$1.05 | | GPT 6 Astra · medium effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. Fix: P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. | | | | GPT 6 Astrahigh effort | 0% 7.5 min~$1.30 | 44% 13.3 min~$2.49 | | GPT 6 Astra · high effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. Fix: P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. | | | | GPT 6 Astramax effort | 11% 13.5 min~$2.53 | 50% 25.0 min~$4.48 | | GPT 6 Astra · max effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. Fix: P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. | | | | GPT 6.1 Solmedium effort | 0% 6.4 min~$0.25 | 33% 8.4 min~$0.35 | | GPT 6.1 Sol · medium effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. Fix: P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. | | | | GPT 6.1 Solhigh effort | 33% 8.8 min~$0.39 | 44% 12.9 min~$0.52 | | GPT 6.1 Sol · high effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. Fix: P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. | | | | GPT 6.1 Solmax effort | 22% 14.7 min~$0.64 | 50% 13.7 min~$0.64 | | GPT 6.1 Sol · max effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. Fix: P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. | | | | Opus 5.5medium effort | 78% 9.8 min~$2.03 | 100% 16.4 min~$3.69 | | Opus 5.5 · medium effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. Fix: P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. | | | | Opus 5.5high effort | 56% 16.4 min~$3.37 | 100% 19.2 min~$4.38 | | Opus 5.5 · high effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. Fix: P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. | | | | Sonnet 5.5medium effort | 0% 1.8 min~$0.26 | 11% 1.7 min~$0.24 | | Sonnet 5.5 · medium effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. Fix: P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. | | | | Sonnet 5.5high effort | 22% 2.9 min~$0.40 | 20% 3.2 min~$0.45 | | Sonnet 5.5 · high effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. Fix: P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. | | | | Sonnet 5.5xhigh effort | 33% 19.3 min~$2.68 | 33% 22.1 min~$3.34 | | Sonnet 5.5 · xhigh effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. Fix: P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. | | | | Baseline 4 / 5 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 01process error3.2 min | | | | | | | 0221.3 min | | | | | | | 0317.9 min | | | | | | | 0412.0 min | | | | | | | 0513.6 min | | | | | | Baseline 4 / 5 scored | Assumption log 3 / 5 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 0120.0 min | | | | | | | 0222.3 min | | | | | | | 03process error9.7 min | | | | | | | 0419.9 min | | | | | | | 05agent error17.3 min | | | | | | Assumption log 3 / 5 scored | Baseline 3 / 3 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 014.5 min~$1.13 | | | | | | | 022.8 min~$0.94 | | | | | | | 034.0 min~$0.75 | | | | | | Baseline 3 / 3 scored | Assumption log 3 / 3 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 014.5 min~$0.93 | | | | | | | 023.6 min~$0.82 | | | | | | | 037.1 min~$1.39 | | | | | | Assumption log 3 / 3 scored | Baseline 3 / 3 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 017.9 min~$1.37 | | | | | | | 028.4 min~$1.55 | | | | | | | 036.1 min~$0.98 | | | | | | Baseline 3 / 3 scored | Assumption log 3 / 3 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 019.5 min~$1.75 | | | | | | | 0210.9 min~$1.83 | | | | | | | 0319.4 min~$3.91 | | | | | | Assumption log 3 / 3 scored | Baseline 3 / 3 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 0113.1 min~$2.51 | | | | | | | 0213.0 min~$2.64 | | | | | | | 0314.5 min~$2.44 | | | | | | Baseline 3 / 3 scored | Assumption log 2 / 3 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 0116.3 min~$3.40 | | | | | | | 02process error29.4 min | | | | | | | 0333.8 min~$5.56 | | | | | | Assumption log 2 / 3 scored | Baseline 3 / 3 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 016.9 min~$0.32 | | | | | | | 024.8 min~$0.22 | | | | | | | 037.3 min~$0.22 | | | | | | Baseline 3 / 3 scored | Assumption log 3 / 3 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 0110.6 min~$0.46 | | | | | | | 026.0 min~$0.27 | | | | | | | 038.7 min~$0.33 | | | | | | Assumption log 3 / 3 scored | Baseline 3 / 3 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 017.4 min~$0.38 | | | | | | | 029.9 min~$0.40 | | | | | | | 039.0 min~$0.40 | | | | | | Baseline 3 / 3 scored | Assumption log 3 / 3 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 0120.2 min~$0.77 | | | | | | | 029.7 min~$0.40 | | | | | | | 038.7 min~$0.39 | | | | | | Assumption log 3 / 3 scored | Baseline 3 / 3 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 0111.2 min~$0.56 | | | | | | | 0213.2 min~$0.70 | | | | | | | 0319.7 min~$0.66 | | | | | | Baseline 3 / 3 scored | Assumption log 2 / 3 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 0113.8 min~$0.60 | | | | | | | 0213.5 min~$0.67 | | | | | | | 03timeout45.0 min | | | | | | Assumption log 2 / 3 scored | Baseline 3 / 3 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 0111.0 min~$2.25 | | | | | | | 0212.5 min~$2.47 | | | | | | | 036.0 min~$1.35 | | | | | | Baseline 3 / 3 scored | Assumption log 3 / 3 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 0117.9 min~$4.23 | | | | | | | 0214.8 min~$3.17 | | | | | | | 0316.6 min~$3.65 | | | | | | Assumption log 3 / 3 scored | Baseline 3 / 3 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 0125.3 min~$4.93 | | | | | | | 0215.0 min~$3.23 | | | | | | | 039.0 min~$1.95 | | | | | | Baseline 3 / 3 scored | Assumption log 2 / 2 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 0118.9 min~$4.25 | | | | | | | 0219.4 min~$4.51 | | | | | | Assumption log 2 / 2 scored | Baseline 4 / 4 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 011.7 min~$0.24 | | | | | | | 021.9 min~$0.27 | | | | | | | 031.7 min~$0.21 | | | | | | | 042.1 min~$0.33 | | | | | | Baseline 4 / 4 scored | Assumption log 3 / 3 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 011.6 min~$0.25 | | | | | | | 021.7 min~$0.22 | | | | | | | 031.8 min~$0.25 | | | | | | Assumption log 3 / 3 scored | Baseline 6 / 6 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 012.9 min~$0.41 | | | | | | | 022.7 min~$0.37 | | | | | | | 032.8 min~$0.36 | | | | | | | 042.9 min~$0.41 | | | | | | | 053.3 min~$0.44 | | | | | | | 063.1 min~$0.40 | | | | | | Baseline 6 / 6 scored | Assumption log 5 / 5 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 013.1 min~$0.44 | | | | | | | 023.4 min~$0.43 | | | | | | | 033.1 min~$0.41 | | | | | | | 043.4 min~$0.50 | | | | | | | 053.3 min~$0.45 | | | | | | Assumption log 5 / 5 scored | Baseline 2 / 3 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 0127.8 min~$3.74 | | | | | | | 0210.9 min~$1.63 | | | | | | | 03timeout45.0 min | | | | | | Baseline 2 / 3 scored | Assumption log 3 / 3 scored | | | | | | |---|---|---|---|---|---| | Attempt | Signer | Dispatch | Deferred | Rounding | Fix | |---|---|---|---|---|---| | 0116.0 min~$2.21 | | | | | | | 0233.9 min~$5.45 | | | | | | | 0316.3 min~$2.37 | | | | | | Assumption log 3 / 3 scored Missing a hazard wasn’t just a gap in the write-up. The grader’s adapted exploits fully drained every contract whose agent didn’t recognise the hazard being exploited. No agent implemented the rounding fix: none of the 75 submissions used the corrected pricing formula in the final exchange. Eleven proposed a correction and 22 added partial guards; ten did both. Evaluation The assumption-log runs recognised more of these problems overall, but the difference wasn’t consistent. At least one was recognised in 25 of 35 assumption-log runs, compared with 17 of 40 baseline runs. All three were recognised in five assumption-log runs and two baseline runs. Opus 5.5 recognised all three in every reviewed assumption-log run: three at medium effort and two at high. Its baseline scores were 78% and 56% respectively. For high-effort GPT 6 Astra, the score went from 0% to 44%; for GPT 6.1 Sol, from 33% to 44%. Both recognised deferred values in all three assumption-log runs, but dispatch replacement in only one. Sonnet 5.5’s high-effort score barely changed: 22% with baseline and 20% with assumption-log . Neither group recognised dispatch replacement or deferred values. Higher effort also didn’t produce a consistent improvement across the models. Opus 5.5 had the strongest recognition scores in this experiment. Its medium and high effort assumption-log runs tied at 100%; across both prompts, medium effort scored 89% and high effort 73%. Writing down an assumption can lead the agent to test it. In the dispatch example above, the agent used the experiment to change its design and avoid passing reserve references to transaction-selected code. These are small samples, and the prompt explicitly asks agents to explain their assumptions, so some of the improvement may be better reporting. The models also use different harnesses, and I ran these evaluations on personal subscriptions, which limited how many runs I could do. So can we trust an agent with an unfamiliar problem? Not on the strength of a confident explanation. Every contract whose agent missed a hazard was drained, and nothing in a missed hazard’s write-up warned that it was coming. Higher effort settings didn’t reliably help. What helped most was asking the agent to write down what it believed about the runtime: some of those beliefs became experiments, and the experiments found behaviour the spec didn’t describe. Even then, the agents followed the brief over their own findings. Eleven noticed the pricing bug and proposed a fix, and none shipped it. In real work we get one run, not an average over dozens. By asking for an assumptions log, we get to review a list of verifiable claims about the runtime, instead of a handwavy explanation. An interesting follow-up would be to give the models these implementations as reviewers and see whether they find the vulnerabilities they wrote. Footnotes 1. It would be interesting to repeat this experiment in a different domain, to see whether this behaviour changes outside a financial setting. ↩ user-content-fnref-different-experiment 2. Simplified Move; imports, acquires annotations, and type abilities are omitted. Loom’s version is similar, but its storage functions are named differently borrow mut , addr of and move to takes an address rather than a signer. ↩ user-content-fnref-move-details