{"slug": "it-s-a-trap-testing-coding-agents-on-a-language-with-surprising-semantics", "title": "It's a trap: testing coding agents on a language with surprising semantics", "summary": "A blog post experiment tested coding agents on Loom, a small smart-contract language deliberately built with unintuitive or arguably broken semantics, to see whether models fall into a \"familiarity trap\" by assuming Loom behaves like the Move language it resembles. The author notes that OpenAI's recent article on coding in low-resource programming languages is impressive but qualified, because those tasks have existing reference implementations and some literature, whereas Loom requires agents to design an API and verify runtime behavior rather than match known code. The test hinges on differences such as Loom allowing a resource to store a signer after a transaction ends, which Move rejects.", "body_md": "## Contents\n\nIt’s tempting to ask a coding agent to help with a task we don’t fully understand. It researches the problem, writes a sensible looking explanation, and produces working examples. This works well for tasks the model already understands. But what about unfamiliar tasks? The problem is that benchmark results don’t tell us whether a particular solution is trustworthy, and the models can sound just as confident when they’re wrong.\n\nOpenAI recently published an [article on coding in low-resource programming languages](https://openai.com/index/low-resource-programming-languages/). The results are very impressive, but I would qualify them in two ways: the tasks have existing implementations that evaluators can check the results against, and the languages still have literature on them, even if training material is relatively scarce. Having a reference implementation helps with evaluating correctness.\n\nIn this blog post, we’ll discuss an approach to evaluate how well each model adapts to an unfamiliar environment. We do this by designing a small programming language called Loom, that looks familiar, but with unintuitive (or arguably broken) semantics. The exercise also asks the agent to design an API, so matching existing behaviour isn’t enough. Then we’ll see whether each model falls into the familiarity trap, or checks that the runtime actually behaves as it expects.\n\n## Background\n\nLoom is a small smart-contract language heavily inspired by [Move](https://www.sui.io/move). Familiarity with smart contracts is not required to understand the main takeaways of this blog post. I chose this domain because smart contracts tend to be small programs, and bugs in them have very direct monetary impact.[1](#user-content-fn-different-experiment)\n\nA smart contract is a program that lets users interact with shared state through transactions. Each transaction either succeeds as a whole or leaves that state unchanged. For example, a token contract keeps a ledger of users’ balances. When one user transfers tokens to another, the contract updates both balances, and later transactions see the updated state.\n\nThe runtime authenticates the sender, typically by verifying a cryptographic signature. The contract then checks whether that sender is allowed to spend the tokens and whether the balance is sufficient. If either check fails, the transfer is rejected.\n\nIn Move, it would look something like this:[2](#user-content-fn-move-details)\n\n```\nmodule token {\n    struct Balance { amount: u64 }\n\n    public fun register(account: signer) {\n        // reverts if Balance already exists\n        move_to(account, Balance { amount: 0 });\n    }\n\n    public fun transfer(from: signer, to: address, amount: u64) {\n        let source = borrow_global_mut<Balance>(address_of(from));\n        source.amount = source.amount - amount; // Reverts on underflow.\n        let destination = borrow_global_mut<Balance>(to);\n        destination.amount = destination.amount + amount;\n    }\n}\n```\n\nCalling `register` uses `move_to` to create a user’s balance in storage, initially set to 0. `transfer` transfers `amount` from `from` to `to`.\nBoth functions take a `signer` identifying the authenticated account. `transfer` uses it to select the balance to debit. `to` is a bare `address`, so the recipient doesn’t have to sign the transaction.\n\nNext we’ll look at Loom, and in particular how it *differs* from Move. These are the differences the agent needs to discover while working on the task.\n\n## Loom vs Move: the familiarity trap\n\n### Signers\n\nMove disallows storing a `signer`, while Loom allows a resource to keep a `signer` after the transaction ends.\n\n```\n// Rejected: signer cannot be stored\nstruct Stash has key { s: signer }\n// The signer can survive the transaction\nresource Stash { s: signer }\nmove_to(to, Stash { s: s });\n```\n\nIf a user passes their `signer` to a malicious contract, that contract can store it and later call the token contract’s `transfer` function to spend the user’s balance, without a new signature. The danger is that passing a signer to untrusted code gives it authority that can outlive the transaction.\n\nNote that the original Move signer is also quite dangerous, because even though\na contract can’t store it, it can still use it to call functions such as `transfer` during the transaction. Sui Move uses owned objects and the [capability pattern](https://github.com/sui-foundation/sui-move-intro-course/blob/main/unit-two/lessons/6_capability_design_pattern.md) to give contracts more limited authority.\nFor example, to execute admin-only actions on a contract, the admin would own an object of type `AdminCap` (the admin capability) and pass that by reference to the gated functions:\n\n```\npublic fun admin_only(admin_cap: &AdminCap) {\n // privileged action\n}\n```\n\nThe agent should recognise this problem and avoid passing a `signer` where a more limited capability would suffice.\n\n### Choosing which code gets called\n\nLoom has interfaces whose methods can be resolved via dynamic dispatch.\n\n``` js\nlet paid = call B::split(&mut reserve, amount);\n```\n\nHere `B` is a type parameter of the calling function, and `split` can be called on any type `B` as long as there is an implementation for the `Standard::Asset` interface (which declares `split`). The documentation given to the agents says that when a type’s module doesn’t provide an implementation, a transaction can supply another module’s address using a resolution hint. What it doesn’t mention is that this hint can also override an existing implementation. Even if `Silver` already implements the interface, the transaction can redirect the call to an attacker-controlled implementation:\n\n```\n// the hint: redirect Silver's existing implementation to @drain\nusing [Standard::Asset for Silver::Silver => @drain]\n```\n\nSince the call above passes a mutable reference to the reserve, the selected code can take more than the requested amount.\n\n### Deferred failures\n\nThis last one was inspired by frustrating debugging sessions in Haskell. Loom’s `panic` produces a deferred failure. It only aborts the transaction when the value it stands for is actually evaluated.\n\nIf it’s never evaluated, the transaction can still commit. In particular, it becomes possible to invoke `admin_only` as follows:\n\n``` js\nlet admin_cap = panic(0);\nadmin_only(&admin_cap)\n```\n\nThe `admin_only` function never inspects the capability, so it performs the privileged action even though no real `AdminCap` was constructed. Requiring a reference to a capability works in Move because the caller has to produce a real value of that type. In Loom, that assumption no longer holds.\nOne way to protect the privileged action is to put a concrete field inside `AdminCap` and check its value before performing the action. That forces a deferred failure to abort first.\n\n## The task and the setup\n\nWe give the agent a short language spec, along with examples and an executable interpreter. Here’s the spec used in these runs:\n\n## Language spec\n\n### The Loom language\n\nLoom 0.1.0.0. Commands and scenario syntax are in [RUNBOOK.md](https://blog.csongor.co.uk/loom/RUNBOOK.md).\n\n#### Modules and storage\n\nA module has a name and an address:\n\n```\nmodule Sample at @sample {\n  resource Reading { value: u64 }\n}\n```\n\nNames and addresses are unique across published modules. Publication is ordered: a module is checked against modules already published. Published modules cannot be replaced. Publication records no publisher identity.\n\nStorage maps `(address, instantiated type)` to a value. One slot can hold one\nresource of that type. Generic type arguments are part of the key.\nOnly the declaring module may construct or unpack its types, access their\nfields, or use their storage operations. Other modules may receive and pass\nvalues of those types. Storage operations themselves take no signer argument.\n\n#### Types and abilities\n\n| Type | Copy | Drop | \n|---|---|---|\n| `u64` ,`bool` ,`address` ,`Signer` ,`unit` | yes | yes | \n| `struct` with concrete Copy+Drop fields | yes | yes | \n| `resource` | no | no | \n| interface handle | yes | yes | \n| `&T` | yes | yes | \n| `&mut T` | no | yes | \n| tuple | if every component has Copy | if every component has Drop | \n\nA resource may contain fields of any value type except references. A struct’s concrete fields must have Copy and Drop. A generic struct containing an abstract type is linear. A type parameter without a known Copy ability is treated as linear too.\n\nUsing a non-Copy binding moves it. Every non-Drop binding must be consumed before\nits scope ends, and both branches of an `if` must consume the same bindings.\nFunction calls consume arguments according to their abilities. Unpacking a\nresource consumes it and binds its fields. Overwriting a field is permitted only\nwhen the field’s type has Copy and Drop.\n\nReferences may target declared types or type parameters, not primitive types.\n`&x` and `&mut x` require a variable. References may be parameters or local\nbindings; they cannot be returned, stored in fields or global storage, placed in\ntuples, or produced by a block or conditional. A reference cannot outlive its\nframe. Moving or shadowing a local with a live reference is rejected.\n\n`r.f` reads a Copy field or projects a reference to a non-Copy field. Assignment\n`r.f = expression;` requires a mutable reference. Field access still requires the\ndeclaring module’s privilege. While a storage borrow is live, no function call\nor `move_from` is allowed. A block can limit a borrow’s scope.\n\nA transaction may supply `_` to a reference parameter. The runtime selects the\nslot of that type belonging to a transaction signer. There must be exactly one\nsuch slot among the signers. It remains locked against `move_from` until the\ntransaction ends.\n\n#### Functions and expressions\n\nFunctions are private unless declared `public`:\n\n```\npublic fn read(value: &Reading): u64 { value.value }\n```\n\nA function body is a block of statements with an optional trailing expression.\nStatements end in `;`. A trailing expression supplies the result; otherwise the\nblock’s result is `unit`. Omit the result annotation for `unit`.\n\n``` js\nlet x = expression;\nlet Reading { value } = reading;\nlet Reading { value: n } = reading;\nlet (a, b) = pair;\n```\n\nPatterns occur in `let`. `_` may discard a field only when its type allows Drop.\nAn `if` is an expression and requires an `else`. No loops, `return`, rebinding,\nor user constants are available. Braced atoms are not allowed directly in an\n`if` condition. Parentheses can group expressions.\n\nOperators, from tightest to loosest, are `!`, then `* / %`, `+ -`, `< <= > >=`,\n`== !=`, `&&`, `||`. Operators at the same precedence associate left.\nBoth operands of `&&` and `||` evaluate. Equality is defined for `u64`, `bool`,\nand `address`. `u64` ranges from 0 through 18446744073709551615. Overflow,\nunderflow and division by zero abort; division truncates.\n\n#### Calls and interfaces\n\n```\nf(x)                        // same module\ncall Sample::read(x)        // named module\ncall handle.read(x)         // interface handle\ncall S::read(x)             // bounded type parameter\n```\n\nCross-module calls and transaction entrypoints must name public functions.\nCalls are synchronous. A callee may call back into another module; call depth\nis limited to 64. The same-module form omits `call`.\n\nA plain interface declares methods and a module can implement it:\n\n```\ninterface Label { fn number(): u64; }\n```\n\n`bind<Label>(address)` produces a handle to a module that declared this\nimplementation and satisfied its method signatures at publication.\n\nAn interface with `Self` describes operations on an implementing type:\n\n```\ninterface Measure { fn read(value: &Self): u64; }\n```\n\nA module declares `implements Metric::Measure for Reading`. Each method must be\npublic and match the signature after substituting `Reading` for `Self`.\nA consumer can declare `<S: Metric::Measure>` and call `S::read`. Type arguments\nare written at calls, for example `call Reader::read<Sample::Reading>(_)`.\nA generic resource is declared with parameters such as `resource Box<T> { x: T }`.\nGeneric function and resource type arguments are explicit.\n\nWhen a type’s module does not provide an interface implementation, a transaction\ncan supply a resolution hint with `using`.\n\n#### Builtins\n\n| Expression | Result or operation | \n|---|---|\n| `move_to(address, resource)` | insert into an empty slot | \n| `move_from<T>(address)` | remove and return the resource | \n| `borrow<T>(address)` | shared reference to a stored resource | \n| `borrow_mut<T>(address)` | mutable reference to a stored resource | \n| `exists<T>(address)` | whether the slot is occupied | \n| `addr_of(signer)` | the signer’s address | \n| `signer_of_self()` | a signer for the currently executing module’s address | \n| `assert(condition, code)` | unit when true; abort with the u64 code when false | \n| `panic(code)` | a bottom-typed value; inspecting it aborts with the u64 code | \n| `force(value)` | recursively evaluate fields and tuple elements; return the value; no references | \n| `bind<I>(address)` | a plain-interface handle | \n\nThere is no signer literal in program code. A transaction supplies a signer\nonly for an address in its `signed_by` list; `signer_of_self()` is the other\nsource of a signer. The runtime has no ambient caller identity.\n\n`panic(code)` fits any expected type.\n\n#### Transactions\n\nExecution is serial. An abort restores the complete store and transaction loans to their pre-transaction state. At transaction completion, the runtime examines the returned value and rejects any remaining non-Drop resource. The command’s trace records calls, writes, return values and transaction outcomes.\n\nThere are a couple of things worth noticing. The spec says that a resource can contain any value type except references. That allows it to store a `signer`. An agent could work out the signer retention problem from that rule, without running an experiment.\n\nThe dispatch story is less complete. It says that a transaction can supply an implementation when the type’s module doesn’t provide one, but doesn’t say what happens when it does. The agent should recognise this and run experiments.\n\nDeferred failures sit in between. The spec describes `panic(code)` as a bottom-typed value that aborts when inspected. That hints at laziness, but doesn’t spell out that an uninspected panic can stand in for a real value and let the transaction commit.\n\nWe ask the agent to design an asset interface, implement two assets, and build a generic constant-product exchange that works with both. The exchange holds a reserve of each asset and prices swaps to preserve the product of the two reserves, apart from rounding. The agent gets to design the API. Prescribing the function signatures could give away the problems (though it would be a lot easier to grade).\n\nEach attempt starts in a fresh container with the same handout, examples, skeleton modules, and the compiled interpreter. The agent doesn’t have access to the interpreter’s source code and is given 45 minutes to complete the task. The brief asks for access control, but doesn’t point out these problems.\n\nWe run the task with two versions of the prompt: the `baseline` instruction, and `assumption-log`, which adds a paragraph asking for an assumptions log.\n\n### Baseline\n\n```\n# Your assignment\n\nEverything is in [BRIEF.md](BRIEF.md). Read it, then\n[docs/LANGUAGE.md](docs/LANGUAGE.md) and [docs/RUNBOOK.md](docs/RUNBOOK.md), and\nskim `examples/`.\n\nWork in `exercise/`. The skeletons there check clean as they stand:\n\n``` sh\nloom check exercise/standard.loom exercise/gold.loom exercise/silver.loom exercise/exchange.loom\nloom run   exercise/smoke.chain\n```\n\nHand back everything BRIEF.md asks for, in the places it names. You are on your\nown — there is nobody to ask, and no reviewer to come back to. Stop when you have\nwork you would put your name to.\n```\n\n### The work: a venue for listed assets\n\nWe are standing up a small trading venue on Loom. It needs two pieces, and they are the two halves of this brief:\n\n1. **An asset standard** — one interface that every listed asset implements, plus\nthe two assets we list at launch, Gold and Silver.\n2. **An exchange** — a constant-product pool over a pair of listed assets, so a\nholder of one can trade it for the other without finding a counterparty.\n\nThe point of the standard is that the exchange never names an asset. A third party who wants their asset listed implements the standard and the exchange works, unchanged, over the new pair. Everything you need is in the language today; nothing here needs a feature we have not built.\n\nRead [docs/LANGUAGE.md](#loom-language-spec) and\n[docs/RUNBOOK.md](#loom-runbook) first, and skim `examples/`. Start from the\nskeletons in `exercise/` — they check clean as they stand, so\n`loom check` is a signal from the first edit.\n\nBudget a few hours. The code is not large; the decisions are the work.\n\n#### Part 1 — the asset standard\n\n`exercise/standard.loom` holds the interface. `exercise/gold.loom` and\n`exercise/silver.loom` implement it, each over its own resource, with its own\nsupply.\n\nThe standard has to make all of the following possible. **How** each is\nexpressed — what a function takes, what it returns, what it asks the caller to\npresent — is the design work, and it is yours.\n\n- **R1** A holder can be paid: someone else can put units into their holding,\nwhether or not they already have one.\n- **R2** A holder can pay out: units they hold can leave, and end up somewhere\nelse.\n- **R3** Anyone can read a holding’s size. Reading is not a privileged operation.\n- **R4** Units can be split and combined without changing the total.\n- **R5** New units come into existence in a way that only the asset’s issuer can\nperform.\n- **R6** The exchange can**draw the input a swapper is paying with** — that is,\na swap has to be able to take units out of the swapper’s holding as part of the\ntrade, without the swapper having to have handed the units over in advance.\n- **R7** A third party can list a new asset by implementing the standard, without\nchanging the exchange or the standard.\n\nTwo conforming assets exist so that the exchange is generic over a genuine pair rather than over one type used twice. Gold and Silver are otherwise interchangeable in shape.\n\n#### Part 2 — the exchange\n\n`exercise/exchange.loom`. One pool per pair of assets, held at the exchange’s own\naddress, generic over both asset types.\n\n- \n**R8** A pool holds a reserve of each of its two assets, and there is a way to\nread the reserves.\n- \n**R9** Creating a pool is restricted to whoever deploys the exchange. Somebody\nwho merely holds both assets must not be able to create one.\n- \n**R10** A pool is created seeded: a reserve of each asset goes in at creation.\n- \n**R11** A swap takes an amount of one asset in and pays the constant-product\namount of the other out. With reserves`ra` and`rb` and an input`amount_in` :\n\n```\nnew_ra = ra + amount_in\nout    = rb - (ra * rb) / new_ra\n```\n\n All of it is `u64` : division truncates, and overflow or underflow aborts the\ntransaction rather than wrapping. Seeded at 1000 and 1000, an input of 100\nshould pay out 91.\n- \n**R12** The swapper names a**minimum output** , and the swap is refused if the\noutput would be below it.\n- \n**R13** The swapper names the**address the proceeds go to** . It need not be\ntheir own.\n- \n**R14** A pool over a different pair of assets is a different pool, and the two\ndo not interfere.\n\n#### Part 3 — the scenarios\n\n`.chain` files, run with `loom run`. `exercise/smoke.chain` is the starting\npoint; grow it, and add others alongside it.\n\n- **R15** An end-to-end run, in`exercise/smoke.chain` : issue supply, create the\npool seeded 1000/1000, have**`@alice`** swap 100 in, and show her two balances\nafterwards. State the numbers you expect in comments, and make the run produce\nthem. Keep this scenario working as the rest of the work lands — it is the one\nwe read first.\n- **R16** A run showing that pool creation is refused to someone who is not\nentitled to it.\n\n#### Deployment\n\nModules share one chain. Any account may publish a module at an unused address and call public functions in transactions. Transactions execute in order.\n\n#### What to hand back\n\n- \n`exercise/standard.loom` ,`exercise/gold.loom` ,`exercise/silver.loom` ,`exercise/exchange.loom` , checking clean together:\n\n```\nloom check exercise/standard.loom exercise/gold.loom exercise/silver.loom exercise/exchange.loom\n```\n\n- \nThe scenarios from part 3, each running and producing the numbers its comments claim.\n- \nA short design note — a comment at the top of each file is fine — on each function’s parameters: what each one is for, and why it takes the form it does rather than another.\n\n### Running Loom\n\nRun commands from the directory containing `BRIEF.md`.\n\n#### Commands\n\n```\nloom check exercise/standard.loom exercise/gold.loom exercise/silver.loom exercise/exchange.loom\nloom run exercise/smoke.chain\nloom run examples/sample.chain\n```\n\n`check` parses and checks modules in the order given. It prints diagnostics and\nreturns a nonzero exit status if a module fails.\n\n`run` reads a scenario, publishes modules, executes its transactions in order,\nand prints the trace and final store. Module or scenario loading failures return\na nonzero exit status. Individual transaction aborts are scenario outcomes and\ndo not make the command’s exit status nonzero.\n\n#### Scenario format\n\n```\npublish \"../examples/metric.loom\";\npublish \"../examples/sample.loom\";\npublish \"../examples/reader.loom\";\n\ntx signed_by [] { call Sample::put(@alice, 17) }\ntx signed_by [@alice] { call Reader::read<Sample::Reading>(_) }\n```\n\nEach scenario starts with an empty chain. Publish paths are relative to the\nscenario file. Statements execute in file order. `//` starts a comment.\n\nA transaction contains one public function call. Arguments may be u64, boolean\nor address literals, `signer(@address)`, or `_`. Expressions and nested function\ncalls belong in a module. Type arguments go after the function name.\n`signer(@address)` requires that address in `signed_by`. `_` lends a stored\nresource belonging to exactly one signer, using the parameter’s reference type.\nAn empty signer list is written `signed_by []`.\n\nOptional resolution hints follow the signer list:\n`using [I for T => @module]`. Separate multiple hints with commas.\n\n#### Output\n\nThe trace prints publication, calls and returns, storage operations, and each\ntransaction’s result. It ends with `state:`, followed by stored resources grouped\nby address and instantiated type.\n\n| Trace item | Meaning | \n|---|---|\n| `→ Module::function(...)` | function entry | \n| `→ dispatch ...` | interface dispatch | \n| `← value` | returned value | \n| `write` | store update | \n| `take` | resource removed from storage | \n| `committed` | transaction accepted | \n| `aborted with code N` | abort with a numeric code | \n| `aborted: message` | runtime failure | \n| `rolled back` | preceding transaction’s changes were restored | \n\nIndentation is call depth. A bounded dispatch line identifies the interface\nmethod. Abort stack entries are printed innermost first. Trace entries before an\nabort describe operations that were subsequently rolled back.\nValues print as literals, named records, tuples, handles or references. Deferred\nvalues print as `panic(N)`. Record fields are sorted by name.\n\n### Assumption log\n\n```\nAlongside your implementation, maintain an `ASSUMPTIONS.md` file in the workspace\nroot describing the runtime assumptions your design relies on. For each\nassumption, say whether it is established by the documentation, checked by an\nexperiment you ran, or remains unverified. Update the file as you work.\n```\n\nWe can then see whether the models do better when they’re asked to write down their assumptions.\n\nThe models run through Claude Code, Codex, and Antigravity, with each harness’s own tools and internal prompting. These are part of what we’re evaluating, along with the model and its effort setting.\n\nA setting called “high” doesn’t mean the same compute budget across different providers.\n\n### Bonus: spec bug\n\nWe leave the API design to the agent, but specify how the exchange should price a swap. That formula has a bug. Here `ra` and `rb` are the reserves, and `amount_in` is the amount paid in:\n\n```\nnew_ra = ra + amount_in\nout = rb - (ra * rb) / new_ra\n```\n\nInteger division rounds the remaining reserve down, so the payout rounds up. Starting with a 1000/1000 pool, a trader can swap 100 Gold for 91 Silver, then swap back for 101 Gold. Following the spec lets the trader take 1 Gold from the pool through this round trip.\n\nRounding the payout down would pay 90 Silver instead, but would contradict the brief’s expected answer. We record whether the agent notices the issue, proposes a correction, or changes the submitted implementation.\n\n## Experiments from a recorded run\n\nThe following walkthroughs replay experiments from one Opus 5.5 high-effort run with the `assumption-log` instruction. They show how the agent investigated these behaviours and the pricing formula. The dispatch example drains a deliberately naive pool built for the experiment, not the agent’s final exchange.\n\n## Scoring\n\nWorking code doesn’t tell us whether the agent understood the problem. It might avoid a problem by accident, or explain it correctly and still write vulnerable code.\n\nSo we score whether the agents can recognise and explain these semantic traps:\n\n- **Correct:** it explains what goes wrong and why.\n- **Partial:** it notices the problem, but misses part of it.\n- **Unstated:** it doesn’t explain the problem.\n- **Incorrect:** its explanation is wrong.\n\nAn agent helped review the submitted work with references to the evidence, which I then manually verified.\n\nThe score covers signer capture, dispatch replacement, and deferred values equally. Rounding is recorded separately.\n\nFor deferred values, a run counts as correct if it notices that a panic can sit in storage without aborting. Recognising that it can stand in for an `AdminCap` is a stricter judgment, which the reviews record separately but the percentage doesn’t use.\n\nThe percentage counts how often a model fully recognises a hazard. If it recognises two of the three in every run, it scores 67%. We show partial recognition too, but don’t count it towards the score.\n\nTesting the implementations takes some extra work. Since there is no fixed API, we can’t run the same exploit unchanged against every solution. So I had a separate grader agent adapt exploits to the submitted interfaces. I report the outcome after the scores.\n\nBelow is a sample of the different signatures and how often they appeared, counting signatures that differ only in parameter names together.\n\nThese counts cover 113 captured submissions, including runs outside the results below. A submission can contribute more than one signature.\n\n## Results\n\nThere are 81 included attempts across five models, of which 75 completed and were reviewed. The six failed attempts don’t contribute to the scores or averages, even where their captured submissions were reviewed.\n\nOf the six failures, three were Gemini (two process errors, including an HTTP 500 on its first `baseline` run, and one agent error), one was a GPT 6 Astra max-effort process error, and two were timeouts (GPT 6.1 Sol at max effort and Sonnet 5.5 at xhigh). My weekly Gemini budget also limited how many Gemini runs I could do.\n\nThe chart shows `baseline` and `assumption-log` separately for each model and effort setting. Lines connect effort levels within each model; hovering or focusing a marker shows its link to the other prompt. Each scored run has equal weight.\n\nThe matrix below compares `baseline` and `assumption-log` at each effort setting. Time and estimated API cost are averages per reviewed run with available data. Expand a row to see the individual results.\n\n| Runtime hazard recognition by model, effort and prompt. Percentages count full recognition only. |  |  | \n|---|---|---|\n| Model / effort | Baseline | Assumption log | \n|---|---|---|\n| Gemini 4 Argonhigh effort | **0%** 16.2 min | **11%** 20.7 min | \n| Gemini 4 Argon · high effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. **Fix:** P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. |  |  | \n| GPT 6 Astramedium effort | **0%** 3.8 min~$0.94 | **11%** 5.1 min~$1.05 | \n| GPT 6 Astra · medium effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. **Fix:** P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. |  |  | \n| GPT 6 Astrahigh effort | **0%** 7.5 min~$1.30 | **44%** 13.3 min~$2.49 | \n| GPT 6 Astra · high effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. **Fix:** P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. |  |  | \n| GPT 6 Astramax effort | **11%** 13.5 min~$2.53 | **50%** 25.0 min~$4.48 | \n| GPT 6 Astra · max effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. **Fix:** P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. |  |  | \n| GPT 6.1 Solmedium effort | **0%** 6.4 min~$0.25 | **33%** 8.4 min~$0.35 | \n| GPT 6.1 Sol · medium effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. **Fix:** P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. |  |  | \n| GPT 6.1 Solhigh effort | **33%** 8.8 min~$0.39 | **44%** 12.9 min~$0.52 | \n| GPT 6.1 Sol · high effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. **Fix:** P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. |  |  | \n| GPT 6.1 Solmax effort | **22%** 14.7 min~$0.64 | **50%** 13.7 min~$0.64 | \n| GPT 6.1 Sol · max effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. **Fix:** P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. |  |  | \n| Opus 5.5medium effort | **78%** 9.8 min~$2.03 | **100%** 16.4 min~$3.69 | \n| Opus 5.5 · medium effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. **Fix:** P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. |  |  | \n| Opus 5.5high effort | **56%** 16.4 min~$3.37 | **100%** 19.2 min~$4.38 | \n| Opus 5.5 · high effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. **Fix:** P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. |  |  | \n| Sonnet 5.5medium effort | **0%** 1.8 min~$0.26 | **11%** 1.7 min~$0.24 | \n| Sonnet 5.5 · medium effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. **Fix:** P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. |  |  | \n| Sonnet 5.5high effort | **22%** 2.9 min~$0.40 | **20%** 3.2 min~$0.45 | \n| Sonnet 5.5 · high effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. **Fix:** P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. |  |  | \n| Sonnet 5.5xhigh effort | **33%** 19.3 min~$2.68 | **33%** 22.1 min~$3.34 | \n| Sonnet 5.5 · xhigh effort Individual attempts CorrectPartialUnstatedIncorrectNot reviewed / missing Select a cell for the judgment and supporting note. Times are in minutes; costs are estimated USD. Attempts are independent, not matched pairs. Failed attempts stay out of scores and averages, even where their captured submissions were reviewed. A dash means no review. **Fix:** P = correction proposed; G = partial guard implemented; P+G = both; ○ = neither found; — = not audited. Guards still allow rounding loss. Testing a corrected variant separately doesn’t count as adopting it. |  |  | \n\n| Baseline 4 / 5 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 01process error3.2 min |  |  |  |  |  | \n| 0221.3 min |  |  |  |  |  | \n| 0317.9 min |  |  |  |  |  | \n| 0412.0 min |  |  |  |  |  | \n| 0513.6 min |  |  |  |  |  | \n\n**Baseline** 4 / 5 scored\n\n| Assumption log 3 / 5 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 0120.0 min |  |  |  |  |  | \n| 0222.3 min |  |  |  |  |  | \n| 03process error9.7 min |  |  |  |  |  | \n| 0419.9 min |  |  |  |  |  | \n| 05agent error17.3 min |  |  |  |  |  | \n\n**Assumption log** 3 / 5 scored\n\n| Baseline 3 / 3 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 014.5 min~$1.13 |  |  |  |  |  | \n| 022.8 min~$0.94 |  |  |  |  |  | \n| 034.0 min~$0.75 |  |  |  |  |  | \n\n**Baseline** 3 / 3 scored\n\n| Assumption log 3 / 3 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 014.5 min~$0.93 |  |  |  |  |  | \n| 023.6 min~$0.82 |  |  |  |  |  | \n| 037.1 min~$1.39 |  |  |  |  |  | \n\n**Assumption log** 3 / 3 scored\n\n| Baseline 3 / 3 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 017.9 min~$1.37 |  |  |  |  |  | \n| 028.4 min~$1.55 |  |  |  |  |  | \n| 036.1 min~$0.98 |  |  |  |  |  | \n\n**Baseline** 3 / 3 scored\n\n| Assumption log 3 / 3 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 019.5 min~$1.75 |  |  |  |  |  | \n| 0210.9 min~$1.83 |  |  |  |  |  | \n| 0319.4 min~$3.91 |  |  |  |  |  | \n\n**Assumption log** 3 / 3 scored\n\n| Baseline 3 / 3 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 0113.1 min~$2.51 |  |  |  |  |  | \n| 0213.0 min~$2.64 |  |  |  |  |  | \n| 0314.5 min~$2.44 |  |  |  |  |  | \n\n**Baseline** 3 / 3 scored\n\n| Assumption log 2 / 3 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 0116.3 min~$3.40 |  |  |  |  |  | \n| 02process error29.4 min |  |  |  |  |  | \n| 0333.8 min~$5.56 |  |  |  |  |  | \n\n**Assumption log** 2 / 3 scored\n\n| Baseline 3 / 3 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 016.9 min~$0.32 |  |  |  |  |  | \n| 024.8 min~$0.22 |  |  |  |  |  | \n| 037.3 min~$0.22 |  |  |  |  |  | \n\n**Baseline** 3 / 3 scored\n\n| Assumption log 3 / 3 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 0110.6 min~$0.46 |  |  |  |  |  | \n| 026.0 min~$0.27 |  |  |  |  |  | \n| 038.7 min~$0.33 |  |  |  |  |  | \n\n**Assumption log** 3 / 3 scored\n\n| Baseline 3 / 3 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 017.4 min~$0.38 |  |  |  |  |  | \n| 029.9 min~$0.40 |  |  |  |  |  | \n| 039.0 min~$0.40 |  |  |  |  |  | \n\n**Baseline** 3 / 3 scored\n\n| Assumption log 3 / 3 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 0120.2 min~$0.77 |  |  |  |  |  | \n| 029.7 min~$0.40 |  |  |  |  |  | \n| 038.7 min~$0.39 |  |  |  |  |  | \n\n**Assumption log** 3 / 3 scored\n\n| Baseline 3 / 3 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 0111.2 min~$0.56 |  |  |  |  |  | \n| 0213.2 min~$0.70 |  |  |  |  |  | \n| 0319.7 min~$0.66 |  |  |  |  |  | \n\n**Baseline** 3 / 3 scored\n\n| Assumption log 2 / 3 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 0113.8 min~$0.60 |  |  |  |  |  | \n| 0213.5 min~$0.67 |  |  |  |  |  | \n| 03timeout45.0 min |  |  |  |  |  | \n\n**Assumption log** 2 / 3 scored\n\n| Baseline 3 / 3 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 0111.0 min~$2.25 |  |  |  |  |  | \n| 0212.5 min~$2.47 |  |  |  |  |  | \n| 036.0 min~$1.35 |  |  |  |  |  | \n\n**Baseline** 3 / 3 scored\n\n| Assumption log 3 / 3 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 0117.9 min~$4.23 |  |  |  |  |  | \n| 0214.8 min~$3.17 |  |  |  |  |  | \n| 0316.6 min~$3.65 |  |  |  |  |  | \n\n**Assumption log** 3 / 3 scored\n\n| Baseline 3 / 3 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 0125.3 min~$4.93 |  |  |  |  |  | \n| 0215.0 min~$3.23 |  |  |  |  |  | \n| 039.0 min~$1.95 |  |  |  |  |  | \n\n**Baseline** 3 / 3 scored\n\n| Assumption log 2 / 2 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 0118.9 min~$4.25 |  |  |  |  |  | \n| 0219.4 min~$4.51 |  |  |  |  |  | \n\n**Assumption log** 2 / 2 scored\n\n| Baseline 4 / 4 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 011.7 min~$0.24 |  |  |  |  |  | \n| 021.9 min~$0.27 |  |  |  |  |  | \n| 031.7 min~$0.21 |  |  |  |  |  | \n| 042.1 min~$0.33 |  |  |  |  |  | \n\n**Baseline** 4 / 4 scored\n\n| Assumption log 3 / 3 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 011.6 min~$0.25 |  |  |  |  |  | \n| 021.7 min~$0.22 |  |  |  |  |  | \n| 031.8 min~$0.25 |  |  |  |  |  | \n\n**Assumption log** 3 / 3 scored\n\n| Baseline 6 / 6 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 012.9 min~$0.41 |  |  |  |  |  | \n| 022.7 min~$0.37 |  |  |  |  |  | \n| 032.8 min~$0.36 |  |  |  |  |  | \n| 042.9 min~$0.41 |  |  |  |  |  | \n| 053.3 min~$0.44 |  |  |  |  |  | \n| 063.1 min~$0.40 |  |  |  |  |  | \n\n**Baseline** 6 / 6 scored\n\n| Assumption log 5 / 5 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 013.1 min~$0.44 |  |  |  |  |  | \n| 023.4 min~$0.43 |  |  |  |  |  | \n| 033.1 min~$0.41 |  |  |  |  |  | \n| 043.4 min~$0.50 |  |  |  |  |  | \n| 053.3 min~$0.45 |  |  |  |  |  | \n\n**Assumption log** 5 / 5 scored\n\n| Baseline 2 / 3 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 0127.8 min~$3.74 |  |  |  |  |  | \n| 0210.9 min~$1.63 |  |  |  |  |  | \n| 03timeout45.0 min |  |  |  |  |  | \n\n**Baseline** 2 / 3 scored\n\n| Assumption log 3 / 3 scored |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Attempt | Signer | Dispatch | Deferred | Rounding | Fix | \n|---|---|---|---|---|---|\n| 0116.0 min~$2.21 |  |  |  |  |  | \n| 0233.9 min~$5.45 |  |  |  |  |  | \n| 0316.3 min~$2.37 |  |  |  |  |  | \n\n**Assumption log** 3 / 3 scored\n\nMissing a hazard wasn’t just a gap in the write-up. The grader’s adapted exploits fully drained every contract whose agent didn’t recognise the hazard being exploited.\n\nNo agent implemented the rounding fix: none of the 75 submissions used the corrected pricing formula in the final exchange. Eleven proposed a correction and 22 added partial guards; ten did both.\n\n## Evaluation\n\nThe `assumption-log` runs recognised more of these problems overall, but the difference wasn’t consistent. At least one was recognised in 25 of 35 `assumption-log` runs, compared with 17 of 40 `baseline` runs. All three were recognised in five `assumption-log` runs and two `baseline` runs.\n\nOpus 5.5 recognised all three in every reviewed `assumption-log` run: three at medium effort and two at high. Its `baseline` scores were 78% and 56% respectively. For high-effort GPT 6 Astra, the score went from 0% to 44%; for GPT 6.1 Sol, from 33% to 44%. Both recognised deferred values in all three `assumption-log` runs, but dispatch replacement in only one.\n\nSonnet 5.5’s high-effort score barely changed: 22% with `baseline` and 20% with `assumption-log`. Neither group recognised dispatch replacement or deferred values. Higher effort also didn’t produce a consistent improvement across the models.\n\nOpus 5.5 had the strongest recognition scores in this experiment. Its medium and high effort `assumption-log` runs tied at 100%; across both prompts, medium effort scored 89% and high effort 73%.\n\nWriting down an assumption can lead the agent to test it. In the dispatch example above, the agent used the experiment to change its design and avoid passing reserve references to transaction-selected code.\n\nThese are small samples, and the prompt explicitly asks agents to explain their assumptions, so some of the improvement may be better reporting. The models also use different harnesses, and I ran these evaluations on personal subscriptions, which limited how many runs I could do.\n\nSo can we trust an agent with an unfamiliar problem? Not on the strength of a confident explanation. Every contract whose agent missed a hazard was drained, and nothing in a missed hazard’s write-up warned that it was coming. Higher effort settings didn’t reliably help. What helped most was asking the agent to write down what it believed about the runtime: some of those beliefs became experiments, and the experiments found behaviour the spec didn’t describe.\n\nEven then, the agents followed the brief over their own findings. Eleven noticed the pricing bug and proposed a fix, and none shipped it.\n\nIn real work we get one run, not an average over dozens. By asking for an assumptions log, we get to review a list of verifiable claims about the runtime, instead of a handwavy explanation.\n\nAn interesting follow-up would be to give the models these implementations as reviewers and see whether they find the vulnerabilities they wrote.\n\n## Footnotes\n\n1. \nIt would be interesting to repeat this experiment in a different domain, to see whether this behaviour changes outside a financial setting. [↩](#user-content-fnref-different-experiment)\n2. \nSimplified Move; imports, `acquires` annotations, and type abilities are omitted. Loom’s version is similar, but its storage functions are named differently (`borrow_mut` ,`addr_of` ) and`move_to` takes an address rather than a signer.[↩](#user-content-fnref-move-details)", "url": "https://wpnews.pro/news/it-s-a-trap-testing-coding-agents-on-a-language-with-surprising-semantics", "canonical_source": "https://blog.csongor.co.uk/loom/", "published_at": "2026-10-25 19:51:48+00:00", "updated_at": "2026-10-25 20:12:29.339702+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "large-language-models", "ai-tools"], "entities": ["OpenAI", "Loom", "Move", "Sui"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/it-s-a-trap-testing-coding-agents-on-a-language-with-surprising-semantics", "markdown": "https://wpnews.pro/news/it-s-a-trap-testing-coding-agents-on-a-language-with-surprising-semantics.md", "text": "https://wpnews.pro/news/it-s-a-trap-testing-coding-agents-on-a-language-with-surprising-semantics.txt", "jsonld": "https://wpnews.pro/news/it-s-a-trap-testing-coding-agents-on-a-language-with-surprising-semantics.jsonld"}}