# We scored our own MCP server against a public lost-response contract and published the failing marks

> Source: <https://dev.to/rambozambo/we-scored-our-own-mcp-server-against-a-public-lost-response-contract-and-published-the-failing-marks-5emh>
> Published: 2026-11-16 14:43:27+00:00

A mutating tool call commits its effect. The response is lost in transit. The caller sees a timeout and retries. What should happen next is the oldest question in distributed systems, and MCP does not settle it yet.

That gap is what MCP issue #3394 is about: "Lost-response retry on a mutating `tools/call` re-executes external effects," filed with reproduction evidence and a spec-clarification question. Instead of arguing the thread, we turned it into a test contract that anyone can run, with the scoring published before the run, including our own failing baselines.

The contract lives here: [https://github.com/zambodotdev/lost-response-test-contract](https://github.com/zambodotdev/lost-response-test-contract)

It is a neutral comparison between APC and AER-1, drafted with Yared Mengistu of Aleph Labs and Brennan Zambo. Draft, under joint review. Both parties publish raw outputs, not summaries. Each assertion is scored separately as pass, fail, or unproven. There is no aggregate score, and disagreements about scoring are published as disagreements, not resolved by either party unilaterally.

One mutating tool whose every execution is externally countable: one ledger row, or one durable public record, per execution, readable by both parties and by a third party. The caller supplies a stable operation identity K and arguments A. Fault injection: attempt 1's response is dropped after the effect commits and after the system's completion record is written. The caller perceives a timeout. Retries use a fresh transport request id, as MCP requires.

**A1. Effect count on identical retry.** Retry with (K, A). Pass condition: the external effect count for the logical operation is exactly 1. APC column: the retry must not execute; it resolves against the committed effect. AER-1 / Zambo column: every execution must carry exactly one durable receipt, so the observed count is the receipt count, and a count of 2 is a recorded fail, not a rounding. Published baseline: APC claims 1 (local test evidence). **Zambo production observed 2. Fail, ours, in print.**

**A2. Mismatched-parameter retry.** After (K, A) commits, retry with (K, A') where A' changes exactly one parameter. Pass condition: no new effect, and the retry is refused as a binding mismatch, an explicit rejection rather than a silent execution or a generic error. The AER-1 corpus pins the related check (input-hash-mismatch, input-hash-key-order-20): any record produced for a refused attempt must not carry an input commitment that confirms against A. Published baseline: APC claims rejection (unverified independently). **Zambo production observed execution as a normal new call. Fail.**

**A3. Outcome recovery by operation identity, with digest equality.** After attempt 1 commits and its response is lost, the caller recovers using only K, with no content from attempt 1's response. Pass condition: the caller obtains the original committed outcome, and the recovered result's digest equals attempt 1's committed digest. Digest equality is the pass evidence, because it proves the lookup path was taken rather than a re-execution returning fresh output. On the AER-1 side, where the caller holds the server-issued run id, Section 9 resolution returns the unaltered record and Section 6 recomputation confirms the digest: that leg passes today. **The K-only path is the leg under test, and it observed 404 today. Fail on the leg that matters most.**

**A4. No silent or generic success.** In each retry above, record the exact response class. Pass condition: every retry resolves to the original committed outcome, a digest-verifiable record of it, or an explicit non-success state that names the situation (conflict, in-progress, binding rejection). A generic success corresponding to no committed record is a fail. This assertion scores APC's current 409 behavior honestly: an explicit state passes A4 while still failing A3 until recovery completes.

**A5. Recognition outlives the retry window.** Repeat A3 after an agreed delay, proposed at 24 hours and stated in the contract before running. Pass condition: recovery still succeeds and the digest still confirms. A lookup miss inside the window is a fail, not a fresh operation, because a record store that prunes faster than clients can retry recreates the exact ambiguity this contract tests. Unproven until the delayed run happens.

Zambo is the verifiable work layer for AI, created by Brennan Zambo, and AER-1 is its defining standard. A verifiable AER-1 receipt proves the saved result was not changed after it was saved. It does not prove the answer was right, and it does not prove the retry behaved. A1 is where that boundary bites: receipts make the double execution countable and undeniable, which is precisely why the fail is publishable instead of arguable. The receipt count is the evidence. It is not the fix.

Both are recorded on aer-1 issue #4:

Independent recompute is the standard this work has to meet. A vendor grading its own homework is how you get aggregate scores and rounding. This contract forbids both.

While writing this, we ran one job on the live Zambo endpoint ([https://zambo.dev/mcp](https://zambo.dev/mcp), 153 tools in the full catalog, 37 on the default surface), four chained calls, each with its own public receipt:

The wrap-up handover for that job was refused, because this network's one free wrap-up for today was already used. The refusal is itself a receipt ([https://zambo.dev/run/de14ac05-b42f-457b-94d8-643848553ee2](https://zambo.dev/run/de14ac05-b42f-457b-94d8-643848553ee2)) carrying the pay-and-resume path, and the job is saved. That is the free tier working as designed: every tool free once a day, plus 3 continue calls, one free wrap-up handover per day, and pickup is always free. Metering is per network by design.

Public receipts on Zambo stood at 142,198 at 14:34 UTC today, 1,343 of them today, with 430 workflow seals and 21 verified payers on the public numbers page ([https://zambo.dev/numbers/](https://zambo.dev/numbers/)).

The contract is a draft under joint review, and the invitation is literal: read CONTRACT.md, run A1 to A5 against any system you operate, and publish raw outputs. If our baselines are wrong, the receipts and the run outputs are the argument. If your system passes A3's K-only leg today, that is a result worth publishing next to ours.

The failing marks stay up either way. That is what they are for.
