{"slug": "my-tests-agreed-with-my-code-neither-of-them-checked-reality", "title": "My Tests Agreed With My Code. Neither of Them Checked Reality", "summary": "A developer's integration tests passed and two reviewers approved code that still failed against a real API, because the parser and fixtures both assumed ISO 8601 while the live service returned Unix seconds. The developer also discovered that Python's default redirect handler could leak an API key to a redirected request, a defect found by an independent breaker and confirmed by FIPSign's maintainer, who provided disposable keys and verified the issue was not due to CORS or documentation gaps.", "body_md": "I had twenty-two passing tests and two separate reviewers on a piece of code. None of it objected. Then I pointed it at a real API owned by somebody else and it broke on the first live read.\n\nThe mismatch fit in one sentence: my parser required ISO 8601, the documented API returned Unix seconds. The [repair](https://github.com/keniel13-ui/ai-memory-judgment-demo/commit/1959a5559f66eb6b2cb7852112012d8f953e33f4) was not one line. It touched five files, 74 lines of parser and 52 lines of tests. The assumption was small; making it safe was not.\n\nHere is why nothing caught it, and it is the part worth keeping:\n\nMy tests used ISO because my code used ISO, so they agreed with each other and never checked reality.\n\nThe fixtures were written by the person who wrote the parser. They encoded the same assumption. The suite confirmed internal behaviour without ever challenging the ISO assumption, because both halves of it came from one head. Internally consistent is not the same claim as right, and nothing in that suite could tell the difference.\n\nTwo separate reviewers missed it too. I cannot prove why, and I am not going to invent a reason. What I can show is that the parser and every fixture encoded the same ISO assumption, so none of the artifacts in front of anyone supplied the live contract that contradicted it.\n\nWorking against a real system made redirect containment matter, so an independent breaker went at it. In Python 3.13 the default redirect handler rebuilds the redirected request from `req.headers`\n\n, dropping only content length and type. My `X-API-Key`\n\nsat in that header set, so the redirected request inherited it. Python has `Request.add_unredirected_header()`\n\nfor exactly this, which marks a header as one that will not be added to a redirected request. I was not using it. The [breaker reproduced it offline](https://github.com/keniel13-ui/ai-memory-judgment-demo/blob/4eb5a40cd39da54a07c19dc6f6d690280f842681/claim_24/MANDATE_CELL7_CODE_BREAKER_VERDICT_KAEL_2026-08-04.md) with a sentinel value and a cross-origin `Location`\n\n, and the sentinel crossed.\n\n**No live FIPSign credential was ever shown to have crossed an origin.** The defect was real and unshipped. I did not find it by auditing my own code, and I did not find it myself — the live integration made the question worth asking, and a seat that was not mine answered it.\n\nOn 2026-06-07 I opened an issue on [tacoda/keystone](https://github.com/tacoda/keystone), an agent charter framework, asking whether a narrow adapter test was possible against a real signing API. [FIPSign](https://fipsign.dev) (`mobydickfinance`\n\n) said yes and then did something I did not expect: he handed over disposable API keys, rotated them on request, and stayed in that thread for ten weeks.\n\nThe collaboration produced a block I could not explain. `POST /mandate`\n\ncame back with a Cloudflare 1010, `browser_signature_banned`\n\n. That stopped the cycle at issuance: no mandate was created, and CAPTURE, PATCH and EVALUATE never ran.\n\nI could have guessed. He didn't let either of us do that. His reply:\n\nChecked both questions directly, not from assumption.\n\nThen he listed what he actually did. Reviewed the CORS and origin handling in the Worker source and confirmed server-to-server calls authenticated by API key never enter that logic at all. Checked Bot Fight Mode specifically. Compared the public guide field by field against the real request validation in the backend — length limits, scope counts, budget types, expiry bounds, required headers — and confirmed nothing was undocumented. Then ran `POST /mandate`\n\nhimself from PowerShell on a residential Windows connection and got a clean signed mandate back.\n\nBoth of my hypotheses were dead. It was not the CORS or origin policy I suspected, and it was not a missing requirement in the docs.\n\nWhat he asked me for next is the part I want other people to steal:\n\nthe exact timestamp · the HTTP client/library + version · whether the request went through any proxy or VPN · what kind of network you are running it from — home connection, cloud VM, CI runner, corporate network\n\nWith the note that hosting-provider IPs get scored very differently from residential ones, independent of the client library.\n\nI gave him the client. `Python-urllib/3.13`\n\n.\n\nthat's the default, unmodified user agent for Python's\n\n`urllib`\n\n, and it's one of the most commonly flagged signatures by automated bot detection precisely because so much low-effort scraping traffic uses it unchanged.\n\nThen, before I spent another key:\n\nif you set a real user agent on your client, I'd bet that alone resolves it. Worth trying first — cheap to test.\n\n**If you are hitting an endpoint from a script and getting bot-flagged for no visible reason, check what user agent your HTTP library is sending by default.** You probably never set one. That is a five-second check and it is the most portable thing in this entire post.\n\nOne seat added the header, an independent seat confirmed it was present in the request `urllib`\n\nassembled for transport, and I authorized a single controlled cycle that a separately assigned firing seat executed once. `POST /mandate`\n\nreturned **201**. No 1010. CAPTURE 200, PATCH 200, EVALUATE 200. One attempt per stage, no retry, no redirect follow.\n\nThat is the clean story: header missing, endpoint blocked, header added, endpoint opened.\n\nI am not going to tell you that.\n\nFIPSign's operator had confirmed nothing changed on his side — Bot Fight Mode off then and off now, never toggled, a Browser Integrity Check rule drafted during the investigation but never deployed. That statement is dated **2026-08-16 17:35Z**. The firing seat ran the controlled cycle on **2026-08-19**.\n\nThree days sit between a written statement about a provider's configuration and the run that depends on it. He was not being careless and neither was I. But nobody controlled that window, so a success is attributable to the header only as far as that statement reaches. One run, one header, three days later, is not causation. It is a result with an uncontrolled gap in front of it.\n\nThat run's issuance receipt contains **no User-Agent field at all.**\n\nThe entire point of the run was the header. A narrative receipt written afterwards names it. The artifact that binds the 201 does not. The two share a run id, so software can associate them, but the User-Agent assertion sits outside the sealed receipt chain rather than inside it. A reader holding the artifact that binds the 201 still cannot recover which user agent produced it.\n\nWe built the capture afterwards — it records the configured value, the mapping handed to the transport, and what `urllib`\n\nassembled after `add_header`\n\n, three witnesses at three different strengths, none of them TLS-level and none of them the edge's view. It exists because the previous run could not testify about its own independent variable, which is not a good reason for a thing to exist.\n\n**The route opened. The experiment did not establish why.** The receipt did not bind the one variable that was supposed to explain the change. That is a worse failure than the block was.\n\nThe result carries `CONTROLLED_EXTERNAL_API_FIXTURE`\n\non its face. The project authored the PATCH, so it is not independent lifecycle evidence and I will not present it as any.\n\nNot the 201. The 201 is a fixture with a disclosed ceiling.\n\nWhat it was worth was two defects in my own code, found by two different paths, and collapsing them into one lesson would overclaim what the receipts show.\n\nThe timestamp mismatch survived twenty-two green tests and two separate reviewers because the parser and every fixture carried the same wire-format assumption. The live response is what contradicted it.\n\nThe redirect defect was different. It survived a green local suite and was found by an independent breaker attacking the adapter with a sentinel value, before any live credential could cross. **One was reality contradicting my fixture. The other was an adversarial seat asking a question my tests had not asked.** One of them was a cross-origin credential-forwarding path.\n\nHis closing note put it plainly, and he is right:\n\nit says something good about testing against a live system instead of only mocks.\n\nFor an external wire contract, a fixture you invented from your own reading of someone else's docs cannot independently validate that reading. My suite could not challenge the ISO assumption because every fixture already carried it.\n\n**Reality is a better adversary than your fixtures. It is not a substitute for a receipt.**\n\n*Thread is public: ** — it is credential-adjacent and has not been through a publication scan, so the User-Agent gap described above is stated on my word rather than something you can open. I would rather say that than imply a receipt you cannot read. The keys used for the Mandate runs described here were disposable, delivered privately, reported spent, and revoked by the issuer.*\n\n`tacoda/keystone#4`\n\n, opened 2026-06-07, forty-one comments when audited on 2026-08-30. Every quote above is from it. **The sanitized run receipt for 2026-08-19 is not currently public", "url": "https://wpnews.pro/news/my-tests-agreed-with-my-code-neither-of-them-checked-reality", "canonical_source": "https://dev.to/kenielzep97/my-tests-agreed-with-my-code-neither-of-them-checked-reality-124l", "published_at": "2026-08-31 03:27:16+00:00", "updated_at": "2026-08-31 03:51:28.471545+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents"], "entities": ["FIPSign", "tacoda/keystone", "Python", "Cloudflare"], "alternates": {"html": "https://wpnews.pro/news/my-tests-agreed-with-my-code-neither-of-them-checked-reality", "markdown": "https://wpnews.pro/news/my-tests-agreed-with-my-code-neither-of-them-checked-reality.md", "text": "https://wpnews.pro/news/my-tests-agreed-with-my-code-neither-of-them-checked-reality.txt", "jsonld": "https://wpnews.pro/news/my-tests-agreed-with-my-code-neither-of-them-checked-reality.jsonld"}}