{"slug": "i-made-claude-code-prove-billing-end-to-end-before-i-let-it-ship", "title": "I Made Claude Code Prove Billing End-to-End Before I Let It Ship", "summary": "A developer used Claude Code to prove billing end-to-end against real infrastructure before shipping a SaaS project, which uncovered a race condition in webhook handling. The agent provisioned temporary infrastructure on Render, configured a Paddle sandbox, and simulated a real payment, revealing that two different subscription webhook events could overlap and cause a unique constraint violation. The fix involved using database savepoints to handle the collision gracefully.", "body_md": "I’ve been using coding agents for a while, but during the last release of a SaaS project I changed one thing about how I used them.\n\nI stopped defining “done” as:\n\nimplementation finished, tests passing\n\nand gave the agent an external condition instead:\n\n**Do not let me ship until you can prove that billing works end-to-end against real infrastructure.**\n\nThat turned out to be much more useful than another round of code generation or code review.\n\nBecause the tests were already green.\n\nThe real system was not.\n\nThe project uses FastAPI, Next.js, Postgres, Redis, Celery and Paddle.\n\nBilling already had tests around checkout, webhook processing, subscriptions, credit grants and idempotency.\n\nAt that point it would have been very easy to say:\n\nLooks good. Ship it.\n\nInstead, I asked Claude Code to prove the actual money path.\n\nThat meant it couldn't stop after reading the code or running pytest.\n\nIt had to create an environment where a real Paddle sandbox transaction could go through the application.\n\nSo it provisioned temporary infrastructure on Render through the API:\n\nThen it configured a temporary Paddle sandbox webhook destination pointing at that deployment.\n\nThe requirement was simple: act like a real customer and show me the resulting state.\n\nThe agent then went through the actual application flow.\n\nIt authenticated through the app, opened checkout, completed a Paddle sandbox payment and inspected what happened afterward.\n\nNot just the HTTP response.\n\nNot just the redirect.\n\nThe resulting state.\n\nIt checked:\n\nBefore payment the account was free, with no paid subscription.\n\nAfter payment it had the expected active subscription and the expected credit balance.\n\nSo far, good.\n\nThen the sandbox exposed a bug.\n\nWe had implicitly been thinking about these events as a sequence:\n\n```\nsubscription.created\n        ↓\nsubscription.activated\n```\n\nThat mental model was wrong.\n\nIn the sandbox, the two webhook handlers could overlap.\n\nBoth could begin while there was still no local subscription row.\n\nConceptually, each handler was doing something like:\n\n```\nsubscription = find_subscription(provider_id)\n\nif subscription is None:\n    create_subscription(provider_id)\n```\n\nPerfectly reasonable if the operations happen sequentially.\n\nNot safe if two different webhook events execute at the same time.\n\nThe actual failure mode was:\n\n```\nHandler A                       Handler B\n\nSELECT subscription             SELECT subscription\n→ nothing                       → nothing\n\nINSERT subscription\n                                INSERT subscription\n\nsuccess                         unique constraint violation\n```\n\nThe database was correctly preventing duplicate subscriptions.\n\nBut one of the webhook handlers was now failing.\n\nAnd this was not an idempotency problem in the usual sense.\n\nThese were two different Paddle events.\n\nMaking each individual event idempotent did not prevent two distinct events from racing over the same underlying subscription.\n\nThe deeper bug was our **sequencing assumption**.\n\nOur tests had covered the individual operations.\n\nThey had covered webhook replay.\n\nThey had covered duplicate credit protection.\n\nWhat they had not challenged was the assumption that two different subscription events would effectively be handled one after another.\n\nThat is the part I found interesting.\n\nThe code was tested.\n\nThe assumption connecting the pieces wasn't.\n\nA real external system invalidated it almost immediately.\n\nThe fix was not to invent a more complicated application-level lock.\n\nWe already had something very good at arbitrating concurrent writes:\n\nthe database.\n\nThe final pattern was roughly:\n\nThe SAVEPOINT mattered.\n\nWithout it, handling the uniqueness collision could poison the larger transaction and interfere with the rest of the webhook processing.\n\nThe losing handler should not become an error path.\n\nIt should simply discover:\n\nanother handler created the same logical subscription first\n\nand continue from the state that now exists.\n\nIn simplified form:\n\n```\nsubscription = find_subscription(provider_id)\n\nif subscription is None:\n    try:\n        with session.begin_nested():\n            subscription = create_subscription(provider_id)\n            session.flush()\n    except IntegrityError:\n        subscription = find_subscription(provider_id)\n```\n\nThe production code has more state handling around this, but that is the important idea.\n\nThe unique constraint becomes part of the concurrency mechanism rather than merely an exception waiting to happen.\n\nThis is the part I now care about most.\n\nAfter changing the code and adding the regression test, the task was still not finished.\n\nClaude redeployed the staging environment and ran another actual sandbox payment.\n\nThis time the webhook events arrived in the opposite order.\n\nThat was useful.\n\nThe fix was no longer depending on the ordering we happened to see during the first run.\n\nThe resulting subscription state was correct and credits were granted exactly once.\n\nI then replayed the real webhook notification and checked that the state and credit ledger remained unchanged.\n\nOnly after that did I consider the billing path proven.\n\nThe temporary Render infrastructure was deleted afterward, along with the temporary Paddle webhook destination. Cleaning up the receiver without cleaning up the sender would have left Paddle retrying a dead endpoint.\n\nA comment on a Reddit discussion about this gave another good example.\n\nSomeone described an agent-run publishing pipeline that considered a post successful because the publishing API returned `200`.\n\nBut sometimes the published content had been silently truncated.\n\nThe fix was to make the agent fetch the published item back and diff it against the source.\n\nThat is the same pattern.\n\nThese are different claims:\n\n```\nthe action completed without error\n```\n\nand:\n\n```\nthe intended state now exists in the real system\n```\n\nThe second one is what we usually care about.\n\nYet a lot of automation, including agentic automation, stops at the first.\n\nThis changed the way I think about using coding agents.\n\nA prompt like:\n\nImplement Paddle billing.\n\nis useful.\n\nImplement Paddle billing and write tests.\n\nis better.\n\nBut the much more interesting version is:\n\nHere is the external condition that would prove billing actually works. Keep going until you can demonstrate it.\n\nThat produces a different workflow:\n\n```\nimplement\n    ↓\ndeploy\n    ↓\ninteract with the real system\n    ↓\ninspect resulting state\n    ↓\ntry to falsify the assumptions\n    ↓\nfix\n    ↓\nrepeat the proof\n```\n\nThe agent is no longer just responsible for producing code.\n\nIt is responsible for producing evidence.\n\nOnce I started thinking this way, I found the same problem in several other parts of the release.\n\nA setup procedure can work perfectly on the developer's machine because the shell already contains environment variables, dependencies or state that the documentation forgot to mention.\n\nSo don't ask:\n\nAre the getting-started docs correct?\n\nMake the agent actually start from a clean clone and execute them.\n\nOne of our setup paths appeared reproducible until we removed configuration inherited from the agent's shell.\n\nThe environment had been silently helping the test pass.\n\nThe interesting test wasn't:\n\n```\ndoes setup work here?\n```\n\nIt was:\n\n```\ndoes setup work with none of our accidental local state?\n```\n\nThe application tests were green.\n\nA fresh:\n\n```\nalembic check\n```\n\nreported 174 schema operations.\n\nSome were metadata details rather than catastrophic defects, but the important point was that “tests pass” and “the deployed schema matches the declared model” were separate claims.\n\nSo we tested the second one directly.\n\nAn agent can write a regression test that passes without really protecting against the original failure.\n\nA simple way to challenge that is:\n\nThat gives you evidence that the test is detecting the actual regression rather than merely exercising nearby code.\n\nThere is a lot of attention around how many lines of code an AI agent can write.\n\nI'm finding the more valuable property is something else.\n\nThey can stay inside verification loops that are tedious enough that humans often stop early.\n\nProvision the environment.\n\nDeploy.\n\nExercise the system.\n\nInspect the database.\n\nRead webhook deliveries.\n\nReplay the event.\n\nChange the implementation.\n\nDeploy again.\n\nRepeat the transaction.\n\nCompare the result.\n\nClean everything up.\n\nNone of those individual steps is especially impressive.\n\nDoing the entire loop without deciding that “probably fine” is good enough is where the value starts to appear.\n\nAll of this happened while preparing the first release of **The Fabrica**, a production FastAPI + Next.js SaaS foundation I built after getting tired of rebuilding the same infrastructure for every product.\n\nIt includes the parts that tend to consume the beginning of a SaaS project:\n\nThe point isn't that those components are difficult to generate.\n\nCoding agents can generate most of them very quickly.\n\nThe difficult part is knowing whether all of those components still behave correctly once they interact with each other and with real external systems.\n\nThat is what I wanted to package.\n\nThe first release is here:\n\nAnd the main thing I took from the release process is this:\n\n**Don't only ask your coding agent to build the thing. Give it a condition that reality can prove false.**", "url": "https://wpnews.pro/news/i-made-claude-code-prove-billing-end-to-end-before-i-let-it-ship", "canonical_source": "https://dev.to/indierob_/i-made-claude-code-prove-billing-end-to-end-before-i-let-it-ship-2h9d", "published_at": "2026-09-08 07:17:14+00:00", "updated_at": "2026-09-08 07:31:38.066461+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "ai-tools", "artificial-intelligence"], "entities": ["Claude Code", "Paddle", "Render", "FastAPI", "Next.js", "Postgres", "Redis", "Celery"], "alternates": {"html": "https://wpnews.pro/news/i-made-claude-code-prove-billing-end-to-end-before-i-let-it-ship", "markdown": "https://wpnews.pro/news/i-made-claude-code-prove-billing-end-to-end-before-i-let-it-ship.md", "text": "https://wpnews.pro/news/i-made-claude-code-prove-billing-end-to-end-before-i-let-it-ship.txt", "jsonld": "https://wpnews.pro/news/i-made-claude-code-prove-billing-end-to-end-before-i-let-it-ship.jsonld"}}