# I Made Claude Code Prove Billing End-to-End Before I Let It Ship

> Source: <https://dev.to/indierob_/i-made-claude-code-prove-billing-end-to-end-before-i-let-it-ship-2h9d>
> Published: 2026-09-08 07:17:14+00:00

I’ve been using coding agents for a while, but during the last release of a SaaS project I changed one thing about how I used them.

I stopped defining “done” as:

implementation finished, tests passing

and gave the agent an external condition instead:

**Do not let me ship until you can prove that billing works end-to-end against real infrastructure.**

That turned out to be much more useful than another round of code generation or code review.

Because the tests were already green.

The real system was not.

The project uses FastAPI, Next.js, Postgres, Redis, Celery and Paddle.

Billing already had tests around checkout, webhook processing, subscriptions, credit grants and idempotency.

At that point it would have been very easy to say:

Looks good. Ship it.

Instead, I asked Claude Code to prove the actual money path.

That meant it couldn't stop after reading the code or running pytest.

It had to create an environment where a real Paddle sandbox transaction could go through the application.

So it provisioned temporary infrastructure on Render through the API:

Then it configured a temporary Paddle sandbox webhook destination pointing at that deployment.

The requirement was simple: act like a real customer and show me the resulting state.

The agent then went through the actual application flow.

It authenticated through the app, opened checkout, completed a Paddle sandbox payment and inspected what happened afterward.

Not just the HTTP response.

Not just the redirect.

The resulting state.

It checked:

Before payment the account was free, with no paid subscription.

After payment it had the expected active subscription and the expected credit balance.

So far, good.

Then the sandbox exposed a bug.

We had implicitly been thinking about these events as a sequence:

```
subscription.created
        ↓
subscription.activated
```

That mental model was wrong.

In the sandbox, the two webhook handlers could overlap.

Both could begin while there was still no local subscription row.

Conceptually, each handler was doing something like:

```
subscription = find_subscription(provider_id)

if subscription is None:
    create_subscription(provider_id)
```

Perfectly reasonable if the operations happen sequentially.

Not safe if two different webhook events execute at the same time.

The actual failure mode was:

```
Handler A                       Handler B

SELECT subscription             SELECT subscription
→ nothing                       → nothing

INSERT subscription
                                INSERT subscription

success                         unique constraint violation
```

The database was correctly preventing duplicate subscriptions.

But one of the webhook handlers was now failing.

And this was not an idempotency problem in the usual sense.

These were two different Paddle events.

Making each individual event idempotent did not prevent two distinct events from racing over the same underlying subscription.

The deeper bug was our **sequencing assumption**.

Our tests had covered the individual operations.

They had covered webhook replay.

They had covered duplicate credit protection.

What they had not challenged was the assumption that two different subscription events would effectively be handled one after another.

That is the part I found interesting.

The code was tested.

The assumption connecting the pieces wasn't.

A real external system invalidated it almost immediately.

The fix was not to invent a more complicated application-level lock.

We already had something very good at arbitrating concurrent writes:

the database.

The final pattern was roughly:

The SAVEPOINT mattered.

Without it, handling the uniqueness collision could poison the larger transaction and interfere with the rest of the webhook processing.

The losing handler should not become an error path.

It should simply discover:

another handler created the same logical subscription first

and continue from the state that now exists.

In simplified form:

```
subscription = find_subscription(provider_id)

if subscription is None:
    try:
        with session.begin_nested():
            subscription = create_subscription(provider_id)
            session.flush()
    except IntegrityError:
        subscription = find_subscription(provider_id)
```

The production code has more state handling around this, but that is the important idea.

The unique constraint becomes part of the concurrency mechanism rather than merely an exception waiting to happen.

This is the part I now care about most.

After changing the code and adding the regression test, the task was still not finished.

Claude redeployed the staging environment and ran another actual sandbox payment.

This time the webhook events arrived in the opposite order.

That was useful.

The fix was no longer depending on the ordering we happened to see during the first run.

The resulting subscription state was correct and credits were granted exactly once.

I then replayed the real webhook notification and checked that the state and credit ledger remained unchanged.

Only after that did I consider the billing path proven.

The temporary Render infrastructure was deleted afterward, along with the temporary Paddle webhook destination. Cleaning up the receiver without cleaning up the sender would have left Paddle retrying a dead endpoint.

A comment on a Reddit discussion about this gave another good example.

Someone described an agent-run publishing pipeline that considered a post successful because the publishing API returned `200`.

But sometimes the published content had been silently truncated.

The fix was to make the agent fetch the published item back and diff it against the source.

That is the same pattern.

These are different claims:

```
the action completed without error
```

and:

```
the intended state now exists in the real system
```

The second one is what we usually care about.

Yet a lot of automation, including agentic automation, stops at the first.

This changed the way I think about using coding agents.

A prompt like:

Implement Paddle billing.

is useful.

Implement Paddle billing and write tests.

is better.

But the much more interesting version is:

Here is the external condition that would prove billing actually works. Keep going until you can demonstrate it.

That produces a different workflow:

```
implement
    ↓
deploy
    ↓
interact with the real system
    ↓
inspect resulting state
    ↓
try to falsify the assumptions
    ↓
fix
    ↓
repeat the proof
```

The agent is no longer just responsible for producing code.

It is responsible for producing evidence.

Once I started thinking this way, I found the same problem in several other parts of the release.

A setup procedure can work perfectly on the developer's machine because the shell already contains environment variables, dependencies or state that the documentation forgot to mention.

So don't ask:

Are the getting-started docs correct?

Make the agent actually start from a clean clone and execute them.

One of our setup paths appeared reproducible until we removed configuration inherited from the agent's shell.

The environment had been silently helping the test pass.

The interesting test wasn't:

```
does setup work here?
```

It was:

```
does setup work with none of our accidental local state?
```

The application tests were green.

A fresh:

```
alembic check
```

reported 174 schema operations.

Some were metadata details rather than catastrophic defects, but the important point was that “tests pass” and “the deployed schema matches the declared model” were separate claims.

So we tested the second one directly.

An agent can write a regression test that passes without really protecting against the original failure.

A simple way to challenge that is:

That gives you evidence that the test is detecting the actual regression rather than merely exercising nearby code.

There is a lot of attention around how many lines of code an AI agent can write.

I'm finding the more valuable property is something else.

They can stay inside verification loops that are tedious enough that humans often stop early.

Provision the environment.

Deploy.

Exercise the system.

Inspect the database.

Read webhook deliveries.

Replay the event.

Change the implementation.

Deploy again.

Repeat the transaction.

Compare the result.

Clean everything up.

None of those individual steps is especially impressive.

Doing the entire loop without deciding that “probably fine” is good enough is where the value starts to appear.

All of this happened while preparing the first release of **The Fabrica**, a production FastAPI + Next.js SaaS foundation I built after getting tired of rebuilding the same infrastructure for every product.

It includes the parts that tend to consume the beginning of a SaaS project:

The point isn't that those components are difficult to generate.

Coding agents can generate most of them very quickly.

The difficult part is knowing whether all of those components still behave correctly once they interact with each other and with real external systems.

That is what I wanted to package.

The first release is here:

And the main thing I took from the release process is this:

**Don't only ask your coding agent to build the thing. Give it a condition that reality can prove false.**