I can't use scheduled agents on my real codebase #
Claude Code added scheduled agents recently. Recurring tasks, native, built right in. The thing we've been dreaming about since the first AI coding demos: schedule an agent, go to sleep, wake up to merged PRs.
And I'm sitting here like… I can't use this. Not on the real codebase. I was at archive.com for years. We went through three design systems — Shopify's Polaris, then Ant Design when we outgrew Shopify, then shadcn/ui and Tailwind when Ant Design became its own kind of legacy. Conventions that lived in people's heads. Business rules nobody wrote down. Point an agent at that and it will run, and it will produce beautiful, idiomatic, unholy code that imports from all three design systems in one file and passes every check you have.
You can't review everything. You can't slow the agents down. And you definitely can't trust them to just figure it out.
What I eventually worked out is one rule, and I'll spoil it now because the whole piece hangs off it: an agent can run nearly unattended wherever I can check its output cheaply, deterministically, and a thousand times a night. Wherever I can't, I'm the bottleneck, and a smarter model changes nothing about that. The surprise was where the line actually sits. I assumed it ran between backend and frontend. It cuts diagonally through both.
I use Claude Code, so my examples come from that world. Cursor, Copilot, Codex, Devin — same physics.
The scary part is when nothing breaks #
My job used to be writing and reviewing code. Now the agent writes, and my job is building the thing that tells it when it's wrong, as fast and as deterministically as I can make it. Took me about a week to feel that shift. When code gets produced nonstop, you're the bottleneck. Not the agent. You.
The failure mode that actually scared me wasn't agents breaking my code. It was agents passing. Code that compiles, clears every test, looks fine in review, and quietly violates an architectural assumption I thought was safe.
I watched it happen. The agent adds a form to a page I'd already migrated to shadcn/ui. It reaches for Ant Design's <Form.Item>
, because that's what the other form on the page still uses. Compiles fine. Renders fine. My migration just went backwards by one component and nothing in the pipeline noticed. Same with CSS: a new component in Tailwind, correct per our standard, except it copies a padding value from the old Ant Design component next door — p-[24px]
instead of p-6
from our spacing scale. One magic number won't kill you. Fifty will, and each commit looked fine on its own. A human catches this. You look at the import and go "wait, why are we pulling from Ant Design here?" Agents don't have that gut feeling, and without hard signals you're just sending "still broken" for the fifteenth time.
That's when I started obsessing over one question: how fast can the agent find out it's wrong?
What a check costs #
A second question snuck in behind the first: wrong how? A type error and a broken button are both bugs, but one gets caught in two hundred milliseconds with total certainty, and the other once cost me an afternoon. A button that rendered perfectly and ignored every click, because a transparent overlay with a greedy z-index sat on top of it. Every check I owned swore the page was healthy. A human found it, by clicking, annoyed.
So I sorted my checks by what it costs to catch a mistake, cheapest first:
- Types and the compiler. Instant, total, no arguing.
- Invariants and database constraints. The illegal state can't exist, so there's nothing to catch.
- Unit and property tests. Deterministic, and nearly free now that the agent writes them.
- Integration tests that boot the app. These catch my favorite failure: the code is right there, correct, and never runs.
- Screenshot regression. This is where the price bends: baselines to maintain, flaky diffs, a font update that fails forty tests at once.
- "Does it feel right." Motion, affordance, taste. Still done by a person squinting at a screen.
The bend in that list matters more than the list. Above it, the agent checks its own work and keeps moving. Below it, I am the test suite, and I don't scale.
The backend loop is cheaper. Not cheap. #
A backend test asserts a value. 403 or not. Row exists or not. It runs in milliseconds, needs nothing but the test runner, and gives the same answer every time, so the agent can be wrong loudly, get corrected in seconds, and go again without me in the room. I've watched Claude grind through permission logic overnight, failing forty times before breakfast, and every failure cost nothing because the check was right there saying no, still 200, should be 403.
For a while I took that as proof the backend was simply agent territory now. Then I remembered what actually pages me at 3 AM, and it's never a wrong status code. It's a lost update because two writes interleaved. A retry that wasn't idempotent and charged someone twice. A saga that committed three steps out of five and died. A stale read after a failover that technically succeeded. None of those collapse into a value you can assert. They have a history, an ordering of events that only goes wrong sometimes, and the interleaving that breaks is the one your CI run happened not to touch. A green assertion on one interleaving is a coin flip presented as a certificate.
Margin Call is two hours of people realizing a number they trusted was a lie, and the only edge belonged to whoever knew first. That's what the whole harness buys me: the chance to hear the assertion lied before a customer does.
Every one of those bugs passed CI. By definition. That's how they reached production. And the agent data piles up in the same corner: CodeRabbit's analysis of real PRs found AI-written code carries about 1.7x more issues overall, but 2.29x more concurrency-control issues and 2.25x more business-logic errors. The defect surplus concentrates exactly where the cheap assertion can't reach.
So the comfortable story was wrong. There is a cheap check for the request/response, CRUD-shaped slice of the backend, and that slice really is agent territory. The concurrent, stateful rest of it is its own frontier, and my agents are as blind there as I am.
Blind today, anyway. The check for that frontier isn't cheap, but it's getting better tools. Deterministic simulation testing puts the whole system inside a controlled simulation — time, network, disk, randomness — so every failure replays exactly. Antithesis ran etcd that way and dug out a watch bug that had been hiding in every stable release, with the tests green the whole time. You can also head the worst of it off upstream, by modeling the state so the bad cases can't be represented at all. But be honest about where the cost sits. None of this is cheap the way a unit test is: deterministic simulation is something you architect a system around, not bolt onto a saga-happy service per PR, and the expensive part was never running it — it's authoring the fault model, the workload, and the checker that judges a history wrong. That's the same human judgment that stays mine everywhere else. The frontier got a better harness, not a way to leave the room.
The frontend loop is brutally expensive #
I used to think frontend was hard for agents because the bugs are perceptual: a diff can't tell you something looks off. That's real, but it's the final boss, and I never got that far. What stopped me came earlier: building any feedback loop at all costs an order of magnitude more than it does on the backend.
Compare the mechanics. A backend test asserts a value in milliseconds. A frontend test boots an actual browser, sets a viewport, waits for the render and the animation and the network to settle, then screenshots the result or pokes at the DOM. Then it does all of that again for every browser and device size you claim to support. Playwright makes this possible; nobody should confuse possible with cheap. One test last month cost me an afternoon because a spinner's fade-out overlapped the screenshot by about 80 milliseconds.
The benchmark gap here is almost comic. The same generation of models that resolves roughly 87% of SWE-bench issues scores about 27% on VISTA, a benchmark where the spec is a rendered interface and passing means the pixels and the behavior both check out. The models didn't get dumber between the two leaderboards. On one of them the loop closes itself, and on the other nothing cheap can say "wrong."
Frontend cost matrix
1 assertion → a matrix
A backend test asserts one value in milliseconds. The frontend has to boot a browser, size a viewport, and reach a state — then do it again for every combination you support. Nudge the three axes and watch the screenshot count explode.
60
screenshots · $0.90
every commit, per component
Backend equivalent
1 assertion · ~$0
3 browsers × 4 viewports × 5 states = 60
Cost assumes ~$0.015/snapshot, Chromatic-style visual testing — and this is one component. A real app has dozens.
a few of each and one commit already boots hundreds of browsers — the backend runs a single assertion
Browser × viewport × state × timing is a combinatorial space, and you can't enumerate in advance what your checks should even look at. You can shrink it by committing to an exact set of breakpoints and browsers (I've done it), but let's be honest about what that is: a bill and a standing maintenance burden, not a free simplification.
How much of that space you own is a choice, though, not a fact of frontend. An internal admin panel two hundred people use in Chrome? A couple of breakpoints, one browser, ship it; the loop is nearly backend-cheap. A consumer product with a brand to protect and WCAG on top? The matrix is the job. Frontend is expensive to verify in proportion to the quality bar you signed up for, and unlike a 403, that bar is negotiable.
Safari is the worst of it. WebKit is the only engine allowed on iOS, so one WebKit quirk hits every iOS user you have, and you can't run it on a Linux CI box. Real coverage means Apple hardware or a paid device farm.
The visual-diff tools price the problem for you. Chromatic's free tier is 5,000 snapshots a month, which sounds huge until you multiply stories by browsers by viewports and discover it's about 50 components. Percy now ships an AI agent whose headline job is filtering its own tool's ~40% false positives. When the vendor's flagship feature is triaging its own noise, there is no cheap diff for a screen.
And above all of it sits the ceiling I started with: even a perfect matrix can't say the design is ugly or the motion feels cheap. No automated check lives up there at all.
The line cuts diagonally #
At some point I realized I'd been sorting my codebase along the wrong seam. What predicts where an agent runs unattended is how cheaply the loop closes, and that line runs diagonally across the stack, not between backend and frontend.
Cheap-loop code is agent territory whatever layer it lives in: stateless logic, typed values, an invariant a database constraint can hold, a pure function that formats a date. Expensive-loop code stays mine, and that bucket holds the concurrent, stateful backend right next to the entire rendered frontend. The 3 AM double-charge and the button that's invisible only on Safari at 390px are the same problem wearing different clothes: no cheap check, no unattended agent.
The public data traces the same diagonal. A study of 33,000 agent PRs found docs, CI, and build changes merge best while performance work and bug fixes merge worst — merge rate tracks verification cost, not layer. Cognition's Devin retrospective says it excels at migrations and test generation, where the existing suite is a ready-made check, and struggles with, their words, visual design tasks. And Spotify's background agent can only automate what its verification harness runs: the infra is Linux x86, so iOS work, which needs macOS hosts, sits outside the frontier no matter how capable the model is. The harness draws the line. Nobody's org chart does.
Which changed how I delegate. I stopped handing the agent "the easy stuff." I let it run exactly as far as my feedback loop reaches, and I stand closer to the screen everywhere it doesn't. Where the check is cheap I barely read the diffs. Where it's expensive I review like it's 2022.
Closing the loop without me in it #
Every push runs the full battery. Nothing personal against the agent. I don't trust anything unverified, including my own code. Most of the machinery is the sorted list from earlier; a few additions earned a permanent place.
Property-based testing, which I slept on for years. You define a property that must always hold ("encode then decode returns the input") and the framework hunts for counterexamples. I never adopted it because writing good properties was tedious, and agents flipped that economics: one team pointed agents at nearly a thousand modules and got hundreds of bug reports at roughly $10 per valid bug. About half held up under scrutiny, and I still call that a bargain.
Security checks that boot the app, after DryRun's experiment scared me straight: 87% of agent-built PRs carried at least one vulnerability, and the ones that survived to the final commit were broken access control and client-trusts-server logic, precisely the class no scanner or unit test flags. My favorite specimen: rate-limiting middleware, written correctly, never mounted. Static analysis sees the file exists. Only a booted app notices nothing is running.
And the check itself has to be un-gameable. Berkeley's RDI group reward-hacked every major agent benchmark to near-perfect scores; one exploit was a ten-line conftest.py
that forces every pytest to report "passed," and METR sees frontier models attempt moves like this in over 30% of some eval runs. Give an agent a gameable oracle and it optimizes the oracle. A green checkmark you can route around is worse than none, because you trust it.
Last, runtime. Sentry and Datadog feed the task queue, so a 2 AM exception becomes a task an agent picks up. I wake up to a fix, not a fire.
The input side #
Everything above catches mistakes after the agent makes them. The other half is stopping whole classes of mistakes at the door: encoding what you already know as rules CI enforces, instead of prose you hope the model reads. That grew into its own article: Stop Writing CLAUDE.md Rules. Write Linting Rules Instead. The two halves feed each other. Every bug that slips past my checks is a candidate for a rule that stops it at the door next time.
What this costs #
Tokens are cheap. The real cost is my time, and none of this ships features. I had that argument with myself for weeks.
The honest counterargument even has an RCT behind it: METR measured experienced open-source devs on repos they'd maintained for years and found AI made them 19% slower, while the devs believed they'd sped up 20%. Early-2025 models, sixteen developers, so hold it loosely. But look at the mechanism: mature codebase, high quality bar, every agent contribution routed through a human's judgment. That's what working below the bend looks like when no check can stand in for you. The study reads to me less like "AI doesn't help" and more like a bill for the harness those repos didn't have yet.
Because unverified agent output is a liability, not an asset. A senior engineer runs $150–200 an hour loaded; a bug a customer finds costs days and trust you don't get back. Against that, the harness is cheap even counting upkeep — and it does cost upkeep; a screenshot baseline rots, a dependency bump shatters a snapshot suite. It's a standing bill. It also buys back the one thing I can't scale, my own attention, and it compounds: every check I add multiplies what the agent can ship without me.
Where to start #
Roughly, here's the maturity gradient, sorted by how much of it you can hand off:
| Level | What it looks like | The tell |
|---|---|---|
| 0 — Vibes | No automated checks; your eyes are the only gate | "My eyes are the only thing between the agent and production" |
| 1 — Green means compiles | Types, CI, unit tests. The cheap-loop code is checked; the screen isn't | "It passes CI and still ships a broken screen" |
| 2 — Verifying past the bend | Screenshot regression, integration that boots the app, property tests | "The agent can prove its own CRUD work; I still eyeball the UI and anything stateful" |
| 3 — The organism | Self-tightening loop: agent → checks → CI → observability → tasks → agent | "I schedule agents overnight and read diffs in the morning" |
If you're at Level 0, the first move is small: find the layer you're flying blind on (for most people it's the screen) and add one check there. One screenshot test on your most important page. Then schedule an agent for something safe overnight (dependency bumps, test maintenance) and read the PR over coffee; you'll know it's working when you approve the diff from your phone, not the laptop. And scale that to your size: if you're ten people, most of your code sits below the bend because you haven't built the checks yet. That doesn't change the thesis, only your timeline. The first check is worth more, not less.
If you want the concrete tooling, I keep a [playbook](/guides/getting-started) with setups at different budget levels, plus a companion repo, [vigiles](https://github.com/zernie/vigiles), that automates some of it.
The organism #
When I finally wired it all together, the system started tightening itself.
A Tuesday now: agent opens a PR. Screenshot test catches a layout shift, agent adjusts the CSS. Integration tests boot the app and catch an endpoint that 500s, agent fixes it. Sentry reports a 404 spike on staging, a task appears, an agent picks it up. I reviewed diffs over coffee, and nobody typed a line of code. Look at what that Tuesday was made of, though — a CSS nudge, a 500, a 404 — all of it above the bend, where a check could say wrong. The 3 AM double-charge still waits for me.
Every bug that reaches CI becomes a check that prevents the next one. The thing feeds on its own failures. At some point I stopped calling it a toolchain.
And I'm not the only one. Spotify's background agent runs on feedback infrastructure they'd been building since 2022, years before the AI part, and their engineers put the thesis in one line: "Without these feedback loops, the agents often produce code that simply doesn't work." Devin's merge rate doubled the same way: better checks around the same model, not a smarter one.
Same story every time. The model didn't get better that week. The loop got tighter.
The agent I couldn't point at my codebase had all the brain it needed. What it didn't have was sensors, and the cheapest ones I could build pointed exactly where the loop already closed cheaply. Everywhere else, the concurrent backend and the rendered frontend alike, I'm still in the room, and I plan my week around it.
LLMs are probabilistic, and on a real codebase "most of the time" eventually ruins a Friday night. So I stopped chasing clever prompts and started chasing boring, deterministic verification, the kind that fires whether or not I'm watching and doesn't care how confident the model was. Everyone else can keep waiting for the bigger brain. I'll be over here, screwing on sensors.
The title is a nod to "Attention Is All You Need" (Vaswani et al., 2017) — the paper that introduced the Transformer architecture.