By Justin Poehnelt Published 10 min read
At the AI Engineer Code Summit, I shared how I finished a stalled Effect migration across a 348,000-line TypeScript codebase and benchmarked 212 agent migrations (slides).
Agents write most of my code now, and a lot of it is slop. Moving to Effect puts errors and dependencies into the return type, Effect<User, UserNotFound | DbError, Database>, so the TypeScript compiler rejects unhandled failures. Getting an existing codebase there with agents left mine half finished for months, with two ways to do everything.
Two months of half-finished Effect #
My production app was three months old, 348,000 lines of TypeScript and Svelte, and about 99% written by agents. I started migrating server-side leaf packages to Effect 3 while keeping each module’s Promise export (getUser) beside the new Effect export ( getUserEffect), using a two-line prompt with no lint rules and no skill:
migrate packages/neo4j-core to https://effect.website/docs
don't change the public interface
Two months later, every file compiled and the test suite passed, while error handling had decayed:
- Agents had mapped one error 29 different ways.
- 264 of 345 error-mapping calls discarded the original cause.
- One catch-all error class,
ServiceFailure, ended up in 844 files, the wayanyspreads through a loose TypeScript migration. - Nested fallbacks swallowed failures, so failed reads returned empty data.
One rule, an ignore list, and two skills #
In late September, I stopped migrating whole packages by prompt and switched to five steps per pattern:
- Pick one convention, such as one tagged error per failure or reading time from Effect’s
Clock. - Ban the old pattern with an
ast-greprule so any new match fails CI. - Put existing matches on the rule’s
ignoreslist by file path to track what remains. - Write a throwaway skill showing how to replace matches and shrink
ignores. - Keep a separate permanent skill describing finished conventions for new code.
ast-grep matches the syntax tree rather than raw text, so formatting and variable names do not change what matches. This rule bans Date.now() inside Effect.gen so tests can control time through Clock. It started with 27 files on ignores and hit zero the same day (a companion rule banning Promise.all inside Effect.gen went from 8 files to zero in six days):
id: no-date-now-in-effect
severity: error
message: Use Clock, not Date.now().
rule:
pattern: Date.now()
inside:
pattern: Effect.gen($$$)
stopBy: end
ignores:
- packages/key-value-cache/src/r2.ts
Keeping the two skills separate lets you delete the migration instructions when you finish. Earlier in the summer, I put Zod-to-Effect Schema migration steps inside my permanent effect skill, and agents kept reading them three weeks after Zod was gone. Now my permanent effect skill describes only finished code (changed 29 times since July), while each migration gets a temporary skill like api-catalog-migration that gets deleted in the PR emptying ignores. HumanLayer uses the same split: effect-program-design for production Effect and migrate-orpc-to-effect to move one async handler at a time.
Agents migrate the slop too #
One page in my app loaded team members with .catch(() => []), so a failed database read rendered an empty list. When an agent migrated that file to Effect, it kept the bug and wrote a comment defending it:
// Before: a failed read becomes an empty list
await listMembers(id).catch(() => []);
// Migrated by an agent: Effect syntax, same hidden failure
yield* listMembers(id).pipe(Effect.orElseSucceed(() => []));
My first rule matched only .catch(() => []), so the migrated bug passed CI until the rule banned both shapes:
rule:
any:
- pattern: $READ.catch(() => [])
- pattern: Effect.orElseSucceed(() => [])
A codemod cannot fix a match like that because someone has to decide what the page renders when the query fails. Moving one module per pull request keeps those product decisions small. Whenever an agent invented a new bad pattern, I added a line to the permanent skill and a matching ast-grep rule:
| Permanent skill addition | Matching ast-grep rule |
|---|---|
| Setup failures are defects | Ban Effect.try around setup code |
| One tagged error per failure | Ban new uses of catch-all ServiceFailure |
Read current time from Clock |
Ban Date.now() insideEffect.gen |
Use HttpClient for outbound HTTP |
Ban Promise.race in server code |
ServiceFailure dropped from 844 files to zero within four days of its rule landing. Across 31 rules, ignored file paths fell from 881 to 61, 14 lists emptied, and 23 PRs shrank an ignore list on our busiest day.
Production errors spiked, then dropped #
Before the cleanup window (September 19 to October 3), Sentry logged about 1,200 production errors a day. Once silent fallbacks started throwing typed failures, daily volume jumped past 16,000 reported events plus thousands dropped under rate limits, then fell below 300 a day by mid-October once those failure paths were handled. One codebase with dozens of changes landing at once cannot isolate what caused that drop, so I built a benchmark to test each piece on its own.
212 benchmark migrations #
I ran 212 unattended Effect v4 migrations in isolated OrbStack Ubuntu VMs (harness and data):
- Real code (32 runs on Claude Opus 5.5):Better Auth
sso(216 matches, 662 tests, 3 runs per setup), Better Authapi-key(88 matches, 193 tests, 3 runs per setup), andjose(255 matches, 540 tests, 2 runs per setup). - Synthetic code (180 runs on Claude Sonnet 5.5): Services I built to test missing shared dependencies, public interface preservation, and a 64-run
2x2x2factorial with Effect’s linter.
Every run got the same prompt (Migrate {{PATHS}} to Effect v4. You are running unattended. Nobody can answer questions during this run, so make the calls yourself and finish.), the shared Effect foundation, and pnpm migration:check running typecheck and tests. I froze the treatments before watching any v2 run:
- Nothing: Foundation and
migration:checkonly. - Skill: Adds a throwaway
effect-v4-migrationskill (old-to-new table, foundation contract, worked example, and “Never” list, with no mention ofast-grep). - Rules: Adds four
ast-greprules (rawthrow,try, cause-dropping catches, and rawAPIErrorthrows) tomigration:check. - Both: Includes both the throwaway skill and the four rules.
After each run, the scorer restored the original test files, counted remaining matches outside test-pinned functions, and ran eight hidden ast-grep rules to count shortcuts:
| 32 Opus 5.5 runs (8 per setup) | Nothing | Skill | Rules | Both |
|---|---|---|---|---|
| Finished runs (0 sites left) | 0 of 8 | 5 of 8 | 8 of 8 | 8 of 8 |
| Mean shortcuts per run (median, range) | 10.1 (6, 1 to 39) | 4.0 (4, 0 to 10) | 16.4 (8.5, 5 to 43) | 2.0 (2, 0 to 5) |
| Median tokens per run | 17.9M | 46.6M | 26.0M | 40.8M |
| Total cost across 8 runs | $71.14 | $109.80 | $82.82 | $113.99 |
| Cost per finished run | none finished | $21.96 | $10.35 | $14.25 |
Rules forced agents to finish #
Across all 32 Opus runs and all 64 Sonnet factorial runs, 96 of 96 runs ended with their own migration:check green. Agents stop when the check passes.
Without rules, Nothing finished 0 of 8 real-code runs (leaving 4.5% to 25% unmigrated), and Skill missed all three sso runs by 1 or 2 of 216 sites because an unmigrated helper still compiles and passes tests. Even a 1-in-8 miss rate leaves about six half-migrated packages in a 50-package repo for future agents to copy. With Rules or Both, every Opus and Sonnet run finished at 100%.
A permanent end-state skill (same Effect v4 conventions, no old-to-new steps) also fell short on sso without rules, stopping with 32 to 40 of 216 sites left because nothing showed how to wrap public helpers whose tests expected a throw:
| Better Auth SSO plugin | Nothing | Permanent skill | Throwaway skill |
|---|---|---|---|
| Sites migrated without rules | 77% | 84% | 99% |
| Sites migrated with rules | 100% | 100% | 100% |
| Mean cost per run, with rules | $12.26 | $18.38 | $16.81 |
Rules alone let shortcuts through #
A rule says what old pattern fails the scan, not what should replace it. On sso (216 sites), Rules alone cleared the scan with the smallest edit that removed throw and try:
Totals across 3 sso runs |
Nothing | Skill | Rules | Both |
|---|---|---|---|---|
Effect.promise on a call that can reject |
30 | 0 | 47 | 0 |
Effect.orDie |
0 | 0 | 15 | 0 |
| Unexplained catch turning failure into success | 8 | 2 | 25 | 0 |
Effect.promise and Effect.orDie turn rejections into untyped crashes. Both remove throw and try from the scan while hiding failures from the type checker. (On the smaller api-key plugin and jose, shortcut counts in Rules stayed close to Nothing.)
None of the eight Rules-only runs in v2 edited sgconfig.yml or appended to ignores. In an earlier discarded round that accepted inline exemption comments (KL-001 to KL-058), agents invented fake IDs ( KL-059 onward) 23 to 37 times in a run, which v2 eliminated by banning unverified comment exemptions.
Missing shared code and broken public interfaces #
When the shared foundation in @better-auth/core lacked a transaction adapter in round 1 on sso, Nothing and Rules built local workarounds inside the plugin, while every skill run stopped and quoted its instruction: “If any of these is missing, stop and report it. Don’t build your own.” Deleting a shared service from the synthetic repo on purpose reproduced the split on Sonnet 5.5: all 16 skill runs reported the missing service (13 stopped with no code edits), while all 16 runs without the skill built a duplicate copy inside the leaf module.
Rules also pushed agents through public interfaces unless the prompt named them. On sso, 3 of 3 runs in Skill, Rules, and Both changed exported helper signatures called by tests. On synthetic code, a // synchronous on purpose comment failed to stop 4 of 8 Rules runs and 5 of 8 Both runs because the prompt’s instruction to remove thrown errors overrode the comment. Listing those functions in the prompt, ignore list, and skill cut interface breakages to 0 of 32 while Rules and Both still finished 8 of 8 runs.
Behavior, tool calls, and token cost #
With one sentence in the skill asking for a comment on every fallback, agents commented 75% (Skill) and 77% ( Both) of fallbacks on real code versus 0% ( Nothing) and 14% ( Rules), and 78% versus 2% and 9% on synthetic code. Skill runs also added more unprompted edge-case comments (median 13.5 and 12 per real-code run vs. 5 in Nothing).
- Sonnet 5.5 without the skill read files for half the run before its first edit (median 51%, with or without rules), versus 28% with the skill.
- Opus 5.5 with the skill read and ran shell commands throughout the run to check edits against reference examples, while
Ruleskept tests running until the end. - Every setup ran
migration:checkat the end. Runningast-grepfrom a post-edit hook per file is the next change I want to test.
| Median per Opus run | Nothing | Skill | Rules | Both |
|---|---|---|---|---|
| Wall-clock minutes | 18 | 24 | 23 | 29 |
| Total tool calls | 117 | 219 | 137 | 178 |
| Shell commands | 76 | 152 | 94 | 140 |
| File edits and writes | 29 | 44 | 33 | 35 |
Dedicated Read tool calls |
6 | 7 | 4 | 2 |
Agents called Read 2 to 7 times per run and used cat, sed, and grep instead (one run wrote 29 Python scripts for bulk edits). All 32 Opus runs used 940 million tokens ($377.76 total), 98% of them cache reads (923M cache reads, 11.9M cache writes, 5.2M output, 7,860 uncached input). Skill runs used 2.6x the median tokens of Nothing (46.6M vs. 17.9M), but Nothing spent $71.14 for zero finished runs while Both finished 8 of 8 at $14.25 per run.
Effect’s linter checks style, not migration progress #
A 64-run 2x2x2 factorial on Sonnet 5.5 (8 runs per cell, $1.56 to $2.02 per run) tested adding @effect/language-service:
| 8 runs per cell (Sonnet 5.5) | Nothing | + linter | Skill | + linter | Rules | + linter | Both | + linter |
|---|---|---|---|---|---|---|---|---|
| Finished runs | 0 | 2 | 8 | 7 | 8 | 8 | 8 | 8 |
| Mean shortcuts per run | 7.6 | 6.5 | 0.8 | 0.9 | 3.5 | 4.2 | 0.8 | 1.6 |
| Mean linter warnings | 2.8 | 0 | 0.4 | 0 | 3.9 | 0 | 0.5 | 0 |
The linter cleared its own warnings every time, and runs with the skill already averaged 0.2 warnings versus 1.7 without it. Completion and shortcuts stayed flat (25 of 32 finished with the linter vs. 24 of 32 without, and 3.3 vs. 3.2 shortcuts) because Effect’s linter checks Effect style rather than plain throw, Effect.promise, or Effect.orDie.
Real-code targets had 2 or 3 runs per cell on one frontier model, I wrote the synthetic services myself, and the public-interface fixes landed together across prompt, rules, and skill. Even with those limits, the gap between setups exceeded run-to-run variation across every target.
Checklist for an agent migration #
Put every completion rule in the check before starting an unattended migration:
- Check command: Run
ast-grep, the type checker, and the original tests in one script outside the paths the agent edits. - Pattern rules: Write one
ast-greprule per old pattern, fail CI on new matches, and track existing files inignores. - Shortcut rules: Ban
Effect.promiseon failable calls,Effect.orDie, cause-dropping mappers, and unverified suppressions. - Public interfaces: List preserved function signatures in the prompt and
ignores, because prompts override code comments. - Throwaway skill: Show old-to-new examples, require a comment on fallbacks, stop on missing shared services, and delete the skill when
ignoresis empty.