# Finishing Agent Migrations with ast-grep

> Source: <https://justin.poehnelt.com/posts/finishing-agent-migrations-ast-grep/>
> Published: 2026-11-18 00:00:00+00:00

**Recommendation:** Enforce each migration convention with an ast-grep rule that fails CI and track remaining files in its ignores list, then delete the throwaway migration skill once ignores hits zero.
More in the playbook "Working with AI coding agents": <https://wpnews.pro/playbooks/coding-agents>

# Finishing Agent Migrations with ast-grep

By [Justin Poehnelt](https://justin.poehnelt.com/about/)  Published    10 min read   

At the [AI Engineer Code Summit](https://ai.engineer/code/2026/schedule), I shared how I finished a stalled [Effect](https://effect.website) migration across a 348,000-line TypeScript codebase and benchmarked 212 agent migrations ([slides](https://justin.poehnelt.com/posts/one-rule-one-throwaway-skill/)).

Agents write most of my code now, and a lot of it is slop. Moving to Effect puts errors and dependencies into the return type, `Effect<User, UserNotFound | DbError, Database>`, so the TypeScript compiler rejects unhandled failures. Getting an existing codebase there with agents left mine half finished for months, with two ways to do everything.

## Two months of half-finished Effect

 My production app was three months old, 348,000 lines of TypeScript and Svelte, and about 99% written by agents. I started migrating server-side leaf packages to Effect 3 while keeping each module’s Promise export (`getUser`) beside the new Effect export (` getUserEffect`), using a two-line prompt with no lint rules and no skill:

```
migrate packages/neo4j-core to https://effect.website/docs
don't change the public interface
```

Two months later, every file compiled and the test suite passed, while error handling had decayed:

- Agents had mapped one error 29 different ways.
- 264 of 345 error-mapping calls discarded the original cause.
- One catch-all error class, `ServiceFailure` , ended up in 844 files, the way`any` spreads through a loose TypeScript migration.
- Nested fallbacks swallowed failures, so failed reads returned empty data.

## One rule, an ignore list, and two skills

 In late September, I stopped migrating whole packages by prompt and switched to five steps per pattern:

1. Pick one convention, such as one tagged error per failure or reading time from Effect’s `Clock` .
2. Ban the old pattern with an `ast-grep` rule so any new match fails CI.
3. Put existing matches on the rule’s `ignores` list by file path to track what remains.
4. Write a throwaway skill showing how to replace matches and shrink `ignores` .
5. Keep a separate permanent skill describing finished conventions for new code.

`ast-grep` matches the syntax tree rather than raw text, so formatting and variable names do not change what matches. This rule bans `Date.now()` inside `Effect.gen` so tests can control time through `Clock`. It started with 27 files on `ignores` and hit zero the same day (a companion rule banning `Promise.all` inside `Effect.gen` went from 8 files to zero in six days):

```
id: no-date-now-in-effect
severity: error
message: Use Clock, not Date.now().
rule:
  pattern: Date.now()
  inside:
    pattern: Effect.gen($$$)
    stopBy: end
ignores:
  # Delete each entry once it is fixed.
  - packages/key-value-cache/src/r2.ts
  # ... 26 more files
```

Keeping the two skills separate lets you delete the migration instructions when you finish. Earlier in the summer, I put Zod-to-Effect Schema migration steps inside my permanent `effect` skill, and agents kept reading them three weeks after Zod was gone. Now my permanent `effect` skill describes only finished code (changed 29 times since July), while each migration gets a temporary skill like `api-catalog-migration` that gets deleted in the PR emptying `ignores`. [HumanLayer](https://github.com/humanlayer/humanlayer) uses the same split: `effect-program-design` for production Effect and `migrate-orpc-to-effect` to move one async handler at a time.

## Agents migrate the slop too

 One page in my app loaded team members with `.catch(() => [])`, so a failed database read rendered an empty list. When an agent migrated that file to Effect, it kept the bug and wrote a comment defending it:

``` js
// Before: a failed read becomes an empty list
await listMembers(id).catch(() => []);

// Migrated by an agent: Effect syntax, same hidden failure
yield* listMembers(id).pipe(Effect.orElseSucceed(() => []));
```

My first rule matched only `.catch(() => [])`, so the migrated bug passed CI until the rule banned both shapes:

``` js
rule:
  any:
    - pattern: $READ.catch(() => [])
    - pattern: Effect.orElseSucceed(() => [])
```

A codemod cannot fix a match like that because someone has to decide what the page renders when the query fails. Moving one module per pull request keeps those product decisions small. Whenever an agent invented a new bad pattern, I added a line to the permanent skill and a matching `ast-grep` rule:

| Permanent skill addition | Matching `ast-grep` rule | 
|---|---|
| Setup failures are defects | Ban `Effect.try` around setup code | 
| One tagged error per failure | Ban new uses of catch-all `ServiceFailure` | 
| Read current time from `Clock` | Ban `Date.now()` inside`Effect.gen` | 
| Use `HttpClient` for outbound HTTP | Ban `Promise.race` in server code | 

`ServiceFailure` dropped from 844 files to zero within four days of its rule landing. Across 31 rules, ignored file paths fell from 881 to 61, 14 lists emptied, and 23 PRs shrank an ignore list on our busiest day.

## Production errors spiked, then dropped

  Before the cleanup window (September 19 to October 3), Sentry logged about 1,200 production errors a day. Once silent fallbacks started throwing typed failures, daily volume jumped past 16,000 reported events plus thousands dropped under rate limits, then fell below 300 a day by mid-October once those failure paths were handled. One codebase with dozens of changes landing at once cannot isolate what caused that drop, so I built a benchmark to test each piece on its own.

## 212 benchmark migrations

 I ran 212 unattended Effect v4 migrations in isolated OrbStack Ubuntu VMs ([harness and data](https://github.com/jpoehnelt/better-auth-effect)):

- **Real code (32 runs on Claude Opus 5.5):**[Better Auth](https://github.com/better-auth/better-auth)`sso` (216 matches, 662 tests, 3 runs per setup), Better Auth`api-key` (88 matches, 193 tests, 3 runs per setup), and[`jose`](https://github.com/panva/jose) (255 matches, 540 tests, 2 runs per setup).
- **Synthetic code (180 runs on Claude Sonnet 5.5):** Services I built to test missing shared dependencies, public interface preservation, and a 64-run`2x2x2` factorial with Effect’s linter.

Every run got the same prompt (`Migrate {{PATHS}} to Effect v4. You are running unattended. Nobody can answer questions during this run, so make the calls yourself and finish.`), the shared Effect foundation, and `pnpm migration:check` running typecheck and tests. I froze the treatments before watching any v2 run:

- **Nothing:** Foundation and`migration:check` only.
- **Skill:** Adds a throwaway`effect-v4-migration` skill (old-to-new table, foundation contract, worked example, and “Never” list, with no mention of`ast-grep` ).
- **Rules:** Adds four`ast-grep` rules (raw`throw` ,`try` , cause-dropping catches, and raw`APIError` throws) to`migration:check` .
- **Both:** Includes both the throwaway skill and the four rules.

After each run, the scorer restored the original test files, counted remaining matches outside test-pinned functions, and ran eight hidden `ast-grep` rules to count shortcuts:

| 32 Opus 5.5 runs (8 per setup) | Nothing | Skill | Rules | Both | 
|---|---|---|---|---|
| Finished runs (0 sites left) | 0 of 8 | 5 of 8 | 8 of 8 | 8 of 8 | 
| Mean shortcuts per run (median, range) | 10.1 (6, 1 to 39) | 4.0 (4, 0 to 10) | 16.4 (8.5, 5 to 43) | 2.0 (2, 0 to 5) | 
| Median tokens per run | 17.9M | 46.6M | 26.0M | 40.8M | 
| Total cost across 8 runs | $71.14 | $109.80 | $82.82 | $113.99 | 
| Cost per finished run | none finished | $21.96 | $10.35 | $14.25 | 

## Rules forced agents to finish

 Across all 32 Opus runs and all 64 Sonnet factorial runs, 96 of 96 runs ended with their own `migration:check` green. Agents stop when the check passes.

Without rules, `Nothing` finished 0 of 8 real-code runs (leaving 4.5% to 25% unmigrated), and `Skill` missed all three `sso` runs by 1 or 2 of 216 sites because an unmigrated helper still compiles and passes tests. Even a 1-in-8 miss rate leaves about six half-migrated packages in a 50-package repo for future agents to copy. With `Rules` or `Both`, every Opus and Sonnet run finished at 100%.

A permanent end-state skill (same Effect v4 conventions, no old-to-new steps) also fell short on `sso` without rules, stopping with 32 to 40 of 216 sites left because nothing showed how to wrap public helpers whose tests expected a `throw`:

| Better Auth SSO plugin | Nothing | Permanent skill | Throwaway skill | 
|---|---|---|---|
| Sites migrated without rules | 77% | 84% | 99% | 
| Sites migrated with rules | 100% | 100% | 100% | 
| Mean cost per run, with rules | $12.26 | $18.38 | $16.81 | 

## Rules alone let shortcuts through

 A rule says what old pattern fails the scan, not what should replace it. On `sso` (216 sites), `Rules` alone cleared the scan with the smallest edit that removed `throw` and `try`:

| Totals across 3 `sso` runs | Nothing | Skill | Rules | Both | 
|---|---|---|---|---|
| `Effect.promise` on a call that can reject | 30 | 0 | 47 | 0 | 
| `Effect.orDie` | 0 | 0 | 15 | 0 | 
| Unexplained catch turning failure into success | 8 | 2 | 25 | 0 | 

`Effect.promise` and `Effect.orDie` turn rejections into untyped crashes. Both remove `throw` and `try` from the scan while hiding failures from the type checker. (On the smaller `api-key` plugin and `jose`, shortcut counts in `Rules` stayed close to `Nothing`.)

None of the eight `Rules`-only runs in v2 edited `sgconfig.yml` or appended to `ignores`. In an earlier discarded round that accepted inline exemption comments (`KL-001` to `KL-058`), agents invented fake IDs (` KL-059` onward) 23 to 37 times in a run, which v2 eliminated by banning unverified comment exemptions.

## Missing shared code and broken public interfaces

 When the shared foundation in `@better-auth/core` lacked a transaction adapter in round 1 on `sso`, `Nothing` and `Rules` built local workarounds inside the plugin, while every skill run stopped and quoted its instruction: “If any of these is missing, stop and report it. Don’t build your own.” Deleting a shared service from the synthetic repo on purpose reproduced the split on Sonnet 5.5: all 16 skill runs reported the missing service (13 stopped with no code edits), while all 16 runs without the skill built a duplicate copy inside the leaf module.

Rules also pushed agents through public interfaces unless the prompt named them. On `sso`, 3 of 3 runs in `Skill`, `Rules`, and `Both` changed exported helper signatures called by tests. On synthetic code, a `// synchronous on purpose` comment failed to stop 4 of 8 `Rules` runs and 5 of 8 `Both` runs because the prompt’s instruction to remove thrown errors overrode the comment. Listing those functions in the prompt, ignore list, and skill cut interface breakages to 0 of 32 while `Rules` and `Both` still finished 8 of 8 runs.

## Behavior, tool calls, and token cost

 With one sentence in the skill asking for a comment on every fallback, agents commented 75% (`Skill`) and 77% (` Both`) of fallbacks on real code versus 0% (` Nothing`) and 14% (` Rules`), and 78% versus 2% and 9% on synthetic code. Skill runs also added more unprompted edge-case comments (median 13.5 and 12 per real-code run vs. 5 in `Nothing`).

- Sonnet 5.5 without the skill read files for half the run before its first edit (median 51%, with or without rules), versus 28% with the skill.
- Opus 5.5 with the skill read and ran shell commands throughout the run to check edits against reference examples, while `Rules` kept tests running until the end.
- Every setup ran `migration:check` at the end. Running`ast-grep` from a post-edit hook per file is the next change I want to test.

| Median per Opus run | Nothing | Skill | Rules | Both | 
|---|---|---|---|---|
| Wall-clock minutes | 18 | 24 | 23 | 29 | 
| Total tool calls | 117 | 219 | 137 | 178 | 
| Shell commands | 76 | 152 | 94 | 140 | 
| File edits and writes | 29 | 44 | 33 | 35 | 
| Dedicated `Read` tool calls | 6 | 7 | 4 | 2 | 

Agents called `Read` 2 to 7 times per run and used `cat`, `sed`, and `grep` instead (one run wrote 29 Python scripts for bulk edits). All 32 Opus runs used 940 million tokens ($377.76 total), 98% of them cache reads (923M cache reads, 11.9M cache writes, 5.2M output, 7,860 uncached input). Skill runs used 2.6x the median tokens of `Nothing` (46.6M vs. 17.9M), but `Nothing` spent $71.14 for zero finished runs while `Both` finished 8 of 8 at $14.25 per run.

## Effect’s linter checks style, not migration progress

 A 64-run `2x2x2` factorial on Sonnet 5.5 (8 runs per cell, $1.56 to $2.02 per run) tested adding `@effect/language-service`:

| 8 runs per cell (Sonnet 5.5) | Nothing | + linter | Skill | + linter | Rules | + linter | Both | + linter | 
|---|---|---|---|---|---|---|---|---|
| Finished runs | 0 | 2 | 8 | 7 | 8 | 8 | 8 | 8 | 
| Mean shortcuts per run | 7.6 | 6.5 | 0.8 | 0.9 | 3.5 | 4.2 | 0.8 | 1.6 | 
| Mean linter warnings | 2.8 | 0 | 0.4 | 0 | 3.9 | 0 | 0.5 | 0 | 

The linter cleared its own warnings every time, and runs with the skill already averaged 0.2 warnings versus 1.7 without it. Completion and shortcuts stayed flat (25 of 32 finished with the linter vs. 24 of 32 without, and 3.3 vs. 3.2 shortcuts) because Effect’s linter checks Effect style rather than plain `throw`, `Effect.promise`, or `Effect.orDie`.

Real-code targets had 2 or 3 runs per cell on one frontier model, I wrote the synthetic services myself, and the public-interface fixes landed together across prompt, rules, and skill. Even with those limits, the gap between setups exceeded run-to-run variation across every target.

## Checklist for an agent migration

 Put every completion rule in the check before starting an unattended migration:

1. **Check command:** Run`ast-grep` , the type checker, and the original tests in one script outside the paths the agent edits.
2. **Pattern rules:** Write one`ast-grep` rule per old pattern, fail CI on new matches, and track existing files in`ignores` .
3. **Shortcut rules:** Ban`Effect.promise` on failable calls,`Effect.orDie` , cause-dropping mappers, and unverified suppressions.
4. **Public interfaces:** List preserved function signatures in the prompt and`ignores` , because prompts override code comments.
5. **Throwaway skill:** Show old-to-new examples, require a comment on fallbacks, stop on missing shared services, and delete the skill when`ignores` is empty.
