{"slug": "pinning-both-ends-letting-ai-rewrite-15k-lines-of-test-tooling", "title": "Pinning Both Ends: Letting AI Rewrite 15k Lines of Test Tooling", "summary": "An engineer rewrote a personal test-generation tool, Tool Y, from about 15,000 lines to about 3,500 lines in 3.5 working days by pinning both ends — the parsed spreadsheet inputs and the byte-for-byte golden files — and letting AI rewrite the middle, with the engineer reading the AI's code after every round. No old golden file changed except those the upstream specification showed were wrong, and the rewrite replaced hand-written per-element-type classes with a single parser, validator and generator that reads the upstream specification's machine-readable element definitions, so a new specification version becomes a data change. The engineer reported that the old tool's spreadsheet-based test cases could silently change output when a valid row was inserted, since combined cases reference rows by number and the tool only checks that the number is in range.", "body_md": "Tool Y, an old test generator, shrank from about 15,000 lines to about 3,500 in 3.5 working days. Not one old golden file changed, except the ones the upstream specification showed were wrong.\n\nThe method is the same as in [Part 1](https://thecloudlet.github.io/technical/refactor-with-ai/part-1-deletion/): pin both ends, then let the AI rewrite the\nmiddle. The input end is the parsed data. The output end is the golden files,\nbyte for byte. The old tool itself was the check, and I read the AI's code after\nevery round.\n\nTool Y was one engineer's personal tool. It reads test specifications from spreadsheets and writes golden files: generated test cases, kept as the known-good set. Another flow, the downstream flow, runs those files as its regression suite.\n\nNobody had maintained tool Y for a long time, so my manager asked me to take it over. The first job was to upgrade it to a newer upstream specification version. The upstream specification is an outside standard. It defines the element types that every test case is built from.\n\nTool Y works, but nobody planned its structure. Every new need added a new script, a new config, or a new copy of an old file. The tree, with names changed:\n\n```\ntool-y/\n├── README.md                manual post-processing steps\n├── config_main/             spreadsheets and JSON for the main CLI\n│   ├── elements.xlsx\n│   ├── sets.json            refers to spreadsheet row numbers\n│   └── ...\n├── gen/                     CLI entry point, plus a flag that acts as another\n│   ├── main.py\n│   ├── elements.py          one hand-written class per element type\n│   ├── utils.py\n│   └── golden_b.py          a second golden path\n├── flow/                    mode dispatch, called back from gen/\n├── post/                    4 scripts, chained through JSON files\n├── config_b/\n│   └── elements.xlsx        same file name as config_main/, different content\n├── gen_b/                   generator B: its own entry point,\n│   ├── main.py              and its own copies of elements.py and utils.py\n│   ├── elements.py\n│   └── utils.py\n└── gen_c/                   generator C: reads a spreadsheet outside the repo\n```\n\nA test case can stand alone or combine several others. These are the modes. Each target, one setup of the downstream flow, has its own generator.\n\nWhat I found when I read the code:\n\nThe worst part was the spreadsheets themselves. Each specification version had its own spreadsheet, and every element type had its own tab in it, far too many to scroll through. The tabs sit along the bottom of the window. Scrolling right through the tabs, I could never reach the one I wanted. Nobody could review a test case in there, let alone maintain one.\n\nThat scale also hides mistakes. Combined test cases point at spreadsheet rows by number, and the tool checks only that the number is in range. So I tried it. I inserted one valid row and ran the same command again. No warning. The case had the same name, but the golden file was different.\n\nThe second problem was the generator itself. Every element type is a hand-written class. The upstream specification already publishes machine-readable definitions of its element types, but tool Y never reads them. So a new specification version means editing the classes one by one, by hand. And nothing checks that a class still matches the specification. The upgrade I was asked to do was exactly this work.\n\nPatching one script at a time would leave the next owner the same problem. So I rewrote it, under three rules I couldn't break:\n\nThe goal: all inputs in git, one parser, one validator, and one generator that reads the upstream specification's machine-readable definitions. Then a new specification version is a data change, and a new test case is a recipe, not code.\n\nBefore I touched any code, I ran the old tool once in every mode and kept the output. These are the old golden files. The new tool has to reproduce them byte for byte.\n\nGenerators B and C still read inputs from outside git. I ran them as they were, then moved those inputs into git, so every later run starts from the same files.\n\nThen I wrote a small compare tool. It takes two output trees and reports every file that differs, down to the field value. CI runs it on every change.\n\nFirst, the spreadsheets needed a replacement. The recipe format had to be easy to read for both a human and a parser, and keep the reason for a case next to the case. And no new dependency just to read a config file. I looked at six:\n\n| Format | Comments | Clean git diff | Parser | Catch | \n|---|---|---|---|---|\n| Spreadsheet (old) | no | no | third-party | binary file, references by row number | \n| JSON | no | yes | stdlib | no comments, so no \"why\" | \n| YAML | yes | yes | third-party | guesses types, e.g. `1.10` →`1.1` | \n| Python as config | yes | yes | it's code | a recipe becomes a program | \n| Racket DSL | yes | yes | easy to write | few engineers read Racket | \n| TOML | yes | yes | stdlib (3.11+) | deep nesting gets awkward | \n\nYAML reads `version: 1.10` as the number `1.1`. I don't want the format to\nguess. I really wanted Racket, but the next owner would have to maintain my\ninterpreter. So I chose TOML: comments, explicit types, and a parser in the\nPython standard library. A recipe is one TOML block per test case:\n\n```\n# Recipes for upstream specification version \"1.10\".\nspec_version = \"1.10\"  # a string, so it stays \"1.10\"\n\n[[case]]\nname = \"case_001\"\ntest_intent = \"<one line: why this case exists>\"\nkind = \"alpha\"\nmode = \"A\"\nsize = 4\nstrict = true\n```\n\nI had the AI write a one-time converter. Each spreadsheet row became a recipe, and every reference by row number became a reference by name.\n\nThen I had the AI write the input pin: parse the same test cases with the old code and the new parser, and check that both give the same data. A small adapter turns the new shape into the old one first. Once it passed, I deleted the converter.\n\nThe validator defines valid input. It also checks each case against the upstream's machine-readable definitions. That is the only check a new specification version gets, because it has no old golden files. It shows the case is legal. It does not show the case tests the right thing.\n\nThis check also found old bugs. Some old test cases failed the upstream definitions. Copying them would only keep the bugs. So I fixed them or dropped them, and took them out of the comparison.\n\nI moved one element type first, to see whether the AI builds this structure:\n`parser → validator → generator → golden files`. Then two. Then batches of 5,\nthen 10, until every element type had moved and every check was green.\n\nThe AI did get things wrong. But the check went red, and the AI fixed its own work before it ever reached me. The pinned ends weren't only my safety net. They were the AI's feedback loop.\n\nAfter each round, I read the code by eye before I let the next round grow. I also checked that the change touched none of the pins: the old golden files, the compare tool, or the parse test. That review was the only guard. No tool stopped the AI from editing the pins. Next time, I will make them read-only with chmod, so an accidental edit fails. CI will also check their hashes against a list kept outside the agent's workspace, because an agent with a shell can run chmod too.\n\nThe pins do not check design. For the first element type, I read the design and\ndecided where each part goes. I dictated the internal functions and the structs\nby hand. I still don't have a good method for that. If anyone has one, [please let me know](https://thecloudlet.github.io/technical/refactor-with-ai/part-2-refactor/#contact).\n\nOnly after every mode matched byte for byte did I delete anything. The old generators went, the second golden path went, and the spreadsheets went.\n\nThen the scaffolding went too: the adapter and the migration tests. Same as Part 1. If it stayed, it would test code that no longer exists. The validator tests and the generator unit tests stay.\n\n| What | Before | After | \n|---|---|---|\n| Generator code (lines) | ~15,000 | ~3,500 | \n| Time spent (working days) | - | 3.5 | \n| Old golden files changed (bytes) | - | 0* | \n| Entry points | 4 | 1 | \n| Inputs outside git | 2 | 0 | \n| Upstream versions | 1 | multiple | \n\n*Every mode and every target, including generators B and C. Old test cases that failed the upstream definitions were fixed or dropped, and left out of the comparison.\n\nThe upgrade I was first asked to do is done, and the downstream flow uses its output.\n\nThe AI did most of the typing again. The pins caught its mistakes, and I read every round. Next time, the pins get a guard of their own.", "url": "https://wpnews.pro/news/pinning-both-ends-letting-ai-rewrite-15k-lines-of-test-tooling", "canonical_source": "https://thecloudlet.github.io/technical/refactor-with-ai/part-2-refactor/", "published_at": "2026-10-23 00:00:00+00:00", "updated_at": "2026-10-25 10:43:56.335364+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "artificial-intelligence"], "entities": ["Tool Y"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/pinning-both-ends-letting-ai-rewrite-15k-lines-of-test-tooling", "markdown": "https://wpnews.pro/news/pinning-both-ends-letting-ai-rewrite-15k-lines-of-test-tooling.md", "text": "https://wpnews.pro/news/pinning-both-ends-letting-ai-rewrite-15k-lines-of-test-tooling.txt", "jsonld": "https://wpnews.pro/news/pinning-both-ends-letting-ai-rewrite-15k-lines-of-test-tooling.jsonld"}}