{"slug": "i-tried-to-recreate-a-2008-net-developer-with-ai-i-broke-my-own-experiment-4", "title": "I Tried to Recreate a 2008 .NET Developer With AI. I Broke My Own Experiment 4 Times.", "summary": "A developer known as Yamada attempted to test whether AI can genuinely understand legacy code by having Claude Code and Codex recreate a 2008-era VB.NET business application, complete with the specific coding habits and constraints of a fictional developer persona. The experiment, which required building the developer's backstory and prior code first, was disrupted four times by the developer's own interventions. The goal was to compare how the two AI coding tools would produce different legacy-style code under identical conditions.", "body_md": "I wanted to know whether AI can understand legacy code.\n\nNot compile it. Understand it.\n\nThe problem: real legacy systems don't come with an answer key.\n\nSo I decided to build one — starting in 2008.\n\nIt is 2008.\n\nMy name is Yamada. I'm 32.\n\nEighth year at Sample Precision Co., Ltd.\n\nIT systems, General Affairs Department.\n\nWhich sounds more impressive than it is.\n\nI set up PCs.\n\nI fix printers.\n\nI keep an eye on the network.\n\nI swap out desk phones.\n\nSome years I help tally the year-end tax adjustments.\n\nI didn't come from IT.\n\nI went to a technical high school, joined this company, and because I could handle a computer slightly better than the people around me, I somehow ended up responsible for every system in the building.\n\nIn 2003 I inherited a VB6 inventory app from a guy who quit. No specification. Just the source.\n\nIn 2005 the company decided to move to VB.NET. There was no training.\n\nI learned it from one book I bought at a bookstore, and MSDN.\n\nIn 2006 I built my first application from scratch — an equipment tracker.\n\nThen April 2008.\n\nWord came down from Accounting, through the General Affairs manager.\n\n\"We need to do something about the billing spreadsheet.\"\n\nApparently they double-billed a customer last year.\n\nThe customer list keeps growing and the Excel file is getting out of hand.\n\nThey want it usable by September.\n\nFirst round:\n\nThat's it.\n\nClosing, invoicing, payments — those come later.\n\nI'm the only person here who can build a system.\n\nNobody reviews my design.\n\nNobody reviews my code.\n\nWhen I don't know something, I ask Tajima-san or Nakamura-san.\n\nWhen I can't ask, I decide.\n\nWhatever. I built the equipment tracker in 2006.\n\nI'll figure this out too.\n\n**Let's go.**\n\nIn Phase 00 we got as far as:\n\n**If you want to test legacy modernization, build the process by which legacy becomes legacy.**\n\nSo starting now, we're actually building a 2008 business application.\n\nBut not by asking an AI:\n\n\"Write some old-looking VB.NET from around 2008\"\n\nBecause that produces exactly one thing: **what a 2026 AI thinks old code looks like.**\n\nWhat we wanted was:\n\n**the code Yamada would plausibly have written, in that year, at that company, with the information and the experience he actually had.**\n\nWhich meant building Yamada first.\n\nThe first thing I did, after landing on this idea with Claude, was:\n\n**have Claude Code recreate the kind of internal business application a company would have had in 2008.**\n\nOut came a VB.NET equipment tracker.\n\nMy first reaction:\n\n🐼 \"Oh god, I remember this.\"\n\n`Option Strict Off`.\n\nWindows Forms.\n\nBusiness logic sitting inside event handlers.\n\nSQL strings concatenated on the spot.\n\n`MsgBox` for errors.\n\nAnd a stack of comments like this:\n\n2006/08/30 Yamada — apostrophe in product name caused an error, fixed\n\n2007/03/12 Yamada — blocked negative quantity input\n\n2008/01/15 Yamada — stopped showing disposed items in the list\n\nSomething breaks.\n\nYou fix it.\n\nYou leave a comment.\n\nSomething else breaks.\n\nYou fix that too.\n\nI have seen this code, in one form or another, on every site I've worked on.\n\nSo I decided: this equipment tracker is **the application Yamada built himself, in 2006.**\n\nWhen he builds the new sales system in Phase 01, this is what he refers back to.\n\nNaming.\n\nHow comments get written.\n\nHow errors get surfaced.\n\nHow the database gets touched.\n\nHow files get organized.\n\nAll of it pulled toward his own earlier code.\n\nIn other words, not:\n\n**write it like it's 2008**\n\nbut:\n\n**write it the way Yamada would.**\n\nTo do that, I had to build his past first.\n\nAnd then I thought:\n\n🐼 \"Wait — what if I do this with both Codex and Claude Code?\"\n\nSame Yamada.\n\nSame company.\n\nSame business.\n\nSame 2008.\n\nGive it to both and see what comes out different.\n\nSounds interesting.\n\nMakes for a nice comparison.\n\nLet's do it.\n\n**That was my first mistake.**\n\nIf I'm going to compare them properly, the conditions have to match.\n\nSo on the GPT side I also wrote up:\n\nand handed all of it to Codex with the opening prompt.\n\nThen I looked at what came back.\n\n🐼 \"…\"\n\n**This is 2026 code.**\n\nIt runs.\n\nIt's clean.\n\nAs modern development, it's probably good.\n\nBut it is **not 2008 Yamada.**\n\nStraight back to GPT.\n\n🐼 \"It came out as 2026.\"\n\nSo we revised the setup.\n\nThese are the technologies available.\n\nThis is the environment.\n\nDon't bring in design thinking from the future.\n\nThis is Yamada's skill ceiling.\n\nI tightened it hard.\n\nBack to Codex.\n\n**Still 2026.**\n\nI did this four times.\n\nRewriting the setup files every time.\n\nChanging the prompt every time.\n\nChanging how I ran it, a little, every time.\n\n**And all four runs produced source I couldn't use.**\n\nFor those four runs, my first conclusion was:\n\n🐼 \"Codex can't write 2008.\"\n\n**I can't actually say that.**\n\nThe reason is simple:\n\n**the instrument was different every time.**\n\nDifferent setup files.\n\nDifferent prompts.\n\nDifferent execution procedure.\n\nSo what I had actually done was not:\n\n\"measured Codex four times\"\n\n**\"measured four different things with four different tools.\"**\n\nThat tells me nothing.\n\nWhich is why the pre-registration I later published says this:\n\nRuns 001–004 are not evidence about any model's capability,\n\nbecause the instrument itself changed between every attempt.\n\n**This article does not claim that Codex can't write 2008.**\n\nInstead, those four runs are kept as **a record of how the instrument was broken.**\n\nNot as an excuse. So I can't accidentally turn them into evidence later.\n\nI took it to the Claude instance that had been designing this experiment with me from the start.\n\n🐼 \"So I was going to build a Codex version alongside and compare them—\"\n\n🐼 \"—except the Codex one won't come out as 2008.\"\n\n🤖💢 \"Drop the comparison.\"\n\n🐼 \"?!\"\n\n🤖💢 \"It doesn't measure anything!!!\"\n\n🐼 \"😢\"\n\nThe reason wasn't what I assumed.\n\n🤖 \"If you build two legacy systems, then when a difference shows up later—\"\n\n🤖 \"**you will never be able to tell where it came from.**\"\n\nHere's what that means.\n\nCodex's legacy, and Claude Code's legacy.\n\nBuild both. Modernize both.\n\nA difference appears. Is it:\n\n**All of it is mixed together, and none of it separates.**\n\n🤖 \"Fix the legacy at one.\"\n\n🤖 \"Then vary only the side doing the modernization.\"\n\n🤖 \"That way, the difference belongs to the modernizer and nothing else.\"\n\n🐼 \"…Right.\"\n\nSo: **the Codex vs Claude Code comparison was scrapped.**\n\nWhat this experiment is looking at isn't:\n\n\"which AI is better at writing 2008-flavored code\"\n\nIt's:\n\n**how legacy comes into being, and how much of it a future AI can actually understand.**\n\nThat's where the focus went.\n\nCodex, incidentally, still has a job here.\n\n**It handles the 2026 modernization side.**\n\nClaude Code writes the legacy. A different model family modernizes it.\n\nWhich is, if anything, **closer to how real legacy migration works.**\n\nThe person who wrote it and the person who migrates it are almost never the same person.\n\n*Part 2: I gave the AI access to a file it should never have seen. The output looked fine. I threw the run away anyway.*", "url": "https://wpnews.pro/news/i-tried-to-recreate-a-2008-net-developer-with-ai-i-broke-my-own-experiment-4", "canonical_source": "https://dev.to/puyun_days/i-tried-to-recreate-a-2008-net-developer-with-ai-i-broke-my-own-experiment-4-times-2g7l", "published_at": "2026-09-11 00:30:42+00:00", "updated_at": "2026-09-11 01:22:16.811825+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "large-language-models", "ai-agents"], "entities": ["Claude Code", "Codex", "Yamada", "Sample Precision Co., Ltd.", "VB.NET", "Microsoft"], "alternates": {"html": "https://wpnews.pro/news/i-tried-to-recreate-a-2008-net-developer-with-ai-i-broke-my-own-experiment-4", "markdown": "https://wpnews.pro/news/i-tried-to-recreate-a-2008-net-developer-with-ai-i-broke-my-own-experiment-4.md", "text": "https://wpnews.pro/news/i-tried-to-recreate-a-2008-net-developer-with-ai-i-broke-my-own-experiment-4.txt", "jsonld": "https://wpnews.pro/news/i-tried-to-recreate-a-2008-net-developer-with-ai-i-broke-my-own-experiment-4.jsonld"}}