{"slug": "when-the-coding-agent-builds-the-world-it-gets-judged-in", "title": "When the Coding Agent Builds the World It Gets Judged In", "summary": "A developer behind DATAMIMIC argues that coding agents which generate their own test fixtures and expectations can produce self-consistent but wrong results, because the same misinterpretation of a requirement flows from implementation to fixture to test. To address this, DATAMIMIC CE 4.1 introduces a model-driven authoring workflow in which an agent captures the intended test world in a typed model.dm.json file, which the engine compiles, validates, and checks against declared expectations, returning structured diagnostics when assumptions fail.", "body_md": "Many years ago I had lots of discussions about AI with a former colleague.\n\nHe was a theoretical physicist and he really loved to dive deep into complex topics and thinking. This was around the time of AlphaGo and Leela Chess.\n\nWe talked about decision making, world models and also consciousness.\n\nOf course I am not trying to connect software testing to consciousness here. The part that stayed with me was much simpler:\n\n**A decision only makes sense relative to some model of the world.**\n\nI never connected this thought to testing until coding agents started writing the implementation, fixtures and tests themselves.\n\nThen it suddenly became very practical.\n\nGive an agent a requirement.\n\nIt writes the implementation.\n\nThen some fixtures.\n\nThen the tests.\n\nEverything is green.\n\nWhat if the agent misunderstood the requirement at the beginning?\n\nThe implementation follows interpretation A.\n\nThe fixtures represent interpretation A.\n\nThe expectations are based on interpretation A.\n\nAnd finally the tests prove that interpretation A is internally consistent.\n\nNothing necessarily disagrees.\n\nThe system can still be wrong.\n\nThis is where I started thinking differently about test data.\n\nTest data is not just some input required to execute a test.\n\nIt defines part of the world in which the implementation has to behave.\n\nFor a small isolated function this distinction may not matter much.\n\nFor a business process with state, relationships, existing data, several systems and rules nobody fully remembers anymore, it matters a lot.\n\nI have no problem with an agent generating test data.\n\nA capable agent can write a Python script, execute it and create very good data.\n\nSomeone pushed back on this with exactly that point:\n\nA smart agent can just write the generation code first and execute it.\n\nYes. Absolutely.\n\nCode vs model is not really what I care about.\n\nI care about where the information comes from.\n\nIdeally the agent building the test world gets things like:\n\nBut not the implementation it will later judge.\n\nOtherwise there is a shortcut.\n\nThe agent can generate data that fits what the implementation actually does instead of challenging it against what it should do.\n\nIt doesn't even need to do this consciously. The same assumption can simply move from implementation to fixture to expectation.\n\nThe result is self-consistent.\n\nStill wrong.\n\nGenerated code can work.\n\nBut when I review arbitrary fixture code I have to understand two things at the same time:\n\nWhat world is this trying to create?\n\nAnd how does the code create it?\n\nI prefer separating those questions.\n\nWith a model-driven approach I can look at the world itself:\n\nThe engine owns the mechanics.\n\nThat makes the test world easier to inspect and discuss.\n\nNot magically correct. Just more visible.\n\nThis is also one reason why we changed the authoring workflow in DATAMIMIC CE 4.1.\n\nI build DATAMIMIC, so my interest here is pretty obvious.\n\nBut the idea itself does not depend on DATAMIMIC. You could build the same separation with Python, SQL or any another data generator.\n\nWhat I want from DATAMIMIC is simpler:\n\nI want to see what world we are building, reproduce it, and keep it separated from the implementation.\n\nIn CE 4.1 an agent can preserve that intent in a typed `model.dm.json`.\n\nA simplified shape looks like this:\n\n```\n{\n  \"version\": \"1\",\n  \"seed\": 42,\n  \"products\": [...],\n  \"expectations\": [...]\n}\n```\n\nTake a small requirement:\n\nCreate 4 customers. Each customer has exactly 2 orders. Customer IDs are unique. Every order references its real customer. Amounts are between 10 and 500.\n\nThe model can make those assumptions explicit:\n\n```\ncustomers = exactly 4\norders per customer = exactly 2\ncustomer.id = unique\norder.customer_id -> customer.id\namount = 10..500\n```\n\nDATAMIMIC compiles the model, validates it, performs a bounded execution and checks the declared expectations.\n\nIf something is wrong, the agent gets structured diagnostics back.\n\nIt changes the model and tries again.\n\nA successful authoring cycle ends with:\n\n```\nverified=true\n```\n\nThis is much more useful to me than:\n\nGenerate some realistic customers and orders.\n\nWhat is realistic?\n\nFour customers or four million?\n\nCan an order exist without a customer?\n\nCan the same customer ID appear twice?\n\nAre negative amounts valid?\n\nThe model forces some of these assumptions into something we can actually inspect.\n\nYou can try the CE yourself:\n\n```\npip install datamimic-ce\ndatamimic capabilities\n```\n\n[https://github.com/rapiddweller/datamimic](https://github.com/rapiddweller/datamimic)\n\nThis is also where I need to be careful with the claim.\n\nWe can build a perfectly deterministic wrong world.\n\nIf the spec is wrong, the model can be wrong.\n\nIf the expectation is wrong, `verified=true` can correctly verify the wrong expectation.\n\nThat actually came up in a Hacker News discussion around this topic.\n\nSomeone described a test where the expectation itself was fabricated. The implementation failed the test, but later they found that the assertion was wrong because the real algorithm behaved globally, not locally as assumed.\n\nThat is exactly the problem.\n\nIndependence from the implementation removes one source of bias.\n\nIt does not make the oracle correct.\n\nDeterminism lets me reproduce the same world.\n\nIt does not tell me whether this is the right world.\n\nAnd an explicit model lets me review the assumptions.\n\nIt doesn't prove them.\n\nThere is another useful pushback I got:\n\nA few rows from a real staging database can sometimes teach you more than a thousand beautiful synthetic fixtures.\n\nI agree.\n\nReal systems contain things nobody put into the original spec.\n\nNulls.\n\nWeird string lengths.\n\nDuplicates.\n\nOld records.\n\nUnexpected distributions.\n\nBroken historical state.\n\nSome migration from six years ago which technically should not exist anymore but still affects today's system.\n\nSo I don't see synthetic data and real data as opposites.\n\nWhat I want is closer to:\n\nobserve the real environment, combine this with the spec and domain constraints, make the assumptions explicit, and then create a deterministic world from that.\n\nThis is also where things become different between a small app and a larger enterprise system.\n\nIn a small greenfield application, the agent may already know almost everything it needs.\n\nThe schema is small.\n\nThe dependencies are obvious.\n\nThere may be one database.\n\nNow take a system that has been running for 15 years.\n\nSeveral databases.\n\nMongoDB somewhere.\n\nServices owned by different teams.\n\nExisting customer data.\n\nPermissions.\n\nSchemas that changed over time.\n\nBusiness rules nobody fully remembers.\n\nIn that kind of system I want more disagreement between implementation and test world.\n\nNot less.\n\nWith our DATAMIMIC EE 4.0 release we moved further into this direction.\n\nThe platform has services that analyse connected environments like SQL databases and MongoDB.\n\nThey inspect schemas and relationships, do planning around dependencies, recommend generators and converters, and use this information to help build the DATAMIMIC model.\n\nThe coding agent does not need to invent the world from scratch.\n\nThere is already evidence.\n\nThe platform gives it access to that evidence under the same project and permission model used by the other DATAMIMIC clients.\n\nThe web UI, IDE integration and agents work against the same project.\n\nFor me that becomes much more interesting than just asking an LLM to generate fixture code.\n\nEven if the test world is independent from the implementation, where does its truth come from?\n\nThe spec?\n\nThe database?\n\nProduction observations?\n\nCustomer examples?\n\nAn API contract?\n\nAnother agent?\n\nA human domain expert?\n\nProbably some combination.\n\nAnd these sources can disagree.\n\nThis is why I don't think the problem is solved by simply running the test-data generation in another agent.\n\nIt removes one dangerous feedback loop.\n\nThat's useful.\n\nBut then we still have to ask whether the evidence used to build that world is good enough.\n\nThe experiment I am preparing is quite simple.\n\nAgent A gets:\n\nIt builds the test world.\n\nIt does not see the implementation.\n\nAgent B gets the implementation task.\n\nThen we test B's implementation against the world built independently by A.\n\nAfter that we compare it with the normal setup where implementation, tests and fixtures are created from the same context.\n\nThe question is not whether the independent approach feels cleaner.\n\nI want to know if it actually catches more errors.\n\nI have been discussing this experiment here:\n\n[https://github.com/BeyondQuality/beyondquality/discussions/48](https://github.com/BeyondQuality/beyondquality/discussions/48)\n\nAnd the earlier field report that pushed me further into this topic is here:\n\n[https://datamimic.io/blog/deterministic-test-data-ai-coding-agents/](https://datamimic.io/blog/deterministic-test-data-ai-coding-agents/)\n\nCoding agents are getting very good at producing software.\n\nGeneration is becoming cheap.\n\nKnowing whether the generated system is actually right is not.\n\nMy current hypothesis is that independence removes one dangerous shortcut. Determinism lets us reproduce the result. The explicit model makes the assumptions visible.\n\nBut none of this proves that the world itself is right.\n\nI also don't know yet how independent it needs to be before it catches materially more implementation errors.\n\nThat's the experiment I want to run.", "url": "https://wpnews.pro/news/when-the-coding-agent-builds-the-world-it-gets-judged-in", "canonical_source": "https://dev.to/ake2l/when-the-coding-agent-builds-the-world-it-gets-judged-in-2ilm", "published_at": "2026-09-16 16:04:18+00:00", "updated_at": "2026-09-16 16:13:41.429140+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools", "mlops"], "entities": ["DATAMIMIC", "DATAMIMIC CE 4.1", "AlphaGo", "Leela Chess"], "alternates": {"html": "https://wpnews.pro/news/when-the-coding-agent-builds-the-world-it-gets-judged-in", "markdown": "https://wpnews.pro/news/when-the-coding-agent-builds-the-world-it-gets-judged-in.md", "text": "https://wpnews.pro/news/when-the-coding-agent-builds-the-world-it-gets-judged-in.txt", "jsonld": "https://wpnews.pro/news/when-the-coding-agent-builds-the-world-it-gets-judged-in.jsonld"}}