{"slug": "we-started-tracking-which-ai-model-wrote-every-line-and-you-should-too", "title": "We started tracking which AI model wrote every line, and you should too", "summary": "Developer Atlas Verifed built a system that tracks which AI model wrote every line of code in their app, recording the coding model, planner model, changed hunks, and plan origin, with enforcement via Cursor hooks, pre-commit checks, and CI. The system, now publicly available and submitted to Cursor Marketplace, aims to provide hard data on model performance, such as identifying models that cause rework or fail to complete tasks, as seen with Opus 4.7's partial implementations.", "body_md": "We've been building the same app for over a year with AI agents @atlas_v_erified.\n\nNot one model. A bunch of them. Testing models, QA models, building models, design models. Frontend, backend, schema, architecture — all of it. Cursor keeps shipping new ones (theirs and other labs) and we keep trying them.\n\nAnd for most of that year period we had no real way to answer a pretty basic question: which model actually wrote this code or which model created this plan?\n\nWe had anecdotes.\n\nTake for example Opus 4.7. Promising, but we learned after heavy use that it would say the work was done and only complete part of it. Partial implementations. We figured that out fast and moved to a different model. But that's still just a feeling. Anecdote, not hard data.\n\nSo I built model tracking into the repo.\n\nEvery time an agent touches a file, it has to leave a byline. Machine-readable. Same session, same sprint.\n\nHere is what we record:\n\nthe coding model that made the edit\n\nthe planner model, if planning was a different model\n\nthe exact hunks that changed\n\nthe plan those changes came from\n\nand if it's a bug fix, which earlier change it's fixing (when we actually know)\n\nOn a normal TypeScript file it looks like this at the top:\n\nHere are a few rules that matter when implemented:\n\nNewest byline only in the file. Next edit replaces it. Git keeps the history.\n\nHunks don't include the provenance block itself, or the line numbers get useless the second the header moves.\n\nIf Cursor doesn't expose the real model id / max mode / context window, we write not_exposed. Agents are not allowed to invent it.\n\nPlanner and coder are separate fields on purpose. A lot of our work is planned by one model and implemented by another.\n\nJSON, lockfiles, binaries can't take comments — those go as one line into .cursor/agent-provenance.jsonl. That sidecar is not a loophole for \"this .ts file was big.\"\n\nDocs weren't enough\n\nNow, here is where it gets interesting. First attempt was the schema and a hard rule in our AGENTS.md (yes, I know that prompts are NOT rules or security or requirements, but it's a fast test).\n\nThe agents ignored it when it got inconvenient. Left hunks: pending. Skipped headers at the end of large plans and sprints.\n\nSo I made it into something they can't skip, and it's working great.\n\nHow we force it\n\nFour layers:\n\nAlways-on Cursor rule — short reminder every turn\n\nCursor hooks — track every edit, nag after writes, and on stop force one cleanup turn if anything is missing\n\nPre-commit — checker blocks the commit\n\nCI — same checker on the PR diff\n\nThe stop hook is the one that actually changed behavior.\n\nAgent thinks it's done → we re-check every file from that session → if provenance is missing, it gets forced back in:\n\nPROVENANCE GATE: You edited files without valid atlas-agent-provenance/v1 attribution. Do not start new feature work. Fix provenance on every file below, then stop.\n\nOne loop max. Pre-commit and CI catch the rest.\n\nRules (or .md) are optional to a model. Hooks and CI are not.\n\nWhy bother\n\nOnce this is all the way through our codebase, we can actually debug where bugs or architectural changes were planned and implement by model.\n\nwhich coding models create the most rework\n\nwhich planner → coder combos ship clean on the first pass\n\nwhen we fix a bug, can we walk it back to the change_id and the model that introduced it\n\nis the \"said it was done, only did half\" failure mode concentrated in specific models\n\nPublicly available\n\nI really wish I had this data back to the start of this project. It would be incredibly valuable to us. Imagine running a plan to review ALL code generated by a specific model (in loops), or tracing back every issue and running data on which models produce the least issues (even segmenting by back-end, front-end, db schema, etc).\n\nSo, I'm going to drop the public repo here so anyone can utilize this, and I have submitted this to Cursor Marketplace for anyone to one-click add (if they approve, 🤞 @cursor_ai).\n\nDrop a comment below with what you are building using Cursor (@cursor_ai). Would love to see what is being built.\n\nPublic Repo", "url": "https://wpnews.pro/news/we-started-tracking-which-ai-model-wrote-every-line-and-you-should-too", "canonical_source": "https://twitter.com/lifeofjer/status/2087608101293916466", "published_at": "2026-08-12 19:05:17+00:00", "updated_at": "2026-08-12 19:12:12.683531+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "ai-tools", "mlops"], "entities": ["Atlas Verifed", "Cursor", "Opus 4.7", "Cursor Marketplace"], "alternates": {"html": "https://wpnews.pro/news/we-started-tracking-which-ai-model-wrote-every-line-and-you-should-too", "markdown": "https://wpnews.pro/news/we-started-tracking-which-ai-model-wrote-every-line-and-you-should-too.md", "text": "https://wpnews.pro/news/we-started-tracking-which-ai-model-wrote-every-line-and-you-should-too.txt", "jsonld": "https://wpnews.pro/news/we-started-tracking-which-ai-model-wrote-every-line-and-you-should-too.jsonld"}}