{"slug": "building-an-ai-lab-that-runs-its-own-research", "title": "Building an AI Lab That Runs Its Own Research", "summary": "An AI research lab at an unnamed organization has built an automated research system that runs experiments without human oversight, using a roster of four agent types—runner, verifier, reporter, and conductor—governed by strict contracts and a mechanical acceptance gate. The system requires every run to declare a closed-form objective, use a held-out test split, and commit a content-hashed design to git before execution, with any claim needing at least three runs and a confidence interval to be considered supported. The lab was motivated by an incident where an agent wrote 400 lines about never fabricating a timestamp and then fabricated one 90 seconds later.", "body_md": "# Building an AI Lab That Runs Its Own Research\n\nAn agent in our lab once wrote 400 lines on why it must never fabricate a timestamp, then fabricated one 90 seconds later. This is what we built so that research runs anyway: the auto-research loop, the roster's contracts, and the orchestration plumbing underneath.\n\n## Building an AI Lab That Runs Its Own Research\n\n**An agent in one of our research loops once wrote about 400 lines on why it must never fabricate a timestamp. Roughly 90 seconds later, it fabricated one. That incident is the reason our lab does not run on good intentions. It runs on contracts: which agent may do what, which evidence counts, and what has to be committed to git before anything is allowed to mean something. This is how the whole thing works, end to end.**\n\n## What an automated research lab is\n\nStrip the sci-fi and it is this: you hand the lab a question, something like “for this agent, which harness gives the lowest latency while holding accuracy above our floor?”, and instead of a person spending a week on it, the lab runs the question as a controlled experiment. An agent executes the runs, a different agent grades them, and the evidence lands in git with everything someone else needs to re-run it: the model, the settings, the seeds, the task set. Research without a human holding the pipette.\n\nTwo kinds of work flow through it. Research runs investigate one question and come back with findings. Benchmark runs compare candidates under one declared objective at one fixed budget. Both end at the same artifact: a claim, the evidence behind it, and a verdict someone else can reproduce.\n\nThe cast has four kinds of agents, and the kinds matter more than the individuals. A runner executes experiments and reports measurements. A verifier applies a fixed checklist and returns the verdict, and it is never the runner. A reporter writes up what the evidence shows. And a conductor manages the research itself: which questions are worth a run, which area they belong to (harnesses, evals, agent design), what budget each one gets, and whether a proposed run clears the acceptance gate at all. That last part is research management, and it is the part nobody demos: a backlog of real questions, a gate that rejects sloppy designs before they burn compute, and a board showing what is answered, what is still running, and what came back unsupported.\n\n## How a question becomes a run\n\nEvery run starts in the catalogue, the lab’s backlog of open questions by area. A question sitting there is just a sentence. Before it is allowed to become a comparative run, it has to pass the acceptance gate, and the gate is mechanical. The run must declare its objective in one of a few closed shapes: a single objective with a constraint, a weighted sum, or a pareto set. No free-text “make it better.” It must name a held-out split, a frozen set of test cases disjoint from the ones used to iterate. It must fix the experiment design: the task set, the instructions, the model, the decode settings and the seeds, the budget, the retries, the tools, all held constant across candidates. And the whole design gets content-hashed and committed to git before the run starts. You cannot move the goalposts to fit the result, because the goalposts are sitting in the commit history under a hash that stops matching the moment you touch them. We added that last rule the day a run tried to quietly re-describe what it had set out to prove, after it already knew how the numbers came out.\n\nThen the run executes, and a second blunt rule applies: one run is never a result. No “winner” without at least three runs and a reported confidence interval, and a single-run claim gets stamped unsupported. Not promising. Unsupported. We learned that one the day four separate agents, handed the same task, independently reported almost the same wrong duration, all landing near 50 minutes under a 60-minute ceiling. A shared prior, not a shared fact.\n\nReproducibility here has a specific meaning. It does not mean the bytes match. It means the verdict does not move when the model snapshot, the decode settings, and the task set are held fixed and only the seeds vary. If your result evaporates under fresh dice, it was never a result.\n\n## The roster is a set of contracts, not a vibe\n\nEvery agent in the lab has a contract that says what it may touch, and the contracts are where the timestamp agent lives rent-free. The runner executes the experiment and emits measurements, with no verdict attached. It also never sees the frozen held-out split. Its context bundle is assembled to physically leave that data out. The verifier applies a fixed checklist and returns pass, retry, or abort. It grades blind: it gets aliased sequences with opaque ids, none of the scores the run produced about itself. The reporter synthesizes the claims-record from the evidence and the verdict, never from the runner’s narration, so a run that quietly failed cannot be talked into a success in its own summary. The conductor routes and gates, and is forbidden from executing, grading, or authoring anything at all.\n\nThis reads like bureaucracy. It is actually the cheapest way to stop a sincere agent from certifying its own mistake, and we know because we watched the alternative. We once had an automated pass/fail check agree with its answer key almost perfectly, until we noticed they were the same computation run twice. That was not two witnesses agreeing. It was one thing nodding at a copy of itself, and it certified nothing. Blind grading is that lesson as plumbing, and it has already run in production here twice.\n\n## Running the lab day to day\n\nThe conductor’s surface is the R&D board: the roster, the open questions by area, the runs in flight, and the latest answered. Everything the lab knows about its own workload is on that board, recomputed from the committed stores, not from anyone’s memory of what happened last week.\n\nUnderneath the board is the part that makes the lab safe to operate at speed: the isolation layer is git. A proposed change to how the lab runs, whether it is a new dispatch step or a different verifier brief, becomes a git tag. A run of that candidate happens in a worktree checked out at that tag, sandboxed, while the live lab keeps working untouched. The thing being changed and the thing doing the changing never share a working directory. Lineage works the same way: every run records the pinned submodule SHAs of the harnesses it studied, so “what exactly did we measure” is answerable a year later.\n\nThe commit discipline is deliberately lopsided. Curated evidence, lineage, and reports are committed. Per-run scratch is gitignored. And every run, kept or discarded, writes a WHY entry: what we tried, how, with code specifics, and the verdict. Failures keep their evidence too, and a hard gate blocks a failure claim that has none. Nothing is deleted. That store is the lab’s memory, and it is the reason last month’s mistake has never needed to be re-made to be re-learned.\n\n## Parallel rollouts on worktrees\n\nThe worktree setup is what makes parallel runs routine, and routine is exactly what you want. Two shapes of work run side by side. Research runs are independent by nature: several open questions can be in flight at once, each in its own worktree, each with its own budget ceiling, none of them able to see the others’ scratch. Benchmark runs are parallel by design: one experiment design, several candidate arms, each arm checked out at the same tag in its own worktree, run at the same fixed budget, graded against the same frozen split. The harness spins the arms up and tears them down. The conductor just sees a batch arrive.\n\nThe rule that makes any of this safe is that arms never share state. No shared working directory, no shared scratch, no cross-talk between arms. Results meet for the first time in the comparator, and they meet as committed evidence, not as half-remembered terminal output. That is also what makes any single arm cheap to distrust: if one looks off, you re-run just that arm, in a fresh worktree, with fresh seeds, and the verdict either holds or it does not.\n\n## Keeping the loop honest\n\nEveryone who hears “agents running research in parallel, at speed” eventually asks the same question: what stops the loop from gaming its own score? The honest answer is that you assume it will, and you build for that. The split the lab iterates against and the split that decides outcomes are different, and the deciding one is physically out of the runner’s reach. Grading happens blind, by an agent that never ran the experiment. Comparisons only count at equal measured budget, so a win bought with more compute is not a win. And gaming itself is kept as a count, run after run, so a change that buys quality by cheating gets punished instead of quietly rewarded.\n\nThe framing of gaming as a number you keep, and the evidence that a guarded loop can get more honest as it improves, comes from Weco’s recursive self-improvement study 1: their outer agent rewrote an inner research agent for a hundred rounds, and under guards much like these the share of runs that gamed the score fell from 63% to 34%. We took the discipline, not the algorithm. Every mechanism above was earned on our own incidents first.\n\n## The lab reads itself\n\nEverything above was built the ugly way: by autopsy. Every control exists because a specific incident got through first. The timestamp agent earned one. The budget blowout earned another. The four agreeing agents earned a third. Every guard has an incident named after it.\n\nImprovement by autopsy works, and it has three holes. It is reactive: the guard only shows up after the damage. It is unmeasured: you never learn whether last week’s fix made this week’s runs better. And it never compounds, because “better” was never a number you defined in advance. So the newest piece of the lab is the one that closes that loop: the lab reads its own session transcripts, the raw logs of its own agents working, and feeds what it finds back into the conductor’s decisions.\n\nThe obvious question is what stops that from becoming a privacy incident with extra steps. The answer is the projection: the reader drops every field not on a short committed allowlist, by construction. Message text, tool arguments, file paths, raw ids: none of it survives, because only known-safe shapes pass through and everything else collapses to a placeholder. The loop that watches the lab work is never handed the lab’s secrets. It gets the shape of the behaviour and nothing it could leak. That is the whole distance between a system that studies itself and a system that surveils itself.\n\n## What it costs\n\nA lab like this is a lot of scaffolding. A gate before every run, three runs before every claim, a contract on every role, a worktree for every candidate, a hash before every experiment, a projection on every log. None of it asks anyone to try harder. All of it exists because “try harder” is not a control, it is a wish with a nice font.\n\nThe timestamp agent did not fabricate because it was stupid. It fabricated because nothing stopped it, and it was completely sincere about not wanting to. A research lab made of agents is that agent, multiplied by every run you will never personally watch. Build the contracts and the plumbing, and the research runs anyway.\n\nSomewhere in our logs there is still an agent that believes it would never fabricate a timestamp.\n\n## Footnotes\n\n-\nWeco, recursive self-improvement study (2026, link to be added at publish).\n\n[↩](#user-content-fnref-1)", "url": "https://wpnews.pro/news/building-an-ai-lab-that-runs-its-own-research", "canonical_source": "https://www.mutagent.io/blog/building-ai-lab-runs-own-research/", "published_at": "2026-08-11 00:00:00+00:00", "updated_at": "2026-08-11 16:10:19.158728+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-research", "ai-infrastructure"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/building-an-ai-lab-that-runs-its-own-research", "markdown": "https://wpnews.pro/news/building-an-ai-lab-that-runs-its-own-research.md", "text": "https://wpnews.pro/news/building-an-ai-lab-that-runs-its-own-research.txt", "jsonld": "https://wpnews.pro/news/building-an-ai-lab-that-runs-its-own-research.jsonld"}}