In the first week of September three arXiv papers landed on the same nerve:
Read together, they say one thing: agents are starting to modify the thing that runs them. Their own tools, their own guardrails, the code they operate on.
Every one of those papers leaves the same question open. Who approves the patch?
We spent a decade learning not to curl | sh from strangers. Then we learned to pin dependencies, sign builds, and review every PR. A self-improving agent that writes a tool and runs it in the same breath undoes all of that in one step — and it does it inside your perimeter, with your credentials.
The tempting answer is "don't let agents write tools." But that throws away the capability that makes agents useful: the thing your agent needs is almost always discovered mid-task. A PDF encryptor that doesn't exist. A library it can't import. A file format nobody wired up. Freeze the toolkit and the agent either fails or — worse — fakes it.
So the real question isn't whether agents should grow. It's how the growth gets governed.
I've spent four months on an open-source runtime called Systemu that takes one specific position on this. Last week I watched it work end-to-end on a boring, real task: "encrypt these PDFs with a password." No such tool existed.
1. The agent asks — it doesn't fail, it doesn't fake.
The running agent issues a REQUEST_HARNESS — "provision what I lack." One card lands in the Inbox:
HIGH — Forge tool: pdf_encrypt
Apply password protection to a PDF file and save the encrypted version to a new path.
What approve does: generates + reviews the tool code, then enables it.
Safe default: Skip
The safe default is Skip. Do nothing, and nothing happens.
2. You review the spec, then the code — before anything exists.
Approving the card doesn't run anything. It generates the tool and shows you the source with a plain warning: this does not prove the code safe — read it before approving. The LLM reviewer can flag problems, but it can only ever push the decision toward deny or escalate. It cannot open a hole.
3. Dependencies gate separately.
The generated tool wanted pypdf. That's its own approval. No silent pip install, ever.
4. The first run is gated too.
Even after the tool is deployed, its first execution lands as an action card marked dangerous-until-proven, with the actual arguments shown. Safe default: Deny. Re-forging the tool re-gates it.
5. Deployed, with provenance.
Minutes later the Build page shows pdf_encrypt — DEPLOYED · AGENT-BUILT · dry-run PASSED. The toolkit grew by one. Every step is in the decision ledger, attributed to the run that made it, revocable in one click.
That's the whole loop: gap → request → govern → grow.
A self-requested capability is more dangerous than a pre-provisioned one. The agent chose it. So it's gated more strictly, not less — the opposite of how most frameworks treat "the agent decided it needs X."
Judgment can only downgrade toward safe. When an ambiguous request needs an LLM judge, the judge may deny or escalate, never grant beyond policy. A judge fault fails to escalation, not to grant. Your safety doesn't depend on the model being right.
That was my fear, so I measured it instead of asserting it. A Capability-Gap Benchmark: tasks that are impossible without acquiring a missing capability, across six capability families (tool, skill, file/secret access, compute budget, sub-agents, MCP servers), across 5 models from 5 vendors, graded by an external oracle rather than the system's own verifier.
Governance isn't the brake on self-improving agents. It's the steering. Preprint on Zenodo.
pip install "systemu[dashboard]"
systemu init && systemu start
MIT licensed. The 22-second walkthrough is at the top of the README: https://github.com/rameswaran-mohan/project-systemu
If your AI patched itself tonight — would you know?