Human-in-the-loop knowledge base for AI agents

A developer describes building a human-in-the-loop knowledge base that separates a curated, permanent source library from untrusted agent notes that expire after 30 days and can only be appended, never edited. Agents query the library through a CLI or API and receive verbatim passages with openable citations and a support score, while a human triage queue with Keep, Add to draft, or Remove verdicts controls what enters the library. The developer reports the library now holds 87 trusted documents and that per-job Profiles with individually revocable keys enforce least privilege.

I stopped letting my agents write to my knowledge base. They can read it; they can leave notes next to it; but the library itself is mine. That split exists because every agent I run loves to create, and none of them want to update. For months I gave my agents a shared Notion workspace. The idea was simple: they work, they write down what they learned, the next agent reads it. What I got was a one-way street. Pages appeared. Nothing got revised, nothing cited a source, and every handoff depended on whatever the previous agent had bothered to write down, in its own words. The moment it got on my nerves: I was working on a client's new feature pages when my agents pulled outdated information from Notion and built a page that wasn't in line with my specs. The specs were right. The page wasn't. The agents had read the wrong thing and written with full confidence. Who writes? Everybody. Who updates? Nobody. Who checks what's current? Me, by hand, which was the job I had tried to automate. So I drew a line. On one side: sources. Documents I chose, organized into projects and folders, permanent until I say otherwise. On the other side: notes. My agents can leave these while they work — the problem, what they tried, what worked, what to check next time. Notes are untrusted by design. They live for 30 days and then they're gone, even the ones I kept. They can't be edited in place append a new one, remove the old one , they never cross between Profiles, and nothing in them ever becomes a source on its own. A note does not become a fact by being saved. The triage is mine. I keep a review queue and each note gets one of three verdicts: Keep, Add to draft, or Remove. Useful ones get collected into a draft, and before that draft joins the library I check its claims and edit it myself. Material enters the source library through one door, and I'm standing in it. Does it work? My library holds 87 docs right now, and I trust all of them. When an agent needs knowledge, it asks through the CLI or API. What comes back is short: verbatim passages, each carrying a citation I can open, plus a single support score. Takibi never writes answer prose. If nothing matches, it returns no passages, and my agents are instructed to say the evidence isn't there instead of filling the gap. When sources disagree, it flags the conflict and returns both sides. I don't want my retrieval picking a winner behind my back. I'd rather see the disagreement and decide. That citation is the whole point. I can follow it and read what the source actually said, including the qualifications the agent left out. Each job gets its own Profile, which is just a defined view over the same workspace. The writing agent reads approved product folders. Nothing else. The transcript uploader files Capkiku captures into each client's Uploads folder and remembers what it already sent — that workflow can add transcripts but can't edit or delete the library, and Uploads permission is off by default everywhere else. Keys rotate or revoke individually. Pause a Profile and all its keys stop. Least privilege isn't a posture here, it's the only reason I sleep: a confused agent can only ever touch what its job needed. Honest version: I'm the bottleneck now, on purpose. The triage queue is real work, and when the library is thin on a topic the move is still to broaden the question, fall back to search, or read the full document. Sometimes I just haven't added the source yet. What I'd change? The queue could rank itself better. Right now every note waits equally, and the obviously stale ones should sink on their own. That's next. You don't need my stack for the core idea. Two folders, a TTL habit, and a weekly triage pass get you most of it: notes older than 30 days get deleted. no archive. if it mattered, it would be a source by now. find ./agent-notes -name ' .md' -mtime +30 -delete The rules that matter: agents append, never edit. Nothing graduates without a human reading it. And the day a note feels too important to expire is the day you promote it, not the day you extend the TTL. My library is small and I trust all of it. My scratch is large and I trust none of it. That sentence took me one dead product to learn. I built this into Takibi Base https://takibibase.com — a shared knowledge base where agents retrieve cited passages from sources I control. If your agents write more than they update, the pattern above works with or without it.