AI coding agents are great at one thing that isn't writing code: asserting. "Tests pass." "The file was updated." "I pushed the fix." And if you've run agents for real work, you know these claims are sometimes... optimistic.
I'm a QA engineer who runs Devin daily. At some point I got tired of manually checking whether the agent's claims matched reality, so I did what a QA engineer does: I built a test harness. Then an eval suite. Then a policy layer. Then a decision layer. Twenty local-first tools later, the whole stack runs on a RAM-constrained laptop with zero telemetry leaving the machine.
This is the short version of what exists, why, and what I learned.
The trigger was simple. An agent session reported "fixed, all green" — and the file on disk disagreed. No malice; agents report from their own narrative, not from ground truth. Classic QA problem: the claim and the state are different artifacts, and only one of them is evidence.
The insight that made everything else possible: agent CLIs already write structured telemetry locally — session files, tool-call state, token usage. The evidence was sitting on disk the whole time. I just had to stop trusting the story and start reading the log.
The tools compose into a sequence: understand → verify → measure → control → judge.
devin-internals-spec — understand. Before you can trust tool output you need to know the contracts: the file formats, exit codes, and behavioral rules of the runtime itself. This is the spec layer — everything downstream reads against it.
devin-qa-pack — verify. Runs a QA audit over actual agent work: file diffs present, tests run, commits exist, pushes landed, verification commands executed. Claims are checked against tool_call_state, not against the agent's narrative. 47 tests, CI on Ubuntu + Windows. This is the tool that started it all.
devin-evals — measure. Once verification works, you can ask the harder question: how good is the agent on this kind of task? Golden tasks, rubric scoring, regression tracking across sessions.
devin-bridge — control. A policy gate between intent and execution — ACP-based control with a --devin-only mode that enforces "this session does exactly what it was scoped to do."
poordjaevin — judge. The piece I'm proudest of. Takes a task description and produces a calibrated confidence score: should I trust this delegation? The calibration story is the interesting part — the judge went from ECE 0.170 to 0.071 through iterative refinement on real session data. (ECE = expected calibration error: when the judge says "80% confident," it should be right ~80% of the time. Most confidence scores don't do this. Now mine roughly does.) It's also a real MCP server — poordjaevin serve — so agents can consult it mid-flight.
Every tool follows the same contract: read local agent state, write local artifacts, expose a stable CLI, degrade gracefully when optional infrastructure is absent. No required daemons, no SaaS dashboard, no telemetry.
Concretely:
devin-history — cross-session memory: searchable SQLite over all past sessions (7+ commands of grep-able agent archaeology)devin-metrics — telemetry aggregation for quality signals over timedevin-memory + devin-search + devin-graph — memory store, retrieval, and a knowledge graph over sessions, projects and decisionsdevin-doctor — environment diagnostics across Windows/Linuxdevin-backup — snapshot + verify + restore of agent statedevin-janitor + devin-redact — cleanup and secret-redaction so state can move safelydevin-office — the fun one: a live circuit-board dashboard that renders real sessions, subagents and tool calls from the local store. Purely visual, read-only, zero telemetry — and it makes for a great demo GIF.
1. The agent's own telemetry is an untapped QA datasource. Session files and tool-call state are structured evidence most people ignore. Reading them turns "trust the agent" into "verify the agent" — and verification is what makes delegation safe at scale.
2. Calibration > confidence. A judge that's confidently wrong is worse than no judge. Iterating on ECE (0.170 → 0.071) took real labeled outcomes, not prompt tweaks. If you build any kind of AI decision layer, measure calibration explicitly.
3. Devin-only mode matters more than integrations. Every tool works standalone with just the agent's CLI present. Obsidian, Slack, MCP — all optional. The ecosystem must survive a locked-down corporate box with nothing but Devin installed, because that's where a lot of real work happens.
Everything is MIT-licensed, cross-platform, and documented in English + PT-BR:
devin-qa-pack (the verifier) or poordjaevin (the calibrated judge — pip install poordjaevin / uv tool install poordjaevin)
Honest question for the comments: if you run AI agents on real work — Copilot, Devin, Cursor, Claude Code — how do you verify their claims today? Manual spot-checks? CI gates? Nothing? I suspect "nothing" is the most common answer, and I built this stack partly because it scared me that it was mine.