{"slug": "track-the-edit-rate-how-we-decide-how-much-to-trust-each-ai-agent-task", "title": "Track the Edit Rate: How We Decide How Much to Trust Each AI Agent Task", "summary": "Justin Pennington, founder of seven-person Florida software firm Infraxio, describes routing agent-completed work into the same review queue as human work in IFX Hub and IFX Bid, tracking an edit rate per task type to calibrate trust. He assigns each task a light, heavy, or human-only review level that shifts as clean or flawed output accumulates, arguing that the agent plus the review loop — not the agent alone — is the system, and that review overhead must pay for itself in saved hours.", "body_md": "The question I get most about running AI agents in a small company is some version of \"can you trust it?\" I think that's the wrong question. You don't trust an agent. You trust an agent *on a specific task*, for now, based on what the review data says.\n\nI'm Justin Pennington, founder of Infraxio, based in Ponte Vedra in St. Johns County, just south of Jacksonville, Florida. We're a seven-person company, down from about thirty before AI, and agents do real work inside our products every day. This is a builder-oriented rewrite of an essay on my site, [AI Agents Do Not Manage Themselves](https://penningtonj.com/blog/ai-agents-do-not-manage-themselves). More about me and what we're building: [infraxio.com/justin-pennington](https://infraxio.com/justin-pennington).\n\nHere's the short version: the agent isn't the system. The agent plus the review loop is the system. If you skip the loop, you end up with one more tool running in the background producing output nobody trusts.\n\nThe first design choice that mattered for us: agent-completed work lands in the same review column as human-completed work. In IFX Hub, when an agent drafts a client summary, assembles data for an invoice, or takes a first pass at a bid section in IFX Bid, it shows up on the board right next to everything else.\n\nThat sounds minor. It isn't. If reviewing agent output means opening a separate dashboard, reviews quietly stop happening. If it's already in the queue the team checks every morning, it gets read.\n\nIt also gives you an audit trail for free. Every agent task carries the same record a person's task does:\n\nThat last field is the one worth building everything around.\n\nEarly on I treated trust as a yes-or-no thing about the agent. In practice it splits by task type.\n\nThe same agent, on the same model, can be very reliable at turning structured data into an internal status update and much less reliable at writing the narrative section of a bid response. The first is close to deterministic: inputs map cleanly to outputs. The second takes judgment about emphasis and tone.\n\nSo instead of one trust setting, we keep a review level per task type:\n\n| Review level | What the reviewer does | Typical fit | \n|---|---|---|\n| Light | Quick scan that the output matches the expected pattern | Structured-data summaries, internal updates | \n| Heavy | Reads closely and rewrites parts | Narrative or client-facing writing | \n| None (human only) | The agent doesn't get the task | Anything where one bad output costs more than the time saved | \n\nTasks move between levels. One that starts on heavy review can drop to light after a run of clean output. One on light review goes back to heavy after a miss. There's no final state. It's ongoing calibration.\n\nBecause every review records what changed, you can ask operations questions instead of guessing:\n\nThe math is plain. An agent that saves four hours a week on drafting and costs two hours of review is a win. One that saves an hour and costs three hours of correction is a loss, and it needs to improve or lose the task.\n\nAt thirty people, a bad process could hide because someone quietly absorbed the extra work. At seven, if review overhead isn't paying for itself, someone notices by Wednesday. That visibility is an advantage if you actually look at the data.\n\nThe obviously wrong outputs aren't the problem; anyone catches those. The dangerous ones look right:\n\nThis is why it's tempting, and wrong, to stop reviewing a task after twenty clean runs. The twenty-first is where a scope shift or an off-system pricing conversation slips through.\n\nThese are the same mistakes a new hire would make. The difference is in the fix. A person learns from the correction. An agent makes the same class of error next time unless you change its input, its prompt, or its scope. With a person you coach. With an agent you adjust the system.\n\nAn agent working from dirty data doesn't stop and say \"this looks off.\" It processes what it sees with the same confidence either way.\n\nThat's why consolidation matters more once agents are doing work. When a deal, its delivery tasks, and its billing live in one place, the agent sees the whole picture. When they live in three systems, either the agent sees one slice or you build integration logic to stitch the context together, and that integration layer becomes one more place for data to go stale.\n\nAgents don't manage themselves. Neither did the people they replaced. The management just looks different now.\n\nI'm curious how other teams handle this. Do you track something like an edit rate on agent output, or is review still mostly gut feel?", "url": "https://wpnews.pro/news/track-the-edit-rate-how-we-decide-how-much-to-trust-each-ai-agent-task", "canonical_source": "https://dev.to/justinpennington/track-the-edit-rate-how-we-decide-how-much-to-trust-each-ai-agent-task-4g46", "published_at": "2026-11-04 16:41:39+00:00", "updated_at": "2026-11-04 16:54:06.304921+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "artificial-intelligence"], "entities": ["Infraxio", "Justin Pennington", "IFX Hub", "IFX Bid"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/track-the-edit-rate-how-we-decide-how-much-to-trust-each-ai-agent-task", "markdown": "https://wpnews.pro/news/track-the-edit-rate-how-we-decide-how-much-to-trust-each-ai-agent-task.md", "text": "https://wpnews.pro/news/track-the-edit-rate-how-we-decide-how-much-to-trust-each-ai-agent-task.txt", "jsonld": "https://wpnews.pro/news/track-the-edit-rate-how-we-decide-how-much-to-trust-each-ai-agent-task.jsonld"}}