{"slug": "measure-agent-review-time-rework-and-cost", "title": "Measure Agent Review Time, Rework and Cost", "summary": "A proposed measurement protocol published on cutler.sg sets out how to count the human cost of agent work by tracking briefing, supervision, review, rework and verification time separately from agent spend and turnaround, with every agent attempt logged under its business task and given an accepted, rejected or not-reviewed disposition. The author, writing as a follow-up to a prior post titled \"26 Minutes vs 33 Seconds,\" states the method is a proposal rather than a completed experiment, and warns that a missed timer entry must be marked as missing data rather than recorded as a zero-minute review. The protocol also requires reporting accepted agent candidates divided by evaluated agent candidates separately from task completion, and labeling manual takeovers explicitly.", "body_md": "# How to Measure the Human Cost of Agent Work\n\n## $ grep -n \"^##\" 2026-10-measure-agent-review-time.md\n\nBefore budgeting the time an agent saves, measure the human work required to get an acceptable result.\n\nIn [26 Minutes vs 33 Seconds](https://cutler.sg/blog/2026-09-agentic-ai-vs-conversational-search-26-minutes), I argued that assumed oversight time is a poor basis for capacity planning. This is the collection protocol I would use to replace that assumption. It is a proposed method, not a report of an experiment I have completed.\n\nThe immediate question is modest: how much human effort and agent spend does this workflow consume per completed task, at an agreed quality bar? Proving it saves time against another method requires an additional comparison.\n\n## [Define the task before starting the timer](#define-the-task-before-starting-the-timer)\n\nChoose one task class: a bounded bug fix, a source-backed research note, a routine data reconciliation. Write down eligibility rules and register every selected task before work starts. Do not collect only the runs somebody considers worth sharing.\n\nFor each task, specify the required result, who can accept it, and what evidence they need. Fix the collection cutoff and a later defect-review date. Acceptance today and a discovered defect tomorrow must be able to coexist in the record.\n\nUse the business task as the unit, keeping every agent attempt beneath it. A rejected first patch followed by two revisions is one task with three attempts. Starting a new chat does not reset its accumulated cost.\n\nGive each candidate attempt its own disposition: accepted, rejected or not reviewed, with a reason and evidence. Report accepted agent candidates divided by evaluated agent candidates separately from task completion, and disclose unreviewed candidates. That yield describes the submitted artifacts, including repeated attempts on the same task.\n\nIn this protocol, a task can also finish after a human takes over. Count that completed task, but label its completion mode explicitly. You are measuring the whole workflow, including rescue work, rather than claiming every accepted result was an accepted agent artifact. Report manual takeovers separately.\n\n## [Keep three clocks separate](#keep-three-clocks-separate)\n\nRecord active human time in categories that explain where capacity goes:\n\n| Category | Include | \n|---|---|\n| Briefing | Defining the request, assembling context and explaining constraints | \n| Supervision | Reading progress, answering questions and redirecting attempts | \n| Review | Inspecting the candidate against its acceptance criteria | \n| Rework | Correcting the result or replacing it after a failed attempt | \n| Verification | Running and inspecting the checks needed for acceptance | \n\nPause the human timer while the agent runs unattended or while you do unrelated work. Task turnaround ends at its terminal outcome: accepted, rejected or abandoned. Record that time separately from each candidate’s disposition. A rejected patch does not stop the task clock if another attempt or manual takeover follows. For pending tasks, report age at the cutoff. Later defect repairs do not move the original endpoint. Waiting in a review queue affects turnaround even when nobody is actively working.\n\nCount all reviewers’ person-minutes. Two people reviewing together consume two people’s time. Conversely, one person cannot log the same five minutes against three agents and call it fifteen minutes of active work. Split or label shared intervals using a consistent rule.\n\nRecord agent execution duration if it helps diagnose latency, but do not add parallel runs together and call the result elapsed turnaround. Human effort, machine runtime and delivery time answer different questions.\n\nA missed timer entry is missing data. Mark it as such; an unrecorded review is not a zero-minute review. If you reconstruct an interval afterwards, label it estimated.\n\nKeep shared human setup and maintenance in a separate cohort overhead record. Report that effort alongside task execution. If allocating it across tasks, declare the rule and count it once; distinguish task-only from overhead-inclusive costs before deciding whether delegation is worthwhile.\n\n## [Attach spend to every attempt](#attach-spend-to-every-attempt)\n\nInclude generation, retries, tool charges and automated verification attributable to the task. An evaluator agent also costs money. Keep failed attempts attached to the same task ID so changing models or sessions does not make them disappear.\n\nPrefer actual provider charges where available. If you estimate from usage, retain the pricing date and relevant billing assumptions, including cache treatment. Keep currency explicit and reconcile the estimate when billing data arrives.\n\nA flat subscription does not produce a precise marginal charge per task. You can report it separately or allocate it by a declared rule, but label the allocation. Apply the same discipline to shared infrastructure. If you convert human minutes into money, state the hourly-rate assumption separately from observed effort.\n\n## [Let failures affect the denominator](#let-failures-affect-the-denominator)\n\nFor a closed cohort with complete effort and charge records, calculate:\n\n```\nhuman minutes per accepted task = all cohort human minutes / accepted tasks\nagent spend per accepted task   = all cohort agent spend / accepted tasks\nacceptance rate                = accepted tasks / started tasks\n```\n\n“All” includes review and rework on rejected or abandoned tasks. “Accepted” means the agreed result was achieved; split the count between agent-assisted completion and manual takeover. Otherwise a successful rescue can look like successful delegation.\n\nHere is **synthetic arithmetic, not collected data**: three started tasks consume 45 human minutes and $6 in agent/tool charges. Two meet the acceptance criteria; one is abandoned. The workflow consumed 22.5 human minutes and $3 per accepted task, with two of three tasks accepted. Dividing only the successful tasks’ costs by two would hide the abandoned work.\n\nWith zero accepted tasks, the per-accepted-task ratios are undefined. Report the spend, effort and outcomes directly.\n\nIf tasks remain pending at the cutoff, show their accrued effort and spend, their count and how long they have been open. Any current unit-cost ratio is provisional. Do not present the easy tasks that finished first as the final cohort result.\n\nMissing measurements also prevent a complete unit-cost claim. Report observed subtotals, estimated amounts and which tasks have missing intervals or charges. A ratio using only observed effort is a lower bound, not the complete cost. Keep those tasks in the outcome counts.\n\nFreeze totals at the collection cutoff. Keep later work linked to the task, identifying defect repairs through the follow-up window. Report additional effort and charges separately; a cumulative follow-up total adds that increment once. Preserve the original acceptance decision alongside any later defect.\n\n## [A baseline answers a different question](#a-baseline-answers-a-different-question)\n\nThis record can expose expensive review or frequent rescue without proving an agent caused either. To claim savings, compare comparable tasks under the same acceptance bar, with a reviewer qualified to apply it.\n\nWhere practical, assign eligible tasks to the agent-assisted and comparison workflows before execution, using randomisation within task classes. [METR’s early-2025 developer study](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) assigned real issues to allow or disallow AI; its methodological relevance here is that assignment preceded the outcome. Its findings belong to that sampled setting, not every later tool or team.\n\nAvoid doing a task with an agent and then timing yourself doing it manually. You already know the solution. Keep task mix, developer familiarity and quality requirements visible; show individual records and distributions when a small sample makes an average misleading.\n\nThe [SPACE framework](https://www.microsoft.com/en-us/research/publication/the-space-of-developer-productivity-theres-more-to-it-than-you-think/) treats developer productivity as multidimensional. Time and cost should sit beside acceptance and defects, not replace them. A shorter review can mean an easier artifact or a weaker review; the timer cannot distinguish those by itself.\n\n## [A record you can use](#a-record-you-can-use)\n\nThis blank template uses one task-level ledger for human time. An optional attempt ID links review to a candidate; briefing and manual rescue need no invented candidate. Each person-minute appears once. Derive category totals and cutoff snapshots from the segments, keeping measured and estimated time separate. Nulls mean unmeasured or unassigned, not zero.\n\n```\ntask_id: null\ntask_class: null\nacceptance_criteria: null\nreviewer: null\nworkflow_version: null\nstarted_at: null\ndisposed_at: null # terminal task outcome; null while pending\ncollection_cutoff: null\ndefect_followup_until: null\nattempts:\n  - attempt_id: null\n    candidate_revision: null\n    model_tool_versions: null\n    disposition: null # accepted, rejected, or not_reviewed\n    reason_and_evidence: null\nhuman_segments:\n  - segment_id: null\n    person: null\n    attempt_id: null # optional\n    category: null # briefing, supervision, review, rework, verification\n    phase: null # initial_work or defect_followup\n    started_at: null\n    ended_at: null\n    basis: null # measured or estimated\ncharges: [] # timestamp, optional attempt_id, phase, amount, currency, basis\nmissing_records: [] # affected work/charge and what remains unknown\ntotals_at_cutoff: null # derived category totals by basis, plus spend\nadditional_after_cutoff: null # same shape; excludes cutoff totals\noutcome: pending # accepted, rejected, abandoned, pending\ncompletion_mode: null # agent_assisted or manual_takeover\nacceptance_evidence: null\nfollowup_defects: [] # discovery time, evidence, linked repair segment IDs\n```\n\nBefore collecting, decide what would make this task class worth delegating: an acceptable human-time budget, quality bar and tolerance for manual takeover. Then use the records to choose whether to continue, change the workflow or stop delegating that class. Knowing where your reviewers’ time went is the first result you need.\n\n$ subscribe --newsletter\n\n## Practical AI engineering, in your inbox\n\nField notes for technical leaders building agents, evaluation systems, governance, and production infrastructure.\n\n## Related\n\n[October 30, 2026 | 5 min read](https://cutler.sg/blog/2026-10-agent-timeout-safe-retry)\n\n### The Agent Timed Out. Did the Action Happen?\n\nA decision table for retrying agent actions when the response is lost, with stable intent, provider-specific idempotency and an explicit unknown outcome.\n\n[October 26, 2026 | 20 min read](https://cutler.sg/blog/2026-10-ai-assisted-pcb-design-kicad-nfc-business-card)\n\n### Back to PCB Design With AI Doing the Legwork: An NFC Business Card in KiCad\n\nI let coding agents research, source, generate and iterate three native KiCad revisions of an NFC business card over three days in October. The files are real and the checks are clean; the confidence came from datasheets, exact part numbers and independent re-examination, not from the renders.\n\n[October 23, 2026 | 7 min read](https://cutler.sg/blog/2026-10-fitness-agent-needs-reliable-data)\n\n### Your Fitness Agent Needs Reliable Data Before Better Advice\n\nA technical follow-up to the Reverse Tamagotchi: physiological days, missing scores and stale rows can change what a fitness assistant thinks it knows.", "url": "https://wpnews.pro/news/measure-agent-review-time-rework-and-cost", "canonical_source": "https://cutler.sg/blog/2026-10-measure-agent-review-time", "published_at": "2026-11-18 12:14:33+00:00", "updated_at": "2026-11-18 13:17:25.623874+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools"], "entities": ["cutler.sg"], "also_reported_by": [], "takeaway": "Log human time in five separate categories (briefing, supervision, review, rework, verification), pause the timer while the agent runs unattended, and count every agent attempt under one business task so a rejected patch plus two revisions reads as one task with three attempts.", "recommendation": {"advice": "Track accepted agent candidates over evaluated candidates separately from task completion, mark missed timer entries as missing data rather than zero, and label manual takeovers so rescue work is counted in the workflow's real cost.", "playbook": "production-ops", "playbook_url": "https://wpnews.pro/playbooks/production-ops"}, "alternates": {"html": "https://wpnews.pro/news/measure-agent-review-time-rework-and-cost", "markdown": "https://wpnews.pro/news/measure-agent-review-time-rework-and-cost.md", "text": "https://wpnews.pro/news/measure-agent-review-time-rework-and-cost.txt", "jsonld": "https://wpnews.pro/news/measure-agent-review-time-rework-and-cost.jsonld"}}