How We Built Charlie, Part 15: What We Learned Building Charlie
What we learned about making AI engineering work owned, bounded, verifiable, and useful to a team.
How evidence-backed completion prevents confident reports from outrunning artifacts, provider state, tests, and CI.
How We Built Charlie continues from Chapter 12: every bounded run needs evidence strong enough for its completion claim.
An engineer asks Charlie to fix a production bug. Twenty minutes later, Charlie replies: “Fixed.”
That sentence may be correct, but the confidence of the summary adds no evidence. The useful answer includes a branch, a commit, the tests that ran, the current CI state, a pull request, and any relevant observability read-back. If the fix could not be verified in production, the result should say so. If a tool timed out after sending a request, the result should preserve the uncertain external outcome instead of converting it into success or failure by tone.
Dependable agent work needs an evidence path from intent to observable state. Charlie records what the model requested, what the tool reported, which durable artifacts were created, what fresh reads show, and which verification systems evaluated the change. Each layer answers a different question. None deserves to be stretched past what it proves.
This is the thesis of “proof, not vibes”: completion is a claim that should be supported by inspectable evidence proportional to the risk of the work.
Models are good at producing coherent summaries. Coherence helps a person understand the work, but it can also hide missing steps. A response can accurately describe the intended patch even when the file was never written. It can report that a pull request was opened after the provider accepted the request but before the pull request number was persisted. It can say tests passed because a command started successfully, even though the test process failed later.
Charlie therefore separates evidence classes that are easy to collapse in ordinary chat:
| Evidence | Supports | Cannot prove alone |
|---|---|---|
| Model intent | The action Charlie attempted to take | That the tool ran or any state changed |
| Tool result | What the tool returned or what error the executor recorded | That an external provider retained the intended state |
| Durable artifact reference | A branch, commit, pull request, issue, comment, message, release, or event exists | That the artifact is correct or currently healthy |
| Provider read-back | The external system currently exposes the expected state | That the code works in all relevant cases |
| Test or evaluation result | A defined behavior passed a defined check on one revision | That live use is healthy beyond the checked coverage |
| Telemetry | The running system showed observed behavior during a time window | That an unseen path is correct or a future release stays healthy |
The strongest completion reports often combine several classes. A code change can be tied to a commit, exercised by targeted tests, checked by CI on that same revision, exposed through a pull request, and followed by Sentry or deployment telemetry after release. The report becomes more useful as the evidence gets closer to the claim.
The executor records tool calls and ordered outcomes so a later reviewer can follow the causal path from request to artifact. Chapter 6 covers the phase and replay mechanics in detail. For proof, the important boundary is simpler: a tool result records what Charlie observed, including errors, deadlines, and interrupted dispatch; it does not automatically establish the provider’s current state.
That distinction keeps uncertainty available for the next evidence step. A stable artifact ID enables provider read-back. A test result can be tied to a revision and environment. CI can confirm configured checks for that revision. The completion claim becomes stronger as independent systems confirm the reported work.
“I created the pull request” is a claim. “Pull request #742” is a reference that a reviewer can inspect. A full commit SHA identifies a revision more precisely than “the latest changes.” A Slack channel and message timestamp identify a visible communication. A Sentry issue URL and event ID tie a diagnosis to a concrete observation.
Durable identifiers support three jobs:
The identifier should be captured from a tool result and, for higher-risk effects, verified by reading the provider back. Branch names and pull request numbers are especially useful because they allow recovery after partial failure. If a create call times out, Charlie can search the expected branch or provider context before deciding whether another create attempt is safe.
Durable IDs still need provenance. A commit SHA reported by the model without a successful git rev-parse
or provider result is text, not proof. A URL assembled from a guessed route may look convincing and point nowhere. Charlie’s operating instructions therefore ask workers to report identifiers only after verifying them through tooling.
Different engineering statements require different verification. A single universal “verified” flag would discard the information reviewers need.
| Claim | Useful supporting evidence |
|---|---|
| ”The file changed” | Diff, file read-back, and commit containing the expected path |
| ”The bug is fixed” | Reproduction before, targeted test or scenario after, and review of the actual patch |
| ”The branch is pushed” | Remote ref read-back at the expected full SHA |
| ”CI passed” | Required check names and conclusions for the target revision |
| ”The pull request is ready” | Pull request URL, current head SHA, mergeability/review state, and relevant checks |
| ”The issue was updated” | Provider read-back of the issue fields or comment with a durable identifier |
| ”The deployment is healthy” | Deployment state plus current service telemetry over an appropriate observation window |
| ”The agent behavior improved” | Versioned evaluation cases, comparable baselines, failure analysis, and results tied to the changed agent revision |
The verification class should also match the failure mode. Formatting checks are useful for a content change but say little about a concurrency bug. A unit test may cover a pure function while missing a provider contract. A visual screenshot can catch overflow and hierarchy problems that typechecking cannot see. Telemetry can reveal a production regression that all pre-merge tests missed.
Proof is therefore compositional. The evidence bundle for a UI change may contain typecheck, a focused interaction test, light and dark screenshots, mobile overflow inspection, and a deployed preview. A scheduler migration may require schema tests, upgrade and rollback reasoning, concurrency cases, and operational telemetry. The same word, “done,” should not flatten those differences.
Verification has a cost. Running every possible test and querying every provider after every action would waste time and can create its own load. The answer is risk-based proof, where evidence depth follows reversibility, blast radius, ambiguity, and the quality of existing safeguards.
| Risk profile | Reasonable evidence pattern |
|---|---|
| Reversible local edit | Read the diff, run the focused formatter or test, and report the changed paths |
| Shared repository change | Add targeted tests, typecheck/build as relevant, commit, push, and verify the remote revision |
| External provider mutation | Capture the returned durable ID, then read back current provider state when duplication or permissions matter |
| Production behavior change | Reproduce, test, pass CI, review deployment state, and inspect relevant telemetry |
| Security- or data-sensitive change | Use explicit human approval, narrowly scoped credentials, audit evidence, and domain-specific validation |
| Ambiguous or partially observed write | Preserve uncertainty, search for existing effects, and escalate rather than retrying a potentially non-idempotent action blindly |
Several factors raise the evidence requirement:
Low-risk work can still benefit from evidence, but the bundle can remain small. A documentation typo does not need a canary deployment. A production data migration should not be closed with a local unit test and a confident paragraph.
The Task scheduler needs a canonical terminal status and a result that later systems can interpret. Charlie’s task-result event distinguishes successful content from error information. It records the shape of the result and can include the result content, but the current event is deliberately compact. It is not an external-effects ledger.
At the entry-agent boundary, newer Charlie agent generations can emit a structured self-report with a trigger, focus, outcome, and a bounded list of important effects. Conceptually:
type EntryTaskSelfReport = {
schema: 'entry_task_self_report_v0';
trigger: ResourceRef;
focus: ResourceRef;
outcome: {
code: OutcomeCode;
summary: string;
};
effects: Array<{
kind: ResourceKind;
summary: string;
ref: string;
}>;
};
The executor validates and canonicalizes compatible reports while preserving ordinary result content when the report is missing or invalid. Unknown values are normalized conservatively, unusable effects are dropped, and parse status is available for telemetry. Worker handoffs use a related convention so the caller can carry durable effects forward without copying entire transcripts.
Structured reporting improves legibility. A scheduler, dashboard, or parent Task can identify the trigger, the resource the work focused on, the outcome category, and the important artifacts. It also keeps the effect list selective: a devbox or temporary file may matter during execution but should not become a customer-facing completion artifact.
The report remains model-authored. It can be checked for shape and compared with transcript evidence, but its existence does not prove that every listed external effect occurred.
Structured effect reports make artifacts easier to query, display, and carry through parent-child handoffs. A row that says “created pull request” is useful metadata tied to a Task, but it remains reported attribution. Provider read-back, the pull request’s current head, CI, or another independent source provides stronger evidence that the artifact exists in the expected state.
Improving the schema improves observability and recovery. Independent evidence is still required before the system upgrades a reported effect into a verified completion claim.
A Task can succeed because the agent completed its requested reasoning and produced a useful terminal result. That status does not automatically certify every effect mentioned in the result.
There are several reasons:
For customer-visible writes, Charlie prefers stable identifiers and current read-back. For repository work, he can verify the branch, full commit SHA, and remote ref. For CI, he can report check names tied to that revision. For Slack, Linear, GitHub, or Sentry, he can preserve provider IDs and links returned by authenticated tools.
When read-back is impossible, the terminal result should narrow the claim. “The request was submitted; provider confirmation was not observed” is less satisfying than “done,” but more useful for recovery.
Tests and provider read-back can show that one Task produced an expected artifact. Improving Charlie also requires evidence across classes of tasks.
An evaluation should tie an agent version to versioned cases, expected behaviors, tool-use constraints, and scored outcomes. Useful cases include straightforward success paths and failures the architecture is designed to prevent: duplicate provider delivery, ambiguous issue inference, interrupted tool dispatch, unsafe retry, stale evidence, mailbox follow-up, and deny-rule enforcement.
Evaluation results should be read with the same discipline as other evidence. A higher aggregate score can hide a regression in a critical class. A prompt change may improve one model and degrade another. A test fixture can become easier than production. The evidence bundle therefore includes the case set, agent revision, model/runtime configuration where relevant, comparison baseline, and failure analysis.
Telemetry then closes part of the gap between evaluation and operation. Dispatch completion sources, self-report parse status, task terminal states, tool errors, provider failures, and recovery paths can reveal behavior that static cases miss. Telemetry supports diagnosis and trend analysis; it does not by itself prove semantic correctness.
A strong Charlie handoff is concise because the evidence is linked, not because the evidence is absent. For a production bug fix, a reviewer should be able to move from the terminal summary to the exact code revision, pull request, tests, CI results, and operational observation without reconstructing the work from prose.
That also improves the next run. Durable references let Charlie recover after interruption, avoid duplicate writes, answer follow-up questions, and compare current provider state with the original result. Typed outcomes distinguish completed work from waiting, insufficient context, a blocked operation, or a required human decision.
The final summary still matters. It explains what changed and why. The evidence gives that explanation something to stand on.
Previous: How We Built Charlie, Part 12: Daemons: Persistent Roles, Bounded Runs. Next: How We Built Charlie, Part 14: Making Agent Work Legible. Browse the full How We Built Charlie series.