{"slug": "how-we-built-charlie-part-13-proof-not-vibes", "title": "How We Built Charlie, Part 13: Proof, Not Vibes", "summary": "Charlie, an AI engineering assistant, now requires evidence-backed completion claims, separating model intent, tool results, durable artifacts, provider read-back, tests, and telemetry to prevent confident reports from outrunning actual state. The approach, detailed in Part 15 of the 'How We Built Charlie' series, ensures each bounded run's completion is supported by inspectable evidence proportional to risk, such as a branch, commit, tests, CI state, and pull request.", "body_md": "### How We Built Charlie, Part 15: What We Learned Building Charlie\n\nWhat we learned about making AI engineering work owned, bounded, verifiable, and useful to a team.\n\nHow evidence-backed completion prevents confident reports from outrunning artifacts, provider state, tests, and CI.\n\n[How We Built Charlie](/blog/series/how-we-built-charlie/) continues from [Chapter 12](/blog/daemons-persistent-roles-bounded-runs/): every bounded run needs evidence strong enough for its completion claim.\n\nAn engineer asks Charlie to fix a production bug. Twenty minutes later, Charlie replies: “Fixed.”\n\nThat sentence may be correct, but the confidence of the summary adds no evidence. The useful answer includes a branch, a commit, the tests that ran, the current CI state, a pull request, and any relevant observability read-back. If the fix could not be verified in production, the result should say so. If a tool timed out after sending a request, the result should preserve the uncertain external outcome instead of converting it into success or failure by tone.\n\nDependable agent work needs an evidence path from intent to observable state. Charlie records what the model requested, what the tool reported, which durable artifacts were created, what fresh reads show, and which verification systems evaluated the change. Each layer answers a different question. None deserves to be stretched past what it proves.\n\nThis is the thesis of “proof, not vibes”: completion is a claim that should be supported by inspectable evidence proportional to the risk of the work.\n\nModels are good at producing coherent summaries. Coherence helps a person understand the work, but it can also hide missing steps. A response can accurately describe the intended patch even when the file was never written. It can report that a pull request was opened after the provider accepted the request but before the pull request number was persisted. It can say tests passed because a command started successfully, even though the test process failed later.\n\nCharlie therefore separates evidence classes that are easy to collapse in ordinary chat:\n\n| Evidence | Supports | Cannot prove alone |\n|---|---|---|\n| Model intent | The action Charlie attempted to take | That the tool ran or any state changed |\n| Tool result | What the tool returned or what error the executor recorded | That an external provider retained the intended state |\n| Durable artifact reference | A branch, commit, pull request, issue, comment, message, release, or event exists | That the artifact is correct or currently healthy |\n| Provider read-back | The external system currently exposes the expected state | That the code works in all relevant cases |\n| Test or evaluation result | A defined behavior passed a defined check on one revision | That live use is healthy beyond the checked coverage |\n| Telemetry | The running system showed observed behavior during a time window | That an unseen path is correct or a future release stays healthy |\n\nThe strongest completion reports often combine several classes. A code change can be tied to a commit, exercised by targeted tests, checked by CI on that same revision, exposed through a pull request, and followed by Sentry or deployment telemetry after release. The report becomes more useful as the evidence gets closer to the claim.\n\nThe executor records tool calls and ordered outcomes so a later reviewer can follow the causal path from request to artifact. [Chapter 6](/blog/the-executor-loop/) covers the phase and replay mechanics in detail. For proof, the important boundary is simpler: a tool result records what Charlie observed, including errors, deadlines, and interrupted dispatch; it does not automatically establish the provider’s current state.\n\nThat distinction keeps uncertainty available for the next evidence step. A stable artifact ID enables provider read-back. A test result can be tied to a revision and environment. CI can confirm configured checks for that revision. The completion claim becomes stronger as independent systems confirm the reported work.\n\n“I created the pull request” is a claim. “Pull request #742” is a reference that a reviewer can inspect. A full commit SHA identifies a revision more precisely than “the latest changes.” A Slack channel and message timestamp identify a visible communication. A Sentry issue URL and event ID tie a diagnosis to a concrete observation.\n\nDurable identifiers support three jobs:\n\nThe identifier should be captured from a tool result and, for higher-risk effects, verified by reading the provider back. Branch names and pull request numbers are especially useful because they allow recovery after partial failure. If a create call times out, Charlie can search the expected branch or provider context before deciding whether another create attempt is safe.\n\nDurable IDs still need provenance. A commit SHA reported by the model without a successful `git rev-parse`\n\nor provider result is text, not proof. A URL assembled from a guessed route may look convincing and point nowhere. Charlie’s operating instructions therefore ask workers to report identifiers only after verifying them through tooling.\n\nDifferent engineering statements require different verification. A single universal “verified” flag would discard the information reviewers need.\n\n| Claim | Useful supporting evidence |\n|---|---|\n| ”The file changed” | Diff, file read-back, and commit containing the expected path |\n| ”The bug is fixed” | Reproduction before, targeted test or scenario after, and review of the actual patch |\n| ”The branch is pushed” | Remote ref read-back at the expected full SHA |\n| ”CI passed” | Required check names and conclusions for the target revision |\n| ”The pull request is ready” | Pull request URL, current head SHA, mergeability/review state, and relevant checks |\n| ”The issue was updated” | Provider read-back of the issue fields or comment with a durable identifier |\n| ”The deployment is healthy” | Deployment state plus current service telemetry over an appropriate observation window |\n| ”The agent behavior improved” | Versioned evaluation cases, comparable baselines, failure analysis, and results tied to the changed agent revision |\n\nThe verification class should also match the failure mode. Formatting checks are useful for a content change but say little about a concurrency bug. A unit test may cover a pure function while missing a provider contract. A visual screenshot can catch overflow and hierarchy problems that typechecking cannot see. Telemetry can reveal a production regression that all pre-merge tests missed.\n\nProof is therefore compositional. The evidence bundle for a UI change may contain typecheck, a focused interaction test, light and dark screenshots, mobile overflow inspection, and a deployed preview. A scheduler migration may require schema tests, upgrade and rollback reasoning, concurrency cases, and operational telemetry. The same word, “done,” should not flatten those differences.\n\nVerification has a cost. Running every possible test and querying every provider after every action would waste time and can create its own load. The answer is risk-based proof, where evidence depth follows reversibility, blast radius, ambiguity, and the quality of existing safeguards.\n\n| Risk profile | Reasonable evidence pattern |\n|---|---|\n| Reversible local edit | Read the diff, run the focused formatter or test, and report the changed paths |\n| Shared repository change | Add targeted tests, typecheck/build as relevant, commit, push, and verify the remote revision |\n| External provider mutation | Capture the returned durable ID, then read back current provider state when duplication or permissions matter |\n| Production behavior change | Reproduce, test, pass CI, review deployment state, and inspect relevant telemetry |\n| Security- or data-sensitive change | Use explicit human approval, narrowly scoped credentials, audit evidence, and domain-specific validation |\n| Ambiguous or partially observed write | Preserve uncertainty, search for existing effects, and escalate rather than retrying a potentially non-idempotent action blindly |\n\nSeveral factors raise the evidence requirement:\n\nLow-risk work can still benefit from evidence, but the bundle can remain small. A documentation typo does not need a canary deployment. A production data migration should not be closed with a local unit test and a confident paragraph.\n\nThe Task scheduler needs a canonical terminal status and a result that later systems can interpret. Charlie’s task-result event distinguishes successful content from error information. It records the shape of the result and can include the result content, but the current event is deliberately compact. It is not an external-effects ledger.\n\nAt the entry-agent boundary, newer Charlie agent generations can emit a structured self-report with a trigger, focus, outcome, and a bounded list of important effects. Conceptually:\n\n```\ntype EntryTaskSelfReport = {\n  schema: 'entry_task_self_report_v0';\n  trigger: ResourceRef;\n  focus: ResourceRef;\n  outcome: {\n    code: OutcomeCode;\n    summary: string;\n  };\n  effects: Array<{\n    kind: ResourceKind;\n    summary: string;\n    ref: string;\n  }>;\n};\n```\n\nThe executor validates and canonicalizes compatible reports while preserving ordinary result content when the report is missing or invalid. Unknown values are normalized conservatively, unusable effects are dropped, and parse status is available for telemetry. Worker handoffs use a related convention so the caller can carry durable effects forward without copying entire transcripts.\n\nStructured reporting improves legibility. A scheduler, dashboard, or parent Task can identify the trigger, the resource the work focused on, the outcome category, and the important artifacts. It also keeps the effect list selective: a devbox or temporary file may matter during execution but should not become a customer-facing completion artifact.\n\nThe report remains model-authored. It can be checked for shape and compared with transcript evidence, but its existence does not prove that every listed external effect occurred.\n\nStructured effect reports make artifacts easier to query, display, and carry through parent-child handoffs. A row that says “created pull request” is useful metadata tied to a Task, but it remains reported attribution. Provider read-back, the pull request’s current head, CI, or another independent source provides stronger evidence that the artifact exists in the expected state.\n\nImproving the schema improves observability and recovery. Independent evidence is still required before the system upgrades a reported effect into a verified completion claim.\n\nA Task can succeed because the agent completed its requested reasoning and produced a useful terminal result. That status does not automatically certify every effect mentioned in the result.\n\nThere are several reasons:\n\nFor customer-visible writes, Charlie prefers stable identifiers and current read-back. For repository work, he can verify the branch, full commit SHA, and remote ref. For CI, he can report check names tied to that revision. For Slack, Linear, GitHub, or Sentry, he can preserve provider IDs and links returned by authenticated tools.\n\nWhen read-back is impossible, the terminal result should narrow the claim. “The request was submitted; provider confirmation was not observed” is less satisfying than “done,” but more useful for recovery.\n\nTests and provider read-back can show that one Task produced an expected artifact. Improving Charlie also requires evidence across classes of tasks.\n\nAn evaluation should tie an agent version to versioned cases, expected behaviors, tool-use constraints, and scored outcomes. Useful cases include straightforward success paths and failures the architecture is designed to prevent: duplicate provider delivery, ambiguous issue inference, interrupted tool dispatch, unsafe retry, stale evidence, mailbox follow-up, and deny-rule enforcement.\n\nEvaluation results should be read with the same discipline as other evidence. A higher aggregate score can hide a regression in a critical class. A prompt change may improve one model and degrade another. A test fixture can become easier than production. The evidence bundle therefore includes the case set, agent revision, model/runtime configuration where relevant, comparison baseline, and failure analysis.\n\nTelemetry then closes part of the gap between evaluation and operation. Dispatch completion sources, self-report parse status, task terminal states, tool errors, provider failures, and recovery paths can reveal behavior that static cases miss. Telemetry supports diagnosis and trend analysis; it does not by itself prove semantic correctness.\n\nA strong Charlie handoff is concise because the evidence is linked, not because the evidence is absent. For a production bug fix, a reviewer should be able to move from the terminal summary to the exact code revision, pull request, tests, CI results, and operational observation without reconstructing the work from prose.\n\nThat also improves the next run. Durable references let Charlie recover after interruption, avoid duplicate writes, answer follow-up questions, and compare current provider state with the original result. Typed outcomes distinguish completed work from waiting, insufficient context, a blocked operation, or a required human decision.\n\nThe final summary still matters. It explains what changed and why. The evidence gives that explanation something to stand on.\n\n*Previous: How We Built Charlie, Part 12: Daemons: Persistent Roles, Bounded Runs. Next: How We Built Charlie, Part 14: Making Agent Work Legible. Browse the full How We Built Charlie series.*", "url": "https://wpnews.pro/news/how-we-built-charlie-part-13-proof-not-vibes", "canonical_source": "https://charlielabs.ai/blog/proof-not-vibes/", "published_at": "2026-08-10 12:13:00+00:00", "updated_at": "2026-08-12 16:16:43.820816+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools"], "entities": ["Charlie"], "alternates": {"html": "https://wpnews.pro/news/how-we-built-charlie-part-13-proof-not-vibes", "markdown": "https://wpnews.pro/news/how-we-built-charlie-part-13-proof-not-vibes.md", "text": "https://wpnews.pro/news/how-we-built-charlie-part-13-proof-not-vibes.txt", "jsonld": "https://wpnews.pro/news/how-we-built-charlie-part-13-proof-not-vibes.jsonld"}}