Publicly auditing my OSS release proved that your documentation

A public audit of the v0.2.1 release of the PlannerCritic engine found that three of five frozen claims were contradicted by public artifacts, including a test count discrepancy (1294 passed, 1 failed, 14 skipped vs. claimed 1295 passed), an internal contradiction (a recorded True Fail = 1 despite a claim of zero true failures), and a metric mismatch (plan_oscillation_detected fired for 5 goals, not 3). The auditor, a stranger, proposed a protocol to verify claims using only public artifacts, and the author concluded that AI workflow documentation must be reconstructible from public evidence to maintain credibility.

Publicly auditing my OSS release proved that your documentation v0.2.1 of my PlannerCritic engine was over. I had the data to back up the release: 73/73 balanced goals approved, 96/97 strict goals escalated, and zero new engine issues discovered during the sweep. I had built a narrative of technical success, and I believed it.Then a stranger decided to audit my claims in public. He didn't just offer feedback; he proposed a rigorous protocol. We froze five specific claims from the release before he inspected them. The rule was simple: verify the claims using only public artifacts. If the evidence didn't match the claim, it was recorded as a divergence—no excuses, no "explaining it away" based on what I knew happened in my local environment. This was a brutal reality check on how we handle AI workflow documentation and LLM agent testing. Most of us release software based on a mix of CI logs, local memory, and general confidence. We know the code works, so we assume our summaries of its performance are also correct. This audit proved that assumption is a trap. The breakdown of the audit We froze five claims. Two were clean. Three were blatant contradictions. The Test Count Discrepancy: I claimed 1295 deterministic tests passed. The public CI run actually showed 1294 passed, 1 failed a flaky SQLite concurrency issue , and 14 skipped. While the failure didn't break the engine, my documentation was mathematically incorrect compared to the public record. The Internal Contradiction: This was the most embarrassing one. In my field-test report, I explicitly wrote "zero true failures." However, just two tables down in Scorecard B, I had recorded True Fail = 1 . I had published two incompatible truths in the same document and never noticed. The Metric Mismatch: I stated that plan oscillation detected fired for 3 goals. The actual stored artifacts showed it actually fired for 5. Why this matters for prompt engineering and agent evaluation The most uncomfortable takeaway isn't that the software failed—the engine actually performed beautifully. The live boundary evaluator returned perfect rates for label flip rate and evidence drift rate . The underlying logic of the PlannerCritic was robust. The failure was in the meta-layer : the bridge between the technical execution and the human-readable report. When we are building complex LLM agents or fine-tuning prompt engineering workflows, we often rely on high-level summaries to communicate success. If your evaluation pipeline produces a metric, but your documentation manual-entry process "rounds" that metric or misinterprets a flag, you lose technical credibility. In a real-world deployment, if your agent's performance metrics are even slightly decoupled from the actual logs, you cannot perform a reliable post-mortem when things go wrong. This experience taught me that an AI workflow is only as reliable as its most visible documentation. If you can't reconstruct your claims from public artifacts, you haven't actually proven anything. Next Your AI agents can report 100% success while actually breaking → /en/threads/7913/ a library of Claude prompt techniques https://tanyan888.com/ , with plenty of directly applicable cases. All Replies （4） @Drew36 /en/users/Drew36/ That's the scary part. If you're only monitoring availability instead of stability, you're basically flying blind....