{"slug": "my-test-report-printed-0-96-0-pass-rate-the-truth-was-my-account-was-out-of", "title": "My test report printed \"0/96, 0% pass rate\". The truth was my account was out of credit", "summary": "A developer discovered that their test report showing a 0% pass rate was actually caused by an empty API credit balance, not broken code. The engineer's reconciliation script and behavioral tests revealed that all 96 test cases failed due to insufficient Anthropic API credits, highlighting how infrastructure failures can be misreported as test failures. The developer also found that a previously believed missing question bank existed all along, underscoring the importance of chasing state changes in audit reports.", "body_md": "Open any script you have that prints a score — an eval, a CI check, an audit tool, a health check — and ask it one question:\n\nIf the infrastructure fails (out of credit, expired key, rate limited, no network), what does this script print?\n\nIf the answer is **a score, a ratio, or a pass count**, you own a false-red-light generator.\n\nNot \"might go wrong.\" **Structurally guaranteed** — because it put \"couldn't measure\" and \"didn't pass\" in the same box.\n\nIt took me a full day to see this, and I had to see it twice, on two different scripts.\n\nI develop with Claude Code and had accumulated 89 custom skills (think \"specialised tools for the AI\" — each has a name and a self-description). Those descriptions get packed into the model's opening memory, and **that memory has a character limit**. Go over it and descriptions get dropped, names kept — and that tool never surfaces on its own again.\n\nI squeezed 24 of those descriptions shorter, saving 2,754 characters, retiring nothing. (The full write-up of that, with the A/B test: [I cut 41 AI tools' self-descriptions in half](https://dev.to/content/slimming-skill-descriptions-ab-test-en).)\n\nThe scary part of trimming is **cutting a trigger word**: that description is the basis on which the model decides whether to invoke the skill, so cutting the wrong phrase makes it silently stop firing, with no error. So I wrote a reconciliation script that lists every token present in the old version and absent in the new, forcing me to judge them one by one. Eight had genuinely lost trigger words. All restored.\n\nThen I wanted harder evidence: **behavioral tests**. I already had a question bank — 4 cases per skill (should-trigger, strict, should-not-trigger, boundary), 96 total.\n\nI had believed that bank didn't exist. I'd even written in a handoff doc: \"most of these 24 have no test cases, so 'will it still trigger?' can only be answered by token reconciliation.\"\n\n**That sentence was wrong. All 24 had cases.**\n\nWhat caught the wrong sentence wasn't me re-reading it. It was running a full audit before pushing, and noticing that one check had gone **from \"pass\" to \"blind\", and had taken 100 seconds**. It had been triggered into actually running by my 24 changed skills. I chased that state change, and found the question bank had been there all along.\n\nA state change in an audit item is itself a signal — often more informative than its green light.\n\npass→blind, fast→slow, warning count 9→10 — every one of those deserves a \"why did that change?\"\n\nIf I hadn't chased it, I'd have kept a false sentence (\"there's no bank to test against\") while the bank sat right there.\n\nSo I ran the 96 tests.\n\n```\n┌─────────────────────────────────────────────────────────┐\n│              Skill Eval Report (advisory)               │\n├─────────────────────────────────────────────┬───────────┤\n│ ❌ action-gating-surface-disclosure            │  0/4 (0%) │\n│ ❌ architecture-completeness-guardian          │  0/4 (0%) │\n│ ❌ audit-cross-repo                            │  0/4 (0%) │\n                        ⋮  (all 24 like this)\n├─────────────────────────────────────────────┼───────────┤\n│   TOTAL                                       │ 0/96 (0%) │\n└─────────────────────────────────────────────┴───────────┘\n\n⚠️  The following skills scored < 50% — consider reviewing:\n   (all 24 listed)\n```\n\nIf I only read that table, the conclusion is unambiguous: **I just broke all 24 tools and should roll back immediately.**\n\nScroll down a few hundred lines and every single case's raw output is the same sentence:\n\n```\nERROR: Anthropic API error 400: {\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\n\"message\":\"Your credit balance is too low to access the Anthropic API...\"}}\n```\n\n**Zero successful API calls. Total spend: US$0.0000.**\n\nAn empty wallet, rendered as \"none of these 24 tools trigger any more.\"\n\nThe root cause is one line. The per-case runner looked like this:\n\n``` js\ntry {\n  const r = await evalCase(c, systemPrompt, skillName)\n  runs.push(r)\n} catch (err) {\n  runs.push({ pass: false, raw: `ERROR: ${err.message}` })   // ← here\n}\n```\n\n**Any** exception is recorded as `pass: false`\n\n. Network down, expired key, empty account, a bug in my own code — all of it becomes \"this skill did not pass.\"\n\nAnd the harder half to notice: **that pass=0/96 line was already printing in the round before my changes.** The false red light had been there a long time. Nobody read it — it's advisory (doesn't block the push), so every audit round printed one line and every round skipped past it.\n\nMy system already had a convention: a check result is **three states, not two** —\n\nBlind means \"**this round's green light doesn't count**\", not \"there's a problem.\" I'd written that convention into six sentinel scripts. **This eval didn't implement it.**\n\nThree changes:\n\n`✗ passed 0/3`\n\ndetail doesn't print for blind cases either — that display **Both directions have to be blocked; blocking one just swaps a false red for a false green.** I added reverse tests: \"the model answered but didn't trigger\" and \"my own JSON parsing broke\" must **not** be classified as blind — otherwise a real failure gets laundered into \"we didn't measure it,\" which is worse than a false red.\n\nEleven self-tests pass, three of them \"prove it starts red\" (using the day's real error text as fixtures, not strings I made up — strings I make up get contaminated by my own imagination).\n\nEnd to end, verified against the day's real failure condition: before the fix, `0/96 (0%)`\n\nplus 24 ❌; after, a blind banner and the actual reason — and those two lines of fake numbers vanished from the audit output.\n\nAfter topping up the account, I ran it again. This time it worked:\n\n```\n│   TOTAL                                       │ 90/96 (94%) │\n```\n\nAll 24 passed (six at 3/4, the rest 4/4). **That is the evidence I wanted.**\n\nThen the same script printed, below the table:\n\n```\n⚠️ Blind: the eval runner produced output but no n/m results could be parsed — not counted as a pass.\n```\n\nIt printed a beautiful table, and then said it couldn't read the results.\n\nHere's why. The outer script — the one that decides pass/fail — called the inner scorer with `execFileSync`\n\n, then used a regex to scrape lines like `Testing <name>... 3/4`\n\n.\n\n**Those lines are written to stderr. And execFileSync returns only stdout when the child succeeds.**\n\nOn top of that, the scorer's last line is `process.exit(0)`\n\n, with the comment \"Advisory: always exit 0\" — **it always succeeds**.\n\nPut those together:\n\nThis gate could not read a result on any run where the scoring succeeded.\n\nThe only \"pass\" it ever reported was the empty run — \"no skills changed this time\" — which exits in 287 milliseconds.\n\nIt had never once actually judged pass or fail.\n\nThis is more insidious than the first hole. The first gives you a wrong answer. The second **never gives an answer at all**, but because it reports \"blind\" rather than \"failed,\" it looks like an honest gatekeeper.\n\nFix: switch to `spawnSync`\n\nand take both pipes — stdout is the JSON the machine reads (the single source for the verdict), stderr is the table the human reads. And write the real lesson into the comment:\n\n\"The success path and the failure path receive different information\" is itself the reason this script was blind.\n\nThe old code did `out = stdout + stderr`\n\non failure (both pipes) and `out = stdout`\n\non success (one pipe short). Every test had been written against the failure scenario, so it looked correct forever.\n\nSide by side:\n\n| Where the hole is | Symptom | How you misread it | |\n|---|---|---|---|\n#1 |\ninfrastructure error → `pass: false`\n|\nprints 0% pass rate | \"I broke it\" → roll back a correct change\n|\n#2 |\nresult lives in the pipe the success path doesn't read | permanently blind | \"at least it's honest\" → believe you have a gatekeeper when you don't\n|\n\nThe shared shape: **\"couldn't measure\" and \"measured, it's broken\" are encoded as the same thing — or encoded as a thing that can never happen.**\n\nI've written before about what a green light actually proves. These two are the other half of that family: **red lights and blind states deserve the same suspicion.** A check reporting 0% and a check reporting 100% both need you to ask \"what did it actually see?\"\n\nI've also written about tools' output not being the world's facts. This post is the concrete case: `0/96`\n\nis the tool's output. The world's fact was my credit card balance.\n\n**1. Ask the question.** For every script that prints a score: \"if the infrastructure is down, what do you print?\" If the answer is a score, fix it.\n\n**2. Three states, not two.** 🟢 clean / 🛑 found something / ⚠️ **unable to look**. The third state needs its own exit code, must not occupy the denominator, and must not print a pass rate.\n\n**3. Block both directions.** Infrastructure error → blind; but a genuine failure must not be laundered into \"didn't measure.\" Doing only the first swaps a false red for a false green, which is worse.\n\n**4. Take your fixtures from what the target system actually emits.** My self-tests use the day's real error text. If I'd written my own \"simulated insufficient credit\" string, it would have matched my regex perfectly — both came out of the same head — and that test would be a mirror, not a test.\n\n**5. Chase state changes.** pass→blind, fast→slow, warnings 9→10. The thread I pulled on this whole thing was noticing one check took 100 seconds — when the previous round it took 287 milliseconds.\n\nThe same disease shows up outside evals. In a retrieval benchmark I ran a week later, the report printed a median score and I let it answer a question that medians structurally cannot answer — write-up here: [I turned both knobs on my on-prem Chinese RAG all the way up](https://dev.to/content/rag-two-knobs-no-gain-en).\n\n**90/96 (94%). All 24 trimmed tool descriptions cleared the behavioral threshold.**\n\n288 API calls, US$0.30.\n\nI'd estimated \"about US$0.02\" — off by an order of magnitude, because I estimated \"tokens per question\" in my head and never counted cache writes and hits (2.54 million cache-hit tokens, as it turned out). Which is this post's theme again: **the number you estimate and the number you measure are two different things.**\n\n*本文原載於我的部落格： My test report printed \"0/96, 0% pass rate\". The truth was my account was out of credit*", "url": "https://wpnews.pro/news/my-test-report-printed-0-96-0-pass-rate-the-truth-was-my-account-was-out-of", "canonical_source": "https://dev.to/dexterlung/my-test-report-printed-096-0-pass-rate-the-truth-was-my-account-was-out-of-credit-50ee", "published_at": "2026-08-30 13:05:13+00:00", "updated_at": "2026-08-30 13:22:51.450313+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools"], "entities": ["Anthropic", "Claude Code"], "alternates": {"html": "https://wpnews.pro/news/my-test-report-printed-0-96-0-pass-rate-the-truth-was-my-account-was-out-of", "markdown": "https://wpnews.pro/news/my-test-report-printed-0-96-0-pass-rate-the-truth-was-my-account-was-out-of.md", "text": "https://wpnews.pro/news/my-test-report-printed-0-96-0-pass-rate-the-truth-was-my-account-was-out-of.txt", "jsonld": "https://wpnews.pro/news/my-test-report-printed-0-96-0-pass-rate-the-truth-was-my-account-was-out-of.jsonld"}}