{"slug": "honest-tests-detect-fake-tests-and-prove-them-with-mutation-testing", "title": "Honest-tests – detect fake tests and prove them with mutation testing", "summary": "Mintwork released honest-tests, an agent skill and CLI distributed as @mintwork/honest-tests, that audits test files for assertions which cannot fail and runs built-in mutation testing to prove which tests are fake. In the tool's demo, an audit of a 10-test pricing file flagged 7 errors and 2 warnings, and mutation testing of src/pricing.ts reported a 15.4% mutation score with 22 of 26 mutants surviving; after the tests were rewritten, the same commands reported no fake tests and a 100% mutation score with 26 of 26 mutants killed. The tool targets agents including Claude Code, Codex and Cursor, and is installed via npx or as a plugin from the mintwork/honest-tests marketplace.", "body_md": "**honest-tests** is an agent skill + CLI that stops Claude Code, Codex, Cursor and friends from writing\nfake tests — and proves which of your existing tests are fake by mutating your code and watching them\nstay green.\n\n```\nnpx @mintwork/honest-tests audit .                       # find tests that can't fail\nnpx @mintwork/honest-tests mutate src/pricing.ts         # prove it: break the code, see who notices\n```\n\nAsk an agent to \"add tests\" and you get a wall of green checkmarks. Look closer:\n\n``` js\nit('handles multiple items', () => {\n  const result = subtotal(items);\n  expect(result).toBe(result);                  // compares a value to itself\n});\n\nit('applies a percentage discount', () => {\n  applyDiscount(100, 10);                       // no assertion at all\n});\n\nit('computes the discounted price', () => {\n  try {\n    expect(applyDiscount(100, 20)).toBe(70);    // WRONG (it's 80) — and still passes\n  } catch {}\n});\n\nit('gives members free shipping', () => {\n  const qualifies = vi.fn().mockReturnValue(true);\n  expect(qualifies(10, true)).toBe(true);       // tests the mock it just configured\n});\n```\n\nEvery one of these passes. Coverage goes up. Nothing is verified.\n\n``` bash\n$ npx @mintwork/honest-tests audit test/pricing.test.ts\n\ntest/pricing.test.ts\n  16:5    error   Compares `result` to itself — this can never fail. tautological-assertion\n          │ expect(result).toBe(result);\n  25:1    error   Test \"applies a percentage discount\" has no assertions — it can only fail by crashing. no-assertions\n          │ it('applies a percentage discount', () => {\n  39:7    error   Assertion is inside a try/catch that swallows errors — when it fails, the catch eats the AssertionError and the test passes. swallowed-assertion\n          │ expect(applyDiscount(100, 20)).toBe(70); // wrong! but the catch hides it\n  53:5    error   Asserts on what mock `qualifies` returns (configured on line 52) — this verifies the mock, not your code. asserts-on-mock\n          │ expect(qualifies(10, true)).toBe(true);\n  ... 5 more\n\n7 of 10 tests can't fail (or don't test what they claim).\n1 file, 10 tests: 7 errors, 2 warnings. Run with --verbose for fixes.\n```\n\nThen the killer demo — **mutation testing, built in**. honest-tests flips operators, booleans,\nconstants and return values in your source (on a temp copy), runs your tests, and shows every change\nnobody noticed:\n\n``` bash\n$ npx @mintwork/honest-tests mutate src/pricing.ts\n\n  22 mutants survived — your tests don't actually check these lines:\n\n  src/pricing.ts:15  if (percent < 0 || percent > 100) {\n      ✗ `<` → `<=`\n      ✗ `||` → `&&`\n      ✗ `>` → `>=`\n      ✗ `100` → `101`\n\n  src/pricing.ts:22  return isMember || total >= 50;\n      ✗ `>=` → `>`\n      ✗ `50` → `51`\n  ...\n\n  Mutation score: 15.4% (4 killed, 0 timed out, 22 survived of 26; baseline 0.5s)\njs\nit('takes the percentage off the amount', () => {\n  expect(applyDiscount(200, 25)).toBe(150);\n});\n\nit('accepts the 0% and 100% boundaries', () => {\n  expect(applyDiscount(200, 0)).toBe(200);\n  expect(applyDiscount(200, 100)).toBe(0);\n});\n\nit('rejects percentages outside 0..100', () => {\n  expect(() => applyDiscount(200, -1)).toThrow(RangeError);\n  expect(() => applyDiscount(200, 101)).toThrow(RangeError);\n});\n\nit('starts free shipping exactly at 50 for non-members', () => {\n  expect(qualifiesForFreeShipping(49.99, false)).toBe(false);\n  expect(qualifiesForFreeShipping(50, false)).toBe(true);\n});\nbash\n$ npx @mintwork/honest-tests audit test && npx @mintwork/honest-tests mutate src/pricing.ts\n✓ 12 tests in 2 files: no fake tests found.\n  Mutation score: 100% (26 killed, 0 timed out, 0 survived of 26; baseline 0.5s)\n  ✓ Every mutation was caught. These tests actually test something.\n```\n\nBoth projects live in [`fixtures/`](https://github.com/mintwork/honest-tests/blob/main/fixtures) — run `npm run demo:bad` / `npm run demo:good` yourself\n(`npm run demo:gif` re-renders the GIF above from the real CLI output).\n\n```\n/plugin marketplace add mintwork/honest-tests\n/plugin install honest-tests@honest-tests\n```\n\nOr drop just the skill into a repo: `npx @mintwork/honest-tests init claude` (writes `.claude/skills/honest-tests/`).\n\nThen ask: *\"write tests for src/cart.ts\"* or *\"audit my tests\"* — or use the slash commands\n`/honest-tests:audit [path]` and `/honest-tests:mutate <file>`.\n\n```\nnpx @mintwork/honest-tests init codex      # appends an honest-tests section to AGENTS.md (idempotent)\nnpx @mintwork/honest-tests init cursor     # writes .cursor/rules/honest-tests.mdc (auto-attaches to test files)\nnpx @mintwork/honest-tests audit . --json                 # machine-readable findings\nnpx @mintwork/honest-tests mutate src/x.ts --min-score 80 # fail CI below 80% mutation score\n```\n\nThe [skill](https://github.com/mintwork/honest-tests/blob/main/skills/honest-tests/SKILL.md) (Agent Skills format) teaches agents one rule — *a test is\nhonest only if it fails when the behavior it names is broken* — plus two workflows:\n\n- **Writing tests:** name the bug each test catches, derive expected values independently, mock only\nboundaries, cover boundaries and error paths, then self-audit with`audit` +`mutate` before saying\n\"done\". Hard rules forbid the classic agent moves: weakening assertions to get green, changing the\nexpected value to match output, mocking the unit under test,`try/catch` around asserts.\n- **Auditing tests** , in phases, like a security audit:\n  1. **Detect** —`honest-tests audit --json` plus a manual pass for smells static analysis can't see.\n  2. **Verify** — every finding is opened and marked*confirmed / false-positive / accepted* ; key\nfindings are proven with surviving mutants.\n  3. **Report** — \"N of M tests can't fail\", confirmed findings, surviving mutants, next steps.\n  4. **Fix** — only when asked; re-run and show the before/after mutation score.\n\nReferences the agent loads on demand: [test smell catalog](https://github.com/mintwork/honest-tests/blob/main/skills/honest-tests/references/test-smells.md),\n[writing honest tests](https://github.com/mintwork/honest-tests/blob/main/skills/honest-tests/references/writing-honest-tests.md),\n[audit procedure](https://github.com/mintwork/honest-tests/blob/main/skills/honest-tests/references/audit-procedure.md),\n[mutation testing](https://github.com/mintwork/honest-tests/blob/main/skills/honest-tests/references/mutation-testing.md).\n\n| Rule | Severity | Catches | \n|---|---|---|\n| `no-assertions` | error | Test body never asserts — passes unless it crashes | \n| `tautological-assertion` | error | `expect(x).toBe(x)` ,`expect(f(1)).toBe(f(1))` ,`expect(true).toBe(true)` ,`toHaveLength(list.length)` | \n| `asserts-on-mock` | error | Asserting on the return value of a mock the test configured ( `vi.fn().mockReturnValue` ,`spyOn` ,`vi.mocked` ,`.mock.results` ) | \n| `asserts-on-own-literal` | error | Asserting on a literal the test just built: `const p = { service: 's3' }; expect(p.service).toBe('s3')` — common in runtime \"type tests\" | \n| `never-failing-assertion` | error | `expect(x).toBeTruthy` (never called), bare`expect(x)` , un-awaited`.resolves` /`.rejects` ,`typeof x` \"defined\",`length >= 0` ,`{}` truthy | \n| `swallowed-assertion` | error | Assertion inside `try` whose`catch` doesn't rethrow | \n| `catch-only-assertion` | error | Assertions only in `catch` — passes when nothing throws | \n| `mocked-unit-under-test` | error | `vi.mock('./cart')` inside`cart.test.ts` ; spying on and replacing the unit's own exports | \n| `focused-test` | error | `.only` ,`fit` ,`fdescribe` left in | \n| `excessive-mocking` | warning | 5+ module mocks in one file | \n| `snapshot-only` | warning | The only assertion is a snapshot | \n| `conditional-assertion` | warning | Assertion behind `if` /`&&` / ternary | \n| `skipped-test` | warning | `.skip` ,`xit` ,`xdescribe` | \n\n`npx @mintwork/honest-tests rules` prints the list. Suppress a deliberate case with a reason:\n\n```\n// honest-tests-disable-next-line no-assertions -- smoke test: module must import cleanly\nit('loads', () => { require('../src/plugin'); });\n```\n\n(`honest-tests-disable-line <rule>` and file-level `/* honest-tests-disable <rule> */` also work.)\n\n| Operator | Example | \n|---|---|\n| binary | `>=` →`>` ,`<` →`<=` ,`===` →`!==` ,`+` →`-` ,`*` →`/` | \n| logical | `&&` ↔`\\|\\|` | \n| assignment | `+=` ↔`-=` | \n| boolean | `true` ↔`false` | \n| negation | `!x` →`x` | \n| number | `50` →`51` ,`0` →`1` | \n| condition | `if (c)` →`if (true)` /`if (false)` | \n| return | `return x` →`return undefined` | \n\n**Safe by design:** the project is copied to a temp dir (`node_modules` symlinked), mutants are written\nonly to the copy and restored after every run, and the copy is removed at the end — even on Ctrl-C. A red\nbaseline aborts the run. Only test files that import the target — directly or through other local\nmodules — are run (vitest/jest), once and with coverage off, so it's fast: 26 mutants in ~12s on the\nfixture. Imports through path aliases (`@/lib/x`) aren't followed yet; pass `--test-cmd` if your tests\nreach the target that way.\n\n```\n--test-cmd \"<cmd>\"   custom command; {files} expands to related test files\n--max-mutants <n>    cap (default 100)          --timeout <ms>   per-run timeout\n--min-score <0-100>  exit 1 below this score    --root <dir>     project root\nhonest-tests audit [paths...]   [--json] [-v] [--ignore <glob>] [--rule <id>] [--disable <id>]\n                                [--fail-on error|warning|never] [--mutate <file> ...]\nhonest-tests mutate <file>      [--json] [--test-cmd <cmd>] [--max-mutants <n>] [--min-score <n>]\nhonest-tests rules              [--json]\nhonest-tests init <claude|codex|cursor> [--dir <repo>]\n```\n\nExit codes: `0` clean, `1` findings at/above `--fail-on` (default `error`) or score below `--min-score`,\n`2` usage/runtime error. JSON output schema is documented in\n[audit-procedure.md](https://github.com/mintwork/honest-tests/blob/main/skills/honest-tests/references/audit-procedure.md#phase-1--detect).\n\nhonest-tests audits its own suite in CI (`npm run dogfood` → zero findings) and mutation-checks its own\nrules. While building v1, `mutate src/rules/asserts-on-mock.ts` scored **50%** against our first rule\ntests — the tool found the gaps in the tests for the tool. After adding the missing negative cases it's\nat 84%, and `never-failing-assertion.ts` is at 100%; the remaining survivors are near-equivalent mutants\nsuch as early-exit optimizations.\n\n-  **pytest adapter** — the rule engine is language-neutral; an adapter only has to emit the`TestFileModel` ([src/adapters/python](https://github.com/mintwork/honest-tests/blob/main/src/adapters/python/index.ts) documents the mapping)\n- Parallel mutant runs and per-test coverage mapping (run only tests that execute the mutated line)\n-  SARIF output for GitHub code scanning; `--changed` mode for PR diffs\n-  More rules: oracle copied from output, re-implemented algorithm in test, weak `toHaveBeenCalled()`\n- Mocha, node:test, Bun, Deno; Go and Rust adapters\n- GitHub Action with a PR comment: \"this PR adds 12 tests, 4 can't fail\"\n\nPRs welcome — especially new rules with a bad/good fixture and pytest adapter work. See\n[CONTRIBUTING.md](https://github.com/mintwork/honest-tests/blob/main/CONTRIBUTING.md).", "url": "https://wpnews.pro/news/honest-tests-detect-fake-tests-and-prove-them-with-mutation-testing", "canonical_source": "https://github.com/mintwork/honest-tests", "published_at": "2026-11-14 22:52:55+00:00", "updated_at": "2026-11-14 23:23:35.757393+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-products"], "entities": ["Mintwork", "honest-tests", "Claude Code", "Codex", "Cursor", "@mintwork/honest-tests"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/honest-tests-detect-fake-tests-and-prove-them-with-mutation-testing", "markdown": "https://wpnews.pro/news/honest-tests-detect-fake-tests-and-prove-them-with-mutation-testing.md", "text": "https://wpnews.pro/news/honest-tests-detect-fake-tests-and-prove-them-with-mutation-testing.txt", "jsonld": "https://wpnews.pro/news/honest-tests-detect-fake-tests-and-prove-them-with-mutation-testing.jsonld"}}