# Honest-tests – detect fake tests and prove them with mutation testing > Source: > Published: 2026-11-14 22:52:55+00:00 **honest-tests** is an agent skill + CLI that stops Claude Code, Codex, Cursor and friends from writing fake tests — and proves which of your existing tests are fake by mutating your code and watching them stay green. ``` npx @mintwork/honest-tests audit . # find tests that can't fail npx @mintwork/honest-tests mutate src/pricing.ts # prove it: break the code, see who notices ``` Ask an agent to "add tests" and you get a wall of green checkmarks. Look closer: ``` js it('handles multiple items', () => { const result = subtotal(items); expect(result).toBe(result); // compares a value to itself }); it('applies a percentage discount', () => { applyDiscount(100, 10); // no assertion at all }); it('computes the discounted price', () => { try { expect(applyDiscount(100, 20)).toBe(70); // WRONG (it's 80) — and still passes } catch {} }); it('gives members free shipping', () => { const qualifies = vi.fn().mockReturnValue(true); expect(qualifies(10, true)).toBe(true); // tests the mock it just configured }); ``` Every one of these passes. Coverage goes up. Nothing is verified. ``` bash $ npx @mintwork/honest-tests audit test/pricing.test.ts test/pricing.test.ts 16:5 error Compares `result` to itself — this can never fail. tautological-assertion │ expect(result).toBe(result); 25:1 error Test "applies a percentage discount" has no assertions — it can only fail by crashing. no-assertions │ it('applies a percentage discount', () => { 39:7 error Assertion is inside a try/catch that swallows errors — when it fails, the catch eats the AssertionError and the test passes. swallowed-assertion │ expect(applyDiscount(100, 20)).toBe(70); // wrong! but the catch hides it 53:5 error Asserts on what mock `qualifies` returns (configured on line 52) — this verifies the mock, not your code. asserts-on-mock │ expect(qualifies(10, true)).toBe(true); ... 5 more 7 of 10 tests can't fail (or don't test what they claim). 1 file, 10 tests: 7 errors, 2 warnings. Run with --verbose for fixes. ``` Then the killer demo — **mutation testing, built in**. honest-tests flips operators, booleans, constants and return values in your source (on a temp copy), runs your tests, and shows every change nobody noticed: ``` bash $ npx @mintwork/honest-tests mutate src/pricing.ts 22 mutants survived — your tests don't actually check these lines: src/pricing.ts:15 if (percent < 0 || percent > 100) { ✗ `<` → `<=` ✗ `||` → `&&` ✗ `>` → `>=` ✗ `100` → `101` src/pricing.ts:22 return isMember || total >= 50; ✗ `>=` → `>` ✗ `50` → `51` ... Mutation score: 15.4% (4 killed, 0 timed out, 22 survived of 26; baseline 0.5s) js it('takes the percentage off the amount', () => { expect(applyDiscount(200, 25)).toBe(150); }); it('accepts the 0% and 100% boundaries', () => { expect(applyDiscount(200, 0)).toBe(200); expect(applyDiscount(200, 100)).toBe(0); }); it('rejects percentages outside 0..100', () => { expect(() => applyDiscount(200, -1)).toThrow(RangeError); expect(() => applyDiscount(200, 101)).toThrow(RangeError); }); it('starts free shipping exactly at 50 for non-members', () => { expect(qualifiesForFreeShipping(49.99, false)).toBe(false); expect(qualifiesForFreeShipping(50, false)).toBe(true); }); bash $ npx @mintwork/honest-tests audit test && npx @mintwork/honest-tests mutate src/pricing.ts ✓ 12 tests in 2 files: no fake tests found. Mutation score: 100% (26 killed, 0 timed out, 0 survived of 26; baseline 0.5s) ✓ Every mutation was caught. These tests actually test something. ``` Both projects live in [`fixtures/`](https://github.com/mintwork/honest-tests/blob/main/fixtures) — run `npm run demo:bad` / `npm run demo:good` yourself (`npm run demo:gif` re-renders the GIF above from the real CLI output). ``` /plugin marketplace add mintwork/honest-tests /plugin install honest-tests@honest-tests ``` Or drop just the skill into a repo: `npx @mintwork/honest-tests init claude` (writes `.claude/skills/honest-tests/`). Then ask: *"write tests for src/cart.ts"* or *"audit my tests"* — or use the slash commands `/honest-tests:audit [path]` and `/honest-tests:mutate `. ``` npx @mintwork/honest-tests init codex # appends an honest-tests section to AGENTS.md (idempotent) npx @mintwork/honest-tests init cursor # writes .cursor/rules/honest-tests.mdc (auto-attaches to test files) npx @mintwork/honest-tests audit . --json # machine-readable findings npx @mintwork/honest-tests mutate src/x.ts --min-score 80 # fail CI below 80% mutation score ``` The [skill](https://github.com/mintwork/honest-tests/blob/main/skills/honest-tests/SKILL.md) (Agent Skills format) teaches agents one rule — *a test is honest only if it fails when the behavior it names is broken* — plus two workflows: - **Writing tests:** name the bug each test catches, derive expected values independently, mock only boundaries, cover boundaries and error paths, then self-audit with`audit` +`mutate` before saying "done". Hard rules forbid the classic agent moves: weakening assertions to get green, changing the expected value to match output, mocking the unit under test,`try/catch` around asserts. - **Auditing tests** , in phases, like a security audit: 1. **Detect** —`honest-tests audit --json` plus a manual pass for smells static analysis can't see. 2. **Verify** — every finding is opened and marked*confirmed / false-positive / accepted* ; key findings are proven with surviving mutants. 3. **Report** — "N of M tests can't fail", confirmed findings, surviving mutants, next steps. 4. **Fix** — only when asked; re-run and show the before/after mutation score. References the agent loads on demand: [test smell catalog](https://github.com/mintwork/honest-tests/blob/main/skills/honest-tests/references/test-smells.md), [writing honest tests](https://github.com/mintwork/honest-tests/blob/main/skills/honest-tests/references/writing-honest-tests.md), [audit procedure](https://github.com/mintwork/honest-tests/blob/main/skills/honest-tests/references/audit-procedure.md), [mutation testing](https://github.com/mintwork/honest-tests/blob/main/skills/honest-tests/references/mutation-testing.md). | Rule | Severity | Catches | |---|---|---| | `no-assertions` | error | Test body never asserts — passes unless it crashes | | `tautological-assertion` | error | `expect(x).toBe(x)` ,`expect(f(1)).toBe(f(1))` ,`expect(true).toBe(true)` ,`toHaveLength(list.length)` | | `asserts-on-mock` | error | Asserting on the return value of a mock the test configured ( `vi.fn().mockReturnValue` ,`spyOn` ,`vi.mocked` ,`.mock.results` ) | | `asserts-on-own-literal` | error | Asserting on a literal the test just built: `const p = { service: 's3' }; expect(p.service).toBe('s3')` — common in runtime "type tests" | | `never-failing-assertion` | error | `expect(x).toBeTruthy` (never called), bare`expect(x)` , un-awaited`.resolves` /`.rejects` ,`typeof x` "defined",`length >= 0` ,`{}` truthy | | `swallowed-assertion` | error | Assertion inside `try` whose`catch` doesn't rethrow | | `catch-only-assertion` | error | Assertions only in `catch` — passes when nothing throws | | `mocked-unit-under-test` | error | `vi.mock('./cart')` inside`cart.test.ts` ; spying on and replacing the unit's own exports | | `focused-test` | error | `.only` ,`fit` ,`fdescribe` left in | | `excessive-mocking` | warning | 5+ module mocks in one file | | `snapshot-only` | warning | The only assertion is a snapshot | | `conditional-assertion` | warning | Assertion behind `if` /`&&` / ternary | | `skipped-test` | warning | `.skip` ,`xit` ,`xdescribe` | `npx @mintwork/honest-tests rules` prints the list. Suppress a deliberate case with a reason: ``` // honest-tests-disable-next-line no-assertions -- smoke test: module must import cleanly it('loads', () => { require('../src/plugin'); }); ``` (`honest-tests-disable-line ` and file-level `/* honest-tests-disable */` also work.) | Operator | Example | |---|---| | binary | `>=` →`>` ,`<` →`<=` ,`===` →`!==` ,`+` →`-` ,`*` →`/` | | logical | `&&` ↔`\|\|` | | assignment | `+=` ↔`-=` | | boolean | `true` ↔`false` | | negation | `!x` →`x` | | number | `50` →`51` ,`0` →`1` | | condition | `if (c)` →`if (true)` /`if (false)` | | return | `return x` →`return undefined` | **Safe by design:** the project is copied to a temp dir (`node_modules` symlinked), mutants are written only to the copy and restored after every run, and the copy is removed at the end — even on Ctrl-C. A red baseline aborts the run. Only test files that import the target — directly or through other local modules — are run (vitest/jest), once and with coverage off, so it's fast: 26 mutants in ~12s on the fixture. Imports through path aliases (`@/lib/x`) aren't followed yet; pass `--test-cmd` if your tests reach the target that way. ``` --test-cmd "" custom command; {files} expands to related test files --max-mutants cap (default 100) --timeout per-run timeout --min-score <0-100> exit 1 below this score --root project root honest-tests audit [paths...] [--json] [-v] [--ignore ] [--rule ] [--disable ] [--fail-on error|warning|never] [--mutate ...] honest-tests mutate [--json] [--test-cmd ] [--max-mutants ] [--min-score ] honest-tests rules [--json] honest-tests init [--dir ] ``` Exit codes: `0` clean, `1` findings at/above `--fail-on` (default `error`) or score below `--min-score`, `2` usage/runtime error. JSON output schema is documented in [audit-procedure.md](https://github.com/mintwork/honest-tests/blob/main/skills/honest-tests/references/audit-procedure.md#phase-1--detect). honest-tests audits its own suite in CI (`npm run dogfood` → zero findings) and mutation-checks its own rules. While building v1, `mutate src/rules/asserts-on-mock.ts` scored **50%** against our first rule tests — the tool found the gaps in the tests for the tool. After adding the missing negative cases it's at 84%, and `never-failing-assertion.ts` is at 100%; the remaining survivors are near-equivalent mutants such as early-exit optimizations. - **pytest adapter** — the rule engine is language-neutral; an adapter only has to emit the`TestFileModel` ([src/adapters/python](https://github.com/mintwork/honest-tests/blob/main/src/adapters/python/index.ts) documents the mapping) - Parallel mutant runs and per-test coverage mapping (run only tests that execute the mutated line) - SARIF output for GitHub code scanning; `--changed` mode for PR diffs - More rules: oracle copied from output, re-implemented algorithm in test, weak `toHaveBeenCalled()` - Mocha, node:test, Bun, Deno; Go and Rust adapters - GitHub Action with a PR comment: "this PR adds 12 tests, 4 can't fail" PRs welcome — especially new rules with a bad/good fixture and pytest adapter work. See [CONTRIBUTING.md](https://github.com/mintwork/honest-tests/blob/main/CONTRIBUTING.md).