Honest-tests – detect fake tests and prove them with mutation testing

honest-tests is an agent skill + CLI that stops Claude Code, Codex, Cursor and friends from writing fake tests — and proves which of your existing tests are fake by mutating your code and watching them stay green.

npx @mintwork/honest-tests audit .                       # find tests that can't fail
npx @mintwork/honest-tests mutate src/pricing.ts         # prove it: break the code, see who notices

Ask an agent to "add tests" and you get a wall of green checkmarks. Look closer:

it('handles multiple items', () => {
  const result = subtotal(items);
  expect(result).toBe(result);                  // compares a value to itself
});

it('applies a percentage discount', () => {
  applyDiscount(100, 10);                       // no assertion at all
});

it('computes the discounted price', () => {
  try {
    expect(applyDiscount(100, 20)).toBe(70);    // WRONG (it's 80) — and still passes
  } catch {}
});

it('gives members free shipping', () => {
  const qualifies = vi.fn().mockReturnValue(true);
  expect(qualifies(10, true)).toBe(true);       // tests the mock it just configured
});

Every one of these passes. Coverage goes up. Nothing is verified.

$ npx @mintwork/honest-tests audit test/pricing.test.ts

test/pricing.test.ts
  16:5    error   Compares `result` to itself — this can never fail. tautological-assertion
          │ expect(result).toBe(result);
  25:1    error   Test "applies a percentage discount" has no assertions — it can only fail by crashing. no-assertions
          │ it('applies a percentage discount', () => {
  39:7    error   Assertion is inside a try/catch that swallows errors — when it fails, the catch eats the AssertionError and the test passes. swallowed-assertion
          │ expect(applyDiscount(100, 20)).toBe(70); // wrong! but the catch hides it
  53:5    error   Asserts on what mock `qualifies` returns (configured on line 52) — this verifies the mock, not your code. asserts-on-mock
          │ expect(qualifies(10, true)).toBe(true);
  ... 5 more

7 of 10 tests can't fail (or don't test what they claim).
1 file, 10 tests: 7 errors, 2 warnings. Run with --verbose for fixes.

Then the killer demo — mutation testing, built in. honest-tests flips operators, booleans, constants and return values in your source (on a temp copy), runs your tests, and shows every change nobody noticed:

$ npx @mintwork/honest-tests mutate src/pricing.ts

  22 mutants survived — your tests don't actually check these lines:

  src/pricing.ts:15  if (percent < 0 || percent > 100) {
      ✗ `<` → `<=`
      ✗ `||` → `&&`
      ✗ `>` → `>=`
      ✗ `100` → `101`

  src/pricing.ts:22  return isMember || total >= 50;
      ✗ `>=` → `>`
      ✗ `50` → `51`
  ...

  Mutation score: 15.4% (4 killed, 0 timed out, 22 survived of 26; baseline 0.5s)
js
it('takes the percentage off the amount', () => {
  expect(applyDiscount(200, 25)).toBe(150);
});

it('accepts the 0% and 100% boundaries', () => {
  expect(applyDiscount(200, 0)).toBe(200);
  expect(applyDiscount(200, 100)).toBe(0);
});

it('rejects percentages outside 0..100', () => {
  expect(() => applyDiscount(200, -1)).toThrow(RangeError);
  expect(() => applyDiscount(200, 101)).toThrow(RangeError);
});

it('starts free shipping exactly at 50 for non-members', () => {
  expect(qualifiesForFreeShipping(49.99, false)).toBe(false);
  expect(qualifiesForFreeShipping(50, false)).toBe(true);
});
bash
$ npx @mintwork/honest-tests audit test && npx @mintwork/honest-tests mutate src/pricing.ts
✓ 12 tests in 2 files: no fake tests found.
  Mutation score: 100% (26 killed, 0 timed out, 0 survived of 26; baseline 0.5s)
  ✓ Every mutation was caught. These tests actually test something.

Both projects live in fixtures/ — run npm run demo:bad / npm run demo:good yourself (npm run demo:gif re-renders the GIF above from the real CLI output).

/plugin marketplace add mintwork/honest-tests
/plugin install honest-tests@honest-tests

Or drop just the skill into a repo: npx @mintwork/honest-tests init claude (writes .claude/skills/honest-tests/).

Then ask: "write tests for src/cart.ts" or "audit my tests" — or use the slash commands /honest-tests:audit [path] and /honest-tests:mutate <file>.

npx @mintwork/honest-tests init codex      # appends an honest-tests section to AGENTS.md (idempotent)
npx @mintwork/honest-tests init cursor     # writes .cursor/rules/honest-tests.mdc (auto-attaches to test files)
npx @mintwork/honest-tests audit . --json                 # machine-readable findings
npx @mintwork/honest-tests mutate src/x.ts --min-score 80 # fail CI below 80% mutation score

The skill (Agent Skills format) teaches agents one rule — a test is honest only if it fails when the behavior it names is broken — plus two workflows:

• Writing tests: name the bug each test catches, derive expected values independently, mock only boundaries, cover boundaries and error paths, then self-audit withaudit +mutate before saying "done". Hard rules forbid the classic agent moves: weakening assertions to get green, changing the expected value to match output, mocking the unit under test,try/catch around asserts.
• Auditing tests , in phases, like a security audit:
  1. Detect —honest-tests audit --json plus a manual pass for smells static analysis can't see.
  2. Verify — every finding is opened and markedconfirmed / false-positive / accepted ; key findings are proven with surviving mutants.
  3. Report — "N of M tests can't fail", confirmed findings, surviving mutants, next steps.
  4. Fix — only when asked; re-run and show the before/after mutation score.

References the agent loads on demand: test smell catalog, writing honest tests, audit procedure, mutation testing.

npx @mintwork/honest-tests rules prints the list. Suppress a deliberate case with a reason:

// honest-tests-disable-next-line no-assertions -- smoke test: module must import cleanly
it('loads', () => { require('../src/plugin'); });

(honest-tests-disable-line <rule> and file-level /* honest-tests-disable <rule> */ also work.)

Safe by design: the project is copied to a temp dir (node_modules symlinked), mutants are written only to the copy and restored after every run, and the copy is removed at the end — even on Ctrl-C. A red baseline aborts the run. Only test files that import the target — directly or through other local modules — are run (vitest/jest), once and with coverage off, so it's fast: 26 mutants in ~12s on the fixture. Imports through path aliases (@/lib/x) aren't followed yet; pass --test-cmd if your tests reach the target that way.

--test-cmd "<cmd>"   custom command; {files} expands to related test files
--max-mutants <n>    cap (default 100)          --timeout <ms>   per-run timeout
--min-score <0-100>  exit 1 below this score    --root <dir>     project root
honest-tests audit [paths...]   [--json] [-v] [--ignore <glob>] [--rule <id>] [--disable <id>]
                                [--fail-on error|warning|never] [--mutate <file> ...]
honest-tests mutate <file>      [--json] [--test-cmd <cmd>] [--max-mutants <n>] [--min-score <n>]
honest-tests rules              [--json]
honest-tests init <claude|codex|cursor> [--dir <repo>]

Exit codes: 0 clean, 1 findings at/above --fail-on (default error) or score below --min-score, 2 usage/runtime error. JSON output schema is documented in audit-procedure.md.

honest-tests audits its own suite in CI (npm run dogfood → zero findings) and mutation-checks its own rules. While building v1, mutate src/rules/asserts-on-mock.ts scored 50% against our first rule tests — the tool found the gaps in the tests for the tool. After adding the missing negative cases it's at 84%, and never-failing-assertion.ts is at 100%; the remaining survivors are near-equivalent mutants such as early-exit optimizations.

• pytest adapter — the rule engine is language-neutral; an adapter only has to emit theTestFileModel (src/adapters/python documents the mapping)
• Parallel mutant runs and per-test coverage mapping (run only tests that execute the mutated line)
• SARIF output for GitHub code scanning; --changed mode for PR diffs
• More rules: oracle copied from output, re-implemented algorithm in test, weak toHaveBeenCalled()
• Mocha, node:test, Bun, Deno; Go and Rust adapters
• GitHub Action with a PR comment: "this PR adds 12 tests, 4 can't fail"

PRs welcome — especially new rules with a bad/good fixture and pytest adapter work. See CONTRIBUTING.md.