Your AI coding agent just announced:
✅ Done! Everything works. All tests passed.
Excellent. Small problem: your project doesn't have any tests.
So you write a better prompt. “Double check your work.” “Follow best practices.” “Act like a senior engineer.” You might as well add “pretty please” and offer it a cookie.
The agent says “Absolutely!” and breaks something else.
Here's my unpopular opinion: we spend too much time trying to write perfect prompts for agents working inside terrible engineering environments.
A better prompt can help. But a prompt can't replace tests, project context, architectural boundaries, or an actual feedback loop.
That's the idea behind harness engineering: design the environment around a coding agent so its work can be guided, checked, and corrected.
And yes, we're building one. In about 15 minutes, with Node.js, no paid API, and no npm dependencies. The harness works with an existing coding agent that can read files and run terminal commands; we're not building another agent framework.
What you'll build:
AGENTS.md to explain the repository and its rules.npm run check) that provides real feedback.
A prompt says: “Don't break existing functionality.”
A harness says: “Here are the regression tests. Run them.”
A prompt says: “Stick to the architecture.”
A harness says: “Read the architecture docs, and here's a check that rejects forbidden imports.”
A prompt says: “Make sure it works.”
A harness says: “Show me the exit code.”
Not the same thing. Also, not enemies: a good prompt tells the agent what you want; a harness gives it an environment in which progress is observable and verifiable.
Task + constraints
|
v
Coding agent <----- failed check output
| ^
v |
Code changes --> npm run check
|
pass / fail
|
Human review
Let's make that real before the buzzword develops its own certification program.
Prerequisites: Node.js 22+, Git, and a coding agent with access to your local repository (Codex, Claude Code, or any equivalent). You can complete the entire exercise manually too.
mkdir agent-harness-lab && cd agent-harness-lab
npm init -y
mkdir -p src test scripts docs .github/workflows
Replace package.json with:
{
"name": "agent-harness-lab",
"private": true,
"type": "module",
"engines": { "node": ">=22" },
"scripts": {
"test": "node --test",
"guardrails": "node scripts/guardrails.js",
"check": "npm run guardrails && npm test"
}
}
Create src/pricing.js:
export function calculateTotal(items, discountPercent = 0) {
const subtotal = items.reduce(
(sum, item) => sum + item.price * item.quantity,
0
);
return Math.round(
(subtotal * (1 - discountPercent / 100)) * 100
) / 100;
}
Looks fine, right? So does a chair with three legs, until someone sits down.
This function accepts negative quantities, NaN prices, and 120% discounts. Let's make those failures impossible to ignore.
Create test/pricing.test.js:
import test from 'node:test';
import assert from 'node:assert/strict';
import { calculateTotal } from '../src/pricing.js';
test('calculates normal totals', () => {
assert.equal(calculateTotal([{ price: 12.5, quantity: 2 }]), 25);
});
test('applies a valid discount', () => {
assert.equal(calculateTotal([{ price: 100, quantity: 1 }], 15), 85);
});
test('rejects negative quantities', () => {
assert.throws(
() => calculateTotal([{ price: 10, quantity: -1 }]),
RangeError
);
});
test('rejects discounts above 100%', () => {
assert.throws(
() => calculateTotal([{ price: 10, quantity: 1 }], 120),
RangeError
);
});
test('rejects non-finite prices', () => {
assert.throws(
() => calculateTotal([{ price: NaN, quantity: 1 }]),
TypeError
);
});
test('does not mutate the cart', () => {
const items = [{ price: 10, quantity: 2 }];
const before = structuredClone(items);
calculateTotal(items);
assert.deepEqual(items, before);
});
Run:
npm test
Expected result: 3 tests pass, 3 fail. That's good news. We've converted a vague request into a measurable target.
And yes, this example uses JavaScript floating point numbers to stay focused on agent workflows. For real money, use integer minor units or a decimal strategy with documented rounding rules.
Create AGENTS.md at the repository root:
## Context
- Node.js 22+, ESM, no runtime dependencies.
- Pricing logic: src/pricing.js
- Tests: test/pricing.test.js
- Architecture notes: docs/architecture.md
## Rules
- Preserve calculateTotal(items, discountPercent = 0).
- Do not change tests or guardrails to make a check pass.
- Do not add packages without approval.
- Do not access secrets, deploy, publish, or push changes.
- Keep changes scoped to the task.
## Verification
- Run: npm run check
- Report the actual result, including failures.
- Review git diff before claiming completion.
## Definition of done
- Requested behavior works.
- npm run check passes.
- No unrelated changes.
- Remaining risks are listed for human review.
Create docs/architecture.md:
The public API is calculateTotal(items, discountPercent = 0).
Production code belongs in src/; tests belong in test/.
The pricing function must be pure and must not perform I/O.
Invalid inputs must throw, not silently produce nonsense totals.
One subtle point: Codex recognizes AGENTS.md as a repository instruction file, but agent products differ. For other tools, verify their convention or explicitly tell the agent to read the file. A markdown filename isn't telepathy.
And this file is guidance, not a security boundary. If an agent has permission to delete production resources, writing “please don't” in Markdown won't revoke that permission.
Create scripts/guardrails.js:
import { readFileSync, readdirSync } from 'node:fs';
const pkg = JSON.parse(readFileSync('package.json', 'utf8'));
for (const field of ['dependencies', 'optionalDependencies', 'peerDependencies']) {
if (Object.keys(pkg[field] ?? {}).length) {
throw new Error(`${field} are not allowed in this lab`);
}
}
for (const file of readdirSync('src').filter((f) => f.endsWith('.js'))) {
const source = readFileSync(`src/${file}`, 'utf8');
// Narrow teaching example: no imports in src/*.js.
if (/\b(?:import|require)\s*(?:\(|[{\'"*])/m.test(source)) {
throw new Error(`Unexpected import in src/${file}`);
}
}
console.log('Architectural guardrails passed.');
This checks two specific constraints: no declared runtime dependencies and no obvious imports in top level src/*.js files. It is not a general security scanner, and regex is not a robust JavaScript parser. For serious projects, use AST aware lint rules and dependency policies.
Now run:
npm run check
The guardrail passes; the three behavior tests still fail. That's the harness doing its job, not an agent having a bad day.
Open the repository with your coding agent and paste:
Read AGENTS.md and docs/architecture.md before editing.
Fix src/pricing.js so all existing pricing tests pass.
Constraints:
- Preserve calculateTotal(items, discountPercent = 0).
- Negative quantities: RangeError.
- Discounts outside 0..100: RangeError.
- Non finite prices: TypeError.
- Keep the function pure.
- Change only src/pricing.js.
- Do not install dependencies or weaken tests.
Run npm run check and inspect its actual output.
If it fails, fix the cause and rerun, up to 3 attempts.
Do not claim success unless the command succeeds.
Report files changed, check results, and remaining risks.
Notice what changed: the prompt is still important, but it now points to an independent contract.
A possible passing implementation is:
export function calculateTotal(items, discountPercent = 0) {
if (!Array.isArray(items)) {
throw new TypeError('items must be an array');
}
if (!Number.isFinite(discountPercent) ||
discountPercent < 0 || discountPercent > 100) {
throw new RangeError('discountPercent must be between 0 and 100');
}
const subtotal = items.reduce((sum, item) => {
if (!item || typeof item !== 'object' || !Number.isFinite(item.price)) {
throw new TypeError('price must be finite');
}
if (!Number.isInteger(item.quantity) || item.quantity < 0) {
throw new RangeError('quantity must be nonnegative');
}
if (item.price < 0) {
throw new RangeError('price must be nonnegative');
}
return sum + item.price * item.quantity;
}, 0);
return Math.round(subtotal * (1 - discountPercent / 100) * 100) / 100;
}
Check it yourself:
npm run check
git diff --check
git diff
Expected after applying that fix: all 6 tests pass. Passing tests don't prove every possible behavior is correct, and git diff still needs human review. But at least we're discussing evidence instead of vibes.
The useful loop is:
Edit -> Run checks -> Read real failure -> Fix -> Run again
|
Stop after 3 attempts
and request review
An agent that retries forever isn't autonomous. It's a subscription renewal strategy.
Create .github/workflows/check.yml:
name: Check
on: [push, pull_request]
permissions:
contents: read
jobs:
verify:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '22'
- run: npm run check
- run: git diff --check
Open a pull request. Now CI runs the same checks as your local agent.
For production use, pin third party GitHub Actions to vetted commit SHAs, enforce required checks and reviews, and protect tests and guardrails (for example with CODEOWNERS and branch rules). An agent can make its own evaluation look green by weakening the evaluator if you let it edit everything.
Also: run agent executed code in a suitably restricted environment. Tests don't replace sandboxing, permissions, or approval gates for destructive actions.
Before, your quality strategy was:
“Please be careful.”
Now you have:
| Problem | Harness response |
|---|---|
| Agent invents test results | Run npm run check and read the actual exit code |
| Agent forgets project conventions | AGENTS.md + architecture notes |
| Agent introduces forbidden imports | Architectural guardrail fails |
| Agent breaks behavior | Regression tests fail |
| Agent declares victory early | CI and human review remain separate gates |
You can also preserve longer tasks using a short docs/HANDOFF.md with the objective, files changed, real verification results, blockers, and next action. Don't paste an entire chat transcript and call it documentation.
Take five comparable tasks. Use the same model, starting repository state, allowed attempts, and time budget. Compare prompt-only versus prompt + harness. Track:
Avoid pretending a single toy example proves a universal performance gain. The point of this lab is to demonstrate the mechanism, not manufacture a benchmark.
Prompt engineering isn't dead. But “write a better prompt” is not a complete software engineering strategy.
If you give an agent no tests, no context, no guardrails, and unlimited permissions, it can fail very efficiently. Congratulations: you've automated chaos.
A harness makes failure visible and gives the agent a structured chance to recover. That doesn't guarantee correctness. It gives you a way to evaluate it.
The goal isn't an agent that confidently announces:
Everything is production ready! 🚀
It's a workflow that can prove:
✓ Architecture checks passed
✓ 6 behavior tests passed
✓ Changes reviewed before merging
→ Ready for human approval
What was the last thing your AI coding agent confidently broke? And more importantly: what single automated check would have caught it?
Tell me in the comments. I collect these stories the way some people collect Pokémon.
Further reading: