- iteratively build a harness as you go
TL;DR I present the motivation for a declarative hooks framework for agent harnesses, with a DSL that agents use to constantly improve their harness. Check it out live at captain-hook.
Please don’t rm… #
One of the unfortunate externalities of prompt engineering working so damn well is we often forget to reach for other tools in our toolbox. The canonical example is the AGENTS.md with “do NOT run rm on any potentially empty variable”… only to watch codex rm -rf "$DEFINITELY_NOT_EMPTY/*". This seems like a bit of a strawman but it’s exactly what we saw back in July.
Of course, this is why the (oft-under-appreciated) hooks system exists. We’ll use Claude Code in our examples, since its hooks are the most robust and familiar, but this applies to any harness. Implementing a hook blocking any command with rm is pretty simple:
{
"hooks": {
"PreToolUse": [
{
"matcher": "Bash",
"hooks": [
{
"type": "command",
"if": "Bash(rm *)",
"command": "echo 'Blocked: rm is not allowed' >&2; exit 2"
}
]
}
]
}
}
This covers our asses, but it’s restrictive and adds little beyond simple sandboxing. What we’d really like is a DSL that adds some sugar on top of this raw JSON and bash-fu. Maybe something like:
from hooks import on, Event, Tool
@on(Event.PreToolUse, only_if=[Tool("Bash")])
def check_command(evt):
if "rm" in evt.tool_input["command"]:
return evt.block("rm is not allowed")
Admittedly, this looks like more work than the first option, but it unlocks some powerful things! For example, let’s add a command-parsing layer:
@on(Event.PreToolUse, only_if=[Command("rm")])
def check_command(evt):
return evt.block("rm is not allowed")
which with a bit of sugar becomes:
from hooks import block_command
block_command(
["rm"],
reason="rm is banned in this repo",
hint="Use `trash` instead if on macOS",
)
or if we wanted to make that hint actionable:
@on(Event.PreToolUse, only_if=[Command("rm")])
def check_command(evt):
return evt.cmd.sub("rm", "trash") if evt.ctx.darwin else evt.block("rm is not allowed")
we can get really fancy now, if we want:
@on(Event.PreToolUse, only_if=[Command("rm")])
def check_command(evt):
if len(evt.cmd.call("rm").target()) > 10:
return evt.warn("You're deleting more than 10 files, FYI")
BLOCKED: rm is banned in this repo. Use `trash` instead if on macOS.BLOCKED: rm is banned in this repo. Use `trash` instead if on macOS.
plain rm call: blocked, with a hint for Claude
Intercepting edits #
Ok, so we can block commands. Whoop-de-doo. Like I said, fancy sandboxing.
Well, not quite. A prompt is advisory: the model is free to ignore it, and empirically it does. A sandbox is a mute wall: the command dies, the agent learns nothing, and it retries blind. A hook is the only one of the three that closes the loop — the hint, the sub, the warn all land back in the model’s context, steering the next action instead of just denying this one.
rm -rf "$BUILD_DIR/*"
rm -rf "$BUILD_DIR/*"
rm -rf "$BUILD_DIR/*"
However, we can build on this DSL to start intercepting more than bash commands. Say, edits.
Most languages have community-standardized linting tools (eslint, black, etc) which mostly act by parsing the AST, looking for forbidden patterns, then reporting back. In theory, this should make it quite easy to add custom linting rules… in reality not so much. Here’s a fun challenge:
With a hooks framework, this becomes stupid simple. If we were writing everything by hand, we would do something like:
import ast
def added_prints(tree: ast.AST) -> set[int]:
return {
node.lineno
for node in ast.walk(tree)
if isinstance(node, ast.Call)
and isinstance(node.func, ast.Name)
and node.func.id == "print"
}
@on(Event.Edit)
def prevent_prints(evt):
if (lines := added_prints(ast.parse(evt.file.open()))):
evt.block(f"Forbidden prints found on lines: {lines}")
if we move the AST parsing into the framework and add a shortcut called lint:
from hooks import lint
lint(
added_prints,
message="This edit added a print() call: {violations}",
)
We use the fantastic ast-grep under the hood to parse across any language, so we can actually skip defining added_prints altogether and go after the JS offenders while we’re at it:
lint(
pattern="console.log($$$)",
message="This edit added a console.log() call: {violations}",
lang="ts",
)
This really works! You can go one step further and just fix it automatically:
from hooks import rewrite_code
rewrite_code("print($MSG)", "logger.info($MSG)")
PreToolUse ❯ rewrite_code("print($MSG)", "logger.info($MSG)")
→ rewrote apply_patch.py:2, apply_patch.py:5
Another great use case for this hook is to prevent the LLM from type-widening when it gets frustrated. However, this one requires some judgment. Luckily, we can bake (small) LLM calls into our framework pretty easily:
@on(Event.Edit)
def block_type_widening(evt):
if evt.edit.introduces("...") and not evt.llm("is this a throwaway file or test?"):
evt.block("Do not use Any as an escape hatch")
or with a bit of sugar:
from hooks import Introduced, llm_nudge
llm_nudge(
"Fire if the code is non-trivial (not a test or throwaway), and there is a stronger type that can be used.",
message="Do not use Any as an escape hatch: {reasoning}.",
contexts=[Introduced(pattern="def $F($$$) -> Any:")],
only_if=[Tool("Edit")],
)
Changing Introduced to Matches would prevent the Any s from getting introduced in the first place.
store.py
widens the return type
1-def load(path: str) -> Config:1+def load(path: str) -> Any:2 raw = read_text(path)3 return Config.parse(raw)
nudge1 match
store.py
nudge 1 match
cache.py
handles cache misses
1 STORE: dict[str, Config] = {}2 3 def fetch(key: str) -> Any:4- return STORE.get(key)4+ hit = STORE.get(key)5+ if hit is None:6+ raise KeyError(key)7+ return hit
silentno match
cache.py
silent no match
llm_nudge(
message="Do not use Any as an escape hatch: {reasoning}.",
contexts=[Introduced(pattern="def $F($$$) -> Any:")],
)
store.py:1 → Do not use Any as an escape hatch: load returns a Config, so the annotation can stay Config.
cache.py → silent
Beyond tool calls #
Stop gates
Another classic CLAUDE.md decree is some variation on “review your code when done implementing”. You could stick that in a Stop hook:
{
"hooks": {
"Stop": [
{
"hooks": [
{
"type": "command",
"command": "echo 'Did you review your code against STYLEGUIDE.md?' >&2; exit 2"
}
]
}
]
}
}
You might even be using /loop, which is basically a Stop hook that says “go look at the original prompt and see if you’re actually done”. These both hint at something powerful: steering Claude at the turn boundary. However, like before, plain-JSON hooks leave you pretty limited. With a DSL, however, the sky is the limit!
@on(Event.Stop)
def ensure_code_reviewed(evt):
if not evt.transcript.read_file("STYLEGUIDE.md"):
return evt.block("Consult the styleguide and review your code before ending your turn.")
for f in evt.ctx.changed_files:
if evt.t.last_read(f) < evt.t.last_write(f):
return evt.block(f"You must review {f.path}.")
@on(Event.Stop)
turn
gate
block
- STYLEGUIDE.mdread
- hooks.pyreviewed
- tests.pyunreviewed
You must review tests.py.You must review tests.py.
block: You must review tests.py.
Approvals
Event.PermissionRequest + a small LLM lets you re-implement Claude Code’s auto mode with your own criteria (the LLM here also gets a copy of claude auto-mode defaults). For example, if we want our subagents to have full read-only autonomy, but apply more scrutiny to our main agent:
from hooks import FromSubagent, llm_approve
llm_approve(
"safe teammate commands",
rubric="Read-only commands and test runs are safe.",
only_if=[FromSubagent()],
)
or, a more fun --dangerously-skip-permissions:
from hooks import LambdaCondition, approve
from datetime import datetime
approve(
"only during daylight hours",
only_if=[LambdaCondition(lambda evt: datetime.now().hour < 20)],
)
llm_approve("safe teammate commands")
rubric: Read-only commands and test runs are safe.
allowruns without asking
ls -la
allow runs without asking
$ ls -la
hook: llm_approve("safe teammate commands")
→ allow · read-only, rubric matches
approve("only during daylight hours")
only_if: LambdaCondition(lambda evt: datetime.now().hour < 20)
4:00 PM
before 8:00 PM: allow · from 8:00 PM: ask
allowruns without asking
4:00 PM
allow runs without asking
hook: approve("only during daylight hours")
datetime.now().hour < 20 → True
→ allow · 4:00 PM, nothing to answer
From responding to steering #
Correcting the LLM helps, but you’ll get more out of your mini-harness (I said the H-word!) by steering proactively, not responsively. This also helps a lot with token costs and overall iteration speed.
A great example that gets triggered for almost every agent at Aneta is the notion of iterative narrowing. By default, when the agent is running a test that fails, it will keep using that test to check its fix. Logical, until that test is a giant integration test that takes 10 minutes to run. So, we give it a skill called iterative-narrowing, and steer it towards that skill whenever we detect it thrashing. A smaller LLM can do the “is it thrashing” classification pretty well, but to avoid slowing down every tool call, we use some cheap NLP signals as a high-pass filter (with a veto clause to bail early if the agent is already doing the right thing).
from hooks import Clause, NlpSignal, Phrase, Signals, UsedSkill
llm_nudge(
"Decide whether the agent is iterating blind (thrashing)"
" or appropriately narrowing/investigating",
message="You are thrashing: {reasoning}. Narrow the failing test with /iterative-narrowing before running it again.",
signals=Signals(
[
NlpSignal(
clauses=[
Clause(verb=Phrase("try", "rerun"), adj=Phrase("again")),
Clause(verb=Phrase.expand("retry", pos="v"), tense="prospective"),
],
weight=2,
),
NlpSignal(
clauses=[
Clause(noun=Phrase("test", "suite"), verb=Phrase("run", "rerun"))
],
weight=1,
),
],
threshold=3,
window=10,
scope="window",
vetoes=[
NlpSignal(
clauses=[
Clause(
verb=Phrase("narrow", "bisect"), noun=Phrase("test", "repro")
)
],
)
],
),
skip_if=(UsedSkill("iterative-narrowing"),),
)
score 2/3
quiet under threshold
- +2Let me try running it againtry/rerun + again
filter 2/3 over the last 10 messages
under threshold
no classifier callfilter 2/3 over the last 10 messages
under threshold
no classifier call
Tests! #
The most annoying part of the settings.json hooks is that you’ve got absolutely no clue if it does what you want until you fire up a session and test it manually. Which is.. a lil scary when it’s your rm hook. It’s GitHub Actions yaml all over again. With a DSL, inline tests are simple to add; running them is as simple as uvx capt-hook test.
block_command(
["rm"],
reason="rm is banned in this repo",
hint="Use `trash` instead if on macOS",
tests={
Input(command='rm -rf "$DEFINITELY_NOT_EMPTY/*"'): Block(),
Input(command="trash old-builds/"): Allow(),
},
)
This is a fairly obvious case but you can imagine how useful this gets as your hook complexity scales.
inline tests
2 tests · 0 failed
block_command(["rm"], reason=…, hint=…)
$ uvx capt-hook test
PASS no_rm:hook_8cf20307:Input(command='rm -rf "$DEFINITELY_NOT_EMPTY/*"')
PASS no_rm:hook_8cf20307:Input(command='trash old-builds/')
2 tests: 2 passed, 0 failed, 0 errors, 0 skipped$ uvx capt-hook test
PASS no_rm:hook_8cf20307:Input(command='rm -rf "$DEFINITELY_NOT_EMPTY/*"')
PASS no_rm:hook_8cf20307:Input(command='trash old-builds/')
2 tests: 2 passed, 0 failed, 0 errors, 0 skipped
Let the agent build the harness #
The framework is great and all, but we don’t write code by hand these days. A good DSL is designed from the ground up to let agents write it. Our framework takes it one step further, using—you guessed it—hooks!
captain-hook · github.com/yasyf/app [watching] PR slots 1/2promote at 3 sessions across 2 days · max 2 open PRsWATCHING(building toward the bar)2█░░hooks.style:nudge_1: stop firing on its own output1/3 sessions · 0/2 days██░block pnpm install from the repo root2/3 sessions · 2/2 daysELIGIBLE(a PR opens next session)1███run the test suite before every commit3/3 sessions · 2/2 daysPR OPEN(awaiting your review)1███use uv instead of pip#12ACCEPTED(PR merged)1███use loguru instead of print#7reviewer idle · next scan at session end
eligible rules wait for the next session and a free PR slot
If you haven’t figured it out by now, this not-so-hypothetical framework is completely real; substitute hooks for captain_hook in all the above code and it actually runs. At the end of each session, a SessionEnd hook kicks off an asynchronous review of your interactions with the agent, keeping track of any corrections or steering you had to do. It stores this in a local SQLite database, deduped by semantic similarity. When it determines a correction is more than a one-off (the default bar: the same deduped correction across 3 distinct sessions spanning at least 2 days, every knob an env var), it opens a PR with a hook. All you have to do is merge, and your harness is now a little better tuned for your workflow. Repeat over weeks and months, and you end up with a lightweight, hyper-tuned harness layered over Claude Code that took you zero effort to build.
session 1 · day 1 · you say: don't run pnpm install from the repo root
block_command(
["pnpm", "install"],
hint="cd into the workspace first",
)SessionEnd fires detached · your prompt is already back
no rule yet · the reviewer is still counting
Try it out yourself: