The task looked trivial on paper: add a rate limiter to a small Python service and update three call sites. I handed it to a coding agent running on MonkeyCode, an open-source project with free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I did what most engineers would do with a ten-million-token allowance: give the agent a generous budget and walk away.
Three hours later I returned to a two-thousand-line diff for what should have been a forty-line change. The test suite was green, which made the failure harder to explain, but the agent's log told a clearer story. The same file had been edited fourteen times, and each edit appeared to revert the previous one before adding something new. The agent was oscillating between two designs, and nothing in my setup was designed to notice.
My first hypothesis was prompt ambiguity, because the task description did leave room for interpretation about where the limiter should live. I rewrote the prompt with explicit constraints, pinned the exact function names, and added a sentence demanding a minimal diff. The second run was faster, but the log showed the same oscillation pattern, which ruled out the prompt as the primary cause.
My second hypothesis was model quality, and I was ready to blame the free tier until I looked at the evidence. The agent's reasoning trace showed that each design change was locally reasonable; the problem was that the agent had no reason to stop exploring. It kept finding marginal improvements, and each one invalidated an assumption from the previous iteration, so the file flipped like a pendulum.
The root cause was not the model and not the prompt; it was the absence of a budget as a first-class constraint. A ten-million-token allowance is generous enough that an agent can treat it as infinite, and my harness gave the agent no termination criterion beyond "finish the task." Without a stopping signal, the agent optimized for an unstated goal: a perfect solution rather than a correct one. The real bug lived in my workflow, which is the most useful kind of bug to find.
The fix was a small Python harness that treats token budget the way a test treats an assertion. It caps the run, measures the diff, and fails loudly when the agent exceeds the cap or oscillates. I wrote it in about forty lines, and it is reproducible on any machine with Python and git.
#!/usr/bin/env python3
"""budget_harness.py - cap an agent run and detect edit oscillation."""
import argparse
import subprocess
import sys
import time
def estimate_tokens(text: str) -> int:
return len(text) // 4
def edits_per_file(diff: str) -> dict[str, int]:
files: dict[str, int] = {}
current = None
for line in diff.splitlines():
if line.startswith("+++ b/"):
current = line[6:]
files[current] = 0
elif current and line.startswith("+") and not line.startswith("+++"):
files[current] += 1
return files
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--task", required=True)
parser.add_argument("--budget-tokens", type=int, default=150_000)
parser.add_argument("--max-edits-per-file", type=int, default=8)
parser.add_argument("--agent-cmd", required=True)
args = parser.parse_args()
if estimate_tokens(args.task) > args.budget_tokens:
sys.exit("Task itself exceeds the token budget.")
start = time.monotonic()
proc = subprocess.run(args.agent_cmd, shell=True, capture_output=True, text=True)
elapsed = time.monotonic() - start
used = estimate_tokens(proc.stdout + proc.stderr)
print(f"elapsed={elapsed:.1f}s estimated_tokens={used}")
if used > args.budget_tokens:
sys.exit(f"Token budget exceeded: {used} > {args.budget_tokens}")
diff = subprocess.run(["git", "diff"], capture_output=True, text=True).stdout
offenders = {f: n for f, n in edits_per_file(diff).items()
if n > args.max_edits_per_file}
if offenders:
sys.exit(f"Oscillation detected: {offenders}")
print("Budget OK, no oscillation detected.")
return 0
if __name__ == "__main__":
raise SystemExit(main())
The harness works in three steps that map directly to the three symptoms I observed. First, it estimates the tokens in the agent's output and exits with a nonzero code when the run exceeds the cap. That turns a silent cost problem into a visible failure. Second, it parses git diff
and counts added lines per file, so a file edited more than eight times becomes an oscillation alarm. Third, it prints the elapsed time and estimated token count on every run, which gives you a baseline for what a normal task should cost.
I ran the same rate-limiter task with a one-hundred-fifty-thousand-token cap and an eight-edit-per-file limit, and the agent finished in a single pass with a forty-one-line diff. The interesting part was the second run, where I deliberately set the cap too low. The harness exited with a clear message, and I could inspect the partial log to see where the agent had started to wander. That failure mode is the one you want, because it converts an unbounded process into a bounded experiment.
The reusable lesson is that a free allowance is still a budget, and a budget that is never asserted is not a budget at all. Treat token limits like test assertions: define them before the run, enforce them during the run, and fail loudly when they are violated. Add an oscillation detector to your diff review, because an agent that edits the same file repeatedly is usually stuck in a local search loop.
This approach has real limitations, and it is not for every team. A hard token cap will break legitimate large refactors that genuinely need hundreds of thousands of tokens, so the harness belongs in a per-task configuration rather than a global policy. The character-based token estimate is a heuristic, not a tokenizer, and it measures only the visible output, not the hidden reasoning tokens that many agents consume. No budget harness fixes model quality; it only makes the cost of a bad model visible before the damage reaches your main branch.
If you want to see the oscillation pattern for yourself, the same reproduction takes about an hour on MonkeyCode's free server. The harness above will show you the failure before the diff does. The script lives in my dotfiles now, and it has caught two more runaway diffs since that first incident. A generous token allowance is a gift, but the only safe way to use a gift is to know exactly where it ends.