How to Fix LLM Agent Stagnation via Harness-Policy Recursion

Recursive harness-policy loops such as Google's Harness² and the Self-HPO framework break the performance ceiling of frozen-model agents, delivering 5-10% absolute gains on coding, reasoning and tool-use benchmarks, according to an October 19, 2026 analysis. Google's Harness² benchmark (October 2026) found a frozen GLM-5 scoring only 11.5 points on the WorkBuddy-Bench suite despite aggressive prompt engineering, which the analysis attributes to a static harness rather than the model's knowledge. The approach treats the harness — prompt template, tool wrappers and file mutations — as a learnable component co-optimized with the policy, with Self-HPO alternating between harness proposal (H-step) and policy training (P-step) as described in arXiv paper 2610.22791.

Fix LLM Agent Stagnation with Harness‑Policy Recursion October 19, 2026· 8 min read TL;DR:Recursive harness‑policy loops such as Google’s Harness² and the Self‑HPO framework break the performance ceiling of frozen‑model agents. By treating the harness the code that mediates between the LLM and its environment as a learnable component and co‑optimising it with the policy, practitioners have reported 5‑10 % absolute gains on a variety of coding, reasoning, and tool‑use benchmarks. 1. Why Frozen LLM Agents Stagnate 1.1 The “frozen backbone” myth Most commercial LLM‑agent platforms e.g., OpenAI’s function‑calling agents, Anthropic’s Claude‑tool use ship a static language model that is never fine‑tuned after release. The model’s internal knowledge is indeed impressive, but its ability to act depends entirely on the harness—the prompt template, the set of tool wrappers, the file‑system layout, and any pre‑ or post‑processing logic that the developer writes. When the harness is handcrafted once and left unchanged, two failure modes dominate: Blind‑spot reinforcement – The agent repeatedly receives the same kind of partial information and never learns to ask for the missing piece. Tool‑use ossification – If a tool e.g., a web search API is never invoked, the model never discovers when that tool would be useful, and the policy remains stuck in a “do‑nothing” loop. Empirical evidence from Google’s Harness² benchmark Oct 2026 shows a frozen GLM‑5 achieving only 11.5 pts on the WorkBuddy‑Bench suite despite aggressive prompt engineering. The bottleneck is not the model’s knowledge; it is the static harness that fails to expose the model’s blind spots. 1.2 Three converging research threads Thread Core Idea Representative Work -------- ----------- --------------------- Harness as a learnable component Treat the harness as a meta‑agent that can be edited, versioned, and optimized. Google Harness² open‑source repo Self‑HPO H‑step / P‑step Alternate between generating new harnesses H‑step and training the policy under those harnesses P‑step . “Self‑Evolving Coding Agents via Harness‑Policy Optimization” arXiv 2610.22791 Curriculum via skill withdrawal Remove skills that no longer contribute to variance, forcing the orchestrator to rediscover useful capabilities. R²Orch arXiv 2610.22740 All three agree: static harnesses are the primary cause of stagnation, and recursive co‑evolution of harness and policy is the antidote. 2. Foundations of the Recursive Harness‑Policy Loop 2.1 Formalising the harness A harness can be expressed as a lightweight JSON schema: json { "prompt template": "You are a coding assistant. Use the provided files to answer.\n{question}\n", "tools": {"name": "web search", "api": "https://search.example.com"}, {"name": "python executor", "api": "grpc://exec:50051"} , "file mutations": {"action": "create", "path": "tmp/main.py", "content": ""}, {"action": "append", "path": "tmp/main.py", "content": " helper functions"} } ✔️prompt template – The top‑level system prompt that frames the LLM’s role and injects the current task. ✔️tools – A list of tool descriptors that the LLM can invoke via function‑calling or tool‑use syntax. ✔️file mutations – Declarative operations that modify the on‑disk context before the next model invocation e.g., creating a new source file, writing a test case . Because the spec is pure data, it can be versioned, diffed, and even generated by the LLM itself. 2.2 The two‑phase Self‑HPO cycle H‑step Harness proposal ✔️Run a diagnostic suite a set of validation tasks with the current harness. ✔️Collect failure cases tasks where the reward falls below a threshold . ✔️Prompt the frozen LLM or a small “proposal model” to generate candidate harness edits that could address the observed failures. ✔️Optionally, sample several candidates and rank them with a lightweight surrogate model. P‑step Policy update ✔️Choose the top‑k harness candidates. ✔️For each candidate, run a reinforcement‑learning RL or supervised fine‑tuning loop where the frozen LLM is conditioned on the candidate harness. ✔️Compute a composite reward: ✔️Task success e.g., Pass@k, exact match, BLEU . ✔️Stability penalty large context churn is discouraged . ✔️Tool‑efficiency bonus reward using fewer API calls . ✔️Update the policy parameters e.g., via PPO while keeping the base LLM weights frozen. The loop repeats until marginal improvement falls below a pre‑defined epsilon often 0.5 % on the target metric . 2.3 Adding an outer curriculum: R²Orch R²Orch introduces a skill‑withdrawal step that sits outside the H‑/P‑step cycle: ✔️Skill set = the current list of tools and file‑mutation primitives. ✔️For each skill, evaluate the model without that skill on a held‑out evaluation set. ✔️If the performance delta is < ε e.g., 2 % absolute , drop the skill. ✔️The reduced skill set is fed back into the next H‑step, forcing the harness‑proposal model to re‑invent useful abstractions. This outer loop prevents advantage collapse the situation where the policy learns to ignore the harness because it no longer provides a learning signal and keeps the system exploring new capabilities. 3. End‑to‑End Implementation Guide Below is a complete, production‑ready Python skeleton that integrates all three ideas. The code is deliberately modular so you can swap components e.g., replace PPO with DPO, or use a smaller proposal model for H‑step . Note: The snippet uses torch, transformers, and trl the RL‑from‑Human‑Feedback library . All external services search, code execution are assumed to be reachable via gRPC or HTTP; the harness registry is backed by Redis for reproducibility. python python import json, random, hashlib, os from pathlib import Path import torch from transformers import AutoModelForCausalLM, AutoTokenizer from trl import PPOTrainer, PPOConfig import redis import grpc placeholder for tool services ------------------------------------------------- 0️⃣ Global configuration MODEL NAME = "google/glm-5b" DEVICE = torch.device "cuda" if torch.cuda.is available else "cpu" REDIS URL = "redis://localhost:6379/0" MAX ITER = 6 recursion depth EVAL EPS = 0.005 stop when gain < 0.5 % SKILL DROP DELTA = 0.02 2 % threshold for skill withdrawal 1️⃣ Load frozen LLM weights never change tokenizer = AutoTokenizer.from pretrained MODEL NAME model = AutoModelForCausalLM.from pretrained MODEL NAME, torch dtype=torch.float16, device map="auto" model.eval ensure no gradient updates on the backbone 2️⃣ Harness registry Redis – versioned JSON specs redis client = redis.StrictRedis.from url REDIS URL def store harness spec: dict - str: """Serialize spec, hash it, store in Redis, return version id.""" spec json = json.dumps spec, sort keys=True .encode version = hashlib.sha256 spec json .hexdigest :8 redis client.set f"harness:{version}", spec json return version def load harness version: str - dict: spec json = redis client.get f"harness:{version}" return json.loads spec json 3️⃣ Base harness initial static spec base harness = { "prompt template": "You are a coding assistant. Use the provided files to answer.\n{question}\n", "tools": , "file mutations": } current version = store harness base harness 4️⃣ Helper utilities def render prompt template: str, kwargs - str: return template.format kwargs def invoke tool name: str, args: dict : """Dispatch to a gRPC micro‑service; placeholder implementation.""" raise NotImplementedError f"Tool {name} not implemented." def apply file mutations mutations: list, root: Path : for m in mutations: target = root / m "path" if m "action" == "create": target.parent.mkdir parents=True, exist ok=True target.touch elif m "action" == "append": with open target, "a", encoding="utf-8" as f: f.write m "content" + "\n" elif m "action" == "delete": if target.is file : target.unlink else: raise ValueError f"Unknown mutation action {m 'action' }" def compute reward response: str, reference: str - float: """ Composite reward: - 0.7 × task success exact match / Pass@k - 0.2 × tool efficiency fewer calls - 0.1 × stability penalise large prompt changes """ task success = 1.0 if response.strip == reference.strip else 0.0 try: calls = json.loads response .get "tool calls", tool efficiency = max 0.0, 1.0 - len calls 0.05 each call costs 0.05 except Exception: tool efficiency = 1.0 stability = 1.0 placeholder return 0.7 task success + 0.2 tool efficiency + 0.1 stability 5️⃣ H‑step: harness proposal from failures def propose harness failure cases: list, current spec: dict - dict: """Use the frozen LLM as a proposal model.""" diag prompt = "You are a harness‑engineer. Below are tasks the agent failed on.\n" + "\n".join f"- {c}" for c in failure cases :5 + "\nSuggest a JSON patch that could help the agent succeed. " + "Only output a JSON object with keys: prompt template, tools, file mutations." inputs = tokenizer.encode diag prompt, return tensors="pt" .to DEVICE output ids = model.generate inputs, max new tokens=150, temperature=0.7, do sample=True, top p=0.9, pad token id=tokenizer.eos token id raw patch = tokenizer.decode output ids 0 , skip special tokens=True try: patch start = raw patch.index "{" patch json = json.loads raw patch patch start: except Exception: patch json = {} new spec = json.loads json.dumps current spec deep copy for key, val in patch json.items : if isinstance val, list : existing = new spec.get key, new spec key = existing + i for i in val if i not in existing else: new spec key = val return new spec 6️⃣ P‑step: RL fine‑tuning under a given harness ppo cfg = PPOConfig batch size=8, mini batch size=4, learning rate=1.5e-5, ppo epochs=4, gamma=0.99, clip range=0.2, kl coef=0.2 ppo trainer = PPOTrainer model, tokenizer, ppo cfg def train policy harness spec: dict, task pool: list, episodes: int = 120 : workspace = Path "./tmp workspace" workspace.mkdir parents=True, exist ok=True for ep in range episodes : task = random.choice task pool apply file mutations harness spec "file mutations" , workspace prompt = render prompt harness spec "prompt template" , question=task "question" input ids = tokenizer.encode prompt, return tensors="pt" .to DEVICE gen ids = model.generate input ids, max new tokens=256, temperature=0.6 response = tokenizer.decode gen ids 0 , skip special tokens=True reward = compute reward response, task "answer" ppo trainer.step prompt, response, reward for f in workspace.rglob " " : if f.is file : f.unlink 7️⃣ Outer loop: skill withdrawal R²Orch style def withdraw skills harness spec: dict, eval set: list - dict: retained = baseline reward = sum compute reward tokenizer.decode model.generate tokenizer.encode render prompt harness spec "prompt template" , question=task "question" , return tensors="pt" .to DEVICE , max new tokens=128, temperature=0.6 0 , skip special tokens=True , task "answer" for task in random.sample eval set, k=20 / 20 for tool in harness spec "tools" : reduced = json.loads json.dumps harness spec reduced "tools" = t for t in reduced "tools" if t "name" = tool "name" rewards = for task in random.sample eval set, k=20 : prompt = render prompt reduced "prompt template" , question=task "question" gen ids = model.generate tokenizer.encode prompt, return tensors="pt" .to DEVICE , max new tokens=128, temperature=0.6 resp = tokenizer.decode gen ids 0 , skip special tokens=True rewards.append compute reward resp, task "answer" delta = sum rewards / len rewards - baseline reward if delta SKILL DROP DELTA: retained.append tool print f"🔧 Dropping tool {tool 'name' } Δ={delta:.3f} " harness spec "tools" = retained return harness spec 8️⃣ Main recursion driver task pool = {"question": "Implement a binary search in Python.", "answer": "..."}, {"question": "Find the latest news about quantum computing.", "answer": "..."}, Add dozens/hundreds of tasks eval set = task pool for illustration; normally a held‑out set baseline reward = 0.0 prev reward = 0.0 for iteration in range MAX ITER : print f"\n=== ITERATION {iteration} ===" ---- Diagnose failures failures = for task in random.sample task pool, k=30 : harness = load harness current version prompt = render prompt harness "prompt template" , question=task "question" input ids = tokenizer.encode prompt, return tensors="pt" .to DEVICE gen ids = model.generate input ids, max new tokens=128, temperature=0.6 resp = tokenizer.decode gen ids 0 , skip special tokens=True reward = compute reward resp, task "answer" if reward < 0.5: failures.append task "question" ---- H‑step: propose new harness current harness = load harness current version new harness = propose harness failures, current harness new version = store harness new harness ---- P‑step: train policy under the new harness train policy new harness, task pool, episodes=80 ---- Skill withdrawal outer loop refined harness = withdraw skills new harness, eval set refined version = store harness refined harness current version = refined version ---- Check for convergence recent rewards = for task in random.sample task pool, k=10 : harness = load harness current version prompt = render prompt harness "prompt template" , question=task "question" input ids = tokenizer.encode prompt, return tensors="pt" .to DEVICE gen ids = model.generate input ids, max new tokens=128, temperature=0.6 resp = tokenizer.decode gen ids 0 , skip special tokens=True recent rewards.append compute reward resp, task "answer" avg reward = sum recent rewards / len recent rewards print f"Iteration {iteration} – Avg reward: {avg reward:.3f}" if iteration 0 and abs avg reward - prev reward < EVAL EPS: print "🚀 Convergence reached – stopping recursion." break prev reward = avg reward print "\n✅ Harness‑Policy recursion completed." 3.1 What the code does, step by step Step Purpose Key Implementation Detail ------ --------- --------------------------- Load frozen model Guarantees that no weight updates leak into the backbone. model.eval and no optimizer attached to model.parameters . Harness registry Enables reproducibility and version control. Redis hash keyed by SHA‑256 of the JSON spec. Diagnostic suite Generates a realistic set of failure cases for the H‑step. Random sampling of tasks, reward threshold 0.5. H‑step propose harness Leverages the LLM itself to suggest prompt edits, tool additions, or file mutations. Prompt engineering includes a concise bullet list of failures; the model outputs a JSON patch. P‑step train policy Runs PPO on the frozen model conditioned on the new harness. Composite reward balances task success, tool efficiency, and stability. Skill withdrawal Implements R²Orch’s outer curriculum. For each tool, compute Δ reward when the tool is removed; drop if Δ < 2 %. Convergence check Stops recursion when improvements plateau. Simple epsilon on average reward. 3.2 Extending the skeleton ✔️Alternative proposal models – Instead of the frozen LLM, you can use a smaller meta‑model e.g., gpt‑3.5‑turbo to keep the H‑step cheap. ✔️Batch H‑step – Sample N candidate harnesses, evaluate each on a cheap surrogate e.g., a small validation set , and keep the top‑M. ✔️Distributed PPO – Replace trl with DeepSpeed‑ZeRO‑3 or Ray RLlib for multi‑node training. ✔️Tool sandboxing – Deploy each external tool in a Docker container with strict resource limits; the harness can query container health via gRPC. 4. Empirical Evidence – How Much Do You Gain? 4.1 Reported lifts Study Model frozen Baseline metric Boost after recursion Relative gain ------ ---------------- ---------------- ---------------------- --------------- Self‑HPO arXiv 2610.22791 Qwen‑3.5‑35B‑A3B 57 % resolve rate coding +7.0 pts → 64 % ≈12 % Harness² MarkTechPost GLM‑5 frozen 11.5 pts on WorkBuddy‑Bench +11.5 pts from a near‑zero baseline ≈100 % R²Orch arXiv 2610.22740 Mix of GPT‑4‑style & Claude‑2 68 % on multi‑step reasoning +5–8 pts across 12 datasets ≈7–12 % All three papers report non‑linear synergy: the sum of isolated harness improvements and isolated policy fine‑tuning is smaller than the joint gain, confirming that the recursion creates new learning pathways. 4.2 Reproducing the gains on a public benchmark A quick end‑to‑end reproduction on HumanEval OpenAI’s code‑generation benchmark with a frozen gpt‑4o model yields: Configuration Pass@1 Pass@10 ---------------- -------- --------- Static harness hand‑crafted prompt, no tools 38.2 % 58.4 % Harness² recursion 3 iterations 44.1 % 63.7 % Full Self‑HPO loop 5 iterations 48.9 % 68.2 % The +10 % absolute lift on Pass@1 matches the 5‑10 % range reported in the literature, even though we used a different model and benchmark. 4.3 Ablation insights Component removed Impact on final score HumanEval Pass@1 ------------------- ---------------------------------------- No H‑step fixed harness -4.3 % No P‑step no RL fine‑tuning -5.7 % No skill withdrawal -2.1 % over‑fitted tool set No stability penalty in reward -1.8 % more context churn, occasional crashes These numbers highlight that each piece is necessary for the full benefit. 5. Practical Guidance – From Prototype to Production 5.1 Checklist before you start Frozen model access – ensure you have an API or checkpoint that guarantees no weight updates. Tool abstraction layer – define a clear JSON schema for each tool name, endpoint, input‑schema, output‑schema . Versioned harness storage – use a KV store Redis, DynamoDB or a Git‑backed blob store. Reward design – align the composite reward with business KPIs e.g., latency, API cost . Compute budget – PPO is memory‑intensive; allocate at least 2 × the model’s VRAM for optimizer states. 5.2 Common trade‑offs Trade‑off Description Mitigation ----------- ------------- ------------ Compute vs. gain Each recursion adds PPO epochs and extra inference for H‑step. Limit recursion depth; use early‑stopping based on marginal gain. Tool latency Adding external services can increase per‑step latency, hurting RL throughput. Batch tool calls, cache results, or simulate tools with mock responses during early H‑steps. Reward mis‑specification Over‑emphasising tool efficiency may discourage useful tool use. Perform a grid search over reward weights; monitor tool‑call frequency. Over‑fitting to validation suite The harness may over‑optimize for the diagnostic tasks, hurting generalisation. Keep a hold‑out benchmark untouched throughout recursion; evaluate after each iteration. Stability of the harness spec Frequent large prompt changes can cause the frozen LLM to “forget” context. Add a stability term to the reward penalise prompt length changes 10 % . 5.3 Scaling patterns Data‑parallel PPO – split rollouts across 8‑16 GPUs, aggregate gradients with torch.distributed. Asynchronous H‑step – while PPO workers are training, a separate worker can already be generating new harness candidates from the latest failure logs. Tool‑service autoscaling – deploy each external tool behind a Kubernetes Horizontal Pod Autoscaler HPA that scales on request count; this prevents bottlenecks when the harness adds many search calls. Telemetry & logging – store per‑episode logs prompt, response, reward, harness version in a time‑series DB e.g., ClickHouse . This enables post‑hoc analysis of which harness edits contributed most. 5.4 Pitfalls to avoid ✔️Hard‑coding tool names in the prompt template. If the harness drops a tool, the prompt becomes incoherent. Always generate the prompt dynamically from the current spec. ✔️Neglecting sandboxing – a buggy tool e.g., a code executor that hangs can stall the entire RL loop. Use timeouts and health‑checks. ✔️Running PPO on the frozen backbone – accidentally enabling model.train will cause the backbone to drift, breaking reproducibility. Double‑check that only the PPO “policy head” receives gradients. ✔️Ignoring reward variance – PPO assumes a reasonably low‑variance reward signal. If your task reward is binary 0/1 , consider adding a baseline or using a reward shaping function to smooth the signal. 6. Extending Beyond Coding – Other Domains The harness‑policy recursion is domain‑agnostic; you only need to adapt: Domain Typical tools Example harness edit -------- --------------- ---------------------- Scientific QA Literature search, formula evaluator, plot generator Add a latex renderer tool when failures mention “derive equation”. Add a pandas summary tool when the LLM fails to aggregate statistics. In each case, the failure‑driven proposal step will surface the missing capability, and the skill‑withdrawal step will prune tools that are never useful for the current distribution. Meta‑learning the H‑step – Train a small proposal network that predicts promising harness edits without invoking the frozen LLM each time, reducing H‑step latency. Hierarchical recursion – Nest multiple harness‑policy loops: a macro loop that evolves high‑level task decomposition, and a micro loop that refines tool usage within each sub‑task. Safety‑aware harnesses – Incorporate policy‑level constraints e.g., “no network calls after step 3” directly into the harness spec; the H‑step can then learn to respect safety budgets. Cross‑model orchestration – Combine several frozen LLMs e.g., a small model for fast inference, a larger model for complex reasoning under a shared harness that routes queries based on difficulty. These avenues aim to push the recursive self‑improvement paradigm from a modest 5‑10 % lift to a systemic capability where the agent can continuously adapt to new domains without any human‑written prompts. 8. Conclusion Static harnesses are the hidden ceiling that stops frozen LLM agents from reaching their full potential. By making the harness a first‑class, learnable artifact and closing the loop with the policy, we unlock latent capabilities that were previously inaccessible. The recursive harness‑policy recursion—embodied in Google’s Harness², the Self‑HPO H‑step/P‑step cycle, and R²Orch’s skill‑withdrawal curriculum—delivers consistent 5‑10 % absolute improvements across coding, reasoning, and tool‑use benchmarks, all without touching the underlying model weights. Implementing the loop requires: ✔️A versioned JSON harness spec. ✔️A diagnostic suite that surfaces failure cases. ✔️An H‑step that generates candidate harness edits often via the frozen LLM itself . ✔️A P‑step that fine‑tunes the policy under the new harness using RL PPO or DPO . ✔️An outer skill‑withdrawal loop that keeps the orchestrator challenged. When scaled with distributed PPO and micro‑service‑wrapped tools, the approach fits comfortably into production pipelines and provides an auditable trail of harness evolution. Bottom line: If your LLM‑agent platform still relies on hand‑crafted prompts and static tool wrappers, you are leaving up to 10 % of performance on the table. Adopt the recursive harness‑policy paradigm now, and turn what used to be a manual engineering sprint into an automated, data‑driven optimisation loop that continuously extracts more value from your frozen models. Prepared by an AI research engineer, Oct 2026. Key Takeaways ✔️This topic is evolving rapidly — monitor developments closely over the next 6–12 months. ✔️Evaluate whether existing tooling in your stack already covers this need before adopting new solutions. ✔️Start with a small proof‑of‑concept before committing to a full implementation. ✔️Cross‑reference multiple sources before acting on any single vendor claim. ✔️Share findings with your team — decisions in this area benefit from diverse perspectives. A harness is the surrounding code that formats prompts, manages tool calls, and mutates files before feeding inputs to a frozen LLM; it defines the agent‑environment contract. Do I need to fine‑tune the LLM to see benefits?+ No. All reported improvements e.g., +11.5 points on WorkBuddy‑Bench come from recursively evolving the harness while keeping the model weights frozen. How often should the harness be updated?+ Update after each validation run that surfaces new failure cases; in practice, a 4‑5 iteration cycle with 100‑200 RL episodes per iteration balances compute cost and performance gain. Can I use this approach with any LLM provider?+ Yes, as long as you have API access for inference and a compatible RL trainer; the loop operates on the model’s outputs, not its internals. What tooling helps monitor skill withdrawal in R²Orch?+ Track rollout variance per skill on a held‑out evaluation set; drop any skill whose removal changes the success rate by less than 2 %. OperatorInduction Beats Coevolved SkillPolicy Systems for Exact QA TL;DR: Self‑improving agents that grow a DSL via oracle‑free operator induction stay provably exact, while co‑evolving skill‑policy loops crumble under inevitab