#

HarnessDev: How LLMs Are Building Their Own Agent Frameworks

#

ByteDance's Breakthrough in Self-Evolving Agent Systems

Published: September 10, 2026 | Reading time: 12 minutes

#

The Revolutionary Research

Last week, ByteDance's Seed team, in collaboration with Singapore University of Technology and Design, Georgia Tech, and other institutions, released HarnessDev — a groundbreaking research project that answers a fundamental question in AI agent engineering:

Can LLMs create their own Agent Harnesses and continuously improve them based on task feedback?

The answer is a resounding yes.

#

What Is an Agent Harness?

An Agent Harness is the core control system that drives an AI agent. It includes:

• Task execution loops
• Tool selection and parameter constraints
• Context management
• State tracking
• Result verification
• Failure recovery mechanisms

Think of it as the "operating system" for an AI agent — without it, the agent is just a language model with no structure or direction.

#

The Two-Phase Process

HarnessDev divides the agent development process into two phases:

Phase 1: Creation

Starting from a Weak Seed Harness (a minimal framework with basic I/O capabilities), the LLM builds a complete agent harness by adding control logic for:

Execution — Task loops, planning, scheduling, and stopping conditions 2. Tools — Tool selection, parameter constraints, input/output handling, and error processing 3. Context — Organization of task information, code, history, and constraints 4. State — Current goals, progress tracking, attempt records, and failure information 5. Lifecycle — Timeout handling, recovery mechanisms, and task cleanup 6. Verification — Testing, result checking, completion determination, and logging

Phase 2: Evolution

Using the created harness as a starting point, the LLM continuously adjusts it based on downstream task feedback, evaluating performance on held-out tasks.

#

Key Findings

1. LLMs Can Build Effective Harnesses

• 18 Code Harnesses were created, adding a total of 17,111 lines of code

Gemini required the fewest changes (1,006 lines) but achieved the highest score on Terminal-Bench 2.1 (68.8 )

• All 18 harnesses implemented Execution Loops; Tools, Lifecycle, and Verification had high completion rates

2. Not All Code Is Used

• Out of 108 component instances in Code Harnesses, only 72 were observed running in real tasks

18 components (all from State and Memory) never appeared in actual execution #

26,679 task trajectories recorded zero checkpoint events, despite some harnesses implementing checkpoint logic

This reveals a critical insight: implementing a mechanism doesn't mean it's actually used.

3. The Verification Gap

In self-evaluation, Opus found that out of 100 runs, the harness reported success 99 times, but only 48 were actually correct. This led to the addition of a Completion Check mechanism.

Similarly, in Data tasks, 441 out of 2,325 executions produced degraded commits, but the harness failed to detect them.

4. Cross-Model Adaptation Challenges

When switching executors, performance varies significantly:

Opus's SWE-Pro Harness : 69.3 (Self-Eval) → 33.0 (Gemini executor) #

Qwen Harness : Improved by 17.6 points on BrowseComp when using Gemini

This shows that harnesses become executor-specific over time, requiring re-tuning when switching models.

5. Evolution Limitations

• 5 evolution trajectories improved on the visible feedback set
• However, improvements on held-out tasks were much smaller (average 3.11 points)
• Only 53.1% of version changes showed consistent direction between feedback and held-out sets

Evaluation fluctuation is approximately ±4.75 points, making it hard to distinguish real improvements from noise

#

The Six Control Capabilities

HarnessDev categorizes agent control into six capabilities:

| Capability | Description | Example | | Execution | Task loops, planning, scheduling | When to stop, how to plan | | Tools | Tool selection, parameters, error handling | Which API to call, how to handle errors | | Context | Organization of task info and history | What context to provide the LLM | | State | Goals, progress, failure records | Current state, attempt history | | Lifecycle | Timeout, recovery, cleanup | Handle failures, recover from errors | | Verification | Testing, result checking, logging | Verify results, log outcomes |

#

Execution Cost Analysis

Different execution strategies significantly impact token consumption:

GPT-5.5 Harness : 29.3M tokens, medal rate 19.1 #

DeepSeek V4 Harness : 208.4M tokens, score 19.6

Same performance, 7x token difference!

Across the entire MLE-bench experiment, token overhead varied by 19x between different harnesses.

This highlights the importance of cost-aware design — a small performance improvement may not justify a large token increase.

#

Code Example: Creating a Simple Harness

#

Why This Matters

For AI Researchers

• Shows LLMs can self-improve their own execution frameworks
• Highlights the gap between implemented andactually used mechanisms
• Reveals the importance of verification andcross-model adaptation

For AI Practitioners

• Demonstrates the value of structured agent design over pure memory
• Shows the importance of cost-aware optimization
• Highlights the need for robust verification mechanisms

For the Industry

• Represents a step toward self-evolving AI systems
• Shows the potential for automated agent development
• Highlights challenges in generalization andadaptation

#

Conclusion

HarnessDev represents a significant step toward self-evolving AI agents. However, several challenges remain:

Implementation vs. Usage Gap — Not all implemented mechanisms are actually used 2. Cross-Model Adaptation — Harnesses become executor-specific 3. Evolution Limitations — Improvements don't always generalize to new tasks 4. Cost Awareness — Performance gains may come with disproportionate token costs

The research provides valuable insights for building more robust, efficient, and self-improving AI agents.

This article is based on research published by ByteDance Seed team on September 8, 2026. Paper: arXiv:2609.01437 | Project: self-developing-agents.github.io

source & further reading

dev.to — original article TS Evidence Graph: Make Every SKILL Instruction 100% Enforced AI agents can sound strategic while reasoning from events that never happened Will Mojo Replace Python for AI Development?