Most agent tuning happens in a loop: tweak the prompt, run it manually, check the output, repeat. Harness Tuner replaces that manual cycle with an automated optimization loop that plans experiments, measures performance on a training set, validates on a held-out set, and never touches the test set until you explicitly promote a candidate.
The tool targets three tuning dimensions: agent tools (which functions the agent can call), prompts (system and user instructions), and model selection (which LLM backend to use). Instead of guessing which change will improve performance, you define evals and let the loop measure.
Harness Tuner is an agentic optimizer. It reads task definitions from Kiln (a structured eval format), plans experiments, runs them, and tracks results. The loop follows a train/val/test discipline borrowed from ML pipelines:
The optimizer is itself an agent. It decides which dimension to tune next based on current performance metrics. If tool selection is the bottleneck, it experiments with different tool combinations. If the prompt is unclear, it generates variations and measures which one improves task completion.
The optimizer maintains a state machine:
The loop never reads the test set during optimization. This prevents the classic ML mistake of tuning to your test data.
Harness Tuner consumes Kiln task definitions. A task is a JSON structure that includes:
Here's a simplified example:
{
"task_id": "summarize_article",
"input": {
"url": "https://example.com/article"
},
"expected_output": {
"summary": "A concise summary of the article",
"key_points": ["point1", "point2"]
},
"tools": ["fetch_url", "extract_text"],
"prompt": "You are a summarization agent. Fetch the article and extract key points."
}
The optimizer runs this task with different configurations and compares outputs to the expected result. Scoring can be exact match, semantic similarity, or a custom function.
LLM outputs are stochastic. Harness Tuner runs each experiment multiple times (configurable, default is 3) and aggregates results. It tracks:
High variance signals that a configuration is unstable. The optimizer penalizes configurations with high variance even if mean performance is good.
The optimizer can add, remove, or swap tools. If an agent has access to fetch_url, search_web, and extract_text, the optimizer might test:
search_web to see if it's a distraction.summarize_text to improve output quality.fetch_url for a cached version to reduce latency.
Each change is measured on the train set. If removing a tool improves performance (e.g., the agent was misusing it), the optimizer keeps the change.
The optimizer generates prompt variations using an LLM. It might:
Prompt changes are risky because they can degrade performance in unexpected ways. The optimizer runs multiple variations in parallel and picks the one with the best train score, then validates it.
The optimizer can swap the underlying LLM. If you start with GPT-4, it might test:
Model selection is measured on cost, latency, and accuracy. The optimizer balances these metrics based on weights you provide.
Automated tuning can overfit to the train set. Harness Tuner includes several guardrails:
The optimizer also logs every experiment. You can inspect the history to see which changes were tried and why they were accepted or rejected.
Harness Tuner runs as a standalone process. You point it at a Kiln task directory, configure the optimization budget (how many experiments to run), and let it work.
harness-tuner optimize \
--task-dir ./tasks \
--budget 50 \
--train-split 0.7 \
--val-split 0.2 \
--test-split 0.1
The optimizer writes results to a SQLite database. You can query it to see experiment history, performance trends, and promoted configurations.
For production, you export the winning configuration as a JSON file and load it into your agent runtime. Harness Tuner doesn't deploy agents; it just finds the best configuration.
The optimizer logs every experiment to a structured format. Each log entry includes:
You can export logs to your observability stack (e.g., Datadog, Grafana) or query them locally. The optimizer also exposes a simple web UI that shows:
| Failure Mode | Cause | Mitigation |
|---|---|---|
| Overfitting to train set | Optimizer runs too many experiments on the same data | Use a large train set and strict validation gate |
| Unstable configurations | High variance in LLM outputs | Increase number of runs per experiment and penalize variance |
| Expensive experiments | Testing large models or slow tools | Set cost and latency budgets; optimizer will skip expensive configs |
| Prompt degradation | Optimizer generates a worse prompt that still scores well on train | Manual review of promoted prompts before production |
| Tool misuse | Optimizer adds a tool that the agent uses incorrectly | Monitor tool call logs and add negative examples to the eval set |
Use it when:
Avoid it when:
Harness Tuner is a closed-loop optimizer for agent configurations. It automates the tedious work of testing tool combinations, prompt variations, and model choices. The train/val/test split prevents overfitting, and the experiment tracking gives you a clear history of what worked and why.
The main risk is garbage-in-garbage-out. If your eval set is bad, the optimizer will tune to the wrong target. Spend time building good evals before you turn on automated tuning.
For teams running production agents, this is a useful primitive. It turns agent tuning from a manual art into a measured process. Just don't expect it to fix a fundamentally broken agent design.